Most current source separation methods only target the voiced component of the singing voice. Besides the unvoiced consonant phonemes, the remaining breathiness is very noticeable to humans and it retains much of the phonetic and timbral information from the singer. We propose a low-latency method for estimating the spectrum of the breathiness component, which is taken into account when isolating the singing voice source from the mixture. The breathiness component is derived from the detected harmonic envelope in pitched vocal sounds. The separation of the voiced components is used in conjunction with an existing iterative approach based on spectrum factorization. Finally, we conduct an objective evaluation that demonstrates the separation improvement, supported also by a number of audio examples.