This paper presents DAIEN-TTS, a zero-shot text-to-speech (TTS) framework that enables ENvironment-aware synthesis through Disentangled Audio Infilling. Built upon F5-TTS, the proposed DAIEN-TTS first incorporates a pre-trained speech–environment separation (SES) module to disentangle the environmental speech into clean speech mel-spectrogram and environmental audio mel-spectrogram. Two random span masks of varying lengths are then applied to both mel-spectrograms, which, together with the text embedding, serve as conditions for infilling the target environmental mel-spectrogram, enabling the simultaneous continuation of personalized speech and time-varying environmental audio. During inference, DAIEN-TTS leverages two environmental speech prompts, a speaker prompt and an environment prompt, to independently control the timbre and the background environment of synthesized speech. To further ensure that the synthesized speech matches the SNR of the environment prompt, we introduce a signal-to-noise ratio (SNR) adaptation strategy. Experimental results demonstrate that DAIEN-TTS generates personalized environmental speech with high naturalness, strong speaker similarity, and high environmental fidelity.
| Text Prompt | Speaker Prompt | Human | F5-TTS (Clean Spk Prompt) | F5-TTS | DAIEN-TTS (w/o CA) | DAIEN-TTS |
| My dog is an alsatian, we call her 'Poppy'. | ||||||
| All that time the Martians must have been getting ready. | ||||||
| This was the strangest of all things that ever came to earth from outer space. | ||||||
| The groupings listed here are from the New Zealand Society. | ||||||
| Then why should they be surprised when they see one? | ||||||
| But as he saw them lying there on the ground, another phrase came to his mind. | ||||||
| Text Prompt | Speaker Prompt | Environment Prompt | Human+Environment | Vocoder Resynthesized | DAIEN-TTS (w/o CA) | DAIEN-TTS |
| The people listed below have featured on Gabonese stamps since independence. | ||||||
| Campbell helped establish branch libraries for Black readers. | ||||||
| On his return to England, he became a successful portrait painter. | ||||||
| He later worked as a journalist, and then became a novelist and playwright. | ||||||
| It is situated across the northeastern rim of the older and larger crater Danjon. | ||||||
| The medal is made of solid gold and is inscribed with the recipient's name. | ||||||
| Examples of contract brewers include Epic Brewing Company, Funk Estate and Yeastie Boys. | ||||||
| However, the music from the former anthem was retained. | ||||||
| The monastery was inhabited by a succession of three different religious orders. | ||||||
| Ladies and gentlemen, this wagon is going to be detached. | ||||||