Abstract
Recent zero-shot text-to-speech (TTS) systems achieve remarkable naturalness and speaker similarity, but typically assume high-quality speaker prompts and either strip away or entangle the acoustic environment with speaker characteristics, which limits their applicability in real-world scenarios.
In this paper, we present DAIEN-TTS, an environment-aware zero-shot TTS framework that disentangles and jointly models speech, background noise, and reverberation, enabling independent control over timbre and acoustic environment through separate speaker and environment prompts.
Built upon the flow-matching-based F5-TTS, DAIEN-TTS employs a speech-environment separation module to decompose environmental speech into speech, noise, and reverberation components, which are then injected into the diffusion transformer to enable environment-aware generation.
The system is first trained on simulated data constructed by mixing clean speech with noise and room impulse responses, with a cross-speaker conditioning strategy to suppress speaker information leakage from the environment branch, and then fine-tuned on real-world speech data to bridge the simulated-to-real domain gap.
At inference, a triple classifier-free guidance mechanism enables fine-grained control over speech, noise, and reverberation, along with an SNR adaptation strategy to align the synthesized speech with the environment prompt.
Experiments on both simulated and real-world test sets show that DAIEN-TTS generates environmental personalized speech with high naturalness, strong speaker similarity, and faithful noise and reverberation reproduction, while offering controllability beyond prior environment-aware TTS systems.