Synthesizing the Lombard Effect: Multi-Level Control of Speech Clarity and Vocal Effort in TTS

Abstract: Humans tend to speak louder and clearer in challenging environments, such as noisy conditions or when addressing hearingimpaired listeners, which is called Lombard effect. To simulate this behavior in speech synthesis systems, we introduce a flow-matching based text-to-speech (TTS) model trained with vocal effort and articulation pseudo-labels. The proposed model achieves continuous and disentangled control of vocal effort and articulation, while also enabling word-level emphasis for clarifying specific segments of an utterance. Experimental results show that these control mechanisms effectively improve clarityrelated acoustic features. Furthermore, speech-in-noise experiments demonstrate that our model successfully simulates the intelligibility gains of human clear speech in noisy conditions.

• The paper introduces a dual-level conditioning strategy that independently modulates articulation and vocal effort in TTS systems.
• It employs encoder-side and decoder-side embeddings to control speaking rate, spectral tilt, and vowel distinctiveness for improved intelligibility.
• Objective and subjective evaluations show reduced word error rates and enhanced CMOS scores, proving effective speech synthesis in noisy conditions.

Multi-Level Lombard TTS: Joint Control of Speech Clarity and Vocal Effort

Introduction

The paper "Synthesizing the Lombard Effect: Multi-Level Control of Speech Clarity and Vocal Effort in TTS" (2606.23176) addresses a longstanding gap in text-to-speech (TTS) systems: the lack of fine-grained, controllable synthesis of speech adaptations characteristic of the Lombard effect. In noisy environments, human speakers naturally modify vocal effort, spectral tilt, articulation, and speaking rate to enhance intelligibility. While traditional TTS systems are trained predominantly on neutral speech, achieving flexible emulation of Lombard adaptations, and especially hyper-articulation, has remained underexplored.

Model Architecture

The proposed framework adopts Matcha-TTS, a conditional flow-matching TTS system that replaces stochastic diffusion with deterministic flow for efficient and high-quality synthesis. Crucially, the authors introduce a dual-level conditioning strategy:

• Encoder-side conditioning: Style and speaker embeddings are injected to modulate duration prediction, enabling control over speaking rate, phoneme durations, and articulation at both global and word-level granularity.
• Decoder-side conditioning: The same embeddings condition the acoustic flow-matching decoder, permitting independent and joint modulation of spectral tilt, energy distribution, and formant clarity without compromising speaker identity preservation.

Embedding spaces for articulation and vocal effort are constructed as continuous axes via learnable projections, supporting smooth interpolation for gradual adaptation to dynamic listening conditions. Word-level control is realized by broadcasting style embeddings temporally, allowing localized clarity enhancement for specific utterance segments.

Training and Feature Conditioning

The Expresso dataset, which includes default, enunciated, fast, and projected speaking styles, provides the basis for pseudo-labeling both articulation (fast–enunciated) and vocal effort (default–projected). To enrich phonetic coverage and stabilize pronunciation modeling, LJ Speech (treated as neutral) is incorporated. Pseudo-labels map into 32-dimensional embeddings for each attribute, concatenated with speaker embeddings for joint, multi-dimensional control. Duration modeling leverages Monotonic Alignment Search, enabling explicit manipulation of phoneme-to-frame durations—a crucial link for temporal adaptation in Lombard speech.

Evaluation Protocol

Objective evaluation employs the Harvard Sentences corpus for phonetic balance, with synthesized speech varying across articulation ($\beta$) and vocal effort ($\alpha$) axes. Baselines use naive signal processing—RMS matching and linear time-stretching—to simulate Lombard-like adaptations, allowing isolation of learned style control effects.

Metrics include:

• Word Error Rate (WER): Using whisper-medium ASR for intelligibility quantification.
• Spectral Tilt: Energy ratio between 5–1kHz versus below 1kHz.
• Mean Vowel Dispersion (MVD): Euclidean distance in F1-F2 space for /i/, /a/, /u/ vowel formants.
• Phoneme Rate: Phonemes per second.
• Speech Intelligibility Index (SII): Percentage of speech audible in noise.

Subjective evaluation uses CMOS for naturalness and intelligibility, with participants blinded to Lombard conditions.

Results

Signal-Level Analysis

The model demonstrates continuous, disentangled control over clarity-related acoustic features:

• WER decreases monotonically with increasing $\beta$, indicating hyper-articulation's dominant role in intelligibility enhancement. Baselines relying on simple rate manipulation show only marginal gains.
• MVD increases with $\beta$, reflecting improved vowel distinctiveness and phonemic separation.
• Spectral Tilt increases chiefly with $\alpha$, revealing Lombard-style spectral redistribution absent from RMS-only baselines.
• Phoneme Rate decreases with hyper-articulation, matching expected temporal expansion.

ASR shows degradation for extreme $\alpha$ values due to distribution mismatch—a phenomenon consistent with prior observations [29].

Speech-in-Noise Experiments

Under SNR-varied conditions (restaurant babble, overlapping speech, white noise):

• Articulation scaling ($\beta$) consistently reduces WER across noise types, especially at intermediate noise levels. Effects plateau past mid-to-high articulation.
• Vocal effort scaling ($\alpha$) is most impactful in energetic masking scenarios (e.g., restaurant babble), with smaller gains in white noise and overlap speech. RMS-normalized evaluation constrains WER gains to spectral redistribution rather than amplitude increase.
• Joint scaling ($\alpha + \beta$) yields maximum robustness under severe noise, particularly SNR = 1, indicating complementary contributions: phonetic clarity from articulation and spectral audibility from vocal effort.
• SII improvements are largest under complex masking, further affirming the dual-axis approach's efficacy.

Word-Level Emphasis

Peak articulation indices ($\beta = 1.5$) applied to target words yield significant intelligibility gains for previously misrecognized words (WER reduction from 17.6% to 3.9%), especially when combined with both hyper-articulation and word-level emphasis. Temporal modulation is highly localized, minimizing leakage and preserving natural speech rhythm.

Human Evaluation

CMOS results demonstrate statistically significant gains in both naturalness and intelligibility (+1.97 for naturalness, +1.13 for intelligibility), with participants consistently favoring the proposed model over naive time-stretching. Subjective analysis underscores that simple rate manipulation is insufficient; true Lombard synthesis requires multi-level control.

Implications and Future Directions

The multi-dimensional framework unlocks flexible and human-like emulation of Lombard strategies in TTS, offering practical tools for context-aware synthesis in conversational agents, hearing-assistive technologies, and interactive dialogue systems. The ability to independently or jointly control vocal effort and articulation enables adaptation to dynamic acoustic environments and listener-specific needs.

Future work could refine token-level control to minimize leakage, develop more robust non-ASR metrics for intelligibility, and explore real-time adaptation for on-the-fly Lombard synthesis. Deeper integration with incremental synthesis and perceptual feedback loops would further advance context sensitivity, closing the gap between human and synthetic speech in real-world scenarios.

Conclusion

The paper presents a rigorous, technically sound approach to synthesizing the Lombard effect in TTS, achieving disentangled, continuous control over vocal effort and articulation at both utterance and word levels. Objective and subjective evaluations confirm substantial gains in intelligibility and naturalness, particularly under noise. This method lays a strong foundation for robust, flexible speech synthesis with practical utility across diverse applications, inviting further exploration of fine-grained control and real-time adaptability in future TTS systems.