Speech Meets ELF: Audio Conditional Continuous-Target Diffusion for Speech Recognition and Translation

Abstract: Speech-to-text (S2T) systems for recognition (ASR) and translation (S2TT) typically generate discrete text tokens. In contrast, continuous-target language modelling performs generation in a continuous space, yet its potential for S2T remains unexplored. To bridge this gap, we propose ELF-S2T, an audio-conditioned continuous-target generative model for S2T. Built upon the pre-trained Embedded Language Flows (ELF) backbone, ELF-S2T processes speech via a frozen Whisper encoder and a single linear projector, prepending the resulting audio condition to the noisy text latent for in-context, flow-matching denoising. To prevent the model from over-relying on its pre-trained text context, we introduce audio forcing during training, and further amplify the audio condition via classifier-free guidance at inference. Experiments on LibriSpeech and CoVoST2 show that ELF-S2T achieves competitive ASR and S2TT performance. Crucially, our error analysis reveals that, although ASR and S2TT errors look very different on the surface, both stem from the same underlying cause, a close distance confusion in the continuous latent space. This finding naturally aligns with the continuous representation generation paradigm, indicating a common semantic mapping process beneath recognition and translation. Our code and pretrained models are publicly available at https://github.com/Sslnon/ELF-S2T.

• The paper introduces ELF-S2T, a pioneering model using audio-conditioned continuous-target diffusion to unify automatic speech recognition and speech-to-text translation.
• It integrates a frozen Whisper encoder and a pre-trained ELF backbone with audio forcing and classifier-free guidance to enhance cross-modal alignment.
• Results show competitive WER and BLEU scores while latent space analysis reveals shared semantic error patterns between recognition and translation tasks.

Audio-Conditioned Continuous-Target Diffusion for Speech Recognition and Translation: An Expert Analysis of ELF-S2T

Overview and Motivation

The paper "Speech Meets ELF: Audio Conditional Continuous-Target Diffusion for Speech Recognition and Translation" (2606.10368) introduces ELF-S2T, a novel paradigm for speech-to-text (S2T) tasks—including automatic speech recognition (ASR) and speech-to-text translation (S2TT)—by leveraging audio-conditioned continuous-target diffusion models. Unlike standard approaches that generate text as discrete tokens, ELF-S2T operates in a continuous text embedding space via diffusion, only committing to discrete tokens at the final step. The method, which fuses a frozen Whisper encoder with a pre-trained Embedded Language Flows (ELF) backbone using a linear projector, aims to improve both modeling fidelity and analytical insight into cross-modal generation.

Figure 1: ELF-S2T formulates S2T as audio-conditioned continuous text generation, denoising from Gaussian noise under the audio condition, with tokens discretized at the final step.

The core motivation is grounded in the theoretical elegance of aligning continuous speech signals directly to a continuous representation of text, thus circumventing the lossiness of discretization and benefiting from the powerful mechanisms of diffusion modeling in generative language tasks.

Methodology: Architecture and Grounding Strategies

Model Architecture

ELF-S2T integrates a frozen Whisper-large-v3 encoder to extract acoustic features, which are projected into the text-latent space and concatenated as a prefix to the noisy text latent fed into the ELF transformer backbone. The entire architecture is initialized from strong pre-trained weights, ensuring powerful language priors, with trainable parameters limited to the linear projector and the finetuned backbone.

Figure 2: ELF-S2T overview. During training, a denoise and decode head are trained with audio forcing; inference denoises from Gaussian noise via audio-guided SDE steps, decoding tokens only at the final step.

This setup enables bidirectional, flow-matching denoising in the joint audio-text space. The discrete text output is produced by decoding only at the final step, supporting both ASR and S2TT tasks within a shared, continuous generative framework.

Conditioning and Grounding Mechanisms

A central challenge in adapting continuous-target diffusion models to S2T is enforcing strong dependence on the acoustic condition, mitigating the tendency of a powerful text backbone to ignore audio cues in favor of internal completion. ELF-S2T introduces two primary mechanisms:

• Audio Forcing: During training, the clarity of the text latent is randomly degraded before decoding supervision. This compels the model to rely on the audio condition to resolve uncertainties in the corrupted latent, closing potential shortcuts that allow skipping the intended cross-modal alignment.
• Classifier-Free Guidance (Audio Guidance): At inference, the model amplifies the effect of the audio condition using classifier-free guidance. Here, conditional and unconditional branches (the latter learned via condition dropout) are interpolated to explicitly strengthen acoustic influence on the generation trajectory.

Together, these mechanisms structurally enforce attention to the speech signal, addressing the challenge of cross-modal grounding in continuous-target architectures.

Results and Ablation Findings

Main Outcomes

ELF-S2T demonstrates competitive performance on both ASR and S2TT in standard benchmarks:

• ASR (LibriSpeech test-clean): Achieves a WER of 5.69% (ELF-L backbone), outperforming discrete-token diffusion systems (e.g., Whisfusion: 8.3%, TransFusion: 6.1%), though not matching supervised autoregressive baselines such as Whisper-large-v3 (1.97%).
• S2TT (CoVoST2 de→en): Achieves a BLEU score of 28.55 and chrF of 54.91, surpassing Whisper-large-v3 (26.23 BLEU, 54.38 chrF), and constituting the first diffusion-based S2T system to report translation metrics.

Effect of Grounding Strategies

Ablation with the ELF-B backbone confirms the efficacy of audio forcing, reducing ASR WER from 11.11% to 10.50%. Sweeps over classifier-free guidance scaling and diffusion steps establish robust sweet spots (audio guidance scale $w=2.0$, $K=128$), with diminishing returns for increased compute.

Figure 3: Performance trends as a function of audio-guidance scale (left) and sampler steps (right) on ASR (WER) and S2TT (BLEU); both tasks share optimal parameter zones.

Scaling the backbone capacity consistently reduces WER and increases BLEU, demonstrating that continuous-target modeling benefits from scale without saturation or destabilization.

Latent Space Error Analysis

A key analytical contribution is the investigation of error phenomena in the continuous text space. Surface-level analysis reveals that ASR errors are dominated by sub-word form corruptions and semantic substitutions (e.g., "begun"→"commenced"), while S2TT errors are primarily semantic drifts at varying granularity—from single swapped words to complete sentence rewrites.

Latent space probing shows that both types of surface error originate from the same geometric deviation: the audio-conditioned mapping positions the generated latent near a semantically related but incorrect hypothesis, with the model’s output lying closer in embedding space to the hypothesized erroneous text than to the reference. The effect size is consistent across both ASR and S2TT (cosine gap $\Delta \approx 0.11$). This demonstrates that fundamentally, both tasks fail by the same mechanism: close-distance confusion in the continuous latent, manifesting as either local or global meaning substitutions.

Notably, the decoder latent and unembed operation are not responsible for these failures—as teacher-forced reconstruction maintains a high cosine similarity ceiling. Rather, improvements must focus on refining the audio-to-latent mapping process, a unifying direction for future work on both recognition and translation.

Theoretical and Practical Implications

ELF-S2T demonstrates that diffusion-based continuous-target models can serve as a viable, unified paradigm for both recognition and translation in S2T. By analyzing errors in a semantic latent space, the study provides an explicit, measurable basis for understanding and improving cross-modal generation. The architectural simplicity—employing a single linear projector and maintaining most of the language backbone frozen or pre-trained—speaks to practical deployability and extensibility.

Practically, continuous-target models offer flexibility for downstream tasks where preserving rich semantic alternatives or operating on continuous representations is beneficial (e.g., edit-resilient transcription, semantic search, downstream multi-modal composition). The explicit role of grounding mechanisms (audio forcing, classifier-free guidance) is instructive for the broader multi-modal community seeking to avoid text-dominant shortcut phenomena.

Theoretically, the results unify the formulation and failure sources for ASR and S2TT, strengthening the case for continuous generative paradigms. Future research may investigate scaling to noisy/low-resource languages, extending to other modalities (e.g., vision-conditioned text), or designing more sophisticated audio-to-latent alignment objectives.

Conclusion

ELF-S2T establishes audio-conditioned continuous-target diffusion as a competitive and insightful approach to speech recognition and translation, rivaling discrete diffusion systems and extending seamlessly to S2TT. The analysis of shared geometric error in the latent space highlights a common underlying process in cross-modal semantic mapping, guiding future research toward targeted architectural and training enhancements. This work positions continuous-target diffusion as an increasingly relevant framework across modalities in generative AI.