Temporal Contrastive Decoding: A Training-Free Method for Large Audio-Language Models

Abstract: Large audio-LLMs (LALMs) generalize across speech, sound, and music, but unified decoders can exhibit a \emph{temporal smoothing bias}: transient acoustic cues may be underutilized in favor of temporally smooth context that is better supported by language priors, leading to less specific audio-grounded outputs. We propose \emph{Temporal Contrastive Decoding} (TCD), a training-free decoding method for unified LALMs that mitigates this effect at inference time. TCD constructs a temporally blurred slow-path view by smoothing the input waveform and re-encoding it, then contrasts next-token logits from the original and slow-path views. The contrastive signal is applied as a token-level logit update restricted to a small candidate set. A self-normalized stability score sets the blur window and update scale, and a step-wise gate based on uncertainty and audio reliance activates the update only when needed. Experiments on MMAU and AIR-Bench show consistent improvements on strong unified LALMs. We further conduct ablations and an architectural applicability study to analyze the contributions of key components and how TCD behaves across large audio-LLM designs.

• The paper introduces TCD, a training-free intervention that mitigates temporal smoothing bias by contrasting original and temporally blurred audio views.
• It employs self-normalized stability scores and step-wise gating to fine-tune next-token predictions, demonstrating improvements of up to 7.7% on temporal reasoning tasks.
• The method shows robust gains on benchmarks like MMAU and AIR-Bench while incurring minimal computational overhead, making it viable for real-time applications.

Temporal Contrastive Decoding: Training-Free Decoding for Unified Large Audio-LLMs

Introduction and Motivation

Temporal smoothing bias is a critical failure mode in Large Audio-LLMs (LALMs) equipped with unified decoders, which tend to over-weight temporally smooth contexts, often governed by strong language priors, at the expense of transient, temporally-localized acoustic cues. This leads to outputs that are less grounded in the fine-grained audio evidence, especially detrimental in reasoning tasks that rely on acoustic events of short duration. The paper introduces Temporal Contrastive Decoding (TCD), a training-free inference-time intervention for unified LALMs that selectively sharpens model sensitivity to transient acoustic evidence without modifications to model parameters or requiring additional training data.

Methodological Framework

TCD augments the autoregressive decoding pipeline with a temporally blurred "slow-path" view of the input audio, constructed by smoothing the waveform with a Hann window and passing it through the same encoder. At every decoding step, TCD computes token-wise logits for both the original and blurred audio views. Their difference, rectified and restricted to a small candidate token set, is used as a contrastive evidence signal (Figure 1).

Figure 1: Overview of Temporal Contrastive Decoding (TCD) — TCD contrasts logits from the original and temporally blurred (“slow-path”) audio views and injects a sparse, gated residual update to sharpen next-token predictions.

The update strength and activation are governed by two core mechanisms:

1. Self-Normalized Stability Score: This per-sample score quantifies temporal flux in encoder hidden states, pooling over layers according to audio-attention intensity. It adapts both the blur magnitude and the scaling of the logit update, enforcing robustness across diverse model architectures and acoustic conditions.
2. Step-wise Gating: The contrastive update is only applied at decoding steps rated as simultaneously audio-reliant (by decoder attention statistics) and uncertain (by entropy over token logits). This ensures that the intervention is conservative, primarily modulating the decoding process when evidence from audio is both relevant and ambiguous.

Ablations demonstrate that TCD's efficacy derives from the structured temporal blur for slow-path construction, the positive-difference (conservative correction), and the gate restricting intervention to informative steps.

Experimental Results

Benchmarks and Models

TCD is evaluated on two demanding benchmarks: MMAU (covering speech, sound, and music in a unified multi-modal framework) and AIR-Bench Foundation (focused on general-purpose speech and sound understanding). It is applied to state-of-the-art unified LALMs, including Qwen2-Audio-Instruct, Qwen2.5-Omni, and Mini-Omni, as well as a set of non-unified or bottlenecked architectures (SALMONN, Audio Flamingo3, DeSTA2.5-Audio, MiMo-Audio) to delineate applicability.

MMAU Results

TCD consistently improves accuracy across all tested unified LALMs. On Qwen2.5-Omni, accuracy climbs from 71.5% to 73.2% overall, with the largest improvement in the music domain (+5.1%). TCD's gains are especially pronounced on tasks requiring recognition of acoustic events with critical temporal cues, e.g., "How many times does the telephone ring?" (see qualitative comparisons in the Appendix).

AIR-Bench Results

On the AIR-Bench Foundation benchmark, TCD increases total accuracy from 64.8% to 66.7% on Qwen2.5-Omni, with the most significant improvements in the Sound domain, again underscoring effectiveness where outputs should hinge on transient temporal structure in the audio.

Temporally Structured Audio Tasks

On tasks explicitly probing temporal reasoning (SLURP, CochlScene, Clotho-AQA), TCD delivers robust gains (up to +7.7%), demonstrating its effectiveness at addressing the temporal smoothing bias in setting where precise timing of events is essential.

Applicability and Limitations

An architectural applicability study reveals a sharp boundary: TCD's positive effects are limited to unified LALMs where the decoder has direct access to temporally-resolved audio representations. In models that map audio into a small set of semantic tokens or heavily aggregate the representation prior to decoding, TCD yields negligible improvements, as the decoder lacks the necessary temporal alignment.

Efficiency and Inference Cost

TCD imposes minimal computational overhead during token generation, with the additional cost localized to a single extra forward pass for the blurred audio during prefill. Memory consumption increases marginally due to doubled key-value caches. The update itself is sparse and lightweight, making TCD compatible with real-time and large-scale inference scenarios.

Practical and Theoretical Implications

TCD advances the space of training-free, inference-only interventions for LALMs. Unlike previous modalities that contrast with non-audio or cross-modal views, TCD focuses explicitly on exploiting differences across temporal scales within the same audio stream, directly targeting the temporal smoothing bias. Empirically, it enables unified LALMs to more robustly leverage transient acoustic evidence without retraining or architectural change.

For practical deployment, TCD is an attractive approach when retraining or collecting additional calibration data is cost-prohibitive or infeasible. Its design—conservative and highly modular—makes it suitable for high-throughput or safety-critical applications where hallucinations and input adherence are focal concerns.

Theoretically, TCD suggests that decoding-time interventions exploiting temporal structure can be as powerful as architectural changes in mediating the balance between language priors and evidence-grounded outputs.

Future Directions

Several lines of research follow from this work: (1) Extending TCD to streaming or online decoding settings, (2) adapting temporal contrastive mechanisms for architectures with heavy semantic bottlenecks, (3) refining gating and stability statistics using more detailed interpretations of audio-attention interplay, and (4) exploring contrastive decoding interventions in additional multimodal contexts.

Conclusion

Temporal Contrastive Decoding is shown to be an effective, training-free method that systematically mitigates temporal smoothing bias in unified LALMs by contrasting next-token predictions conditioned on the original and blurred audio input. Its principled design, empirical effectiveness, and efficient inference profile make it a robust addition to the LALM decoding toolkit, especially in settings requiring precise, temporally-grounded language generation from complex audio streams (2604.15383).