AUDIORATIO in SpeechLLMs

• AUDIORATIO is an attention-derived metric that measures the ratio of decoder attention on audio tokens versus auto-regressive text, providing a clear signal for hallucination detection in SpeechLLMs.
• It is computed per decoder head and time step by aggregating audio attention, excluding prompt tokens, and then averaging these ratios to form feature vectors for a logistic regression classifier.
• Empirical evaluations show that AUDIORATIO improves performance in ASR and S2TT tasks, though its effectiveness depends on task-specific calibration and head selection.

AUDIORATIO is an attention-derived metric for inference-time hallucination detection in Speech LLMs (SpeechLLMs), introduced in “Detecting Hallucinations in SpeechLLMs at Inference Time Using Attention Maps” (Waldendorf et al., 21 Apr 2026). In its primary and most specific usage, it quantifies, for each decoder attention head and decoding step, the proportion of attention allocated to input audio tokens relative to the sum of attention allocated to audio tokens and the auto-regressive text prefix. The metric operationalizes whether decoding is grounded in source audio or increasingly driven by self-generated text, and it is one of four audio-centric attention metrics proposed alongside AUDIOCONSISTENCY, AUDIOENTROPY, and TEXTENTROPY for hallucination detection in automatic speech recognition (ASR) and speech-to-text translation (S2TT) (Waldendorf et al., 21 Apr 2026).

1. Formal definition and architectural setting

The AUDIORATIO metric is defined in a unified transformer setting in which the input sequence $X$ is a concatenation of audio tokens $x_{a,1}, \dots, x_{a,N}$ and text prompt tokens $x_{t,1}, \dots, x_{t,M}$, while the decoder predicts an auto-regressive output sequence $Y=\{y_1,\dots,y_{t-1}\}$. At decoding step $t$, each attention head $(l,h)$ produces attention weights $a_{t,i}^{l,h}$ over the concatenated prefix of audio, prompt, and previously generated tokens, with $\sum_i a_{t,i}^{l,h}=1$ and $a_{t,i}^{l,h}\ge 0$ (Waldendorf et al., 21 Apr 2026).

The metric first aggregates attention mass over audio positions and over the auto-regressive text prefix (ART). For head $(l,h)$ at step $x_{a,1}, \dots, x_{a,N}$0,

$x_{a,1}, \dots, x_{a,N}$1

and

$x_{a,1}, \dots, x_{a,N}$2

AUDIORATIO is then

$x_{a,1}, \dots, x_{a,N}$3

A defining design choice is that attention to prompt tokens $x_{a,1}, \dots, x_{a,N}$4 is excluded from the denominator. The prompt is treated as instructions rather than content to transcribe or translate, so the ratio is intended to compare dependence on acoustic evidence against dependence on the model’s own generated prefix. Because the denominator excludes prompt attention, extreme values can arise when much attention is assigned to the prompt, and the downstream classifier is expected to learn how to interpret that regime (Waldendorf et al., 21 Apr 2026).

The intuitive interpretation is local but not scalar-thresholded. High $x_{a,1}, \dots, x_{a,N}$5 indicates that a head is attending more to audio than to previously generated text; low $x_{a,1}, \dots, x_{a,N}$6 indicates stronger reliance on the generated prefix. In the paper’s framing, hallucinations can emerge when decoding “runs away” on self-generated text that is only weakly constrained by the input audio, but AUDIORATIO is not used as a single global threshold. Instead, it is treated as a structured feature over layers and heads (Waldendorf et al., 21 Apr 2026).

2. Inference-time computation and classifier integration

AUDIORATIO is computed during normal decoding. For each decoding step $x_{a,1}, \dots, x_{a,N}$7, each layer $x_{a,1}, \dots, x_{a,N}$8, and each head $x_{a,1}, \dots, x_{a,N}$9, the model’s attention row $x_{t,1}, \dots, x_{t,M}$0 is read out, the two partial sums $x_{t,1}, \dots, x_{t,M}$1 and $x_{t,1}, \dots, x_{t,M}$2 are formed, and the ratio $x_{t,1}, \dots, x_{t,M}$3 is evaluated whenever the denominator is nonzero. The per-step values are then averaged over all output steps,

$x_{t,1}, \dots, x_{t,M}$4

yielding one scalar per head for the entire utterance (Waldendorf et al., 21 Apr 2026).

The resulting $x_{t,1}, \dots, x_{t,M}$5 array is flattened into a feature vector. For Qwen-2-Audio, AudioRatio alone yields 1024 features; for Voxtral-3B, it yields 960. These vectors are used as input to a lightweight logistic regression classifier for inference-time hallucination detection. The main models use L2 logistic regression with $x_{t,1}, \dots, x_{t,M}$6, max_iter=5000, and solver lbfgs; an L1 configuration with $x_{t,1}, \dots, x_{t,M}$7 and solver liblinear is used for stable-feature selection across folds (Waldendorf et al., 21 Apr 2026).

The computational overhead is modest relative to SpeechLLM inference. The transformer already computes the attention maps; the extra work consists primarily of storing or streaming the relevant attention rows, summing over the audio and ART index ranges, averaging across time, and evaluating logistic regression. The paper characterizes the naive complexity as $x_{t,1}, \dots, x_{t,M}$8, while emphasizing that only two partial sums per row are required in practice (Waldendorf et al., 21 Apr 2026).

Feature-budget experiments show that using all heads is not necessary. Around $x_{t,1}, \dots, x_{t,M}$9 heads per metric, performance becomes robust; after approximately $Y=\{y_1,\dots,y_{t-1}\}$0 heads, gains plateau. Using roughly $Y=\{y_1,\dots,y_{t-1}\}$1 selected heads improves out-of-domain generalization relative to using all heads, and this is presented as a practical operating point for deployment (Waldendorf et al., 21 Apr 2026).

3. Hallucination detection performance

The hallucination-detection pipeline is supervised, so the meaning of AUDIORATIO in practice depends on the labeling protocol. For ASR, hallucinations are defined by manual annotation on 1,950 VoxPopuli development examples and then approximated automatically by a high-precision criterion based on Word Error Rate and Semantic Hallucination Score: $Y=\{y_1,\dots,y_{t-1}\}$2 The threshold is chosen to maximize precision $Y=\{y_1,\dots,y_{t-1}\}$3 at the expense of recall $Y=\{y_1,\dots,y_{t-1}\}$4. For S2TT, hallucinations are defined as the worst $Y=\{y_1,\dots,y_{t-1}\}$5 of outputs by XCOMET-XL score (Waldendorf et al., 21 Apr 2026).

In in-domain ASR on VoxPopuli, AUDIORATIO alone is a strong signal. For Qwen-2-Audio, an AudioRatio-only logistic regression reaches $Y=\{y_1,\dots,y_{t-1}\}$6 and PR-AUC $Y=\{y_1,\dots,y_{t-1}\}$7, compared with PR-AUC $Y=\{y_1,\dots,y_{t-1}\}$8 for the Mean Entropy baseline; PRR@10% is $Y=\{y_1,\dots,y_{t-1}\}$9 versus $t$0. For Voxtral-3B, AudioRatio-only reaches $t$1, PR-AUC $t$2, and PRR@10% $t$3, compared with baseline PR-AUC $t$4 and PRR@10% $t$5. The paper highlights this as an improvement of up to $t$6 PR-AUC (Waldendorf et al., 21 Apr 2026).

Out-of-domain behavior is more nuanced. On CALLHOME ASR, Qwen-2-Audio with AudioRatio-only yields predicted hallucination rate $t$7 against true $t$8, indicating severe threshold miscalibration; its $t$9 trails Perplexity or Mean Entropy at $(l,h)$0, although PR-AUC remains $(l,h)$1, close to its VoxPopuli value. Voxtral-3B shows $(l,h)$2, PR-AUC $(l,h)$3, and PRR@30% $(l,h)$4, roughly matching baseline PR-AUC but not clearly exceeding it without multi-metric feature selection (Waldendorf et al., 21 Apr 2026).

Transfer across tasks is weak. If the logistic regression is trained on ASR and evaluated on Fleurs S2TT, PR-AUC drops to $(l,h)$5–$(l,h)$6. When trained on S2TT labels instead, AudioRatio-only performs strongly again: for Qwen-2-Audio, PR-AUC $(l,h)$7 versus $(l,h)$8 for the best baseline and PRR@10% $(l,h)$9 versus $a_{t,i}^{l,h}$0; for Voxtral-3B, PR-AUC $a_{t,i}^{l,h}$1 versus $a_{t,i}^{l,h}$2 and PRR@10% $a_{t,i}^{l,h}$3 versus $a_{t,i}^{l,h}$4. The results establish AUDIORATIO as a useful task-specific feature rather than a task-invariant hallucination score (Waldendorf et al., 21 Apr 2026).

4. Relation to the companion metrics and to head specialization

AUDIORATIO is one of a four-metric suite. AUDIOCONSISTENCY measures the Pearson correlation between audio-only attention distributions at steps $a_{t,i}^{l,h}$5 and $a_{t,i}^{l,h}$6,

$a_{t,i}^{l,h}$7

and targets heads whose attention should move forward over time rather than collapse onto fixed regions. AUDIOENTROPY computes entropy over the renormalized audio-only attention distribution; TEXTENTROPY does the same over text-input positions rather than over the generated prefix. The distinction is important: AUDIORATIO captures input-versus-prefix balance, whereas the entropy and correlation metrics capture how attention is distributed and how it evolves temporally (Waldendorf et al., 21 Apr 2026).

Within this suite, AUDIORATIO is the strongest single metric on in-domain ASR, especially for Voxtral-3B. In stable-feature selection on Voxtral-3B, 39 of the 99 retained heads come from AudioRatio, compared with 36 from AudioConsistency, 20 from AudioEntropy, and 4 from TextEntropy. This head distribution indicates that balancing attention between acoustic evidence and self-conditioning is central to the learned detector (Waldendorf et al., 21 Apr 2026).

Head importance is markedly task-specific. The “Top 50” heads per metric overlap only partially between ASR and S2TT: for AudioRatio the overlap is $a_{t,i}^{l,h}$8 for Qwen-2-Audio and $a_{t,i}^{l,h}$9 for Voxtral-3B. This indicates that the most diagnostic heads for hallucination detection depend on the task, even within the same model architecture. A plausible implication is that AUDIORATIO is best viewed as a family of head-localized features rather than as a universal global statistic (Waldendorf et al., 21 Apr 2026).

A frequent misconception is that AUDIORATIO simply means “low ratio implies hallucination.” The empirical design rejects that simplification. The classifier operates on a full vector of per-head, time-aggregated ratios, and the same raw value can be informative or uninformative depending on head function, layer, model, and task. Prompt-focused heads, early-frame heads, and diagonally aligned heads do not share a single diagnostic threshold (Waldendorf et al., 21 Apr 2026).

5. Ratio-based audio descriptors in related fields

Outside SpeechLLM hallucination detection, closely related “audio ratio” constructions recur across audio research. In room acoustics, the direct-to-reverberant ratio is defined from the acoustic impulse response decomposition $\sum_i a_{t,i}^{l,h}=1$0 as

$\sum_i a_{t,i}^{l,h}=1$1

and serves as a room descriptor for dereverberation, robust ASR, and acoustic characterization (Eaton et al., 2015).

In binaural localization, the rectified binaural ratio is a complex time-frequency ratio between two microphone channels after noise whitening and bias correction. Under a Gaussian source-plus-noise model it follows a complex $\sum_i a_{t,i}^{l,h}=1$2-distribution whose mean equals the relative transfer function, yielding a statistically principled spatial feature for robust RTF and TDOA estimation (Deleforge et al., 2016).

In spatial-audio impairment analysis, the decomposition into spatial and residual errors yields the energy-ratio metrics

$\sum_i a_{t,i}^{l,h}=1$3

which separate geometrical distortion from non-spatial degradation (Watcharasupat et al., 2023).

These constructions are not identical to AUDIORATIO in SpeechLLMs, but they share a structural theme: a ratio isolates the balance between two competing components—direct versus reverberant energy, one channel versus another, spatial versus residual error, or acoustic evidence versus self-conditioning. This suggests a broader methodological family of ratio-based descriptors that convert latent balance relationships into compact, diagnostically useful observables.

6. Limitations, interpretation, and significance

AUDIORATIO is model-dependent, task-specific, and label-dependent. The paper explicitly reports better out-of-domain generalization for Voxtral-3B than for Qwen-2-Audio, and it shows that ASR-trained detectors fail to transfer to S2TT. It also notes that different labeling schemes—WER+SHS for ASR and XCOMET-XL for S2TT—change what is operationally counted as a hallucination. Consequently, the feature should not be interpreted as a universal measure of “truthfulness” in speech generation (Waldendorf et al., 21 Apr 2026).

It is also most effective for severe hallucinations rather than subtle errors. The reported ASR labels are intentionally high precision, and recall is limited; for S2TT, combining AUDIORATIO with the other attention metrics consistently outperforms using AUDIORATIO alone. This pattern indicates that ratio-based grounding cues are highly informative but not exhaustive (Waldendorf et al., 21 Apr 2026).

Its research significance lies in demonstrating that decoder attention patterns contain model-internal signals unavailable to generic text-only uncertainty methods. The metric adapts the input-versus-prefix attention idea associated with Lookback-Lens to a speech setting by focusing specifically on audio tokens and by excluding instructional prompt tokens from the denominator. In that sense, AUDIORATIO is both a concrete feature and a statement about SpeechLLM failure modes: hallucination is partly observable as a redistribution of attention away from acoustic grounding and toward self-conditioning (Waldendorf et al., 21 Apr 2026).

The term therefore has a precise technical meaning in current SpeechLLM research, but it also occupies a broader conceptual niche shared with other ratio-based audio descriptors. In its primary usage, however, AUDIORATIO denotes the per-head, per-step ratio of attention to audio versus attention to the auto-regressive text prefix, temporally aggregated and fed to a lightweight classifier for inference-time hallucination detection in speech-capable LLMs (Waldendorf et al., 21 Apr 2026).