AuraScore: Metric for Audio-Visual Reasoning

• AuraScore is a metric designed to evaluate audio-visual reasoning by combining answer correctness with detailed assessments of factual consistency (FCS) and logical inference (CIS).
• It decomposes evaluation into discrete components, where a correct answer is further scrutinized through grounding in perceptual evidence and valid abstraction of reasoning steps.
• The metric helps differentiate genuine multimodal understanding from models that achieve high accuracy by relying on hallucinated evidence or flawed inferential chains.

AuraScore is a decomposed reasoning-fidelity metric introduced alongside the AURA benchmark for audio-visual reasoning. It is defined as a per-instance score in $[0,1]$ that rewards a model only if it produces the correct answer and supports that answer with reasoning that is both perceptually grounded and logically sound. In the AURA formulation, answer accuracy is necessary but not sufficient: reasoning is further evaluated through a Factual Consistency Score (FCS), which measures grounding in perceptual audio-visual evidence, and a Core Inference Score (CIS), which measures the logical validity of the abstract reasoning steps. The metric was proposed in response to the observation that existing audio-visual benchmarks often emphasize final answer accuracy while overlooking whether the intermediate reasoning process is faithful or hallucinatory (Galougah et al., 10 Aug 2025).

1. Benchmark setting and motivation

AuraScore is embedded in AURA, short for Audio-visual Understanding and Reasoning Assessment, a benchmark designed to evaluate the cross-modal reasoning capabilities of Audio-Visual LLMs (AV-LLMs) and Omni-modal LLMs (OLMs). The benchmark includes questions across six challenging cognitive domains: causality, timbre and pitch, tempo and AV synchronization, unanswerability, implicit distractions, and skill profiling. These questions are explicitly designed to be unanswerable from a single modality, thereby requiring a logical path grounded jointly in audio and video rather than a uni-modal shortcut (Galougah et al., 10 Aug 2025).

The central motivation for AuraScore is that final-answer metrics can conflate genuine multimodal reasoning with two distinct error modes: correct answers supported by hallucinated perceptual evidence, and correct facts arranged into invalid inference chains. Within this framework, AuraScore functions as a diagnostic complement to answer accuracy. It does not merely ask whether a model chose the right option; it asks whether the model’s reasoning trace mirrors the annotated perceptual evidence and whether its inferential structure is valid.

This design reflects a broader shift from answer-only evaluation toward process-sensitive evaluation. In the AURA benchmark, the metric is used to expose a “reasoning gap” between what a model answers and how it reasons about the answer.

2. Formal definition and mathematical structure

AuraScore decomposes into three quantities for each question instance $i$:

• $\mathrm{ACC}_i \in \{0,1\}$, the binary answer-accuracy
• $\mathrm{FCS}_i \in [0,1]$, the Factual Consistency Score
• $\mathrm{CIS}_i \in [0,1]$, the Core Inference Score

Its gating structure is strict. If the answer is incorrect, the score is zero:

$$\text{If } \mathrm{ACC}_i = 0,\ \mathrm{AuraScore}_i = 0.$$

If the answer is correct, AuraScore averages the two reasoning sub-scores:

$$\mathrm{AuraScore}_i = \mathrm{ACC}_i \times \frac{\mathrm{FCS}_i + \mathrm{CIS}_i}{2}.$$

Equivalently,

$$\mathrm{AuraScore}_i = \begin{cases} 0, & \text{if } \mathrm{ACC}_i = 0 \ \frac{\mathrm{FCS}_i + \mathrm{CIS}_i}{2}, & \text{if } \mathrm{ACC}_i = 1 \end{cases}$$

The Factual Consistency Score is defined using atomic factual elements extracted from the ground-truth and model reasoning traces. Let $E^{gt}$ denote the set of atomic factual elements extracted from the ground-truth reasoning trace, and $E^{mod}$ the corresponding set extracted from the model trace. Then

$i$0

This definition makes FCS a recall-like grounding measure relative to the ground-truth trace. It quantifies whether the reasoning mentions the entities, attributes, and relations that are actually present in the annotated perceptual explanation.

The Core Inference Score is defined on sanitized reasoning traces. Let $i$1 denote the sanitized, abstracted form of the ground-truth reasoning trace, and $i$2 the sanitized form of the model reasoning trace. If $i$3 is the entailment probability returned by a pretrained natural language inference model, then

$i$4

CIS therefore measures whether the abstract inferential pattern in the model’s explanation is entailed by the abstract inferential pattern in the ground-truth explanation. The decomposition separates evidential faithfulness from inferential validity rather than collapsing both into a single textual similarity score (Galougah et al., 10 Aug 2025).

3. Evaluation pipeline

The evaluation procedure is defined per question-model pair and proceeds in four stages.

First, answer correctness is checked by extracting the model’s selected MCQ answer $i$5 and comparing it to the ground-truth answer $i$6 via a simple exact-match or GPT-4-based correctness prompt. If the answer is incorrect, $i$7 and AuraScore is assigned zero immediately.

Second, FCS is computed through fact decomposition and element matching. GPT-4o is prompted to break both the ground-truth reasoning and the model reasoning into lists of atomic factual propositions, such as “subject = violinist,” “pitch high,” or “on the right.” The two lists are then compared, the matched propositions are counted, and the score is calculated as the number of matched propositions divided by the total number of propositions in the ground-truth list.

Third, CIS is computed through sanitization and NLI. GPT-4o is prompted to remove all specific visual and audio details, leaving only the abstract inference, formulated schematically as “because $i$8 implies $i$9.” The sanitized ground-truth trace is treated as premise and the sanitized model trace as hypothesis. These are passed into a supervised NLI cross-encoder, with the paper giving nli-deberta-v3-base as an example, and the predicted entailment probability becomes the CIS value.

Fourth, the metric aggregates the components. If $\mathrm{ACC}_i \in \{0,1\}$0, the final score is $\mathrm{ACC}_i \in \{0,1\}$1; otherwise it is zero (Galougah et al., 10 Aug 2025).

This procedure makes AuraScore a hybrid metric: answer correctness is discrete, factual grounding is overlap-based after LLM-mediated decomposition, and inferential validity is probabilistic through NLI on sanitized traces.

4. Worked example

The paper provides an illustrative cross-modal causal-reasoning instance. The video shows that a blender turns on and fruit is pureed. The question is: “What caused the fruit to be pureed?” The ground-truth answer is: “The blender was turned on.”

The annotated ground-truth reasoning trace is:

1. “We hear a whirring motor sound.”
2. “We see the blender blades spinning.”
3. “Spinning blades crush the fruit into puree.”

The model’s answer is correct, so $\mathrm{ACC}_i \in \{0,1\}$2. Its reasoning trace is:

1. “There is a whirring noise.”
2. “You can see something spinning.”
3. “Therefore the fruit is being crushed.”

For FCS, the ground-truth factual set is

$\mathrm{ACC}_i \in \{0,1\}$3

and the model factual set is

$\mathrm{ACC}_i \in \{0,1\}$4

Because all three facts match in essence, the paper assigns

$\mathrm{ACC}_i \in \{0,1\}$5

For CIS, the sanitized traces are:

• $\mathrm{ACC}_i \in \{0,1\}$6: “If you hear a whirring motor and see blades spinning, then fruit is crushed.”
• $\mathrm{ACC}_i \in \{0,1\}$7: “If you hear whir and see spinning, then fruit is crushed.”

The NLI entailment probability is given as $\mathrm{ACC}_i \in \{0,1\}$8, so $\mathrm{ACC}_i \in \{0,1\}$9. The final score is therefore

$\mathrm{FCS}_i \in [0,1]$0

The example shows the intended behavior of the metric. The model is not rewarded merely for naming the blender; it is rewarded because its trace preserves both the factual grounding and the inferential structure of the annotated explanation (Galougah et al., 10 Aug 2025).

5. Relation to raw accuracy and empirical findings

AuraScore is explicitly presented as more faithful than raw multiple-choice accuracy. The underlying argument is that conventional answer accuracy admits two failure modes. The first is hallucination: a model guesses the right answer but invents facts not present in the video. The second is flawed logic: a model lists correct facts but strings them into an invalid inference. AuraScore addresses both by requiring factual consistency with annotated perceptual evidence and logical consistency at the level of core inference (Galougah et al., 10 Aug 2025).

Empirically, evaluations on AURA revealed a substantial discrepancy between answer-level success and reasoning-level fidelity. The paper reports that state-of-the-art models can achieve high accuracy, up to $\mathrm{FCS}_i \in [0,1]$1 on some tasks, while their Factual Consistency and Core Inference scores remain below $\mathrm{FCS}_i \in [0,1]$2. The detailed exposition also states that many models scored $\mathrm{FCS}_i \in [0,1]$3–$\mathrm{FCS}_i \in [0,1]$4 ACC yet fell below $\mathrm{FCS}_i \in [0,1]$5 on FCS and CIS. In both formulations, the point is the same: correctness of the final answer substantially overestimates true multimodal reasoning capability.

A common misreading of answer-only benchmarks is that a correct output implies successful cross-modal integration. AuraScore was constructed to reject that equivalence. In the AURA setting, high accuracy without strong FCS and CIS is treated as evidence of unfaithful or illogical reasoning rather than as evidence of robust audio-visual understanding.

6. Homonymous later usage in audio question answering

A plausible source of confusion is that a later paper introduced a distinct metric named “AURA score” for holistic audio question answering evaluation. That metric was proposed for Audio Question Answering (AQA), not for reasoning-trace evaluation on audio-visual multiple-choice questions, and its formulation differs materially from AuraScore in AURA (Dixit et al., 6 Oct 2025).

Metric	Primary setting	Scoring components
AuraScore	Audio-visual reasoning in AURA	$\mathrm{FCS}_i \in [0,1]$6, $\mathrm{FCS}_i \in [0,1]$7, $\mathrm{FCS}_i \in [0,1]$8
AURA score	Open-ended AQA in AQEval	$\mathrm{FCS}_i \in [0,1]$9, $\mathrm{CIS}_i \in [0,1]$0, $\mathrm{CIS}_i \in [0,1]$1, Normalize

The later AQA metric is defined for a question $\mathrm{CIS}_i \in [0,1]$2, audio clip $\mathrm{CIS}_i \in [0,1]$3, candidate response $\mathrm{CIS}_i \in [0,1]$4, and reference answer $\mathrm{CIS}_i \in [0,1]$5 as

$\mathrm{CIS}_i \in [0,1]$6

Here, $\mathrm{CIS}_i \in [0,1]$7 is an LLM-based contextual correctness score derived from a 3-point rating, $\mathrm{CIS}_i \in [0,1]$8 is a hypothesis generated from the question and response, and $\mathrm{CIS}_i \in [0,1]$9 is an audio-entailment score obtained by thresholding cosine similarity between CLAP audio and text embeddings. The final value is min–max normalized to $$\text{If } \mathrm{ACC}_i = 0,\ \mathrm{AuraScore}_i = 0.$$0.

The later work evaluates this metric on AQEval, a benchmark of $$\text{If } \mathrm{ACC}_i = 0,\ \mathrm{AuraScore}_i = 0.$$1 annotated examples assembled from ClothoAQA and OpenAQA, and reports Pearson correlation with human judgments of $$\text{If } \mathrm{ACC}_i = 0,\ \mathrm{AuraScore}_i = 0.$$2 for AURA, compared with $$\text{If } \mathrm{ACC}_i = 0,\ \mathrm{AuraScore}_i = 0.$$3 for an LLM-only baseline, $$\text{If } \mathrm{ACC}_i = 0,\ \mathrm{AuraScore}_i = 0.$$4 for METEOR, $$\text{If } \mathrm{ACC}_i = 0,\ \mathrm{AuraScore}_i = 0.$$5 for ROUGE-L, $$\text{If } \mathrm{ACC}_i = 0,\ \mathrm{AuraScore}_i = 0.$$6 for BLEU, and approximately $$\text{If } \mathrm{ACC}_i = 0,\ \mathrm{AuraScore}_i = 0.$$7 for BERTScore via MACE. Its purpose is to improve holistic evaluation of open-ended AQA responses, especially under paraphrase, partial correctness, and hallucination.

The similarity in names should therefore not be taken to imply identity of formulation or task. The earlier AuraScore is a decomposed, answer-gated reasoning-fidelity metric for audio-visual reasoning traces; the later AURA score is a normalized combination of contextual textual judgment and audio entailment for open-ended audio question answering (Dixit et al., 6 Oct 2025).