Audio-Grounded Fact Verification

• Audio-Grounded Fact Verification (AGFV) is a multi-modal process that uses both transcripts and acoustic features to assess the factuality of spoken claims.
• It integrates text encoding and audio preprocessing with deep learning models to detect and verify claims in dynamic debate and podcast environments.
• Benchmarks like MAD and LiveFC report detection F1 scores around 82% and verification accuracies of 72–74%, highlighting both its potential and challenges.

Audio-Grounded Fact Verification (AGFV) refers to the problem of automatically identifying and assessing the factuality of claims made in spoken, multi-turn dialogues by exploiting both the linguistic content (transcript) and acoustic characteristics (audio waveform) of the utterances. AGFV is central to the analysis of misinformation in streaming audio (e.g., debates, podcasts), where conversational dynamics, speaker interplay, and prosodic features collectively determine the spread and interpretation of claims. Recent benchmarks and systems, including MAD (Chun et al., 17 Aug 2025) and LiveFC (V et al., 14 Aug 2024), provide concrete instantiations of AGFV methodology, annotation, and evaluation, supporting both research and deployment scenarios.

1. Formal Framework and Subtasks

A multi-turn audio dialogue is represented as

$D = \{(T_i, A_i, p_i)\}_{i=1}^N,$

where $T_i$ is the transcript, $A_i$ the audio signal, and $p_i$ the speaker for turn $i$. AGFV is decomposed into two primary subtasks (Chun et al., 17 Aug 2025):

• Check-worthy claim detection: Learning $C: D \to \{0,1\}^N$, which assigns each utterance a binary label $y_i$ indicating whether it asserts or elaborates a verifiable claim.
• Claim verification: For the subset of turns with $y_i=1$, inferring veracity $$V: \{(T_i, A_i)\mid y_i=1\} \to \{\mathrm{true},\mathrm{false},\mathrm{unverifiable}\}$$ at the sentence level. Aggregation across a dialogue enables dialogue-level verdicts.

In a streaming or live context, as operationalized by LiveFC, these tasks are embedded in a pipeline involving real-time ASR, speaker diarization, segment-to-speaker mapping, claim detection/normalization, evidence retrieval, and natural language inference-based claim verification (V et al., 14 Aug 2024).

2. Datasets: MAD and Domain-Specific Corpora

The MAD benchmark is the first dataset aligned with audio-grounded, multi-turn spoken dialogue fact-checking. Its construction involves (Chun et al., 17 Aug 2025):

• Core claims: 600 political claims (300 true, 300 false) sampled from the LIAR corpus.
• Dialogue synthesis: 4–6 turn dialogues generated by LLMs (Gemini 2.5 Pro) conditioned on speaker roles, claim-introduction styles, and dialogue scenarios.
• Annotation: Human annotators label check-worthy sentences (1,748 of 4,915 total), and veracity at both sentence and dialogue levels.
• Acoustic realization: Audio synthesized using XTTS-v2; supports extraction of MFCCs, prosodic features, and more.

Key corpus statistics are summarized as:

Property	Value
Number of dialogues	600 (300 True, 300 False)
Turns per dialogue	4–6 (mean 4.45, SD 0.61)
Sentences per dialogue	5–15 (mean 8.19, SD 2.23)
Check-worthy sentences	1,748 (35.6% of total)

Editor's term: "MAD-style AGFV" refers to AGFV tasks using synthetic, speaker-profiled, and scenario-driven dialogues.

LiveFC deploys in-house datasets for offline training, emphasizing political debates and real streaming data, and evaluates claim detection/veracity modules using human-annotated and expert-verified debates (V et al., 14 Aug 2024).

3. Feature Extraction and Multimodal Modeling

Both textual and acoustic modalities are critical for AGFV. The MAD benchmark details the following pipelines (Chun et al., 17 Aug 2025):

• Text encoding: Tokenization (AutoTokenizer, max 512 tokens) followed by transformer encoders (RoBERTa-base, DeBERTa-v3-base).
• Audio preprocessing: Resampling (24 kHz), VAD, and windowing.
• Acoustic features:
  • MFCCs: Computed via STFT, mel filtering, log transformation, and DCT.
  • Prosody: Pitch ($f_0$), energy ($E$), and voiced/unvoiced segment durations.

Proposed modeling strategies include:

• Text-only models: Embedding $h_{\text{text}} = \mathrm{TextEncoder}(T_i)$ and softmax classification.
• Audio-only models: $h_{\text{audio}} = \mathrm{AudioEncoder}(A_i)$.
• Multimodal fusion: Concatenation and nonlinear mixing via $$h_{\text{fused}} = \mathrm{ReLU}(W_f[h_{\text{text}};h_{\text{audio}}] + b_f)$$.

LiveFC currently operates in a predominantly text-grounded mode, with claims normalized for co-reference and disfluencies prior to claim detection, and plans extension to audio-metadata features for richer verification (V et al., 14 Aug 2024).

4. System Architectures and Pipelines

LiveFC embodies an end-to-end, real-time AGFV platform optimized for low-latency streaming scenarios (V et al., 14 Aug 2024). Its architecture consists of:

1. Audio ingestion and segmentation into 5 s buffers;
2. ASR (Whisper-large-v3) with VAD;
3. Speaker diarization (Diart, pyannote/embeddings) with overlap-aware assignment and incremental clustering;
4. Claim detection and normalization (Mistral-7b for de-co-referencing, fine-tuned XLM-RoBERTa-Large for classification);
5. Evidence retrieval across web, Wikipedia, Semantic Scholar, and Elasticsearch-indexed fact-checks;
6. Claim decomposition (LLM prompting for sub-queries);
7. NLI-based verification (fine-tuned XLM-RoBERTa-Large on FEVER, MNLI, X-Fact, and FactiSearch examples);
8. Front-end visualization (Streamlit UI for segment-level verdicts, speaker tracking, and evidence display);
9. Back-end (FastAPI microservices, WebSockets, Elasticsearch).

Modular and streaming design enables throughput of ~30 segments/minute and per-segment latency of ~2 s.

5. Evaluation Protocols and Benchmark Results

Standard AGFV evaluation focuses on classification metrics at both claim detection and verification stages (Chun et al., 17 Aug 2025, V et al., 14 Aug 2024).

• Check-worthy claim detection: Precision, recall, and F1 for the positive (check-worthy) class.
• Claim verification: Accuracy and macro-F1 for true/false/unverifiable labels at sentence and dialogue levels.

MAD baseline results (mean ± SD over three runs) (Chun et al., 17 Aug 2025):

Model	Detection Acc (%)	Detection F1 (%)	Verification Acc (%)	Verification F1 (%)
RoBERTa-base	86.95 ± 1.0	82.71 ± 1.34	72.60 ± 4.04	68.69 ± 7.67
DeBERTa-v3-base	86.36 ± 1.73	81.94 ± 1.53	74.42 ± 3.29	73.28 ± 2.65
Llama 3 (8B)+QLoRA	–	–	70.62 ± 3.73	68.50 ± 6.45

• Empirical findings: Transformer encoders excel in check-worthy detection (F1 ≈ 82%) but verification plateaus at 72–74% accuracy. Aggregation at dialogue level offers marginal gains for encoder-only models; performance degrades for LLM-embedding aggregation.

In LiveFC's end-to-end US presidential debate trial (V et al., 14 Aug 2024):

• Precision: 82.6%
• Recall: 85.8%
• Macro-F1: 83.9%
• Weighted F1: 87.3%

Qualitative annotation shows moderate-to-high inter-annotator agreement on evidence completeness (EC=3.46/5, α=0.76), usefulness (EU=3.60/5, α=0.65), and topic relevance (TR=4.37/5, α=0.51).

6. Error Modes, Limitations, and Open Challenges

AGFV systems encounter specific error types and open challenges (Chun et al., 17 Aug 2025, V et al., 14 Aug 2024):

• Cross-turn coreference: Failure to resolve pronouns or referent statistics.
• Prosodic cues: Missed nuances in sarcasm or certainty (not captured in text-only pipelines).
• Disfluencies and overlap: Interruptions and fillers degrade both ASR and downstream accuracy.
• Speaker bias: Sarcastic or emotionally inflected claims may flip literal meaning.
• Scalability: Basic models remain limited to short (≤6 turn) dialogues; real podcasts exceed this by large margins.
• Multimodal fusion: Robustly integrating acoustic and textual evidence remains an open research area; initial results set a high unimodal baseline.

Certain system constraints are evident: LiveFC is currently limited to HLS (m3u8) inputs and binary (Supported/Refuted) verdicts; mixtures/contextual veracity are not supported. Extension to additional languages and domains, incorporation of non-textual evidence, and deeper fusion of audio features are active areas of development.

7. Future Directions

Progress in AGFV will require advances in several areas (Chun et al., 17 Aug 2025, V et al., 14 Aug 2024):

• Paralinguistic modeling: Automatic extraction of prosody, speaker confidence, and tonal cues to capture implicit speaker stance and sarcasm.
• Speaker interaction modeling: Tracking reinforcement and contestation of misinformation through conversational structure.
• Robustness to real speech phenomena: Improved handling of disfluencies, overlapped speech, and varied dialogue lengths.
• Generalization across domains and languages: Extending annotation and evidence retrieval to non-political, multilingual contexts is essential for broad adoption.
• Explainable verification: Incorporating evidence retrieval and justification generation into end-user interfaces to foster interpretability and user trust.
• Scalable deployment: Real-time operation across heterogeneous audio sources, efficient model serving, and integration with live media platforms.

MAD (Chun et al., 17 Aug 2025) constitutes a rigorous, structured benchmark for research, while LiveFC (V et al., 14 Aug 2024) demonstrates the feasibility and challenges of live, operational AGFV systems. The persistent performance ceiling of current systems underlines the need for innovation in multimodal reasoning and conversational understanding.