---
title: 'AV-LLMs: Audio-Video Transformer Models'
url: https://www.emergentmind.com/topics/audio-video-large-language-models-av-llms
type: topic
---

# AV-LLMs: Audio-Video Transformer Models

Audio-Video Large Language Models (AV-LLMs) are a class of transformer-based architectures that ingest synchronized audio and visual streams—typically video frames and raw or processed audio—enabling multimodal language understanding, open-ended generation, and cross-modal reasoning over temporally extended, richly annotated datasets. AV-LLMs have become foundational for tasks requiring deep, fine-grained integration of dynamic vision and audio, including video captioning, audio-visual question answering (AVQA), speech-augmented comprehension, audio-visual speech recognition (AVSR), and reasoning-intensive multi-step QA grounded in complex or adversarial scenarios. This article systematically details (1) core model architectures and alignment strategies, (2) benchmark design and evaluation, (3) advances in reasoning, preference optimization, and hallucination mitigation, (4) key empirical results and ablation findings, and (5) current efficiency and scalability challenges.

## 1. Architectural Design and Multimodal Fusion Strategies

AV-LLMs are generally constructed by bridging frozen modality-specific encoders (vision and audio) to a large language model through one or more alignment modules. Canonical architectures include Video-LLaMA, video-SALMONN, Dolphin, C3LLM, and Zero-AVSR, featuring combinations of Q-Formers, adapters, and sophisticated token fusion policies [2306.02858][2406.15704][2504.02061][2405.16136][2503.06273].

- **Encoders:** Visual streams are processed by ViT-style image encoders (e.g., EVA-CLIP, InstructBLIP), while audio uses models such as Whisper, BEATs, or ImageBind-AST, often operating on mel-spectrograms and synchronized at frame or segment-level granularity.
- **Alignment modules:** Cross-modal alignment is achieved through several strategies. Q-Formers (cross-attention modules) learn to aggregate temporally or semantically meaningful feature sets from each modality and project them into the LLM hidden space. Adapter-based approaches—including LoRA adapters—allow efficient fine-tuning with a small parameter footprint.
- **Temporal and spatial fusion:** Audio and visual tokens are fused via concatenation, interleaved merging, or fine-grained cross-attention at various stages. Dolphin [2504.02061] utilizes a multi-scale adapter for spatial alignment and interleaved merging for temporal synchronization, while video-SALMONN’s MRC Q-Former pools features at multiple resolutions and enforces causal self-attention across windows [2406.15704]. The LLaVA-AV-SSM model compresses audio tokens using a Mamba-based SSM, achieving scalable attention across long temporal contexts [2509.17901].
  
## 2. Benchmarking, Datasets, and Evaluation Protocols

AV-LLM development is coupled with benchmark design to probe multi-modal reasoning, robustness, and compositional capability:

- **Generic AVQA and music-focused tasks:** Datasets such as AVQA, Music-AVQA, and the more challenging AVQA-Hard and Music-AVQA-Hard support benchmarking under conditions where visual shortcuts are removed [2509.17901]. Metrics are typically accuracy, answer-exact match, and n-gram-based semantic scores.
- **Instruction and alignment datasets:** AVU [2504.02061] and SAVEn-Vid [2411.16213] provide millions of paired video-audio-caption tuples for instruction tuning, including negative/rejection and modality-focused subsets.
- **Reasoning and robustness benchmarks:** RivaBench [2502.11775] and AVTrustBench [2501.02135] assess compositional reasoning, adversarial perturbation, and modality-specific dependency using circular evaluation and calibrated preference metrics.
- **Specialized emotion and AVSR tasks:** AV-EMO-Reasoning [2510.07355] utilizes continuous emotion regression from jointly annotated audio-video dialog, while Zero-AVSR [2503.06273] benchmarks zero-shot speech recognition in low-resource and unseen languages.

## 3. Reasoning, Preference Optimization, and Hallucination Mitigation

Advanced AV-LLMs integrate explicit training or inference-time techniques to address the logical complexity and reliability of multi-modal generation:

- **Preference optimization:** Multi-round DPO (MrDPO) [2506.15220] and process DPO (pDPO) [2502.11775] optimize LoRA-adapted policies on preference pairs decomposed into event-level or step-level feedback, leading to substantial reductions in missing/hallucinated content and higher QA accuracy. CAVPref extends DPO with explicit audio/video calibration terms and distributional robustness to counter bias toward a dominant modality [2501.02135].
- **Hallucination suppression:** AVCD [2505.20862] introduces trimodal contrastive decoding, masking non-dominant modalities based on attention statistics and combining logits from multiple corrupted/uncorrupted modality passes to penalize hallucinated generations. This yields significant gains on hallucination-focused benchmarks such as AVHBench.
- **Modality bias correction:** Fork-Merge Decoding (FMD) [2505.20873] forks audio and video tokens through early decoder layers, merging them for joint reasoning in upper layers by a weighted sum, rectifying modality under-utilization. Dolphin applies unpaired mixed training and diversity loss, while AVU integrates rejection-tuning to further limit hallucination [2504.02061].
- **Efficiency and scaling:** AccKV [2511.11106] replaces naive selective KV caching with layer-adaptive focusing and cross-calibration, minimizing memory/FLOP requirements by evicting or merging less-attended tokens dynamically per layer.

## 4. Key Empirical Findings and Ablation Insights

Extensive head-to-head comparisons across benchmarks and ablation studies clarify the incremental and synergistic gains of architectural and training advances:

| Model/Technique         | AVQA-Hard Accuracy | Music-AVQA-Hard | AVHBench Gain vs. Base | Video Captioning Error Reduction |
|------------------------|--------------------|-----------------|------------------------|----------------------------------|
| LLaVA-AV-SSM + audio+Mamba [2509.17901]    | 71.6% (+4.5pp)  | 36.8% (+1.2pp) | –                      | –                                |
| AVCD (contrastive decoding) [2505.20862]     | –                | –               | +6–11%                 | –                                |
| video-SALMONN 2 + MrDPO [2506.15220]         | –                | –               | –                      | −28% vs. base                    |
| video-SALMONN-o1 + pDPO [2502.11775]         | +3–8% (over base)| –               | –                      | –                                |
| CAVPref [2501.02135]                        | –                | –               | +20–30% across tasks   | –                                |

- **Ablation on fusion strategies:** Interleaved merging and bidirectional cross-attention outperform naive concatenation (Dolphin), while fork-merge and Mamba compression outperform non-interleaving or uncompressed sequences.
- **Failure modes:** Audio hallucinations are prevalent if audio signals are ignored (32% hallucination rate [2401.09774]); bag-of-words shortcutting persists in compositional QA tasks. Modal bias in attention (e.g., 70% to video) is empirically corrected by FMD or AccKV.
- **Emergent abilities and robust gains:** video-SALMONN family models show emergent speech-visual co-reasoning (e.g., lip-reading, cross-modal identification). Dolphin and AVTrustBench studies reveal that robust preference optimization yields large gains on compositional and adversarial subtasks.

## 5. Efficiency, Scalability, and Practical Model Deployment

Addressing the computational cost of long audio-video streams, especially in long-form video or conversational settings, has driven technical optimization:

- **Token compression:** Causal SSM (Mamba) in LLaVA-AV-SSM compresses 25 Hz audio into 1 Hz token streams, permitting hour-long video inference within feasible memory bounds [2509.17901].
- **KV cache management:** AccKV’s layer-adaptive focusing and cross-calibration reduce KV cache memory by up to 90% and total inference latency by ~600 ms per 1k tokens, with <2% accuracy loss on standard benchmarks [2511.11106].
- **Plug-and-play strategies:** Both AVCD and Fork-Merge Decoding are training-free and can be ablated in existing architectures without modification, supporting efficient inference and experimentation across proprietary or frozen model weights [2505.20862][2505.20873].
- **Modality-agnostic extensions:** Techniques for dynamic fusion, caching, and preference calibration generalize to additional modalities and tri-modal or higher scenarios, including speech, sensor, and text channels [2511.11106][2504.02061].

## 6. Open Challenges, Limitations, and Future Directions

Despite robust advances, AV-LLMs face critical bottlenecks in their path to human-level multimodal comprehension:

- **Benchmark limitations:** Most “video understanding” benchmarks can still be solved by visual-only inference; genuine audio sensitivity is only exposed by filtering single-frame answerable items (AVQA-Hard, Music-AVQA-Hard) [2509.17901]. AVTrustBench and RivaBench set a new standard by directly targeting adversarial, compositional, and modality-ablation settings [2501.02135][2502.11775].
- **Hallucination and modality entanglement:** Audio hallucination rates remain high in AV-LLMs [2401.09774]; even advanced models must balance out contrastive decoding, attention re-weighting, or explicit calibration losses to avoid text-only shortcuts [2505.20862][2501.02135].
- **Data diversity and supervision:** Most instruction datasets are English-focused and limited in real-world conversational or non-Western contexts. Speech-heavy tasks (e.g., AVSR) still lag behind specialized speech models unless large-scale, language-agnostic corpora and adaptable romanizer-LLM pipelines are introduced [2503.06273].
- **Scalability and real-time adaptation:** End-to-end adaptation requires learning more scalable fusion and streaming strategies, variable context windowing, and real-time streaming AV processing (as in Dolphin’s planned real-time extension) [2504.02061].

A central implication is that further progress will rely on advances in dataset scope, compositional evaluation, dynamic fusion algorithms, and robust, modality-calibrated training objectives, complemented by scalable, modular architectures that can flexibly ingest and align rich, temporally extended audio-visual streams.

Source: https://www.emergentmind.com/topics/audio-video-large-language-models-av-llms