CAT: Multimodal Clue Aggregator for AVQA

• CAT is a multimodal model that integrates audio and visual clues through a question-aware aggregation mechanism for precise audio-visual question answering.
• It employs a specialized clue aggregator to localize and condense relevant audio-video features, improving the accuracy of identified objects and events.
• The model uses ambiguity-aware direct preference optimization (ADPO) to refine outputs, resulting in concise and non-ambiguous answers in dynamic scenes.

CAT is a multimodal LLM designed to answer questions in dynamic audio-visual scenarios by explicitly gathering question-relevant clues from both modalities and by training the LLM to prefer precise, non-ambiguous answers (Ye et al., 2024). It targets Audio-Visual Question Answering (AVQA) settings in which the relevant evidence is distributed across time-varying visual regions and audio events, and it addresses two core limitations of prior audio-video MLLMs: simple bridging of global audio and video embeddings without low-level interaction with the question, and training on loosely aligned multimodal corpora that yields ambiguous responses and poor localization of specific audio or sound events.

1. Problem setting and design rationale

Answering questions about rich, time-varying scenes with synchronized video and audio requires grounding the question in the relevant spatiotemporal visual regions and audio events, while avoiding vague or overly verbose responses that obscure the specific facts needed for AVQA (Ye et al., 2024). CAT is introduced around three contributions.

First, CAT adds a clue aggregator that conditionally attends to audio-video features given the question and distills them into compact, query-relevant clue tokens for the LLM. This is intended to replace the common pattern in which global modality tokens are passed into an LLM with limited question-conditioned localization.

Second, CAT is trained on a mixed multimodal dataset, including an audio-visual joint instruction dataset named AVinstruct, so that the model can be applied directly in audio-visual scenarios and can better model cross-semantic correlations.

Third, CAT introduces AI-assisted ambiguity-aware direct preference optimization (ADPO). In this stage, ambiguous outputs are rewritten into precise alternatives, and the model is retrained to favor the non-ambiguity response. The reported effect is improved localization of specific audio-visual objects and more concise answers.

A plausible implication is that CAT treats audio-visual reasoning not only as multimodal fusion, but also as a precision-control problem: the model must identify the relevant evidence and then suppress ambiguity in the final textual response.

2. Architecture and clue aggregation

CAT uses a frozen ImageBind encoder as a universal feature extractor for video and audio, and LLaMA2-7B as the LLM backbone with LoRA adapters for parameter-efficient tuning (Ye et al., 2024). Given a video $V$ and audio $A$, ImageBind produces hidden features

$$h_v = \mathrm{ImageBind}(V) \in \mathbb{R}^{T \times d_h},\qquad h_a = \mathrm{ImageBind}(A) \in \mathbb{R}^{1 \times d_h},$$

where $T$ is the compressed temporal length of the video. Two linear projectors map these into visual and audio tokens in the LLM input space, $x^{vid} \in \mathbb{R}^{1\times 4096}$ and $x^{aud} \in \mathbb{R}^{1\times 4096}$.

The core architectural novelty is the clue aggregator (CA). It operates in two steps. In the first step, a shared-parameter perceiver mixes question tokens with modality features through self-attention, cross-attention, and feedforward layers. With question embedding $h_t$, the forward-order block performs question-aware localization,

$$\mathcal{B}_1(h_t; X) = \mathrm{XA}\!\left(h_t,\, \mathrm{FFN}\big(\mathrm{SA}(X)\big)\right),$$

and the reverse-order block consolidates modality representations,

$$\mathcal{B}_2\big(X; \mathcal{B}_1(h_t;X)\big) = \mathrm{SA}\!\left(\mathrm{FFN}\!\Big(\mathrm{XA}\big(\mathrm{FFN}(\mathrm{SA}(X)),\ \mathcal{B}_1(h_t;X)\big)\Big)\right).$$

Applying these blocks yields question-aware features $h_v'$ and $A$0. The audio and video perceivers share parameters to promote cross-modal associations.

In the second step, CAT uses a Q-Former with $A$1 learnable queries to aggregate the question-aware features into $A$2 clue tokens. After a clues projector, the result is

$A$3

Conceptually, the aggregation weights over video time steps and audio events are query-conditioned:

$A$4

These clue tokens summarize compact, question-relevant evidence for the LLM.

Fusion is performed explicitly at the prompt level: $$h_v = \mathrm{ImageBind}(V) \in \mathbb{R}^{T \times d_h},\qquad h_a = \mathrm{ImageBind}(A) \in \mathbb{R}^{1 \times d_h},$$8 All three token branches plus text are fed to the frozen LLaMA2-7B backbone with LoRA adapters enabled. CAT does not introduce an explicit spatiotemporal grounding loss; localization emerges from the question-conditioned attention and the preference learning that penalizes ambiguous responses.

3. Training data, instruction tuning, and ADPO

Training is organized into three stages (Ye et al., 2024). In Stage I, CAT performs pre-alignment with frozen ImageBind and frozen LLaMA2. The visual projector is trained on WebVid 2.5M, and the audio projector is trained on WavCaps. The reported setup uses AdamW with cosine learning-rate decay and warm-up, learning rate $A$5, batch size 128, and 1 epoch.

In Stage II, CAT performs audio-visual joint instruction tuning. The central resource is AVinstruct, collected from raw videos with audio from YouTube and VGGSound. QA pairs are adapted from the training splits of Music-AVQA and AVQA; BLIP2 and Whisper produce video and audio descriptions; GPT synthesizes question-guided audio-visual descriptions to strengthen alignment and reasoning. During this stage, projectors are frozen, while the clue aggregator and LoRA adapters are fine-tuned jointly on AVinstruct and 100k video instruction samples from Video-ChatGPT’s collection. The reported LoRA configuration is $A$6, $A$7, with learning rate $A$8, batch size 128, and 1 epoch.

The third stage is Ambiguity-aware Direct Preference Optimization (ADPO). The trained model is run on the instruction-tuning training set to collect ambiguous outputs, which are treated as negative responses; GPT rewrites them into precise, concise positive responses that preserve the original semantics. CAT then retrains the LoRA parameters with classic DPO plus a small supervised fine-tuning term to stabilize cases where the positive and negative responses are close. The paper reports $A$9 and $$h_v = \mathrm{ImageBind}(V) \in \mathbb{R}^{T \times d_h},\qquad h_a = \mathrm{ImageBind}(A) \in \mathbb{R}^{1 \times d_h},$$0 by default; ADPO training uses LoRA only with $$h_v = \mathrm{ImageBind}(V) \in \mathbb{R}^{T \times d_h},\qquad h_a = \mathrm{ImageBind}(A) \in \mathbb{R}^{1 \times d_h},$$1, $$h_v = \mathrm{ImageBind}(V) \in \mathbb{R}^{T \times d_h},\qquad h_a = \mathrm{ImageBind}(A) \in \mathbb{R}^{1 \times d_h},$$2, learning rate $$h_v = \mathrm{ImageBind}(V) \in \mathbb{R}^{T \times d_h},\qquad h_a = \mathrm{ImageBind}(A) \in \mathbb{R}^{1 \times d_h},$$3, batch size 1, and 1 epoch.

The practical training configuration is also unusually compact in hardware terms: all phases are reported on 1× NVIDIA A100 GPU.

4. Evaluation and benchmark results

CAT is evaluated on video-based text generation, zero-shot video QA, closed-ended AVQA, and open-ended AVQA (Ye et al., 2024). The reported results emphasize that the main gains are strongest on audio-visual tasks, especially AVQA.

The video-centric and AVQA results can be summarized as follows.

Benchmark	CAT result	Comparison reported
Video-ChatGPT metrics	Cr=61.6, De=62.0, Ct=69.8, Te=56.2, Cs=57.8	Outperforms prior 7B models such as Video-ChatGPT and VISTA-LLaMA
MSRVTT-QA	Acc=62.1, Score=3.5	Exceeds VISTA-LLaMA at Acc=60.5, Score=3.3
ActivityNet-QA	Acc=50.2, Score=3.5	Best among listed 7B baselines
Music-AVQA (fine-tuned)	overall=84.3, audio=84.9, visual=86.1, audio-visual=83.2	Surpasses VAST 80.7 and CAD 78.2
AVQA multiple-choice (fine-tuned)	Acc=92.0	Outperforms PSTP-Net 90.2 and HCRN+HAVF 89.0
Music-AVQA (zero-shot)	CAT-7B=48.6	Higher than ChatBridge-13B 47.6 and OneLLM-7B 43.0

For open-ended AVQA, the gains remain consistent.

Benchmark	CAT result	Comparison reported
AVSD	CIDEr=79.0	Higher than OneLLM-7B 74.5 and ChatBridge-13B 75.4
VALOR	CIDEr=32.4	Higher than OneLLM-7B 29.2 and ChatBridge-13B 24.7

One operationally important result is that CAT reaches these audio-visual gains with only ~5.8M trainable parameters via LoRA in Music-AVQA fine-tuning. This suggests that the reported improvement is attributed primarily to the clue-aggregation and preference-optimization design rather than to large-scale backbone modification.

5. Ablations, qualitative behavior, and interpretation

The ablation study isolates the importance of the clue stream and the ambiguity-aware optimization (Ye et al., 2024). On Video-ChatGPT metrics, clues alone already outperform plain audio+video input: clues alone achieve Cr=58.6, De=57.4, Te=55.8, whereas plain audio+video gives Cr=40.6, De=44.8, Te=35.2. Combining all three branches is best, with Cr=61.6, De=59.0, Te=56.2. This directly supports the claim that question-conditioned clue extraction is not merely redundant with raw audio and video tokens.

Internal ablations on the clue aggregator show that removing visual attention hurts more than removing audio attention, highlighting vision’s richer context, and that using only $$h_v = \mathrm{ImageBind}(V) \in \mathbb{R}^{T \times d_h},\qquad h_a = \mathrm{ImageBind}(A) \in \mathbb{R}^{1 \times d_h},$$4 degrades performance relative to the full $$h_v = \mathrm{ImageBind}(V) \in \mathbb{R}^{T \times d_h},\qquad h_a = \mathrm{ImageBind}(A) \in \mathbb{R}^{1 \times d_h},$$5 design. The model therefore depends both on question-aware localization and on subsequent representation consolidation.

ADPO also generalizes beyond CAT itself. On AVSD, Video-LLaMA improves from CIDEr 63.7 to 69.2 with ADPO, while CAT improves from 74.8 to 79.0. This indicates that the ambiguity-aware preference scheme is not tied exclusively to the clue aggregator, though CAT obtains the strongest reported result when both components are combined.

The qualitative analysis is consistent with the quantitative pattern. CAT resolves confusion in multi-sound scenes, such as counting sounding instruments, and in visually cluttered indoor environments by focusing on question-relevant regions and audio events. In side-by-side examples, CAT identifies “bagpipes” where baselines produce vague descriptions, and it answers “How many instruments are sounding?” where baselines fail. The paper’s interpretation is that CAT improves both localization and answer specificity, producing short, accurate answers without extraneous text.

6. Limitations, reproducibility, and applications

The reported limitations are concrete and mostly tied to sensing quality, temporal compression, and supervision regime (Ye et al., 2024). If audio is noisy or visual objects are heavily occluded, the perceiver may mis-attend, degrading localization and precision. Long videos and fast motion remain difficult because temporal compression plus limited visual tokens can miss fine-grained temporal events; the paper suggests that increasing $$h_v = \mathrm{ImageBind}(V) \in \mathbb{R}^{T \times d_h},\qquad h_a = \mathrm{ImageBind}(A) \in \mathbb{R}^{1 \times d_h},$$6 or adding stronger temporal modeling could help.

The data construction strategy also introduces domain considerations. AVinstruct is derived from web data (YouTube, VGGSound) and task training sets such as Music-AVQA and AVQA. This suggests that distribution shifts may induce errors, and that GPT rewriting in ADPO may inherit its own biases. A further limitation is the absence of explicit grounding supervision: grounding emerges from attention and preference learning rather than from a dedicated spatiotemporal grounding loss.

Reproducibility is supported through released code and data at https://github.com/rikeilong/Bay-CAT. The repository provides model code, training recipes, and the collected AVinstruct instructions. The practical inference workflow is fixed: encode video and audio with ImageBind, run the clue aggregator to obtain $$h_v = \mathrm{ImageBind}(V) \in \mathbb{R}^{T \times d_h},\qquad h_a = \mathrm{ImageBind}(A) \in \mathbb{R}^{1 \times d_h},$$7, construct the prompt with <Q></Q>, <video>, <audio>, and <clues>, and generate the final answer with LLaMA2-7B and LoRA adapters enabled.

The application scope given in the paper includes audio-visual question answering, video-grounded dialogue, multimodal retrieval and captioning, and surveillance/event analysis where precise identification of objects and sounds is critical. A plausible implication is that CAT is most useful in settings where ambiguity is itself a system failure mode: not merely answering about a scene, but answering in a form that preserves object-level and event-level specificity.