--- title: 'CAT: Multimodal Clue Aggregator for AVQA' url: https://www.emergentmind.com/topics/cat type: topic --- # CAT: Multimodal Clue Aggregator for AVQA CAT is a multimodal large language model designed to answer questions in dynamic audio-visual scenarios by explicitly gathering question-relevant clues from both modalities and by training the language model to prefer precise, non-ambiguous answers [2403.04640]. It targets Audio-Visual Question Answering (AVQA) settings in which the relevant evidence is distributed across time-varying visual regions and audio events, and it addresses two core limitations of prior audio-video MLLMs: simple bridging of global audio and video embeddings without low-level interaction with the question, and training on loosely aligned multimodal corpora that yields ambiguous responses and poor localization of specific audio or sound events. ## 1. Problem setting and design rationale Answering questions about rich, time-varying scenes with synchronized video and audio requires grounding the question in the relevant spatiotemporal visual regions and audio events, while avoiding vague or overly verbose responses that obscure the specific facts needed for AVQA [2403.04640]. CAT is introduced around three contributions. First, CAT adds a **clue aggregator** that conditionally attends to audio-video features given the question and distills them into compact, query-relevant clue tokens for the language model. This is intended to replace the common pattern in which global modality tokens are passed into an LLM with limited question-conditioned localization. Second, CAT is trained on a **mixed multimodal dataset**, including an audio-visual joint instruction dataset named **AVinstruct**, so that the model can be applied directly in audio-visual scenarios and can better model cross-semantic correlations. Third, CAT introduces **AI-assisted ambiguity-aware direct preference optimization (ADPO)**. In this stage, ambiguous outputs are rewritten into precise alternatives, and the model is retrained to favor the non-ambiguity response. The reported effect is improved localization of specific audio-visual objects and more concise answers. A plausible implication is that CAT treats audio-visual reasoning not only as multimodal fusion, but also as a precision-control problem: the model must identify the relevant evidence and then suppress ambiguity in the final textual response. ## 2. Architecture and clue aggregation CAT uses a frozen **ImageBind** encoder as a universal feature extractor for video and audio, and **LLaMA2-7B** as the LLM backbone with **LoRA adapters** for parameter-efficient tuning [2403.04640]. Given a video $V$ and audio $A$, ImageBind produces hidden features $$ h_v = \mathrm{ImageBind}(V) \in \mathbb{R}^{T \times d_h},\qquad h_a = \mathrm{ImageBind}(A) \in \mathbb{R}^{1 \times d_h}, $$ where $T$ is the compressed temporal length of the video. Two linear projectors map these into visual and audio tokens in the LLM input space, $x^{vid} \in \mathbb{R}^{1\times 4096}$ and $x^{aud} \in \mathbb{R}^{1\times 4096}$. The core architectural novelty is the **clue aggregator (CA)**. It operates in two steps. In the first step, a shared-parameter perceiver mixes question tokens with modality features through self-attention, cross-attention, and feedforward layers. With question embedding $h_t$, the forward-order block performs question-aware localization, $$ \mathcal{B}_1(h_t; X) = \mathrm{XA}\!\left(h_t,\, \mathrm{FFN}\big(\mathrm{SA}(X)\big)\right), $$ and the reverse-order block consolidates modality representations, $$ \mathcal{B}_2\big(X; \mathcal{B}_1(h_t;X)\big) = \mathrm{SA}\!\left(\mathrm{FFN}\!\Big(\mathrm{XA}\big(\mathrm{FFN}(\mathrm{SA}(X)),\ \mathcal{B}_1(h_t;X)\big)\Big)\right). $$ Applying these blocks yields question-aware features $h_v'$ and $h_a'$. The audio and video perceivers share parameters to promote cross-modal associations. In the second step, CAT uses a **Q-Former** with $K=48$ learnable queries to aggregate the question-aware features into $K$ clue tokens. After a clues projector, the result is $$ x^{cue} \in \mathbb{R}^{48 \times 4096}. $$ Conceptually, the aggregation weights over video time steps and audio events are query-conditioned: $$ \alpha_t \propto \exp\big(\phi(q)^\top \psi(h_{v,t}')\big),\qquad \alpha_e \propto \exp\big(\phi(q)^\top \psi(h_{a,e}')\big). $$ These clue tokens summarize compact, question-relevant evidence for the LLM. Fusion is performed explicitly at the prompt level: ```text USER: …question… x^{cue} ``` All three token branches plus text are fed to the frozen LLaMA2-7B backbone with LoRA adapters enabled. CAT does not introduce an explicit spatiotemporal grounding loss; localization emerges from the question-conditioned attention and the preference learning that penalizes ambiguous responses. ## 3. Training data, instruction tuning, and ADPO Training is organized into three stages [2403.04640]. In **Stage I**, CAT performs pre-alignment with frozen ImageBind and frozen LLaMA2. The visual projector is trained on **WebVid 2.5M**, and the audio projector is trained on **WavCaps**. The reported setup uses AdamW with cosine learning-rate decay and warm-up, learning rate $2\mathrm{e}{-5}$, batch size 128, and 1 epoch. In **Stage II**, CAT performs audio-visual joint instruction tuning. The central resource is **AVinstruct**, collected from raw videos with audio from **YouTube** and **VGGSound**. QA pairs are adapted from the training splits of **Music-AVQA** and **AVQA**; **BLIP2** and **Whisper** produce video and audio descriptions; **GPT** synthesizes question-guided audio-visual descriptions to strengthen alignment and reasoning. During this stage, projectors are frozen, while the clue aggregator and LoRA adapters are fine-tuned jointly on AVinstruct and **100k video instruction samples** from Video-ChatGPT’s collection. The reported LoRA configuration is $r=64$, $\alpha=128$, with learning rate $2\mathrm{e}{-5}$, batch size 128, and 1 epoch. The third stage is **Ambiguity-aware Direct Preference Optimization (ADPO)**. The trained model is run on the instruction-tuning training set to collect ambiguous outputs, which are treated as negative responses; GPT rewrites them into precise, concise positive responses that preserve the original semantics. CAT then retrains the LoRA parameters with classic DPO plus a small supervised fine-tuning term to stabilize cases where the positive and negative responses are close. The paper reports $\beta=0.1$ and $\lambda=0.1$ by default; ADPO training uses LoRA only with $r=64$, $\alpha=16$, learning rate $4\mathrm{e}{-6}$, batch size 1, and 1 epoch. The practical training configuration is also unusually compact in hardware terms: all phases are reported on **1× NVIDIA A100 GPU**. ## 4. Evaluation and benchmark results CAT is evaluated on video-based text generation, zero-shot video QA, closed-ended AVQA, and open-ended AVQA [2403.04640]. The reported results emphasize that the main gains are strongest on audio-visual tasks, especially AVQA. The video-centric and AVQA results can be summarized as follows. | Benchmark | CAT result | Comparison reported | |---|---:|---| | Video-ChatGPT metrics | Cr=61.6, De=62.0, Ct=69.8, Te=56.2, Cs=57.8 | Outperforms prior 7B models such as Video-ChatGPT and VISTA-LLaMA | | MSRVTT-QA | Acc=62.1, Score=3.5 | Exceeds VISTA-LLaMA at Acc=60.5, Score=3.3 | | ActivityNet-QA | Acc=50.2, Score=3.5 | Best among listed 7B baselines | | Music-AVQA (fine-tuned) | overall=84.3, audio=84.9, visual=86.1, audio-visual=83.2 | Surpasses VAST 80.7 and CAD 78.2 | | AVQA multiple-choice (fine-tuned) | Acc=92.0 | Outperforms PSTP-Net 90.2 and HCRN+HAVF 89.0 | | Music-AVQA (zero-shot) | CAT-7B=48.6 | Higher than ChatBridge-13B 47.6 and OneLLM-7B 43.0 | For open-ended AVQA, the gains remain consistent. | Benchmark | CAT result | Comparison reported | |---|---:|---| | AVSD | CIDEr=79.0 | Higher than OneLLM-7B 74.5 and ChatBridge-13B 75.4 | | VALOR | CIDEr=32.4 | Higher than OneLLM-7B 29.2 and ChatBridge-13B 24.7 | One operationally important result is that CAT reaches these audio-visual gains with **only ~5.8M trainable parameters via LoRA** in Music-AVQA fine-tuning. This suggests that the reported improvement is attributed primarily to the clue-aggregation and preference-optimization design rather than to large-scale backbone modification. ## 5. Ablations, qualitative behavior, and interpretation The ablation study isolates the importance of the clue stream and the ambiguity-aware optimization [2403.04640]. On Video-ChatGPT metrics, **clues alone** already outperform plain audio+video input: clues alone achieve **Cr=58.6, De=57.4, Te=55.8**, whereas plain audio+video gives **Cr=40.6, De=44.8, Te=35.2**. Combining all three branches is best, with **Cr=61.6, De=59.0, Te=56.2**. This directly supports the claim that question-conditioned clue extraction is not merely redundant with raw audio and video tokens. Internal ablations on the clue aggregator show that removing visual attention hurts more than removing audio attention, highlighting vision’s richer context, and that using only $\mathcal{B}_1$ degrades performance relative to the full $\mathcal{B}_1+\mathcal{B}_2$ design. The model therefore depends both on question-aware localization and on subsequent representation consolidation. ADPO also generalizes beyond CAT itself. On AVSD, **Video-LLaMA** improves from **CIDEr 63.7 to 69.2** with ADPO, while **CAT** improves from **74.8 to 79.0**. This indicates that the ambiguity-aware preference scheme is not tied exclusively to the clue aggregator, though CAT obtains the strongest reported result when both components are combined. The qualitative analysis is consistent with the quantitative pattern. CAT resolves confusion in multi-sound scenes, such as counting sounding instruments, and in visually cluttered indoor environments by focusing on question-relevant regions and audio events. In side-by-side examples, CAT identifies **“bagpipes”** where baselines produce vague descriptions, and it answers **“How many instruments are sounding?”** where baselines fail. The paper’s interpretation is that CAT improves both localization and answer specificity, producing short, accurate answers without extraneous text. ## 6. Limitations, reproducibility, and applications The reported limitations are concrete and mostly tied to sensing quality, temporal compression, and supervision regime [2403.04640]. If audio is noisy or visual objects are heavily occluded, the perceiver may mis-attend, degrading localization and precision. Long videos and fast motion remain difficult because temporal compression plus limited visual tokens can miss fine-grained temporal events; the paper suggests that increasing $K$ or adding stronger temporal modeling could help. The data construction strategy also introduces domain considerations. AVinstruct is derived from **web data (YouTube, VGGSound)** and task training sets such as **Music-AVQA** and **AVQA**. This suggests that distribution shifts may induce errors, and that GPT rewriting in ADPO may inherit its own biases. A further limitation is the absence of explicit grounding supervision: grounding emerges from attention and preference learning rather than from a dedicated spatiotemporal grounding loss. Reproducibility is supported through released code and data at **https://github.com/rikeilong/Bay-CAT**. The repository provides model code, training recipes, and the collected AVinstruct instructions. The practical inference workflow is fixed: encode video and audio with ImageBind, run the clue aggregator to obtain $x^{cue}$, construct the prompt with ``, `