SightSound-R1: Audio Reasoning Distillation

• SightSound-R1 is a cross-modal distillation framework that transfers advanced chain-of-thought reasoning from vision-language models to audio-language models.
• It leverages audio-visual question answering datasets and rigorous audio-grounded fact verification to filter hallucinated or inconsistent reasoning traces.
• The framework combines supervised fine-tuning using LoRA with GRPO reinforcement learning, yielding improved accuracy and interpretability on AVQA benchmarks.

SightSound-R1 is a cross-modal reasoning distillation framework enabling the transfer of advanced stepwise reasoning capabilities from state-of-the-art vision–LLMs (LVLMs) to large audio–LLMs (LALMs). The motivating insight is that reasoning in auditory scenes is bottlenecked by scarce large-scale chain-of-thought (CoT) audio datasets, while LVLMs are already proficient at compositional and causal inference over multimodal inputs. SightSound-R1 addresses this limitation by exploiting audio–visual question answering (AVQA) datasets and constructing a pipeline that extracts, verifies, and distills audio-focused reasoning traces from visually grounded teachers into auditory students.

1. Architectural Overview

SightSound-R1 operates in three sequential stages, each indexed by explicit mechanisms and associated mathematical formalism.

1. Teacher Chain-of-Thought Generation and Test-Time Scaling
   • An LVLM (e.g., Qwen2.5-VL-32B) receives an audio-specific prompt alongside silent video and a question. The prompt is designed to orient the teacher's CoT synthesis toward audio-relevant aspects, despite its inability to access the true audio signal.
   • Multiple independent CoT reasoning trajectories $\{\mathcal{R}_i\}_{i=1}^n$ are sampled for each video–question pair using self-consistency techniques (e.g., temperature scaling, diverse sampling).
   • For each trajectory, the inferred answer is extracted. Only when all sampled CoTs produce unanimous answers—i.e., $|\mathrm{unique}(\mathcal{A})| = 1$—is the reasoning retained:

   $$|\mathrm{unique}(\mathcal{A})| = 1 \implies (v, q, \mathcal{R}) \in \mathcal{D}_{\mathrm{reason}}$$

• This stage filters out inconsistent or noisy teacher rationales, maximizing reliability for downstream transfer.

1. Audio-Grounded Fact Verification (AGFV)
   • The teacher’s CoT traces, being visually derived, can hallucinate or misattribute audio properties. Thus, each trace $r \in \mathcal{R}$ is subjected to audio-grounded validation against the true audio segment $a$ by a pre-trained LALM verifier (e.g., GPT-4o-audio).
   • The AGFV is a binary operator: $$\mathrm{AGFV}(r, a) \in \{\mathrm{accept}, \mathrm{reject}\}$$. Only factually warranted reasoning is retained, aggregated as:

   $$\mathcal{D}_{\mathrm{FC}} = \{(a, q, r): r \in \mathcal{R}, \mathrm{AGFV}(r, a) = \mathrm{accept}\}$$

• This procedure corrects hallucinations and aligns teacher rationales to actual auditory content.

1. Student LALM Training: Supervised Fine-Tuning (SFT) and GRPO
   • The LALM student (e.g., Qwen2-Audio-7B-Instruct) is trained on the fact-checked corpus via SFT, optimizing:

   $$\mathcal{L}_{\mathrm{SFT}}(\theta_{\mathrm{LoRA}}) = \mathbb{E}_{(x,y) \sim \mathcal{D}_{\mathrm{FC}}}\left[-\sum_{t=1}^{|y|}\log \pi_{\theta_{\mathrm{base}} \oplus \theta_{\mathrm{LoRA}}}(y_t | x, y_{<t})\right]$$

• This step tunes only low-rank adaptation (LoRA) parameters, keeping the student’s base model weights frozen.
• Subsequently, Group Relative Policy Optimization (GRPO), a reinforcement learning (RL)-based algorithm, refines student outputs with respect to answer correctness and CoT format tags. For sampled group outputs $\{o_i\}_{i=1}^G$, each receives a normalized advantage signal:

  $$\hat{A}_{i,t} = \frac{r_i - \mathrm{mean}(r)}{\mathrm{std}(r)}$$

• The policy objective incorporates a clipped ratio and KL regularization:

  $$J_{\mathrm{GRPO}}(\theta) = \mathbb{E}_{q \sim P(Q), \{o_i\} \sim \pi_{\theta_{\mathrm{old}}}}\left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \min \left\{ \frac{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(o_{i,t}|q,o_{i,<t})} \hat{A}_{i,t}, \mathrm{clip}\left(\frac{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(o_{i,t}|q,o_{i,<t})}, 1-\epsilon, 1+\epsilon \right) \hat{A}_{i,t} \right\} - \beta D_{\mathrm{KL}}\left[\pi_{\theta} || \pi_{\mathrm{ref}}\right] \right]$$

• Rewards $r_i$ reflect both answer correctness and proper CoT format (presence of > … and <answer>…</answer> tags).

2. Cross-Modal Reasoning Pipeline Characteristics

SightSound-R1 explicitly targets the transfer of epistemic reasoning skills from visual to auditory domains by leveraging chain-of-thought supervision that would otherwise be unavailable for audio-only models. The CoT traces produced by LVLMs typically exhibit compositional, causal, and temporal reasoning steps relevant to AVQA tasks (e.g., inferring the number of sounds, identifying event causality, or explaining sequential dynamics). The audio-focused prompts steer the LVLM teacher to produce traces attentive to sound-related phenomena, thus maximizing transfer utility.

The AGFV stage is crucial given that LVLM rationales may conflate visual events with presumed auditory cues absent in the actual signal; the LALM verifier rectifies such mismatches. A plausible implication is that further improvements could be made by iterating AGFV using an ensemble of audio verifiers with different strengths (e.g., pretrained vs. finetuned LALMs), although this was not reported as an experiment.

The RL-finetuning stage with GRPO moves beyond classical SFT by simultaneously rewarding response structure and content, which empirically incentivizes interpretable, multi-step explanations in the LALM student.

3. Empirical Performance

Experimental results were reported on standard and challenging audio–visual question answering benchmarks, including MMAU Test-mini and MUSIC-AVQA. Key outcomes:

• On MMAU Test-mini, SightSound-R1 achieves 66.1% accuracy in Sound questions, outperforming baseline label-only distillation and pretrained LALM inference.
• On MUSIC-AVQA, the framework reaches 59.5% accuracy overall, with strong performance on temporal (62.7%) and comparative reasoning (63.3%) tasks.
• In all cases, SightSound-R1 surpasses both direct inference and pure label-distilled counterparts. The improvements are evident in tasks requiring multi-step, context-aware reasoning about auditory phenomena (e.g., “Does the rhythm increase after the bell rings?” or “Are there more voices before or after the violin starts?”).

These results demonstrate not only higher answer accuracy but also gains in reasoning interpretability and stepwise structure—critical for explainable AI and human-aligned auditory analysis.

4. Technical Deployment and Implementation Considerations

The SightSound-R1 pipeline is compatible with any LVLM teacher capable of audio-oriented chain-of-thought rationalization under silent video plus question prompts. In practice, Qwen2.5-VL-32B is used for CoT generation and Qwen2-Audio-7B-Instruct for LALM student distillation. The AGFV can employ existing audio LLMs (e.g., GPT-4o-audio) for assertive fact verification.

Training is resource-efficient for SFT, since only LoRA adapters are updated, minimizing the burden on base model storage and compute. RL finetuning (GRPO) introduces additional sample complexity due to candidate response generation and reward computation but aligns with contemporary policy optimization workflows.

The CoT selection strategy (self-consistency voting) requires evaluation of multiple teacher traces per sample, doubling or tripling the test-time cost on the teacher but reducing the need for human annotation and boosting precision of reasoning supervision for the student.

The design is directly extensible to other multimodal QA settings (e.g., text–audio, image–audio, or multi-turn dialog about auditory scenes) with only minimal modifications to the prompt engineering and verification modules.

5. Applications and Broader Significance

By enabling LALMs to acquire interpretable, stepwise auditory reasoning patterns, SightSound-R1 is significant for research in audio question answering, explainable auditory scene analysis, and multimodal reasoning transfer. The capability to reason about sound in sequential, compositional manner directly impacts domains such as:

• Automated audio-visual content analysis and retrieval,
• Assistive technologies for the hearing-impaired or audio scene annotation,
• Multi-agent auditory reasoning (e.g., robotics interpreting environmental sounds),
• Cross-modal explainability and diagnostic QA.

A key insight is that multimodal data can bootstrap reasoning capabilities in modalities lacking annotated reasoning supervision, with modality-bridging verification ensuring fidelity. This suggests further research into cross-modal distillation not only for reasoning but also for generation, detection, and localization tasks where one modality is more data-rich.

6. Limitations and Future Directions

Reported limitations include dependency on the reliability of teacher CoT outputs and the AGFV filter; hallucination correction is limited by verifier accuracy. Future work may focus on scaling to larger audio–visual corpora, exploring multi-turn dialog reasoning, adversarial settings, more granular CoT supervision, and integrating multimodal multi-teacher ensembles.

Extending SightSound-R1 to domains such as audio–text or audio–robotics reasoning, and iterating with human-in-the-loop fact verification, is a plausible direction. Optimizing the sampling and filtering procedure to balance diversity (essential for generalization) and answer consensus (crucial for supervision reliability) remains an open question.

---

In sum, SightSound-R1 establishes a scalable cross-modal distillation paradigm, effectively bridging the reasoning gap between vision- and audio-LLMs, and sets a precedent for principled multimodal knowledge transfer in the presence of data annotation bottlenecks (Wang et al., 19 Sep 2025).