Fork-Merge Decoding for AV-LLMs

• FMD is an inference-time algorithm that addresses modality bias by separating early unimodal reasoning from later joint fusion.
• It employs a fork phase for isolated audio and visual processing, followed by a merge phase that combines these modalities without retraining.
• Empirical results show FMD enhances cross-modal accuracy and reduces hallucinations using attention-guided fusion in both token-wise and channel-wise settings.

Fork-Merge Decoding (FMD) is an inference-time algorithm designed to address modality bias in audio-visual LLMs (AV-LLMs). Modality bias arises when joint processing of audio and visual inputs in existing AV-LLMs causes the model to over-rely on the most informative or easily learned modality (often visual), leading to suboptimal utilization of complementary cues and, in some cases, hallucinations. FMD introduces modality-specific reasoning by initially processing audio-only and video-only inputs through the early decoder layers—a stage termed the "fork phase"—and subsequently merging the resulting hidden states for joint reasoning in the later layers, known as the "merge phase." This approach requires no additional training or architectural modification and is compatible with a variety of decoding paradigms, improving balanced multimodal understanding in AV-LLMs such as VideoLLaMA2 and video-SALMONN without decreasing inference efficiency (Jung et al., 27 May 2025).

1. Motivation and Theoretical Foundations

Modern AV-LLMs typically concatenate audio and video features alongside text tokens, jointly encoding these modalities from the decoder's input layer onward. This joint fusion, while efficient for unified multimodal reasoning, often induces a bias: the model prefers the "dominant" or most discriminative modality, thus compromising the fidelity of contributions from the other modality. The consequence is an imbalanced feature attribution and potential hallucinations in tasks requiring nuanced audio-visual integration.

Fork-Merge Decoding addresses this challenge by enforcing unimodal reasoning prior to fusion. By isolating modalities during the initial decoder layers, FMD forces the model to extract salient information from both audio and video streams independently. The subsequent merge enables the model to leverage cross-modal complementarities in higher layers, resulting in more robust and balanced decision making. This functional separation requires no retraining or architecture changes and operates entirely at inference via input manipulation and strategically timed merging of states (Jung et al., 27 May 2025).

2. Algorithmic Details

Inputs and Parameters

• Decoder: Pretrained AV-LLM decoder φ with $L_{\text{total}}$ layers.
• Inputs: Video-frame embeddings $x^{(v)}$ (length $M$), audio embeddings $x^{(a)}$ (length $N$), and text embeddings $x^{(l)}$ (length $L$).
• Fork depth: $L_{\text{fork}}$, number of early layers assigned to unimodal reasoning (e.g., 5 for VideoLLaMA2, 8 for video-SALMONN).
• Fusion weight: $\alpha$ (e.g., 0.8; optionally attention-guided).

Procedural Steps (Token-wise Fusion Case)

1. Input Construction: Concatenate embeddings: $x = x^{(v)} \oplus x^{(a)} \oplus x^{(l)}$.
2. Masked Inputs: Create two variants:
   • $x^{(v)}$0
   • $x^{(v)}$1
3. Fork Phase: Independently process both masked sequences through early decoder layers:
   • $x^{(v)}$2
   • $x^{(v)}$3
4. Fusion Weight Calculation (Optional): Attention-guided $x^{(v)}$4 derived from last-layer attention distributions:
   • $x^{(v)}$5
   • $x^{(v)}$6
   • $x^{(v)}$7
5. Merge Phase: Form $x^{(v)}$8 by combining corresponding token positions:
   • For $x^{(v)}$9 (video): $M$0
   • For $M$1 (audio): $M$2
   • For $M$3 (text): $M$4
6. Joint Decoding: Continue decoding from $M$5 to $M$6 layers:
   • $M$7
   • Output logits for next-token prediction.

For channel-wise fusion (video-SALMONN), $M$8 if $M$9 (audio-visual), and $x^{(a)}$0 for text tokens.

3. Mathematical Formalization

Let $x^{(a)}$1 denote the concatenated input. Masked versions are $x^{(a)}$2 and $x^{(a)}$3.

Early layer outputs:

• $x^{(a)}$4
• $x^{(a)}$5

Attention-guided fusion (Eqns. 5–7 in the original paper):

• $x^{(a)}$6
• $x^{(a)}$7
• $x^{(a)}$8

Merge equations:

• For $x^{(a)}$9: $N$0
• For $N$1: $N$2
• For $N$3: $N$4

Final output:

• $N$5 logits $N$6 next token.

4. Hyperparameterization and Ablation Findings

FMD depends primarily on two key hyperparameters: the fork depth $N$7 and the fusion weight $N$8. Their empirical tuning determines the algorithm’s balance between unimodal and multimodal integration.

• Fork depth ($N$9): For VideoLLaMA2 ($x^{(l)}$0), $x^{(l)}$1 was optimal. For video-SALMONN ($x^{(l)}$2), $x^{(l)}$3 yielded best results. Deeper merging (higher $x^{(l)}$4) increases audio attention but can reduce video-task performance; too shallow merging leads to insufficient cross-modal fusion.
• Fusion weight ($x^{(l)}$5): Set to 0.8, derived as a rounded mean over 100 AVHBench samples. Ablation indicates exclusion strategies ($x^{(l)}$6 or $x^{(l)}$7) provide modest gains or none, uniform averaging ($x^{(l)}$8) decreases performance, while attention-guided fusion consistently yields improvements.
• Fusion type: Both token-wise (VideoLLaMA2) and channel-wise (video-SALMONN) fusion are supported. Optimal strategy depends on the model architecture.

Table: Summary of Key Hyperparameter Choices and Effects

Hyperparameter	Best Value (VideoLLaMA2)	Best Value (video-SALMONN)
Fork Depth ($x^{(l)}$9)	5	8
Fusion Weight ($L$0)	0.8 (attention-guided)	0.8 (attention-guided)

5. Quantitative Performance and Comparative Results

FMD consistently improves accuracy and balance across multiple task categories. The following summarizes the reported improvements on benchmark datasets using VideoLLaMA2 (token-wise fusion) and video-SALMONN (channel-wise fusion):

Task	VideoLLaMA2 Vanilla	VideoLLaMA2 FMD	video-SALMONN Vanilla	video-SALMONN FMD
AVQA	82.46	82.74 (+0.28)	36.36	36.89 (+0.53)
MUSIC-AVQA	81.30	81.57 (+0.27)	44.48	44.78 (+0.30)
A→V	80.02	80.34 (+0.32)	68.69	71.05 (+2.36)
V→A	77.03	77.70 (+0.67)	62.39	65.31 (+2.92)
AV-match	57.75	58.89 (+1.14)	49.46	51.49 (+2.03)
Captioning	2.84	2.94 (+0.10)	1.83	1.89 (+0.06)

Relative to other decoding schemes (such as DoLa, VCD, SID), FMD zero-out masking achieves the strongest overall gains, particularly in cross-modal transfer tasks (e.g., A→V, V→A). Input perturbation ablation shows that zero-masking (rather than additive noise) better isolates modalities and yields superior performance (Jung et al., 27 May 2025). Inference efficiency is maintained ($L$1 tokens/sec), with FMD comparable to or faster than many baseline schemes.

6. Layerwise Attention, Fusion Strategies, and Empirical Insights

Analysis of layerwise attention reveals that the point of merging (i.e., $L$2) modulates the model’s modality attribution: deeper merges bias attention towards audio, while early merges favor video. FMD narrows the audio-video attention gap, promoting balanced feature exploitation. Attention-guided fusion outperforms both hard exclusion and uniform averaging approaches, as demonstrated empirically.

Ablation across merge layers indicates that $L$3 in VideoLLaMA2 achieves the best trade-off for both unimodal and multimodal performance. FMD offers robust plug-and-play capabilities, yielding consistent accuracy improvements across varying model, task, and input settings without adverse effects on generation speed or architectural complexity (Jung et al., 27 May 2025).

7. Applications and Implications

FMD is applicable wherever balanced utilization of audio and visual cues is critical, such as audio-visual question answering, cross-modal reasoning, and AV-matching tasks in video-language understanding. Its inference-only, training-free nature makes it especially attractive for deployment settings where retraining or architectural modification is impractical. The approach generalizes across AV-LLMs with both token-wise and channel-wise fusion pipelines and offers empirically validated improvements on standard benchmarks. A plausible implication is that similar fork-merge paradigms could extend to other multimodal domains, mitigating bias and enhancing the interpretability of large-scale fusion models (Jung et al., 27 May 2025).