CounterFlow: A Two-Phase Inference-Time Sampling for Counterfactual Video Foley Generation

Abstract: We investigate Counterfactual Video Foley Generation, which aims to adopt a sound-source identity that contradicts the visual evidence while remaining temporally synchronized to a silent video. Existing Video&Text-to-Audio (VT2A) models struggle with this, often remaining anchored to the visually implied sound source when video and text contents disagree. We present ConterFlow, an inference-time dual-phase sampling scheme for pretrained flow-matching VT2A models. Phase 1 builds a video-derived temporal structure while suppressing the visually implied source; Phase 2 drops video conditioning to focus entirely on shaping audio timbre toward the target prompt. ConterFlow substantially improves counterfactual Video Foley generation compared to naive negative prompting and state-of-the-art baselines. To evaluate replacement quality, we propose a metric leveraging a text-audio co-embedding space to measure both target-prompt evidence and residual visually implied source leakage. Video demonstrations and code are available at https://gyubin-lee.github.io/counterflow-demo/

• The paper introduces a two-phase sampling approach that first establishes temporal alignment through video conditioning and then controls sound identity via negative prompting.
• It employs decomposed guidance separating video- and text-conditioned gradients to effectively suppress visually implied source sounds, achieving a 92% replacement success rate.
• Quantitative and qualitative results on VGGSound-Sparse demonstrate state-of-the-art performance in audio quality, synchronization, and targeted identity replacement.

CounterFlow: Two-Phase Inference-Time Sampling for Counterfactual Video Foley Generation

Problem Formulation and Motivation

Counterfactual Video Foley Generation demands a generative model that, given a silent video depicting an event, a source prompt describing the visible sound source, and a conflicting target prompt, synthesizes audio that is temporally aligned to the video while displaying the sound identity characterized by the target prompt rather than the visually implied source. Standard VT2A (Video{content}Text-to-Audio) systems, despite conditioning on both video and text, exhibit a pronounced bias towards the sound source inferred from visual content, restricting controllable counterfactual audio synthesis. Such limitations hamper creative applications in post-production, virtual environments, and interactive multimedia, where the ability to impose arbitrary, even contradictory sound sources, is crucial.

Methodology: Two-Phase Guided Sampling

The proposed CounterFlow scheme decomposes the generative process of pretrained flow-matching VT2A models into two discrete inference-time phases, each employing distinct conditioning and guidance strategies to separate temporal alignment from source identity control.

During Phase 1, CounterFlow applies video conditioning to establish a temporally aligned structure based on the video input but introduces a decomposed guidance approach. Inspired by compositional guidance [liu2022composable], the velocity field is steered by: (1) extracting temporal structure from the video with the text prompt masked; (2) explicitly promoting the target text identity and simultaneously suppressing the source identity through a negative prompt.

Figure 1: CounterFlow’s two-phase scheme first creates a temporally aligned structure with decomposed guidance, then removes video conditioning and uses negative text prompting for identity control.

Transitioning to Phase 2 (after a predefined sampling step), video conditioning is dropped, and the model employs negative prompting with the source text— forcing generation to match only the target sound identity, using the previously established timing blueprint. The process is agnostic to backbone architecture and can be applied to any compatible pretrained VT2A flow-matching model without requiring retraining.

Mathematically, Phase 1's guidance combines separate video- and text-conditioned gradients, while Phase 2 employs standard classifier-free guidance (CFG) with the source prompt as a negative constraint. The guidance weights and the transition step are tunable and influence the trade-off between alignment and successful source replacement.

Quantitative and Qualitative Results

Experiments leverage the VGGSound-Sparse Clean subset, constructing all possible source-target prompt conflict pairs (4,961 triplets), enforcing strong identity clashes. The main comparison evaluates FAD, IS, CLAP, and DeSync for audio quality, identity matching, and synchronization, respectively. Notably, two new evaluation metrics are introduced to directly index counterfactual success: $\Delta$FLAM (the frame-level, prompt-contrastive metric quantifying the difference in evidence for target vs. source sound identities in the generated waveform using FLAM [wu2025flam]), and its positive-ratio to measure replacement success rate.

CounterFlow achieves the highest $\Delta$FLAM (0.2641), indicating strong suppression of visually implied sound identity, and a positive-$\Delta$FLAM ratio of 0.92, corresponding to 92% successful replacements, while maintaining state-of-the-art FAD (23.55), IS (7.92), and CLAP (0.2840) scores. Baselines such as CAFA, even with added negative prompting, are unable to avoid generating mixed or residual source evidence, exposing deficiencies in naive integration strategies. ReWaS is especially weak in source suppression.

Qualitative visualizations further reinforce these findings: the source-class FLAM activations are consistently low during the event, while the target-class activations are synchronized and high, resulting in authentic counterfactual perception.

Figure 2: FLAM frame-level visualization of a “dog barking” to “lion roaring” counterfactual exemplifies temporal alignment and identity replacement.

Ablations

Ablation studies dissect the contribution of each design element. Removing decomposed guidance in Phase 1 yields near-zero $\Delta$FLAM and CLAP, showing that vanilla CFG cannot disentangle video and text when conditions conflict. Omitting source-based negative prompting in Phase 1 likewise degrades replacement metrics. Notably, reversing the phase order severely deteriorates FAD and DeSync, corroborating the necessity of constructing the temporal scaffold first before fine-grained identity modulation.

A sweep over phase-transition step $N_{\mathrm{trans}}$ quantifies the trade-off: increasing $N_{\mathrm{trans}}$ improves temporal alignment but decreases $\Delta$FLAM; earlier transitions favor identity replacement at the cost of alignment.

Implications and Future Directions

Practically, CounterFlow enables post-hoc control of Foley sound in arbitrary videos, empowering content creators and interactive applications. Theoretically, the results substantiate a general principle: early-stage conditioning sets structure; late-stage guidance can override semantic identity—a paradigm shift for controllable generation with multimodal, possibly adversarial, conditions.

The proposed method’s model-agnostic, inference-only design augurs broad applicability across future VT2A architectures, including those leveraging more expressive CLAP or FLAM embeddings. Future development should address strict temporal gating (e.g., enforcing silence during visual inactivity) and explore explicit adversarial training on counterfactual samples for robustness. Applying CounterFlow at scale or in human-in-the-loop creative pipelines could redefine workflows in audiovisual media, synthetic data creation, and beyond.

Conclusion

CounterFlow introduces a principled, inference-time, two-phase sampling protocol for counterfactual video Foley generation, effectively separating temporal alignment from sound identity guidance in pretrained VT2A systems. Empirical evaluation demonstrates significant improvement in replacing visually implied sound with the target prompt while maintaining temporal coherence and audio quality. The decomposed guidance approach and FLAM-based evaluation set new baselines for controllable, adversarial sound generation. Future work will generalize this strategy across architectures and address temporal gating fidelity.