---
title: Counterfactual Video Foley Generation
url: https://www.emergentmind.com/topics/counterfactual-video-foley-generation
type: topic
---

# Counterfactual Video Foley Generation

Counterfactual video Foley generation is a Video&Text-to-Audio (VT2A) task in which a silent video, a source caption aligned with the visible event, and a conflicting target caption are used to synthesize audio that preserves the timing and rhythm implied by the video while adopting the sound identity specified by the target and suppressing the visually implied source. In the formulation studied by "CounterFlow: A Two-Phase Inference-Time Sampling for Counterfactual Video Foley Generation" [2605.18916], the central difficulty is that pretrained VT2A systems are optimized for agreement between video and text and therefore tend to revert to the source suggested by the visuals when the modalities disagree.

## 1. Task definition and conceptual lineage

Counterfactual video Foley generation differs from standard VT2A in a precise way. Standard VT2A takes a video \(V\) and a matching text prompt \(T\) and aims to generate audio consistent with both. Counterfactual VT2A instead takes a silent video \(V\), a source text \(T_{\mathrm{src}}\) that describes the visible event, and a target text \(T_{\mathrm{tar}}\) that deliberately contradicts it, then seeks audio \(\hat{A}\) that is temporally synchronized with \(V\), sounds like \(T_{\mathrm{tar}}\), and contains much less of the sound identity implied by \(T_{\mathrm{src}}\) [2605.18916].

A canonical example is a video of a cat walking whose motion specifies when footsteps occur, while the desired output is "metal robot footsteps" rather than cat paw sounds. The problem is therefore not merely controllable video-to-audio synthesis; it is replacement under conflict. The required output must preserve event timing and coarse rhythmic structure while changing timbre and source identity.

This formulation has antecedents in earlier controllable Foley work. "Conditional Generation of Audio from Video via Foley Analogies" frames conditional Foley as creating a soundtrack that differs from a scene’s true sound while still plausibly matching on-screen events, using a conditioning audio-video exemplar rather than text [2304.08490]. CAFA formulates Foley as a video-and-text-to-audio problem in which text can refine semantics and introduce creative variations even when they contradict the visual cues [2504.06778]. CounterFlow specializes this broader controllable setting to the stricter requirement that the original visually implied source should be actively suppressed rather than merely coexisting with the target [2605.18916].

## 2. Why cross-modal conflict is difficult for VT2A models

The technical obstacle is a modality conflict internal to the learned conditioning structure of VT2A backbones. In pretrained systems such as MMAudio and HunyuanVideo-Foley, video features encode not only event timing but also source identity, and training data usually pairs videos with matching text. As a result, the model learns to reinforce semantic agreement rather than resolve contradictions. Under conflict, the video condition often dominates the sampling trajectory, so the model keeps generating the sound source implied by the visuals even when the text demands another source [2605.18916].

This failure has three components. First, there is **visual dominance**: the video embedding strongly anchors the generation to the visible source. Second, there is **entanglement of timing and identity**: the same video condition carries both when events happen and what object appears to produce them. Third, there is **training mismatch**: pretrained VT2A models rarely observe conflicting video-text pairs during training and are therefore not optimized for replacement.

Several baseline behaviors follow from this structure. CAFA and MultiFoley are designed for controllable Foley under conflicting video/text conditions, but generated audio often contains both the original source and the target. MMAudio and HunyuanVideo-Foley are even more strongly biased toward the visually implied sound and tend to ignore or weaken contradictory text. Naive negative prompting can improve suppression somewhat, but it also degrades text-audio relevance for the target and worsens synchronization because the guidance is not disentangled by modality [2605.18916].

A common misconception is that high text-audio similarity alone demonstrates successful counterfactual replacement. The CounterFlow analysis rejects that interpretation: audio can receive a favorable CLAP score because target evidence is present somewhere in the clip while the original source remains audible as leakage. This is why replacement quality requires a metric that measures target evidence against residual source evidence rather than target evidence alone [2605.18916].

## 3. Flow-matching backbone and the CounterFlow inference scheme

CounterFlow assumes a pretrained flow-matching VT2A backbone, specifically MMAudio large\_44k\_v2. Audio is encoded by an audio VAE into a latent \(Z_t\), video is encoded into \(c_{\mathrm{vid}}\), source and target texts are encoded into \(c_{\mathrm{src}}\) and \(c_{\mathrm{tar}}\), and null embeddings \(\emptyset_{\mathrm{vid}}\) and \(\emptyset_{\mathrm{txt}}\) disable conditioning when needed. The learned conditional velocity field is written as
\[
v_\theta(Z_t, c_{\mathrm{vid}}, c_{\mathrm{txt}}, t),
\]
and inference integrates the ODE
\[
\frac{dZ_t}{dt} = v_\theta(Z_t, c_{\mathrm{vid}}, c_{\mathrm{txt}}, t)
\]
from Gaussian noise using deterministic Euler steps with \(N=25\) [2605.18916].

Vanilla classifier-free guidance couples video and target text:
\[
v_i^{\mathrm{CFG}} = v_i(\emptyset_{\mathrm{vid}}, \emptyset_{\mathrm{txt}})
+ w\left[v_i(c_{\mathrm{vid}}, c_{\mathrm{tar}}) - v_i(\emptyset_{\mathrm{vid}}, \emptyset_{\mathrm{txt}})\right].
\]
This coupling is precisely what becomes problematic when \(c_{\mathrm{vid}}\) and \(c_{\mathrm{tar}}\) contradict each other.

CounterFlow replaces this with a two-phase inference-time procedure. The stated observation is that early steps primarily establish macro temporal structure, whereas later steps refine sound identity and timbre. With \(N_{\mathrm{trans}}=17\), Phase 1 occupies steps \(i \in [0, N_{\mathrm{trans}})\) and Phase 2 occupies steps \(i \in [N_{\mathrm{trans}}, N]\) [2605.18916].

In **Phase 1**, the goal is to build video-derived temporal structure while suppressing visual identity and pushing toward the target prompt. The guidance is
\[
\begin{aligned}
v_i^{(1)} &= v_i(\emptyset_{\mathrm{vid}}, \emptyset_{\mathrm{txt}}) \\
&+ w_{\mathrm{vid}}\Big(v_i(c_{\mathrm{vid}}, \emptyset_{\mathrm{txt}}) - v_i(\emptyset_{\mathrm{vid}}, \emptyset_{\mathrm{txt}})\Big) \\
&+ w_{\mathrm{txt}}\Big(v_i(\emptyset_{\mathrm{vid}}, c_{\mathrm{tar}}) - v_i(\emptyset_{\mathrm{vid}}, c_{\mathrm{src}})\Big),
\end{aligned}
\]
with \(w_{\mathrm{vid}}=3.0\) and \(w_{\mathrm{txt}}=5.0\). The video-only term captures event timing without explicit textual identity, while the text contrast term both attracts the target and repels the source [2605.18916].

In **Phase 2**, video conditioning is removed entirely so that the latent, which already contains video-derived timing, can be refined toward target timbre without being pulled back toward the visible source:
\[
v_i^{(2)} = v_i(\emptyset_{\mathrm{vid}}, \emptyset_{\mathrm{txt}})
+ w_{\mathrm{cfg}}\Big(v_i(\emptyset_{\mathrm{vid}}, c_{\mathrm{tar}}) - v_i(\emptyset_{\mathrm{vid}}, c_{\mathrm{src}})\Big),
\]
with \(w_{\mathrm{cfg}}=4.5\). Audio is then obtained by decoding \(Z_{t_N}\) through the audio VAE [2605.18916].

This decomposition is not a retraining strategy but an inference-time modification. The paper explicitly states that CounterFlow does not retrain MMAudio; it changes only how a pretrained flow-matching VT2A model is sampled. A plausible implication is that the method depends on the latent trajectory already containing separable temporal and timbral phases, which the ablations support.

## 4. Evaluation methodology, metrics, and empirical behavior

The evaluation protocol is designed around the replacement problem rather than generic audio quality alone. CounterFlow uses VGGSound-Sparse Clean, described as a Synchformer subset containing 451 test videos, each with a single sound source and 12 unique sound-source captions. For each video, its caption is used as \(T_{\mathrm{src}}\) and paired with the other 11 captions as \(T_{\mathrm{tar}}\), yielding \(451 \times 11 = 4{,}961\) triplets \((c_{\mathrm{vid}}, c_{\mathrm{tar}}, c_{\mathrm{src}})\) [2605.18916].

To evaluate replacement quality, the paper introduces a FLAM-based metric. For generated audio \(A\) and text prompt \(c\), framewise FLAM probabilities \(p_{\mathrm{FLAM}}(c,A,l)\) are computed, then summarized by
\[
P_{\mathrm{FLAM}}(c,A)=\max_l p_{\mathrm{FLAM}}(c,A,l).
\]
Replacement quality is measured by
\[
\Delta \mathrm{FLAM} = P_{\mathrm{FLAM}}(c_{\mathrm{tar}},A)-P_{\mathrm{FLAM}}(c_{\mathrm{src}},A),
\]
and the positive-\(\Delta\)FLAM ratio is
\[
r_{>0}=\frac{1}{M}\sum_{m=1}^{M}\mathbf{1}\!\left[\Delta\mathrm{FLAM}^{(m)} > 0\right].
\]
If \(\Delta\mathrm{FLAM}>0\), target evidence exceeds source evidence in the generated audio [2605.18916].

The remaining metrics are FAD for overall audio quality, IS for diversity/quality, CLAP for target text-audio similarity, and DeSync from Synchformer for audio-visual misalignment. The empirical pattern is consistent across these measures. CounterFlow attains the highest CLAP among methods that also suppress the source, and the highest \(\Delta\)FLAM and positive-\(\Delta\)FLAM ratio, while maintaining best or near-best FAD and IS. CAFA and ReWaS often produce mixtures in which both target and source are present. Naive negative prompting improves \(\Delta\)FLAM somewhat but hurts CLAP and DeSync, demonstrating the trade-off produced by non-decomposed guidance [2605.18916].

The ablations clarify what the two phases contribute. Vanilla CFG in Phase 1 yields near-zero \(\Delta\)FLAM and CLAP, indicating reversion to the visually implied source. Removing negative prompting in Phase 1 degrades replacement metrics, which the paper interprets as evidence that video features themselves encode source identity. Swapping the phase order—text contrast first, then video—severely degrades FAD and DeSync, indicating that early video-conditioned updates are crucial for timing [2605.18916].

Hyperparameter sweeps over \(N_{\mathrm{trans}}\) show an explicit trade-off. Earlier transition to Phase 2 improves \(\Delta\)FLAM but worsens DeSync; later transition improves DeSync but worsens \(\Delta\)FLAM. The chosen \(N_{\mathrm{trans}}=17\) is described as a knee in this trade-off curve. The paper also notes failure cases: the model sometimes emits sound during visually silent frames, and rare or out-of-domain target sounds remain harder to realize cleanly [2605.18916].

## 5. Related paradigms of controllable and alternative Foley

Counterfactual Foley generation sits within a broader family of controllable soundtrack synthesis, but the conditioning interface varies substantially across methods. One direction uses **reference audio** rather than source–target text contrast. AC-Foley is an audio-conditioned V2A model that conditions on video, text, and reference audio and is described as enabling fine-grained sound synthesis, timbre transfer, and zero-shot sound generation; for a fixed video, varying the reference audio yields different acoustic realizations while preserving visual timing [2603.15597]. This corresponds to counterfactual control through exemplar timbre rather than explicit source suppression.

A second direction is **track-wise compositional generation**. "Step-by-Step Video-to-Audio Synthesis via Negative Audio Guidance" generates individual tracks \(x^{(k)}\) for prompts \(C_k\) and mixes them as
\[
\hat{x} = \mathrm{normalize}\!\left(\sum_{k=1}^K x^{(k)}\right).
\]
Its Negative Audio Guidance steers each new track away from previously generated audio, allowing add, remove, and replace operations at the stem level [2506.20995]. This makes counterfactual editing natural in multi-event scenes, though the paper frames the method as step-by-step video-to-audio rather than source replacement within a single track.

A third direction expands from Foley to **complete soundtrack generation**. Foley-Omni jointly models speech, sound effects, and music within a shared latent generation process and uses structured text fields such as `[WORDS]`, `[AUDIO]`, and `[MUSIC]` to generate different complete soundtracks for the same video [2606.03672]. FoleyDesigner goes further toward production workflows by decomposing clips into symbolic Foley scripts, extracting spatio-temporal cues, and performing multi-track mixing with stereo-to-5.1 integration; its modular script-based formulation makes alternative event structures straightforward to generate for the same video [2604.05731].

These paradigms differ in what they treat as the principal control variable: text conflict and source suppression in CounterFlow, reference timbre in AC-Foley, stem-level exclusion in Negative Audio Guidance, and script-level or soundtrack-level restructuring in Foley-Omni and FoleyDesigner. This suggests that counterfactual Foley is best understood not as one model family but as a control objective that can be implemented through distinct conditioning regimes.

## 6. Limitations, applications, and open research directions

The immediate applications identified for counterfactual Foley are film and television post-production, game audio, AR/VR, immersive media, and data augmentation for audio-visual learning [2605.18916]. In all of these settings, the attractive property is the ability to preserve the timing implied by an already approved visual sequence while changing what the scene sounds like.

The limitations are also structurally clear. CounterFlow relies on a strong pretrained VT2A backbone and inherits its domain coverage. If the backbone has weak coverage of rare or out-of-domain targets, replacement may remain incomplete or artifact-prone. The method also incurs the computational cost of ODE-based guided sampling, though the paper describes it as similar to standard high-quality VT2A inference. Temporal gating is imperfect, and the model may still generate events in visually silent spans [2605.18916].

The broader literature identifies several adjacent open problems. Foley-Flow emphasizes that segment-level masked audio-visual alignment and dynamic conditional flow improve rhythmic synchronization, suggesting that stronger local timing representations could reduce false positives and improve temporal gating [2603.08126]. Diff-Foley shows that CAVP-aligned video features and multi-condition guidance are effective for synchronization, which suggests a pathway for extending counterfactual generation with more explicit attribute control [2306.17203]. FoleyCrafter separates semantic conditioning from temporal control via a semantic adapter and timestamp-based controller, suggesting that finer-grained counterfactual editing may benefit from explicit event masks or onset schedules rather than purely latent phase separation [2407.01494].

Ethically, the same methods that enable creative post-production can also be used to generate misleading audio for real-world video. The CounterFlow analysis notes plausible concerns including misrepresenting documentary or surveillance footage and generating manipulative soundtracks. A plausible implication is that counterfactual Foley systems will require context-aware deployment constraints or disclosure mechanisms when used outside clearly fictional or editorial contexts [2605.18916].

At a methodological level, two unresolved questions recur across the cited work. The first is whether future systems should pursue **single-track replacement**, as in CounterFlow, or **multi-track decomposition**, as in Negative Audio Guidance and FoleyDesigner. The second is whether the most effective control signal is **conflicting text**, **reference audio**, or **structured multimodal scripts**. Existing results indicate that each interface captures a different part of the problem: text is strong for semantic replacement, reference audio is strong for timbral precision, and script or stem structures are strong for complex scene composition [2605.18916].

Source: https://www.emergentmind.com/topics/counterfactual-video-foley-generation