Counterfactual Video Foley Generation

Updated 5 July 2026

Counterfactual video Foley generation is a VT2A task that uses a silent video, a source caption, and a conflicting target caption to create alternative audio while preserving event timing.
The approach employs a two-phase inference scheme, where early steps build temporal structure and later steps refine target timbre, effectively suppressing the source sound.
Leveraging a pretrained flow-matching backbone with classifier-free guidance, the method uses negative prompting and tailored sampling to enhance target fidelity over visual cues.

Counterfactual video Foley generation is a Video&Text-to-Audio (VT2A) task in which a silent video, a source caption aligned with the visible event, and a conflicting target caption are used to synthesize audio that preserves the timing and rhythm implied by the video while adopting the sound identity specified by the target and suppressing the visually implied source. In the formulation studied by "CounterFlow: A Two-Phase Inference-Time Sampling for Counterfactual Video Foley Generation" (Lee et al., 18 May 2026), the central difficulty is that pretrained VT2A systems are optimized for agreement between video and text and therefore tend to revert to the source suggested by the visuals when the modalities disagree.

1. Task definition and conceptual lineage

Counterfactual video Foley generation differs from standard VT2A in a precise way. Standard VT2A takes a video $V$ and a matching text prompt $T$ and aims to generate audio consistent with both. Counterfactual VT2A instead takes a silent video $V$ , a source text $T_{\mathrm{src}}$ that describes the visible event, and a target text $T_{\mathrm{tar}}$ that deliberately contradicts it, then seeks audio $\hat{A}$ that is temporally synchronized with $V$ , sounds like $T_{\mathrm{tar}}$ , and contains much less of the sound identity implied by $T_{\mathrm{src}}$ (Lee et al., 18 May 2026).

A canonical example is a video of a cat walking whose motion specifies when footsteps occur, while the desired output is "metal robot footsteps" rather than cat paw sounds. The problem is therefore not merely controllable video-to-audio synthesis; it is replacement under conflict. The required output must preserve event timing and coarse rhythmic structure while changing timbre and source identity.

This formulation has antecedents in earlier controllable Foley work. "Conditional Generation of Audio from Video via Foley Analogies" frames conditional Foley as creating a soundtrack that differs from a scene’s true sound while still plausibly matching on-screen events, using a conditioning audio-video exemplar rather than text (Du et al., 2023). CAFA formulates Foley as a video-and-text-to-audio problem in which text can refine semantics and introduce creative variations even when they contradict the visual cues (Benita et al., 9 Apr 2025). CounterFlow specializes this broader controllable setting to the stricter requirement that the original visually implied source should be actively suppressed rather than merely coexisting with the target (Lee et al., 18 May 2026).

The technical obstacle is a modality conflict internal to the learned conditioning structure of VT2A backbones. In pretrained systems such as MMAudio and HunyuanVideo-Foley, video features encode not only event timing but also source identity, and training data usually pairs videos with matching text. As a result, the model learns to reinforce semantic agreement rather than resolve contradictions. Under conflict, the video condition often dominates the sampling trajectory, so the model keeps generating the sound source implied by the visuals even when the text demands another source (Lee et al., 18 May 2026).

This failure has three components. First, there is visual dominance: the video embedding strongly anchors the generation to the visible source. Second, there is entanglement of timing and identity: the same video condition carries both when events happen and what object appears to produce them. Third, there is training mismatch: pretrained VT2A models rarely observe conflicting video-text pairs during training and are therefore not optimized for replacement.

Several baseline behaviors follow from this structure. CAFA and MultiFoley are designed for controllable Foley under conflicting video/text conditions, but generated audio often contains both the original source and the target. MMAudio and HunyuanVideo-Foley are even more strongly biased toward the visually implied sound and tend to ignore or weaken contradictory text. Naive negative prompting can improve suppression somewhat, but it also degrades text-audio relevance for the target and worsens synchronization because the guidance is not disentangled by modality (Lee et al., 18 May 2026).

A common misconception is that high text-audio similarity alone demonstrates successful counterfactual replacement. The CounterFlow analysis rejects that interpretation: audio can receive a favorable CLAP score because target evidence is present somewhere in the clip while the original source remains audible as leakage. This is why replacement quality requires a metric that measures target evidence against residual source evidence rather than target evidence alone (Lee et al., 18 May 2026).

3. Flow-matching backbone and the CounterFlow inference scheme

CounterFlow assumes a pretrained flow-matching VT2A backbone, specifically MMAudio large_44k_v2. Audio is encoded by an audio VAE into a latent $Z_t$ , video is encoded into $T$ 0, source and target texts are encoded into $T$ 1 and $T$ 2, and null embeddings $T$ 3 and $T$ 4 disable conditioning when needed. The learned conditional velocity field is written as

$T$ 5

and inference integrates the ODE

$T$ 6

from Gaussian noise using deterministic Euler steps with $T$ 7 (Lee et al., 18 May 2026).

Vanilla classifier-free guidance couples video and target text: $T$ 8 This coupling is precisely what becomes problematic when $T$ 9 and $V$ 0 contradict each other.

CounterFlow replaces this with a two-phase inference-time procedure. The stated observation is that early steps primarily establish macro temporal structure, whereas later steps refine sound identity and timbre. With $V$ 1, Phase 1 occupies steps $V$ 2 and Phase 2 occupies steps $V$ 3 (Lee et al., 18 May 2026).

In Phase 1, the goal is to build video-derived temporal structure while suppressing visual identity and pushing toward the target prompt. The guidance is

$V$ 4

with $V$ 5 and $V$ 6. The video-only term captures event timing without explicit textual identity, while the text contrast term both attracts the target and repels the source (Lee et al., 18 May 2026).

In Phase 2, video conditioning is removed entirely so that the latent, which already contains video-derived timing, can be refined toward target timbre without being pulled back toward the visible source: $V$ 7 with $V$ 8. Audio is then obtained by decoding $V$ 9 through the audio VAE (Lee et al., 18 May 2026).

This decomposition is not a retraining strategy but an inference-time modification. The paper explicitly states that CounterFlow does not retrain MMAudio; it changes only how a pretrained flow-matching VT2A model is sampled. A plausible implication is that the method depends on the latent trajectory already containing separable temporal and timbral phases, which the ablations support.

4. Evaluation methodology, metrics, and empirical behavior

The evaluation protocol is designed around the replacement problem rather than generic audio quality alone. CounterFlow uses VGGSound-Sparse Clean, described as a Synchformer subset containing 451 test videos, each with a single sound source and 12 unique sound-source captions. For each video, its caption is used as $T_{\mathrm{src}}$ 0 and paired with the other 11 captions as $T_{\mathrm{src}}$ 1, yielding $T_{\mathrm{src}}$ 2 triplets $T_{\mathrm{src}}$ 3 (Lee et al., 18 May 2026).

To evaluate replacement quality, the paper introduces a FLAM-based metric. For generated audio $T_{\mathrm{src}}$ 4 and text prompt $T_{\mathrm{src}}$ 5, framewise FLAM probabilities $T_{\mathrm{src}}$ 6 are computed, then summarized by

$T_{\mathrm{src}}$ 7

Replacement quality is measured by

$T_{\mathrm{src}}$ 8

and the positive- $T_{\mathrm{src}}$ 9FLAM ratio is

$T_{\mathrm{tar}}$ 0

If $T_{\mathrm{tar}}$ 1, target evidence exceeds source evidence in the generated audio (Lee et al., 18 May 2026).

The remaining metrics are FAD for overall audio quality, IS for diversity/quality, CLAP for target text-audio similarity, and DeSync from Synchformer for audio-visual misalignment. The empirical pattern is consistent across these measures. CounterFlow attains the highest CLAP among methods that also suppress the source, and the highest $T_{\mathrm{tar}}$ 2FLAM and positive- $T_{\mathrm{tar}}$ 3FLAM ratio, while maintaining best or near-best FAD and IS. CAFA and ReWaS often produce mixtures in which both target and source are present. Naive negative prompting improves $T_{\mathrm{tar}}$ 4FLAM somewhat but hurts CLAP and DeSync, demonstrating the trade-off produced by non-decomposed guidance (Lee et al., 18 May 2026).

The ablations clarify what the two phases contribute. Vanilla CFG in Phase 1 yields near-zero $T_{\mathrm{tar}}$ 5FLAM and CLAP, indicating reversion to the visually implied source. Removing negative prompting in Phase 1 degrades replacement metrics, which the paper interprets as evidence that video features themselves encode source identity. Swapping the phase order—text contrast first, then video—severely degrades FAD and DeSync, indicating that early video-conditioned updates are crucial for timing (Lee et al., 18 May 2026).

Hyperparameter sweeps over $T_{\mathrm{tar}}$ 6 show an explicit trade-off. Earlier transition to Phase 2 improves $T_{\mathrm{tar}}$ 7FLAM but worsens DeSync; later transition improves DeSync but worsens $T_{\mathrm{tar}}$ 8FLAM. The chosen $T_{\mathrm{tar}}$ 9 is described as a knee in this trade-off curve. The paper also notes failure cases: the model sometimes emits sound during visually silent frames, and rare or out-of-domain target sounds remain harder to realize cleanly (Lee et al., 18 May 2026).

Counterfactual Foley generation sits within a broader family of controllable soundtrack synthesis, but the conditioning interface varies substantially across methods. One direction uses reference audio rather than source–target text contrast. AC-Foley is an audio-conditioned V2A model that conditions on video, text, and reference audio and is described as enabling fine-grained sound synthesis, timbre transfer, and zero-shot sound generation; for a fixed video, varying the reference audio yields different acoustic realizations while preserving visual timing (Fang et al., 16 Mar 2026). This corresponds to counterfactual control through exemplar timbre rather than explicit source suppression.

A second direction is track-wise compositional generation. "Step-by-Step Video-to-Audio Synthesis via Negative Audio Guidance" generates individual tracks $\hat{A}$ 0 for prompts $\hat{A}$ 1 and mixes them as

$\hat{A}$ 2

Its Negative Audio Guidance steers each new track away from previously generated audio, allowing add, remove, and replace operations at the stem level (Hayakawa et al., 26 Jun 2025). This makes counterfactual editing natural in multi-event scenes, though the paper frames the method as step-by-step video-to-audio rather than source replacement within a single track.

A third direction expands from Foley to complete soundtrack generation. Foley-Omni jointly models speech, sound effects, and music within a shared latent generation process and uses structured text fields such as [WORDS], [AUDIO], and [[MUSIC](https://www.emergentmind.com/topics/multiple-signal-classification-music)] to generate different complete soundtracks for the same video (Tao et al., 2 Jun 2026). FoleyDesigner goes further toward production workflows by decomposing clips into symbolic Foley scripts, extracting spatio-temporal cues, and performing multi-track mixing with stereo-to-5.1 integration; its modular script-based formulation makes alternative event structures straightforward to generate for the same video (Li et al., 7 Apr 2026).

These paradigms differ in what they treat as the principal control variable: text conflict and source suppression in CounterFlow, reference timbre in AC-Foley, stem-level exclusion in Negative Audio Guidance, and script-level or soundtrack-level restructuring in Foley-Omni and FoleyDesigner. This suggests that counterfactual Foley is best understood not as one model family but as a control objective that can be implemented through distinct conditioning regimes.

6. Limitations, applications, and open research directions

The immediate applications identified for counterfactual Foley are film and television post-production, game audio, AR/VR, immersive media, and data augmentation for audio-visual learning (Lee et al., 18 May 2026). In all of these settings, the attractive property is the ability to preserve the timing implied by an already approved visual sequence while changing what the scene sounds like.

The limitations are also structurally clear. CounterFlow relies on a strong pretrained VT2A backbone and inherits its domain coverage. If the backbone has weak coverage of rare or out-of-domain targets, replacement may remain incomplete or artifact-prone. The method also incurs the computational cost of ODE-based guided sampling, though the paper describes it as similar to standard high-quality VT2A inference. Temporal gating is imperfect, and the model may still generate events in visually silent spans (Lee et al., 18 May 2026).

The broader literature identifies several adjacent open problems. Foley-Flow emphasizes that segment-level masked audio-visual alignment and dynamic conditional flow improve rhythmic synchronization, suggesting that stronger local timing representations could reduce false positives and improve temporal gating (Mo et al., 9 Mar 2026). Diff-Foley shows that CAVP-aligned video features and multi-condition guidance are effective for synchronization, which suggests a pathway for extending counterfactual generation with more explicit attribute control (Luo et al., 2023). FoleyCrafter separates semantic conditioning from temporal control via a semantic adapter and timestamp-based controller, suggesting that finer-grained counterfactual editing may benefit from explicit event masks or onset schedules rather than purely latent phase separation (Zhang et al., 2024).

Ethically, the same methods that enable creative post-production can also be used to generate misleading audio for real-world video. The CounterFlow analysis notes plausible concerns including misrepresenting documentary or surveillance footage and generating manipulative soundtracks. A plausible implication is that counterfactual Foley systems will require context-aware deployment constraints or disclosure mechanisms when used outside clearly fictional or editorial contexts (Lee et al., 18 May 2026).

At a methodological level, two unresolved questions recur across the cited work. The first is whether future systems should pursue single-track replacement, as in CounterFlow, or multi-track decomposition, as in Negative Audio Guidance and FoleyDesigner. The second is whether the most effective control signal is conflicting text, reference audio, or structured multimodal scripts. Existing results indicate that each interface captures a different part of the problem: text is strong for semantic replacement, reference audio is strong for timbral precision, and script or stem structures are strong for complex scene composition (Lee et al., 18 May 2026).