ConterFlow: Inference Sampling for Video Foley

• ConterFlow is an inference-time sampling method for counterfactual video Foley generation that generates audio following video timing while altering sound identity.
• It employs a two-phase sampling scheme: Phase 1 uses video conditioning with decomposed guidance for temporal structure, and Phase 2 refines timbre via text-only contrastive guidance.
• The method effectively suppresses visually implied sounds in favor of target audio prompts, showing improved ΔFLAM and CLAP metrics compared to conventional VT2A systems.

ConterFlow, spelled “CounterFlow” in the paper, is an inference-time sampling method for counterfactual video Foley generation built on top of a pretrained flow-matching Video&Text-to-Audio (VT2A) model, specifically MMAudio large_44k_v2. It addresses a setting in which a silent video $V$, a source text $T_\mathrm{src}$ describing the visually implied sound, and a conflicting target text $T_\mathrm{tar}$ are given, and the objective is to generate audio that remains temporally synchronized to the video while adopting the target sound identity and suppressing the visually implied source. The method changes how sampling is performed from the pretrained model and does not retrain the backbone. Its central mechanism is a two-phase inference-time sampling scheme that uses video conditioning early to establish temporal structure and then removes video conditioning later so that the trajectory can refine timbre and identity toward the target prompt without being pulled back toward the visible source (Lee et al., 18 May 2026).

1. Problem formulation and task definition

Counterfactual video Foley generation is defined by three inputs: a silent video $V$, a source text $T_\mathrm{src}$ that describes the visually implied sound, and a target text $T_\mathrm{tar}$ that describes a different, conflicting sound source. The generation goal has three parts: the audio should follow the temporal progression of the video, reflect the target sound identity, and suppress the visually implied sound associated with $T_\mathrm{src}$. The paper’s example is a video showing a dog barking while the desired output is a lion roaring that still follows the timing of the dog’s mouth movements (Lee et al., 18 May 2026).

The difficulty arises because existing VT2A and Foley systems, including MMAudio, HunyuanVideo-Foley, CAFA, and MultiFoley, are trained under regimes in which video and text usually agree. Under that training distribution, video features carry strong object-specific information, described as visual evidence, and in flow-matching generative models this signal tends to dominate the sampling trajectory when video and text disagree. As a result, conflicting prompting often yields either the visually implied source or a mixture of source and target rather than a clean counterfactual replacement.

Within this formulation, three notions are central. Visual evidence refers to information in the video features $c_\mathrm{vid}$ that encodes what object or action is present and therefore what sound is expected. Sound-source identity denotes the semantic identity of the sound source, such as dog bark or lion roar, as described by text or implied by video. Counterfactual means that the output audio remains temporally consistent with the visible event while deliberately adopting a different source identity than the one suggested by the video.

A key negative result motivating the method is that naive negative prompting, for example using the source text as a negative prompt, often weakens all conditioning rather than isolating source suppression. The reported effect is degradation in audio quality, text relevance, and synchronization. This suggests that counterfactual replacement cannot be reduced to standard prompt subtraction alone.

2. Backbone model, latent dynamics, and conditioning structure

ConterFlow operates on a pretrained flow-matching VT2A backbone. The paper uses MMAudio large_44k_v2, which takes a video condition $c_\mathrm{vid}$, a text condition $$c_\mathrm{txt} \in \{c_\mathrm{tar}, c_\mathrm{src}\}$$, the current latent state $T_\mathrm{src}$0, and predicts a velocity field in an audio latent space,

$T_\mathrm{src}$1

Generation is posed as integrating this velocity field over continuous time. Sampling proceeds along an ODE timestep grid $T_\mathrm{src}$2 from an initial latent $T_\mathrm{src}$3. At each step $T_\mathrm{src}$4, the model predicts

$T_\mathrm{src}$5

The implementation uses deterministic Euler ODE sampling with $T_\mathrm{src}$6 steps and switches from the first phase to the second at $T_\mathrm{src}$7 (Lee et al., 18 May 2026).

The backbone supports null embeddings that disable individual modalities: $T_\mathrm{src}$8 for video-disabled conditioning and $T_\mathrm{src}$9 for text-disabled conditioning. This capability is structurally important because ConterFlow explicitly decomposes video and text influence rather than always combining them in a single conditional branch.

The paper treats the backbone architecture itself largely as a black box. Audio is modeled in a latent space via a VAE encoder/decoder, the video $T_\mathrm{tar}$0 is passed through MMAudio’s video encoder to produce $T_\mathrm{tar}$1, and text prompts are passed through MMAudio’s text encoder to produce $T_\mathrm{tar}$2 and $T_\mathrm{tar}$3. The final latent $T_\mathrm{tar}$4 is decoded by the audio VAE decoder to yield the waveform. ConterFlow therefore modifies only the guidance logic used during sampling, not the training procedure or backbone architecture.

3. Two-phase inference-time sampling scheme

The method divides sampling into two phases with distinct objectives. Phase 1 uses video conditioning strongly to establish coarse temporal structure while suppressing the visually implied source. Phase 2 removes video conditioning entirely and focuses on refining timbre and identity toward the target prompt.

Phase 1: video-derived temporal structure with decomposed guidance

For timesteps $T_\mathrm{tar}$5, the goal is to exploit video for event timing while avoiding the degradation that occurs when the backbone is asked to process the conflicting pair $T_\mathrm{tar}$6. The paper contrasts this with standard classifier-free guidance, which would combine video and target text inside a single conditional branch and thereby encourage the backbone to prioritize video.

ConterFlow replaces vanilla CFG with the following decomposed guidance: $T_\mathrm{tar}$7

Here the unconditional term provides a baseline trajectory, the video-only term contributes video-derived temporal structure without introducing text identity, and the text-only contrastive term pushes the latent toward the target prompt while pushing it away from the source prompt. The reported weights are $T_\mathrm{tar}$8 and $T_\mathrm{tar}$9 (Lee et al., 18 May 2026).

A central design point is that the model never evaluates the conflicting pair $V$0 within a single forward pass. Instead, video and text are separated into different conditional branches whose contributions are linearly combined. The paper attributes low-fidelity and visually anchored outputs to asking the backbone to resolve such incompatible conditioning directly. In the authors’ interpretation, Phase 1 supplies “where/when” from video and “which source” from text, with explicit suppression of the visually implied source.

Phase 2: text-only identity shaping via negative prompting

For timesteps $V$1, the method removes video conditioning altogether: $V$2

In this stage, the temporal structure established during Phase 1 is assumed to be largely fixed, and the sampling trajectory is refined using text-only contrastive guidance. The reported weight is $V$3. There is no continuous blending function; the transition is piecewise, occurring at the fixed step index $V$4. After the final step, the resulting latent is decoded by the audio VAE decoder into waveform audio (Lee et al., 18 May 2026).

The paper’s intuition is explicit: early steps are best used for video-based temporal structuring, whereas later steps are better suited for identity refinement without video. A plausible implication is that the method relies on a temporal separation in the generative trajectory between coarse event placement and finer timbral specification.

4. Evaluation protocol and counterfactual metrics

Evaluation is conducted on the VGGSound-Sparse Clean subset, described as single-source videos comprising 451 test videos and 12 sound-source captions. For each video, its annotated caption is used as the source prompt $V$5, and the other 11 captions are paired as target prompts $V$6, producing 4,961 triplets $V$7 in which the video sound identity and target text conflict (Lee et al., 18 May 2026).

The study reports standard metrics: FAD for audio quality, Inception Score for audio quality and diversity, CLAP for text-audio similarity between the target prompt and generated audio, and DeSync for video-audio temporal misalignment. The paper explicitly notes that CLAP alone can be misleading, because a model that produces both target and source sounds may score well on target relevance despite failing at clean counterfactual replacement.

To address this, the paper proposes a FLAM-based metric based on frame-level language-audio modeling. Let $V$8 denote the generated audio, $V$9 any text prompt, and $T_\mathrm{src}$0 the frame-wise probability that the content corresponding to text $T_\mathrm{src}$1 is present at frame $T_\mathrm{src}$2. The clip-level evidence is defined as

$T_\mathrm{src}$3

Using this quantity, the paper defines the counterfactual replacement score

$T_\mathrm{src}$4

which increases when target evidence exceeds source evidence, and the success rate

$T_\mathrm{src}$5

which measures the fraction of clips for which target evidence is greater than source evidence.

These definitions are significant because they explicitly encode both sides of the replacement problem: strong evidence for the target prompt and weak evidence for the visually implied source. The paper presents this as a direct measurement of replacement quality, rather than merely prompt presence.

5. Comparative results and ablation findings

The baseline set includes CAFA, ReWaS, negative-prompting variants of both, the full ConterFlow method, and a variant labeled “w/o P2 neg.”. The following values are reported for the main counterfactual setup (Lee et al., 18 May 2026):

Method	FAD↓	IS↑	ΔFLAM↑	(+)Ratio↑	CLAP↑	DeSync↓
CAFA	24.81	5.931	0.1289	0.8258	0.2371	0.5888
CAFA + neg.	31.46	7.606	0.2573	0.8835	0.1801	0.6431
ReWaS	75.18	4.223	0.0560	0.6184	0.1084	1.078
ReWaS + neg.	79.52	4.703	0.1905	0.7130	0.0947	1.103
ConterFlow	23.55	7.915	0.2641	0.9200	0.2840	0.6695
w/o P2 neg.	23.29	7.790	0.2373	0.9170	0.2849	0.6261

In the reported interpretation, ConterFlow achieves the best $T_\mathrm{src}$6, the best positive-$T_\mathrm{src}$7 ratio, and the best CLAP, while maintaining FAD comparable to or better than CAFA and much better than ReWaS. DeSync is described as close to CAFA and much better than ReWaS. The comparison with naive negative prompting is particularly important: adding negative prompts to CAFA or ReWaS improves $T_\mathrm{src}$8 and the success ratio, but damages CLAP and DeSync, consistent with the paper’s claim that naive negative prompting weakens both target and video conditioning.

A qualitative example, “dog barking $T_\mathrm{src}$9 lion roaring,” is used to illustrate the difference between mixed and clean replacement. In the description of Figure 1, CAFA exhibits both source and target evidence during the event, whereas ConterFlow shows low FLAM scores for the source prompt and high FLAM scores for the target prompt over the event duration, indicating cleaner replacement while respecting timing. The paper states that it does not describe a formal human listening test.

Ablations isolate the roles of Phase 1 decomposition, Phase 1 negative prompting, and phase ordering:

Method	FAD↓	ΔFLAM↑	DeSync↓	CLAP↑
ConterFlow (full)	23.55	0.2641	0.6695	0.2840
w/o P1 decomp. CFG	24.36	0.0278	0.2390	0.0894
w/o P1 neg.	21.00	0.0534	0.4362	0.2608
Phase swap (P1 ↔ P2)	52.33	0.2367	0.9989	0.2817

The paper’s conclusions from these ablations are direct. Removing Phase 1 decomposed guidance collapses $T_\mathrm{tar}$0 to near zero and sharply reduces CLAP, indicating that vanilla CFG with joint video-target conditioning fails to impose counterfactual identity. Removing Phase 1 negative prompting also sharply reduces $T_\mathrm{tar}$1, which the paper interprets as evidence that video features alone carry strong source identity and therefore require explicit suppression. Swapping the phase order degrades FAD and DeSync severely, confirming that the method depends on using early steps for temporal structuring and later steps for identity refinement (Lee et al., 18 May 2026).

The paper also sweeps the transition step $T_\mathrm{tar}$2. Increasing $T_\mathrm{tar}$3, thereby lengthening Phase 1, improves DeSync but reduces $T_\mathrm{tar}$4. The selected value $T_\mathrm{tar}$5 is described as a “knee” in the trade-off between synchronization and counterfactual controllability. This suggests a tunable balance between temporal fidelity and source replacement strength.

6. Limitations, scope, and disambiguation

The paper identifies two explicit limitations. First, ConterFlow sometimes generates sound during visually silent intervals, which is described as limited temporal gating: video cues are not used to fully suppress audio outside active events. Second, the method is demonstrated mainly on MMAudio; although it is described as conceptually model-agnostic, broader validation on other VT2A backbones is left for future work (Lee et al., 18 May 2026).

The future directions listed in the paper follow directly from these limitations. One is to train models so that generation is exclusively anchored to active visual cues, with improved silence modeling between events. Another is to test the same two-phase decomposed-guidance framework on other VT2A backbones to assess generalizability. A plausible implication is that the core contribution is less a new architecture than a reusable inference-time control strategy for pretrained flow-matching VT2A systems.

The name “ConterFlow” is ambiguous in the literature provided here. In the paper that introduces the method, the spelling is explicitly “CounterFlow” (Lee et al., 18 May 2026). The same string is also discussed in relation to FlowCast, a conditional flow-matching model for precipitation nowcasting, in the sense that “ConterFlow” may be used informally to refer to conditional flow-based forecasting ideas (Ribeiro et al., 12 Nov 2025). Separately, ContentFlow is an SDN architecture for mapping content names to IP flows over OpenFlow and is described as “sometimes miswritten as ‘ConterFlow’” (Chanda et al., 2013). In the present context, however, the term denotes the counterfactual video Foley generation method based on two-phase inference-time sampling for pretrained flow-matching VT2A models.