---
title: 'ConterFlow: Inference Sampling for Video Foley'
url: https://www.emergentmind.com/topics/conterflow
type: topic
---

# ConterFlow: Inference Sampling for Video Foley

ConterFlow, spelled “CounterFlow” in the paper, is an inference-time sampling method for **counterfactual video Foley generation** built on top of a pretrained **flow-matching Video&Text-to-Audio (VT2A)** model, specifically **MMAudio large\_44k\_v2**. It addresses a setting in which a **silent video** \(V\), a **source text** \(T_\mathrm{src}\) describing the visually implied sound, and a conflicting **target text** \(T_\mathrm{tar}\) are given, and the objective is to generate audio that remains temporally synchronized to the video while adopting the target sound identity and suppressing the visually implied source. The method changes **how sampling is performed** from the pretrained model and does **not retrain** the backbone. Its central mechanism is a **two-phase inference-time sampling scheme** that uses video conditioning early to establish temporal structure and then removes video conditioning later so that the trajectory can refine timbre and identity toward the target prompt without being pulled back toward the visible source [2605.18916].

## 1. Problem formulation and task definition

Counterfactual video Foley generation is defined by three inputs: a silent video \(V\), a source text \(T_\mathrm{src}\) that describes the **visually implied** sound, and a target text \(T_\mathrm{tar}\) that describes a **different, conflicting** sound source. The generation goal has three parts: the audio should follow the **temporal progression of the video**, reflect the **target sound identity**, and **suppress the visually implied sound** associated with \(T_\mathrm{src}\). The paper’s example is a video showing a dog barking while the desired output is a lion roaring that still follows the timing of the dog’s mouth movements [2605.18916].

The difficulty arises because existing VT2A and Foley systems, including **MMAudio, HunyuanVideo-Foley, CAFA, and MultiFoley**, are trained under regimes in which video and text usually agree. Under that training distribution, video features carry strong **object-specific information**, described as **visual evidence**, and in flow-matching generative models this signal tends to dominate the sampling trajectory when video and text disagree. As a result, conflicting prompting often yields either the **visually implied source** or a **mixture** of source and target rather than a clean counterfactual replacement.

Within this formulation, three notions are central. **Visual evidence** refers to information in the video features \(c_\mathrm{vid}\) that encodes what object or action is present and therefore what sound is expected. **Sound-source identity** denotes the semantic identity of the sound source, such as dog bark or lion roar, as described by text or implied by video. **Counterfactual** means that the output audio remains temporally consistent with the visible event while deliberately adopting a different source identity than the one suggested by the video.

A key negative result motivating the method is that naive **negative prompting**, for example using the source text as a negative prompt, often weakens **all** conditioning rather than isolating source suppression. The reported effect is degradation in audio quality, text relevance, and synchronization. This suggests that counterfactual replacement cannot be reduced to standard prompt subtraction alone.

## 2. Backbone model, latent dynamics, and conditioning structure

ConterFlow operates on a pretrained **flow-matching VT2A** backbone. The paper uses **MMAudio large\_44k\_v2**, which takes a video condition \(c_\mathrm{vid}\), a text condition \(c_\mathrm{txt} \in \{c_\mathrm{tar}, c_\mathrm{src}\}\), the current latent state \(Z_t\), and predicts a **velocity field** in an audio latent space,
\[
v_\theta(Z_t, c_\mathrm{vid}, c_\mathrm{txt}, t).
\]

Generation is posed as integrating this velocity field over continuous time. Sampling proceeds along an ODE timestep grid \(\{t_i\}_{i=0}^N\) from an initial latent \(Z_{t_0} \sim \mathcal{N}(0, I)\). At each step \(i\), the model predicts
\[
v_i(c_\mathrm{vid}, c_\mathrm{txt}) = v_\theta(Z_{t_i}, c_\mathrm{vid}, c_\mathrm{txt}, t_i).
\]
The implementation uses **deterministic Euler ODE sampling** with \(N=25\) steps and switches from the first phase to the second at \(N_\mathrm{trans}=17\) [2605.18916].

The backbone supports **null embeddings** that disable individual modalities: \(\emptyset_\mathrm{vid}\) for video-disabled conditioning and \(\emptyset_\mathrm{txt}\) for text-disabled conditioning. This capability is structurally important because ConterFlow explicitly decomposes video and text influence rather than always combining them in a single conditional branch.

The paper treats the backbone architecture itself largely as a black box. Audio is modeled in a latent space via a **VAE encoder/decoder**, the video \(V\) is passed through MMAudio’s **video encoder** to produce \(c_\mathrm{vid}\), and text prompts are passed through MMAudio’s **text encoder** to produce \(c_\mathrm{tar}\) and \(c_\mathrm{src}\). The final latent \(Z_{t_N}\) is decoded by the **audio VAE decoder** to yield the waveform. ConterFlow therefore modifies only the **guidance logic used during sampling**, not the training procedure or backbone architecture.

## 3. Two-phase inference-time sampling scheme

The method divides sampling into two phases with distinct objectives. **Phase 1** uses video conditioning strongly to establish **coarse temporal structure** while suppressing the visually implied source. **Phase 2** removes video conditioning entirely and focuses on refining **timbre and identity** toward the target prompt.

### Phase 1: video-derived temporal structure with decomposed guidance

For timesteps \(i \in [0, N_\mathrm{trans})\), the goal is to exploit video for event timing while avoiding the degradation that occurs when the backbone is asked to process the conflicting pair \((c_\mathrm{vid}, c_\mathrm{tar})\). The paper contrasts this with standard classifier-free guidance, which would combine video and target text inside a single conditional branch and thereby encourage the backbone to prioritize video.

ConterFlow replaces vanilla CFG with the following **decomposed guidance**:
\[
\begin{aligned}
v_i^{(1)} =\; v_i(\emptyset_\mathrm{vid}, \emptyset_\mathrm{txt}) &+ w_{\mathrm{vid}}\left(v_i(c_\mathrm{vid}, \emptyset_\mathrm{txt}) - v_i(\emptyset_\mathrm{vid}, \emptyset_\mathrm{txt})\right) \\
&+ w_{\mathrm{txt}}\left(v_i(\emptyset_\mathrm{vid}, c_\mathrm{tar}) - v_i(\emptyset_\mathrm{vid}, c_\mathrm{src})\right).
\end{aligned}
\tag{1}
\]

Here the unconditional term provides a baseline trajectory, the video-only term contributes **video-derived temporal structure** without introducing text identity, and the text-only contrastive term pushes the latent toward the target prompt while pushing it away from the source prompt. The reported weights are \(w_\mathrm{vid}=3.0\) and \(w_\mathrm{txt}=5.0\) [2605.18916].

A central design point is that the model **never** evaluates the conflicting pair \((c_\mathrm{vid}, c_\mathrm{tar})\) within a single forward pass. Instead, video and text are separated into different conditional branches whose contributions are linearly combined. The paper attributes low-fidelity and visually anchored outputs to asking the backbone to resolve such incompatible conditioning directly. In the authors’ interpretation, Phase 1 supplies **“where/when”** from video and **“which source”** from text, with explicit suppression of the visually implied source.

### Phase 2: text-only identity shaping via negative prompting

For timesteps \(i \in [N_\mathrm{trans}, N]\), the method removes video conditioning altogether:
\[
v_i^{(2)} = v_i(\emptyset_\mathrm{vid}, \emptyset_\mathrm{txt}) + w_{\mathrm{cfg}}\left(v_i(\emptyset_\mathrm{vid}, c_\mathrm{tar}) - v_i(\emptyset_\mathrm{vid}, c_\mathrm{src})\right).
\tag{2}
\]

In this stage, the temporal structure established during Phase 1 is assumed to be largely fixed, and the sampling trajectory is refined using **text-only contrastive guidance**. The reported weight is \(w_\mathrm{cfg}=4.5\). There is no continuous blending function; the transition is **piecewise**, occurring at the fixed step index \(i=N_\mathrm{trans}\). After the final step, the resulting latent is decoded by the audio VAE decoder into waveform audio [2605.18916].

The paper’s intuition is explicit: early steps are best used for **video-based temporal structuring**, whereas later steps are better suited for **identity refinement** without video. A plausible implication is that the method relies on a temporal separation in the generative trajectory between coarse event placement and finer timbral specification.

## 4. Evaluation protocol and counterfactual metrics

Evaluation is conducted on the **VGGSound-Sparse Clean subset**, described as single-source videos comprising **451 test videos** and **12 sound-source captions**. For each video, its annotated caption is used as the source prompt \(c_\mathrm{src}\), and the other **11 captions** are paired as target prompts \(c_\mathrm{tar}\), producing **4,961 triplets** \((c_\mathrm{vid}, c_\mathrm{tar}, c_\mathrm{src})\) in which the video sound identity and target text conflict [2605.18916].

The study reports standard metrics: **FAD** for audio quality, **Inception Score** for audio quality and diversity, **CLAP** for text-audio similarity between the target prompt and generated audio, and **DeSync** for video-audio temporal misalignment. The paper explicitly notes that **CLAP alone can be misleading**, because a model that produces both target and source sounds may score well on target relevance despite failing at clean counterfactual replacement.

To address this, the paper proposes a **FLAM-based metric** based on **frame-level language-audio modeling**. Let \(A\) denote the generated audio, \(c\) any text prompt, and \(p_\mathrm{FLAM}(c, A, l)\) the frame-wise probability that the content corresponding to text \(c\) is present at frame \(l\). The clip-level evidence is defined as
\[
P_\mathrm{FLAM}(c, A) = \max_l [p_\mathrm{FLAM}(c, A, l)].
\]
Using this quantity, the paper defines the counterfactual replacement score
\[
\Delta\mathrm{FLAM} = P_\mathrm{FLAM}(c_\mathrm{tar}, A) - P_\mathrm{FLAM}(c_\mathrm{src}, A),
\]
which increases when target evidence exceeds source evidence, and the success rate
\[
r_{>0} = \frac{1}{M} \sum_{m=1}^{M} \mathbf{1}\left[\Delta\mathrm{FLAM}^{(m)} > 0\right],
\]
which measures the fraction of clips for which target evidence is greater than source evidence.

These definitions are significant because they explicitly encode both sides of the replacement problem: strong evidence for the target prompt and weak evidence for the visually implied source. The paper presents this as a direct measurement of **replacement quality**, rather than merely prompt presence.

## 5. Comparative results and ablation findings

The baseline set includes **CAFA**, **ReWaS**, negative-prompting variants of both, the full **ConterFlow** method, and a variant labeled **“w/o P2 neg.”**. The following values are reported for the main counterfactual setup [2605.18916]:

| Method | FAD↓ | IS↑ | ΔFLAM↑ | (+)Ratio↑ | CLAP↑ | DeSync↓ |
|---|---:|---:|---:|---:|---:|---:|
| CAFA | 24.81 | 5.931 | 0.1289 | 0.8258 | 0.2371 | 0.5888 |
| CAFA + neg. | 31.46 | 7.606 | 0.2573 | 0.8835 | 0.1801 | 0.6431 |
| ReWaS | 75.18 | 4.223 | 0.0560 | 0.6184 | 0.1084 | 1.078 |
| ReWaS + neg. | 79.52 | 4.703 | 0.1905 | 0.7130 | 0.0947 | 1.103 |
| **ConterFlow** | **23.55** | 7.915 | **0.2641** | **0.9200** | **0.2840** | 0.6695 |
| w/o P2 neg. | 23.29 | 7.790 | 0.2373 | 0.9170 | 0.2849 | **0.6261** |

In the reported interpretation, ConterFlow achieves the best **\(\Delta\mathrm{FLAM}\)**, the best **positive-\(\Delta\mathrm{FLAM}\) ratio**, and the best **CLAP**, while maintaining **FAD** comparable to or better than CAFA and much better than ReWaS. **DeSync** is described as close to CAFA and much better than ReWaS. The comparison with naive negative prompting is particularly important: adding negative prompts to CAFA or ReWaS improves \(\Delta\mathrm{FLAM}\) and the success ratio, but damages **CLAP** and **DeSync**, consistent with the paper’s claim that naive negative prompting weakens both target and video conditioning.

A qualitative example, “dog barking \(\rightarrow\) lion roaring,” is used to illustrate the difference between mixed and clean replacement. In the description of Figure 3, **CAFA** exhibits both source and target evidence during the event, whereas **ConterFlow** shows low FLAM scores for the source prompt and high FLAM scores for the target prompt over the event duration, indicating cleaner replacement while respecting timing. The paper states that it does not describe a formal human listening test.

Ablations isolate the roles of Phase 1 decomposition, Phase 1 negative prompting, and phase ordering:

| Method | FAD↓ | ΔFLAM↑ | DeSync↓ | CLAP↑ |
|---|---:|---:|---:|---:|
| ConterFlow (full) | 23.55 | 0.2641 | 0.6695 | 0.2840 |
| w/o P1 decomp. CFG | 24.36 | 0.0278 | 0.2390 | 0.0894 |
| w/o P1 neg. | 21.00 | 0.0534 | 0.4362 | 0.2608 |
| Phase swap (P1 ↔ P2) | 52.33 | 0.2367 | 0.9989 | 0.2817 |

The paper’s conclusions from these ablations are direct. Removing **Phase 1 decomposed guidance** collapses \(\Delta\mathrm{FLAM}\) to near zero and sharply reduces CLAP, indicating that vanilla CFG with joint video-target conditioning fails to impose counterfactual identity. Removing **Phase 1 negative prompting** also sharply reduces \(\Delta\mathrm{FLAM}\), which the paper interprets as evidence that video features alone carry strong source identity and therefore require explicit suppression. Swapping the phase order degrades **FAD** and **DeSync** severely, confirming that the method depends on using early steps for temporal structuring and later steps for identity refinement [2605.18916].

The paper also sweeps the transition step \(N_\mathrm{trans}\). Increasing \(N_\mathrm{trans}\), thereby lengthening Phase 1, improves **DeSync** but reduces **\(\Delta\mathrm{FLAM}\)**. The selected value \(N_\mathrm{trans}=17\) is described as a **“knee”** in the trade-off between synchronization and counterfactual controllability. This suggests a tunable balance between temporal fidelity and source replacement strength.

## 6. Limitations, scope, and disambiguation

The paper identifies two explicit limitations. First, ConterFlow sometimes **generates sound during visually silent intervals**, which is described as limited **temporal gating**: video cues are not used to fully suppress audio outside active events. Second, the method is demonstrated mainly on **MMAudio**; although it is described as conceptually model-agnostic, broader validation on other VT2A backbones is left for future work [2605.18916].

The future directions listed in the paper follow directly from these limitations. One is to train models so that generation is **exclusively anchored to active visual cues**, with improved silence modeling between events. Another is to test the same two-phase decomposed-guidance framework on **other VT2A backbones** to assess generalizability. A plausible implication is that the core contribution is less a new architecture than a reusable inference-time control strategy for pretrained flow-matching VT2A systems.

The name “ConterFlow” is ambiguous in the literature provided here. In the paper that introduces the method, the spelling is explicitly **“CounterFlow”** [2605.18916]. The same string is also discussed in relation to **FlowCast**, a conditional flow-matching model for precipitation nowcasting, in the sense that “ConterFlow” may be used informally to refer to conditional flow-based forecasting ideas [2511.09731]. Separately, **ContentFlow** is an SDN architecture for mapping content names to IP flows over OpenFlow and is described as “sometimes miswritten as ‘ConterFlow’” [1302.1493]. In the present context, however, the term denotes the counterfactual video Foley generation method based on two-phase inference-time sampling for pretrained flow-matching VT2A models.

Source: https://www.emergentmind.com/topics/conterflow