SpecMaskFoley: Video-Guided Foley Synthesis

• The paper demonstrates how adapting a pretrained audio transformer with a ControlNet framework achieves state-of-the-art foley synthesis with high semantic and temporal accuracy.
• SpecMaskFoley uses a Frequency-Aware Temporal Feature Aligner to effectively match high-rate video cues with lower-resolution spectrogram tokens.
• The method outperforms from-scratch approaches by delivering superior audio fidelity and synchronization while significantly reducing training data and computational costs.

SpecMaskFoley is a neural method for foley synthesis that adapts a pretrained spectral masked generative transformer toward generating audio that is both semantically and temporally synchronized with video. The system leverages ControlNet-based conditioning on high-dimensional video features and introduces a frequency-aware temporal feature aligner to resolve modality discrepancies, achieving state-of-the-art results across standard foley generation metrics. SpecMaskFoley demonstrates that pretrained audio generative models, when properly steered, can match or surpass from-scratch trained baselines in foley generation both in quality and synchronization, with significantly reduced data and computational requirements (Zhong et al., 22 May 2025).

1. Motivation and Problem Formulation

Foley synthesis requires automatic generation of realistic sound effects that are temporally and semantically matched to corresponding video content, such as precisely-aligned footfalls or collisions in film or game scenes. From-scratch video-to-audio generative models have demonstrated strong results (e.g., VATT, V-AURA, Frieren, MMAudio), but demand large-scale datasets (tens of thousands of hours), substantial engineering, and high GPU costs. In contrast, pre-existing text-to-audio (TTA) models, such as AudioLDM and SpecMaskGIT, encode high-level priors for timbre, global structure, and other salient audio features. Adaptation of such models appears attractive, but prior ControlNet-based steering typically utilized handcrafted and low-dimensional conditions (e.g., energy curves, onset stamps), leading to suboptimal audio fidelity and synchronization.

Two primary deficiencies were identified:

• ControlNet approaches offered limited synchronization and audio variety due to impoverished conditions.
• From-scratch methods, while flexible, were prohibitively expensive to train and required specialized architectures for cross-modal alignment.

SpecMaskFoley targets these issues by steering the pretrained SpecMaskGIT model using semantically rich, temporally fine-grained video features via an efficient and unified ControlNet-based framework, with a specialized aligner to bridge between video and audio token representations (Zhong et al., 22 May 2025).

2. Spectral Masked Generative Transformer (SpecMaskGIT) Backbone

The core of SpecMaskFoley is built on the SpecMaskGIT, a "masked generative transformer" for Mel-spectrogram token sequences. A typical input is a 10-second, 22.05 kHz audio segment, converted into an 80-bin Mel-spectrogram (848 time frames). This is tokenized using SpecVQGAN into a grid of shape $F=5$ (frequency) $\times T=53$ (time), where each token encodes a vector-quantized codebook index (256 possible values).

SpecMaskGIT uses MaskGIT-style parallel generative modeling. A subset of tokens in the grid $\mathbf{z} \in \{1,...,K\}^{F\times T}$ is masked at random: $$\tilde{\mathbf{z}} = M(\mathbf{z}, \theta),\quad M(\mathbf{z}, \theta)_{i,j} = \begin{cases} [\texttt{MASK}] & \text{w.p. }\theta \ \mathbf{z}_{i,j} & \text{otherwise} \end{cases}$$ A Vision Transformer variant (24 layers, 8 heads, hidden size 768) jointly predicts the full distribution over masked tokens per step using standard self-attention: $$\text{Attention}(Q, K, V) = \text{softmax}(QK^T/\sqrt{d})V$$ With cross-entropy training loss: $$\mathcal{L}_{\mathrm{ce}} = \mathbb{E}_{\mathbf{z},\,\theta} \left[-\sum_{(i,j)\in\mathrm{mask}} \log p_\phi(\mathbf{z}_{i,j}|M(\mathbf{z},\theta))\right]$$ This backbone, pretrained on AudioSet ($\sim2$M clips), provides strong audio prior for downstream adaptation (Zhong et al., 22 May 2025).

3. ControlNet Branch and Conditioning Signal

To facilitate video-guided audio generation, SpecMaskFoley adopts a ControlNet configuration. The first 12 transformer blocks of the SpecMaskGIT are duplicated to create a parallel "ControlNet" branch. At each block, a residual control signal derived from aligned video features $C_{\mathrm{align}}$ is injected: $$\tilde h^\ell = h^{\ell-1} + W_c^\ell C_{\mathrm{align}}$$ where $W_c^\ell$ is a learnable, zero-initialized linear projection at each layer. Outputs from these branches are merged back into the main stream via summation, allowing adaptive steering without corrupting the pretrained model at initialization. This ControlNet integration enables the system to respond to video-derived guidance and thus both synchronize and semantically match generated audio to visual events (Zhong et al., 22 May 2025).

4. Frequency-Aware Temporal Feature Aligner (FT-Aligner)

A core architectural challenge is the alignment of temporally high-rate video features and the lower-resolution, 2D structure of spectrogram tokens. The FT-Aligner module addresses this by transforming video features into a grid compatible with the audio token latent space.

Video-derived conditioning comprises:

• High-rate temporal features from Synchformer ($\times T=53$0, 25Hz)
• Low-rate CLIP semantic features ($\times T=53$1, 8Hz), temporally pooled to $\times T=53$2
• Fused as $\times T=53$3

These features are projected to length $\times T=53$4 and hidden dimension $\times T=53$5 via 1D convolution and adaptive average pooling: $\times T=53$6 The result is broadcast along frequency ($\times T=53$7) to yield

$\times T=53$8

providing all spectrogram tokens at each timestep with identical aligned temporal conditioning. This strategy eliminates the need for complex multi-branch or cross-attention conditioning architectures commonly adopted in prior arts (Zhong et al., 22 May 2025).

5. Training and Inference Objectives

SpecMaskFoley adopts cross-entropy over masked tokens ($\times T=53$9) as the primary loss, as described above. During inference, classifier-free guidance is extended to incorporate both text and video conditions: $\mathbf{z} \in \{1,...,K\}^{F\times T}$0 where $\mathbf{z} \in \{1,...,K\}^{F\times T}$1 denotes unconditioned logits, $\mathbf{z} \in \{1,...,K\}^{F\times T}$2 is conditioned on video features, $\mathbf{z} \in \{1,...,K\}^{F\times T}$3 conditions on both video and CLAP-derived text features, and $\mathbf{z} \in \{1,...,K\}^{F\times T}$4 is the adjustable CFG scale (up to 3). CLAP text conditioning is randomly dropped 90% of training steps for robustness. Zero-initialization of all ControlNet linear projections mitigates catastrophic forgetting at training onset (Zhong et al., 22 May 2025).

6. Experimental Protocol and Empirical Results

SpecMaskFoley was pretrained on AudioSet and finetuned with ControlNet on VGGSound (~180K 10s clips). Standardized acoustic and cross-modal metrics were used:

• Audio quality: Frechét Distance (FD, PaSST), Frechét Audio Distance (FAD, VGGish), KL divergence
• Semantic alignment: cosine similarity (ImageBind)
• AV synchronization: DeSync (measured by Synchformer, lower is better)
• Inference speed: per-clip timing on H100 GPU

Representative test results on VGGSound are summarized below:

Method	FD	FAD	KL	IB	DeSync (s)	Inf. time (s)
SpecMaskFoley	109	1.03	1.76	26.4	0.65	0.47
ReWaS	141	1.79	2.82	14.8	1.06	–
FoleyCrafter	140	2.51	2.23	25.7	1.23	–
VATT	132	2.77	1.41	25.0	1.20	–
V-AURA	218	2.88	2.07	27.6	0.65	–
Frieren	106	1.34	2.86	22.8	0.85	–
MMAudio	70.2	0.79	1.59	29.1	0.48	–

SpecMaskFoley achieves better AV synchronization and audio fidelity than both prior ControlNet variants and strong from-scratch baselines like VATT and V-AURA. Ablation reveals major drops in all metrics when ControlNet or the video CFG term is disabled. Synthesis quality remains robust with as few as 4–6 MaskGIT steps, and saturates by 12 steps—faster and more efficient than baseline systems.

Qualitatively, the method produces crisp, realistic sound events (e.g., footsteps, object collisions) with tightly aligned onsets and enhanced semantic congruence relative to video cues. A live demonstration is available at https://zzaudio.github.io/SpecMaskFoley_Demo/ (Zhong et al., 22 May 2025).

7. Limitations and Prospects

Several limitations persist:

• The coarse temporal resolution of the SpecVQGAN latent grid implies each token resets approximately every 188 ms in a 10s audio clip, potentially blurring very rapid sound onsets or high-precision cues.
• Complex multi-source audio events remain challenging for a fixed single-branch ControlNet approach; rare or overlapping scenes may not be handled optimally.

Suggested future directions include migrating to 1-D maskGIT backbones for finer time resolution, incorporating spatial audio and multi-view video conditioning, and enhancing video–audio alignment via adaptive CFG scales or hybrid cross-attention schemes. The out-of-the-box use of a pretrained backbone with a single ControlNet branch and FT-Aligner represents a significant simplification over previous methods, yielding fast training and efficient, high-quality synthesis (Zhong et al., 22 May 2025).