---
title: 'StereoFoley: Object-Aware Stereo Audio'
url: https://www.emergentmind.com/topics/stereofoley
type: topic
---

# StereoFoley: Object-Aware Stereo Audio

Searching arXiv for StereoFoley and related stereo/binaural video-to-audio papers.
StereoFoley denotes a line of video-to-audio research concerned with generating stereo or binaural Foley that is semantically aligned with visible actions, temporally synchronized to on-screen events, and spatially anchored to the apparent source location. In its most specific usage, the term refers to the framework "StereoFoley: Object-Aware Stereo Audio Generation from Video" [2509.18272], which formulates stereo Foley as an object-aware generation problem for ordinary 2D video. Closely related systems include FoleySpace, which targets binaural spatial audio generation via explicit 3D sound-source trajectories [2508.12918], and FoleyDesigner, which addresses film-oriented immersive stereo Foley through a workflow-inspired multi-stage framework [2604.05731]. Across these works, the common technical premise is that mono video-to-audio generation is insufficient for immersive audiovisual realism because it omits spatial cues, object correspondence, and, in some cases, professional post-production constraints.

## 1. Conceptual scope and problem definition

StereoFoley extends conventional video-to-audio generation by adding an explicit spatial dimension to the usual questions of semantic content and temporal alignment. In the formulation of StereoFoley, traditional systems primarily answer “what sound should this video make?” and “when should the sound occur?”, whereas stereo Foley additionally asks “where in the stereo field should that sound come from?” [2509.18272]. The practical motivation is cinematic mixing behavior: objects are panned based on screen position, sounds may remain audible when objects leave the frame, background ambience is distributed more broadly across channels, and visible size or distance affects loudness [2509.18272].

This problem setting is distinct from generic stereo synthesis. The key issue is not merely producing two channels, but producing left/right structure that is visually grounded. FoleySpace makes this distinction explicit by arguing that prior “stereo” methods such as AudioX and ThinkSound use stereo VAEs or stereo output channels without explicitly enforcing visual-spatial consistency, while its own goal is “spatially consistent binaural audio” guided by estimated source position and depth [2508.12918]. FoleyDesigner similarly defines StereoFoley in film terms as requiring frame-accurate timing, spatio-visual consistency, and professional acoustic quality simultaneously [2604.05731].

A recurring claim across the literature is that dataset quality is a central bottleneck. StereoFoley states that publicly available resources such as VGGSound and AudioSet do not provide professionally mixed, spatially reliable stereo supervision, and reports that about 27% of VGGSound clips are effectively mono, with nearly identical left/right channels [2509.18272]. This motivates synthetic or augmented supervision pipelines in both StereoFoley and FoleySpace, while FoleyDesigner introduces a dedicated film-oriented stereo dataset with spatial metadata [2604.05731].

## 2. Core architecture of StereoFoley

StereoFoley is organized as a two-stage framework: **StereoFoley-base** and **StereoFoley-obj** [2509.18272]. The base model is a large latent diffusion video-to-audio model trained to generate synchronized stereo audio from video and text/audio conditioning. The object-aware variant is obtained by fine-tuning the base model on a synthetic stereo dataset designed to encode object-position-dependent spatial structure [2509.18272]. This decomposition reflects the paper’s central thesis that the main bottleneck is data rather than architectural novelty.

The architecture uses three conditioning streams: an audio encoder or codec, a text encoder, and a video encoder, with a Diffusion Transformer (DiT) as the generative backbone [2509.18272]. The inputs are defined as stereo audio
$$
x_{\text{audio} \in \mathbb{R}^{(T  \times f_s) \times 2}
$$
text
$$
x_{\text{text} \in \mathbb{V}^L
$$
and video
$$
x_{\text{video} \in \mathbb{R}^{(T \times f_v) \times H \times W \times 3}.
$$
Stereo audio is encoded by an in-house stereo codec into latent tokens
$$
z \in \mathbb{R}^{T_z \times D_z},
$$
while text and audio are mapped into a shared CLAP embedding
$$
c \in \mathbb{R}^{D_c}.
$$
Video is encoded by SyncFormer to yield
$$
c_v \in \mathbb{R}^{T_v \times D_v}
$$
with the design constraint
$$
T_v = T_z,
$$
so that video and audio latent time grids align directly [2509.18272].

The generative model is a DiT with self-attention, cross-attention, gated MLP blocks, layer normalization, and skip connections [2509.18272]. CLAP features are injected through cross-attention, whereas SyncFormer video features are added directly to the noisy audio latent. The diffusion training objective uses the \(v\)-objective:
$$
\mathcal{L}_{\text{Diffusion} = \left\| v_\theta(z_t, t, \{c, c_v\}) - v \right\|_2^2.
$$
This is presented as a comparatively simple synchronization design relative to earlier systems that require more elaborate architectural alignment machinery [2509.18272].

The base model is trained on 9.56-second clips with audio resampled to 48 kHz stereo and video standardized to 25 fps at \(224 \times 224\). The paper reports audio embeddings of size \(224 \times 256\), video embeddings of size \(224 \times 768\), a DiT with 24 attention layers, 256 channels, 1536-dimensional embeddings, and about 1.1B parameters total. Optimization uses training from scratch for up to 800 epochs, batch size 512, 8 NVIDIA A100 GPUs, AdamW, learning rate \(10^{-4}\), weight decay \(10^{-3}\), and 2500 warmup steps. Training further uses classifier-free guidance with dropout 0.1, alternating audio/text CLAP conditioning with probability 0.5 each, and inference with 100 denoising steps and CFG scale 6.0 [2509.18272].

## 3. Object-aware stereo supervision and synthetic data generation

The distinctive contribution of StereoFoley lies in its synthetic object-aware stereo data pipeline, which converts video analysis outputs into stereo supervision obeying cinematic mixing conventions [2509.18272]. The pipeline has four stages: video scene analysis, object detection and segmentation, audio generation and synchronization, and stereo spatialization and mixing.

In the scene analysis stage, Gemini 2.5-Flash is prompted with enumerated frames and task instructions to produce scene summaries. Each scene is represented as
$$
s_i = \{T_i, O_{i,j}, B_i\},
$$
where \(T_i\) denotes scene start and end time, \(O_{i,j}\) are sentences describing visible sound-making objects, and \(B_i\) is a background description [2509.18272]. This structured scene analysis is then used to guide open-vocabulary detection.

During detection and segmentation, the system parses scene text into nouns and verbs, filters generic plurals such as “people” or “cars,” focuses on single trackable objects, and groups detections into visual clusters \(O_{i,j} \rightarrow c_{i,j}\) [2509.18272]. SAM2 is then used for video segmentation. For each cluster \(c_{i,j}\), the per-frame pixel mass and centroid are computed as
$$
m_{i,j}[k] = \lVert \Omega_{i,j,k} \rVert,\: (\bar w_{i,j}[k],\bar h_{i,j}[k]) = \frac{1}{\lVert \Omega_{i,j,k}\rVert} \sum_{(w,h)\in\Omega_{i,j,k} \left(\tfrac{2w}{W}-1,\,\tfrac{2h}{H}-1\right),
$$
where \(\Omega_{i,j,k}\) is the pixel set for the cluster in frame \(k\) [2509.18272]. These sequences act as spatial control signals.

Audio is synthesized object-wise using an in-house text-to-audio model conditioned on the object description \(O_{i,j}\), and then synchronized with the visual clip by StereoFoley-base. Background ambience is also synthesized, but without strict synchronization because it is not tied to a specific visible event [2509.18272]. This two-stage synthesis-plus-synchronization design is described as more stable than direct conditioning on segmented video alone.

Stereo spatialization is then applied by using horizontal object position for panning, pixel mass as a loudness or distance proxy, and explicit handling for off-frame persistence [2509.18272]. Trajectories and masses are smoothed using weighted moving average, missing detections are interpolated, and off-frame segments are extrapolated using recent velocity. The explicit preservation of object permanence means that when an object leaves the frame, its sound does not vanish immediately.

The panning and loudness model is defined by a clipped horizontal coordinate
$$
\hat w_{i,j}[n] = \Pi_{[-1,1]} \big(\bar w_{i,j}[n]\big),
$$
equal-power gains
$$
g^L_{i,j}[n] = \cos(\tfrac{\pi}{4}(\hat w_{i,j}[n]+1)),\qquad g^R_{i,j}[n] = \sin(\tfrac{\pi}{4}(\hat w_{i,j}[n]+1)),
$$
and normalized mass term
$$
v_{i,j}[n] = \frac{m_{i,j}[n]}{m_{\max} \cdot \frac{1}{1+\max(0,|\bar w_{i,j}[n]|-1)^2} }
$$
with
$$
m_{\max}=\max_k (\tilde m_{i,j}[k]).
$$
The final stereo rendering for each object is
$$
s^L_{i,j}[n] = v_{i,j}[n]\,g^L_{i,j}[n]\,a_{i,j}[n], \qquad s^R_{i,j}[n] = v_{i,j}[n]\,g^R_{i,j}[n]\,a_{i,j}[n].
$$
These object streams are mixed together with background ambience to form the synthetic stereo supervision used to fine-tune StereoFoley-base into StereoFoley-obj [2509.18272].

A closely related but technically different strategy appears in FoleySpace, which derives binaural supervision from HRIR-based rendering rather than panning heuristics. There, monaural 8-second clips from VGGSound are convolved with HRIRs from ten subjects in the HUTUBS HRTF database under fine-grained and coarse-grained motion scenarios, producing dynamic binaural training samples with explicit trajectory variation [2508.12918]. This suggests a broader methodological split in the field between heuristic stereo rendering and HRIR-conditioned binaural synthesis.

## 4. Spatial control formulations in related systems

StereoFoley’s object-aware panning formulation represents one approach to spatial grounding. FoleySpace provides a more explicit geometric model by estimating sound source location in each frame with YOLO-World, obtaining planar coordinates \((w_k, h_k)\), predicting depth with DepthMaster, and reading
$$
d_k = \mathbf{D}_{h_k, w_k}.
$$
These 2D-plus-depth observations are mapped into a listener-centered 3D trajectory through
$$
\delta = \frac{2S_y}{W}
$$
and
$$
\left\{\begin{aligned}
x_k &= \delta \tilde{d}_k,\ y_k &= \delta \left(w_k - \frac{W}{2}\right),\ z_k &= -\delta \left(h_k - \frac{H}{2}\right),
\end{aligned}\right.
$$
where
$$
\tilde{d}_k = \gamma\left( \frac{d_k - \min(\mathbf{D})}{\max(\mathbf{D}) - \min(\mathbf{D})} \right), \qquad \gamma = \frac{W}{2}.
$$
The resulting trajectory
$$
\mathcal{T} = \{ (x_k, y_k, z_k) \}_{k=1}^{K}
$$
is smoothed using frame-to-frame motion magnitude
$$
\Delta_k = \left\| (x_{k+1}, y_{k+1}, z_{k+1}) - (x_k, y_k, z_k) \right\|_2,
$$
with outliers above the 95% threshold removed and linearly interpolated [2508.12918]. The diffusion model is then conditioned on monaural audio from MMAudio and the smoothed 3D trajectory.

FoleyDesigner adopts a film-production-oriented control representation. It extracts keyframes \(\mathcal{K} = \{I_1,\dots,I_N\}\), bounding boxes \(\mathcal{B} = \{b_1,\dots,b_N\}\), and depth maps \(\mathbf{D}_i \in \mathbb{R}^{H \times W}\), computes average depth \(d_i\), and converts horizontal center \(x_i\) into azimuth
$$
\theta_i = \arctan\left(\frac{x_i - W/2}{d_i}\right) \cdot \frac{180^\circ}{\pi} + 90^\circ.
$$
Temporal activity is encoded as
$$
\mathbf{c} = \{c_1,\ldots,c_T\} \in \{0,1\}^T,
$$
and the per-frame spatio-temporal control signal is
$$
\mathbf{p}_t = c_t \cdot \mathbf{x}_t,\quad \mathcal{P}=\{\mathbf{p}_t\}_{t=1}^T
$$
with \(\mathbf{x}_t=(d_t,\theta_t)\) [2604.05731]. These controls are injected into a DiT-based latent diffusion model built on Stable Audio Open.

The position encoding in FoleyDesigner uses Fourier features
$$
\gamma(\mathbf{p}_t) = \left[ \cos(2\pi \mathbf{B}\mathbf{p}_t); \sin(2\pi \mathbf{B}\mathbf{p}_t) \right] \in \mathbb{R}^{2m}
$$
and a temporally modulated version
$$
\tilde{\gamma}(\mathbf{p}_t) = c_t \cdot \gamma(\mathbf{p}_t) + \epsilon \cdot \gamma(\mathbf{p}_t)
$$
with \(\epsilon=0.1\), followed by a convolutional positional encoder producing
$$
\mathbf{E}_{\text{pos} = \text{PosEncoder}(\{\tilde{\gamma}(\mathbf{p}_t)\}_{t=1}^T) \in \mathbb{R}^{T' \times d_{\text{emb} }.
$$
Cross-attention injection occurs at layers \(\ell \in \{3,7,11,15,19,23\}\) via
$$
\mathbf{z}'_{\ell} = \text{InjBlock}(\mathbf{z}_{\ell}, \text{LN}(\mathbf{E}_{\text{pos})).
$$
Relative to StereoFoley, this indicates a stronger emphasis on controllable film production and dense multi-event conditioning rather than the paper’s explicit claim that the primary barrier is supervision data [2604.05731].

## 5. Data resources and evaluation methodology

A central issue in StereoFoley research is the mismatch between desired spatial supervision and available corpora. StereoFoley uses VGGSound as the main training set and an optional VGGSound plus AudioSet setup, but emphasizes the limitations of VGGSound for stereo learning. The paper reports about 200K examples in VGGSound, with about 27% effectively mono, and uses AudioSet as a larger supplemental source of roughly 2M clips after filtering out overlap, speech, and music [2509.18272]. For fine-tuning, it constructs **VGG-obj**, a modified VGGSound subset chosen to satisfy geometric and scene criteria, including no more than 3 scenes, at least one scene with a single trackable object, object motion beyond 15% from center or stationary off-center placement, and size thresholds excluding objects \(\leq 2\%\) or \(\geq 60\%\) of frame area. Only about 18% of VGGSound meets these criteria, and the authors estimate fewer than 10% of those clips, about 1.8% of VGGSound overall, actually contain clearly object-aware stereo in the original audio [2509.18272].

FoleySpace addresses the data issue by synthesizing dynamic binaural trajectories from 10,000 monaural VGGSound clips and HRIRs from ten subjects in HUTUBS. The HUTUBS database provides 440 measured HRIR directions on a spherical grid. Two motion schemes are used: a fine-grained spatio-temporal scheme with azimuth range \([90^\circ, 270^\circ]\), elevation range \([-40^\circ, 40^\circ]\), and direction variation rate \(M=200\), and a coarse-grained scheme using 15 fixed directions with direction changes once per second and \(M=8\) [2508.12918]. Distance variation is approximated by time-domain resampling to simulate source distances in the 0–6 m range; in the coarse scheme, depth is discretized into \(\{1,2,3,4,5\}\) meters.

FoleyDesigner introduces **FilmStereo**, described as the first professional stereo audio dataset containing spatial metadata, precise timestamps, and semantic annotations for eight common Foley categories [2604.05731]. The paper reports 166 hours in the main text, while the supplement reports 42.3 hours across 14,784 samples after curation and statistics reporting. The dataset includes stereo audio, spatial captions, temporal annotations or timestamps, and spatial metadata such as azimuth, depth, and reverberation-related information [2604.05731]. This discrepancy between reported duration figures is present in the source description itself and should therefore be treated as part of the documented record rather than resolved here.

Evaluation also diverges across systems. StereoFoley follows MMAudio’s protocol and reports FD with PANNs, VGGish, and PaSST backbones; KL with PANNs and PaSST; IS; IB-score; DeSync; and Stereo-Score [2509.18272]. However, it explicitly notes that these metrics are not ideal for stereo because the feature extractors operate at lower sample rates, are defined for mono audio, and require downmixing, making them insensitive to stereo spatial detail. To address this, the paper introduces **bin-alignment score (BAS)**:
$$
BAS = \tfrac{1}{K} \sum_{k=1}^K \mathbf{1}[\mathrm{bin}(\bar{w}[k]) = \mathrm{bin}(\bar{a}[k])].
$$
BAS compares quantized object horizontal position and audio energy center position over left, center, and right bins, with the center defined as \(\pm 5\%\), and is reported separately for in-frame, off-frame, and combined segments [2509.18272].

FoleySpace uses subjective metrics PSS, SA, TA, SC, and AQ in a user study with 24 participants on 32 videos, alongside objective metrics including \(\text{FD}_{\text{PANNs}}\), \(\text{FD}_{\text{VGG}}\), \(\text{KL}_{\text{PANNs}}\), \(\text{KL}_{\text{PaSST}}\), IS, IB-score, DeSync, and MAE on azimuth and elevation for coarse localization accuracy [2508.12918]. FoleyDesigner evaluates audio quality with IS, KL divergence, FAD, and CLAP score; spatio-temporal alignment with GCC-MAE, CRW-MAE, FSAD, and IoU; and cinematic Foley quality with ImageBind Score, AV-Sync, Sonic Richness Score, and Cinematic Clarity Score [2604.05731].

## 6. Empirical findings and reported performance

StereoFoley reports that StereoFoley-base is competitive with MMAudio and Kling-Foley on standard video-to-audio metrics, despite using a simpler temporal alignment strategy based on matched latent time grids [2509.18272]. The paper highlights Stereo-Score values of 0.21 for StereoFoley-base and 0.24 for StereoFoley-obj, and states that adding AudioSet yields only modest improvement. It further notes that StereoFoley-obj slightly degrades some standard metrics, which it attributes to partial replacement of real data by synthetic VGG-obj during fine-tuning [2509.18272]. The principal gain appears in the object-awareness metric BAS, which improves from 0.23 for StereoFoley-base to 0.33 for StereoFoley-obj.

The listening study in StereoFoley comprises 1,341 ratings from 131 raters, with 49 disqualified raters. Mean opinion scores for stereo alignment are reported separately for in-frame only, leave/enter-frame events, and all cases. StereoFoley-obj achieves 3.54 in-frame, 3.37 for enter/exit events, and 3.46 overall; StereoFoley-base achieves about 3.05, 2.82, and 2.93 respectively; MMAudio is around 2.2 overall [2509.18272]. A Kruskal–Wallis test with Bonferroni-corrected pairwise comparisons shows StereoFoley-obj significantly outperforming all others with \(p < 0.001\), MMAudio underperforming with \(p < 0.001\), StereoFoley-base and original VGGSound being statistically indistinguishable with \(p = 0.80\), and no significant difference between in-frame and off-frame conditions with \(p > 0.05\). The paper interprets this as evidence that BAS correlates well with human perception.

FoleySpace reports best or near-best subjective scores overall, especially for perceived spatiality and spatial alignment. The table values given for FoleySpace are PSS 3.72, SA 3.85, TA 3.79, SC 3.81, and AQ 3.69 [2508.12918]. The paper also describes waveform examples in which left and right channel amplitudes change dynamically to match source motion across the screen, whereas baselines often show static or identical stereo energy. An ablation finds that using offline vocabularies in YOLO-World instead of manual text labels yields only small MAE changes, suggesting the method can function without human-provided labels at inference. Another reported ablation shows HRIR-based binaural synthesis to be far superior to RIR-based synthesis on FD, KL, and IS, supporting the importance of human-ear filtering cues [2508.12918].

FoleyDesigner reports best CLAP 0.679 and best FAD 1.88, as well as best GCC-MAE 48.79, best CRW-MAE 34.23, best FSAD 0.138, and best IoU 32.2 [2604.05731]. For film Foley quality it reports ImageBind 0.402, SRS 8.27, CCS 6.2, and AV-Sync 0.726. An ablation removing spatio-temporal cues changes GCC from 62.02 to 48.79, CRW from 55.89 to 34.23, FSAD from 0.297 to 0.138, and FAD from 2.14 to 1.88, which the paper attributes to the value of its position-aware injection mechanism [2604.05731].

A concise comparison of representative reported results is given below.

| System | Primary spatial mechanism | Representative reported results |
|---|---|---|
| StereoFoley [2509.18272] | Synthetic object-aware stereo supervision with panning and mass-based loudness | BAS 0.23 \(\rightarrow\) 0.33; MOS 3.46 overall for StereoFoley-obj |
| FoleySpace [2508.12918] | 2D detection + depth \(\rightarrow\) 3D trajectory + HRIR-trained binaural diffusion | PSS 3.72; SA 3.85; TA 3.79; SC 3.81; AQ 3.69 |
| FoleyDesigner [2604.05731] | Spatio-temporal cue injection into latent diffusion + workflow-aware refinement | CLAP 0.679; FAD 1.88; GCC-MAE 48.79; FSAD 0.138 |

These results are not directly interchangeable because the tasks, datasets, and evaluation protocols differ. A plausible implication is that the field currently lacks a single unified benchmark for stereo or binaural Foley, which complicates direct cross-paper comparison.

## 7. Limitations, misconceptions, and research trajectory

A common misconception is that StereoFoley is simply “mono-to-stereo conversion” or a post-processing method. The defining claim of StereoFoley is instead that object-aware stereo generation can be learned end-to-end from video, provided that supervision is spatially informative [2509.18272]. Similarly, FoleySpace is not described as merely producing stereo output channels; its novelty is the explicit conditioning on estimated 3D sound-source trajectories derived from video [2508.12918]. FoleyDesigner likewise frames stereo Foley as a broader production-pipeline problem involving analysis, decomposition, controllable synthesis, and refinement [2604.05731].

The limitations reported across these systems are substantial and partly complementary. StereoFoley acknowledges that its synthetic stereo pipeline uses amplitude panning and simple loudness heuristics rather than full physically based acoustics; relies on heuristic object tracking and scene parsing; is tested on 2D video rather than true 3D scenes; uses objective metrics that are not fully suited to stereo evaluation; and may incur slight reductions in some standard audio quality metrics after fine-tuning on synthetic data [2509.18272]. FoleySpace depends on sound source localization by an open-vocabulary detector and monocular depth estimation, assumes a mostly single dominant sound source in the main evaluation setting, uses a simplified listener-centered 3D coordinate transformation, and trains on synthetically constructed HRIR-convolved mono audio rather than complex real acoustic recordings [2508.12918]. FoleyDesigner explicitly notes degraded performance in scenes with densely overlapping concurrent sound events, failures in spatial localization when multiple objects interact simultaneously, and a need for improved multi-object tracking and hierarchical reasoning [2604.05731].

These limitations indicate several active directions. One is richer supervision: StereoFoley argues that data is the main bottleneck, FoleySpace demonstrates the utility of HRIR-based dynamic binaural datasets, and FoleyDesigner introduces film-oriented stereo annotations with spatial metadata. Another is improved spatial modeling: StereoFoley uses cinematic panning heuristics, FoleySpace uses explicit 3D listener-centered trajectories, and FoleyDesigner uses depth–azimuth controls injected at selected diffusion layers. This suggests a progression from heuristic stereo imaging toward more structured spatial representations, although the current literature still reflects multiple competing abstractions rather than a settled standard.

At a broader level, StereoFoley research sits at the intersection of video understanding, controllable generative audio, spatial hearing, and post-production practice. The papers considered here collectively support the view that high-quality stereo or binaural Foley requires not only semantic sound generation and synchronization, but also explicit treatment of source location, motion, persistence, and scene structure [2509.18272][2508.12918][2604.05731].

Source: https://www.emergentmind.com/topics/stereofoley