StereoFoley: Object-Aware Stereo Audio

• StereoFoley is a video-to-audio framework that generates immersive stereo Foley by aligning audio with visible actions, temporal events, and spatial object positions.
• It employs a two-stage model—a base latent diffusion network and an object-aware fine-tuning variant using synthetic stereo data and cinematic panning heuristics.
• Empirical results demonstrate improved spatial alignment and immersive audio realism over traditional mono synthesis despite reliance on heuristic spatial cues.

Searching arXiv for StereoFoley and related stereo/binaural video-to-audio papers. StereoFoley denotes a line of video-to-audio research concerned with generating stereo or binaural Foley that is semantically aligned with visible actions, temporally synchronized to on-screen events, and spatially anchored to the apparent source location. In its most specific usage, the term refers to the framework "StereoFoley: Object-Aware Stereo Audio Generation from Video" (Karchkhadze et al., 22 Sep 2025), which formulates stereo Foley as an object-aware generation problem for ordinary 2D video. Closely related systems include FoleySpace, which targets binaural spatial audio generation via explicit 3D sound-source trajectories (Zhao et al., 18 Aug 2025), and FoleyDesigner, which addresses film-oriented immersive stereo Foley through a workflow-inspired multi-stage framework (Li et al., 7 Apr 2026). Across these works, the common technical premise is that mono video-to-audio generation is insufficient for immersive audiovisual realism because it omits spatial cues, object correspondence, and, in some cases, professional post-production constraints.

1. Conceptual scope and problem definition

StereoFoley extends conventional video-to-audio generation by adding an explicit spatial dimension to the usual questions of semantic content and temporal alignment. In the formulation of StereoFoley, traditional systems primarily answer “what sound should this video make?” and “when should the sound occur?”, whereas stereo Foley additionally asks “where in the stereo field should that sound come from?” (Karchkhadze et al., 22 Sep 2025). The practical motivation is cinematic mixing behavior: objects are panned based on screen position, sounds may remain audible when objects leave the frame, background ambience is distributed more broadly across channels, and visible size or distance affects loudness (Karchkhadze et al., 22 Sep 2025).

This problem setting is distinct from generic stereo synthesis. The key issue is not merely producing two channels, but producing left/right structure that is visually grounded. FoleySpace makes this distinction explicit by arguing that prior “stereo” methods such as AudioX and ThinkSound use stereo VAEs or stereo output channels without explicitly enforcing visual-spatial consistency, while its own goal is “spatially consistent binaural audio” guided by estimated source position and depth (Zhao et al., 18 Aug 2025). FoleyDesigner similarly defines StereoFoley in film terms as requiring frame-accurate timing, spatio-visual consistency, and professional acoustic quality simultaneously (Li et al., 7 Apr 2026).

A recurring claim across the literature is that dataset quality is a central bottleneck. StereoFoley states that publicly available resources such as VGGSound and AudioSet do not provide professionally mixed, spatially reliable stereo supervision, and reports that about 27% of VGGSound clips are effectively mono, with nearly identical left/right channels (Karchkhadze et al., 22 Sep 2025). This motivates synthetic or augmented supervision pipelines in both StereoFoley and FoleySpace, while FoleyDesigner introduces a dedicated film-oriented stereo dataset with spatial metadata (Li et al., 7 Apr 2026).

2. Core architecture of StereoFoley

StereoFoley is organized as a two-stage framework: StereoFoley-base and StereoFoley-obj (Karchkhadze et al., 22 Sep 2025). The base model is a large latent diffusion video-to-audio model trained to generate synchronized stereo audio from video and text/audio conditioning. The object-aware variant is obtained by fine-tuning the base model on a synthetic stereo dataset designed to encode object-position-dependent spatial structure (Karchkhadze et al., 22 Sep 2025). This decomposition reflects the paper’s central thesis that the main bottleneck is data rather than architectural novelty.

The architecture uses three conditioning streams: an audio encoder or codec, a text encoder, and a video encoder, with a Diffusion Transformer (DiT) as the generative backbone (Karchkhadze et al., 22 Sep 2025). The inputs are defined as stereo audio

$x_{\text{audio} \in \mathbb{R}^{(T \times f_s) \times 2}$

text

$x_{\text{text} \in \mathbb{V}^L$

and video

$x_{\text{video} \in \mathbb{R}^{(T \times f_v) \times H \times W \times 3}.$

Stereo audio is encoded by an in-house stereo codec into latent tokens

$z \in \mathbb{R}^{T_z \times D_z},$

while text and audio are mapped into a shared CLAP embedding

$c \in \mathbb{R}^{D_c}.$

Video is encoded by SyncFormer to yield

$c_v \in \mathbb{R}^{T_v \times D_v}$

with the design constraint

$T_v = T_z,$

so that video and audio latent time grids align directly (Karchkhadze et al., 22 Sep 2025).

The generative model is a DiT with self-attention, cross-attention, gated MLP blocks, layer normalization, and skip connections (Karchkhadze et al., 22 Sep 2025). CLAP features are injected through cross-attention, whereas SyncFormer video features are added directly to the noisy audio latent. The diffusion training objective uses the $v$-objective:

$\mathcal{L}_{\text{Diffusion} = \left\| v_\theta(z_t, t, \{c, c_v\}) - v \right\|_2^2.$

This is presented as a comparatively simple synchronization design relative to earlier systems that require more elaborate architectural alignment machinery (Karchkhadze et al., 22 Sep 2025).

The base model is trained on 9.56-second clips with audio resampled to 48 kHz stereo and video standardized to 25 fps at $224 \times 224$. The paper reports audio embeddings of size $x_{\text{text} \in \mathbb{V}^L$0, video embeddings of size $x_{\text{text} \in \mathbb{V}^L$1, a DiT with 24 attention layers, 256 channels, 1536-dimensional embeddings, and about 1.1B parameters total. Optimization uses training from scratch for up to 800 epochs, batch size 512, 8 NVIDIA A100 GPUs, AdamW, learning rate $x_{\text{text} \in \mathbb{V}^L$2, weight decay $x_{\text{text} \in \mathbb{V}^L$3, and 2500 warmup steps. Training further uses classifier-free guidance with dropout 0.1, alternating audio/text CLAP conditioning with probability 0.5 each, and inference with 100 denoising steps and CFG scale 6.0 (Karchkhadze et al., 22 Sep 2025).

3. Object-aware stereo supervision and synthetic data generation

The distinctive contribution of StereoFoley lies in its synthetic object-aware stereo data pipeline, which converts video analysis outputs into stereo supervision obeying cinematic mixing conventions (Karchkhadze et al., 22 Sep 2025). The pipeline has four stages: video scene analysis, object detection and segmentation, audio generation and synchronization, and stereo spatialization and mixing.

In the scene analysis stage, Gemini 2.5-Flash is prompted with enumerated frames and task instructions to produce scene summaries. Each scene is represented as

$x_{\text{text} \in \mathbb{V}^L$4

where $x_{\text{text} \in \mathbb{V}^L$5 denotes scene start and end time, $x_{\text{text} \in \mathbb{V}^L$6 are sentences describing visible sound-making objects, and $x_{\text{text} \in \mathbb{V}^L$7 is a background description (Karchkhadze et al., 22 Sep 2025). This structured scene analysis is then used to guide open-vocabulary detection.

During detection and segmentation, the system parses scene text into nouns and verbs, filters generic plurals such as “people” or “cars,” focuses on single trackable objects, and groups detections into visual clusters $x_{\text{text} \in \mathbb{V}^L$8 (Karchkhadze et al., 22 Sep 2025). SAM2 is then used for video segmentation. For each cluster $x_{\text{text} \in \mathbb{V}^L$9, the per-frame pixel mass and centroid are computed as

$x_{\text{video} \in \mathbb{R}^{(T \times f_v) \times H \times W \times 3}.$0

where $x_{\text{video} \in \mathbb{R}^{(T \times f_v) \times H \times W \times 3}.$1 is the pixel set for the cluster in frame $x_{\text{video} \in \mathbb{R}^{(T \times f_v) \times H \times W \times 3}.$2 (Karchkhadze et al., 22 Sep 2025). These sequences act as spatial control signals.

Audio is synthesized object-wise using an in-house text-to-audio model conditioned on the object description $x_{\text{video} \in \mathbb{R}^{(T \times f_v) \times H \times W \times 3}.$3, and then synchronized with the visual clip by StereoFoley-base. Background ambience is also synthesized, but without strict synchronization because it is not tied to a specific visible event (Karchkhadze et al., 22 Sep 2025). This two-stage synthesis-plus-synchronization design is described as more stable than direct conditioning on segmented video alone.

Stereo spatialization is then applied by using horizontal object position for panning, pixel mass as a loudness or distance proxy, and explicit handling for off-frame persistence (Karchkhadze et al., 22 Sep 2025). Trajectories and masses are smoothed using weighted moving average, missing detections are interpolated, and off-frame segments are extrapolated using recent velocity. The explicit preservation of object permanence means that when an object leaves the frame, its sound does not vanish immediately.

The panning and loudness model is defined by a clipped horizontal coordinate

$x_{\text{video} \in \mathbb{R}^{(T \times f_v) \times H \times W \times 3}.$4

equal-power gains

$x_{\text{video} \in \mathbb{R}^{(T \times f_v) \times H \times W \times 3}.$5

and normalized mass term

$x_{\text{video} \in \mathbb{R}^{(T \times f_v) \times H \times W \times 3}.$6

with

$x_{\text{video} \in \mathbb{R}^{(T \times f_v) \times H \times W \times 3}.$7

The final stereo rendering for each object is

$x_{\text{video} \in \mathbb{R}^{(T \times f_v) \times H \times W \times 3}.$8

These object streams are mixed together with background ambience to form the synthetic stereo supervision used to fine-tune StereoFoley-base into StereoFoley-obj (Karchkhadze et al., 22 Sep 2025).

A closely related but technically different strategy appears in FoleySpace, which derives binaural supervision from HRIR-based rendering rather than panning heuristics. There, monaural 8-second clips from VGGSound are convolved with HRIRs from ten subjects in the HUTUBS HRTF database under fine-grained and coarse-grained motion scenarios, producing dynamic binaural training samples with explicit trajectory variation (Zhao et al., 18 Aug 2025). This suggests a broader methodological split in the field between heuristic stereo rendering and HRIR-conditioned binaural synthesis.

4. Spatial control formulations in related systems

StereoFoley’s object-aware panning formulation represents one approach to spatial grounding. FoleySpace provides a more explicit geometric model by estimating sound source location in each frame with YOLO-World, obtaining planar coordinates $x_{\text{video} \in \mathbb{R}^{(T \times f_v) \times H \times W \times 3}.$9, predicting depth with DepthMaster, and reading

$z \in \mathbb{R}^{T_z \times D_z},$0

These 2D-plus-depth observations are mapped into a listener-centered 3D trajectory through

$z \in \mathbb{R}^{T_z \times D_z},$1

and

$z \in \mathbb{R}^{T_z \times D_z},$2

where

$z \in \mathbb{R}^{T_z \times D_z},$3

The resulting trajectory

$z \in \mathbb{R}^{T_z \times D_z},$4

is smoothed using frame-to-frame motion magnitude

$z \in \mathbb{R}^{T_z \times D_z},$5

with outliers above the 95% threshold removed and linearly interpolated (Zhao et al., 18 Aug 2025). The diffusion model is then conditioned on monaural audio from MMAudio and the smoothed 3D trajectory.

FoleyDesigner adopts a film-production-oriented control representation. It extracts keyframes $z \in \mathbb{R}^{T_z \times D_z},$6, bounding boxes $z \in \mathbb{R}^{T_z \times D_z},$7, and depth maps $z \in \mathbb{R}^{T_z \times D_z},$8, computes average depth $z \in \mathbb{R}^{T_z \times D_z},$9, and converts horizontal center $c \in \mathbb{R}^{D_c}.$0 into azimuth

$c \in \mathbb{R}^{D_c}.$1

Temporal activity is encoded as

$c \in \mathbb{R}^{D_c}.$2

and the per-frame spatio-temporal control signal is

$c \in \mathbb{R}^{D_c}.$3

with $c \in \mathbb{R}^{D_c}.$4 (Li et al., 7 Apr 2026). These controls are injected into a DiT-based latent diffusion model built on Stable Audio Open.

The position encoding in FoleyDesigner uses Fourier features

$c \in \mathbb{R}^{D_c}.$5

and a temporally modulated version

$c \in \mathbb{R}^{D_c}.$6

with $c \in \mathbb{R}^{D_c}.$7, followed by a convolutional positional encoder producing

$c \in \mathbb{R}^{D_c}.$8

Cross-attention injection occurs at layers $c \in \mathbb{R}^{D_c}.$9 via

$c_v \in \mathbb{R}^{T_v \times D_v}$0

Relative to StereoFoley, this indicates a stronger emphasis on controllable film production and dense multi-event conditioning rather than the paper’s explicit claim that the primary barrier is supervision data (Li et al., 7 Apr 2026).

5. Data resources and evaluation methodology

A central issue in StereoFoley research is the mismatch between desired spatial supervision and available corpora. StereoFoley uses VGGSound as the main training set and an optional VGGSound plus AudioSet setup, but emphasizes the limitations of VGGSound for stereo learning. The paper reports about 200K examples in VGGSound, with about 27% effectively mono, and uses AudioSet as a larger supplemental source of roughly 2M clips after filtering out overlap, speech, and music (Karchkhadze et al., 22 Sep 2025). For fine-tuning, it constructs VGG-obj, a modified VGGSound subset chosen to satisfy geometric and scene criteria, including no more than 3 scenes, at least one scene with a single trackable object, object motion beyond 15% from center or stationary off-center placement, and size thresholds excluding objects $c_v \in \mathbb{R}^{T_v \times D_v}$1 or $c_v \in \mathbb{R}^{T_v \times D_v}$2 of frame area. Only about 18% of VGGSound meets these criteria, and the authors estimate fewer than 10% of those clips, about 1.8% of VGGSound overall, actually contain clearly object-aware stereo in the original audio (Karchkhadze et al., 22 Sep 2025).

FoleySpace addresses the data issue by synthesizing dynamic binaural trajectories from 10,000 monaural VGGSound clips and HRIRs from ten subjects in HUTUBS. The HUTUBS database provides 440 measured HRIR directions on a spherical grid. Two motion schemes are used: a fine-grained spatio-temporal scheme with azimuth range $c_v \in \mathbb{R}^{T_v \times D_v}$3, elevation range $c_v \in \mathbb{R}^{T_v \times D_v}$4, and direction variation rate $c_v \in \mathbb{R}^{T_v \times D_v}$5, and a coarse-grained scheme using 15 fixed directions with direction changes once per second and $c_v \in \mathbb{R}^{T_v \times D_v}$6 (Zhao et al., 18 Aug 2025). Distance variation is approximated by time-domain resampling to simulate source distances in the 0–6 m range; in the coarse scheme, depth is discretized into $c_v \in \mathbb{R}^{T_v \times D_v}$7 meters.

FoleyDesigner introduces FilmStereo, described as the first professional stereo audio dataset containing spatial metadata, precise timestamps, and semantic annotations for eight common Foley categories (Li et al., 7 Apr 2026). The paper reports 166 hours in the main text, while the supplement reports 42.3 hours across 14,784 samples after curation and statistics reporting. The dataset includes stereo audio, spatial captions, temporal annotations or timestamps, and spatial metadata such as azimuth, depth, and reverberation-related information (Li et al., 7 Apr 2026). This discrepancy between reported duration figures is present in the source description itself and should therefore be treated as part of the documented record rather than resolved here.

Evaluation also diverges across systems. StereoFoley follows MMAudio’s protocol and reports FD with PANNs, VGGish, and PaSST backbones; KL with PANNs and PaSST; IS; IB-score; DeSync; and Stereo-Score (Karchkhadze et al., 22 Sep 2025). However, it explicitly notes that these metrics are not ideal for stereo because the feature extractors operate at lower sample rates, are defined for mono audio, and require downmixing, making them insensitive to stereo spatial detail. To address this, the paper introduces bin-alignment score (BAS):

$c_v \in \mathbb{R}^{T_v \times D_v}$8

BAS compares quantized object horizontal position and audio energy center position over left, center, and right bins, with the center defined as $c_v \in \mathbb{R}^{T_v \times D_v}$9, and is reported separately for in-frame, off-frame, and combined segments (Karchkhadze et al., 22 Sep 2025).

FoleySpace uses subjective metrics PSS, SA, TA, SC, and AQ in a user study with 24 participants on 32 videos, alongside objective metrics including $T_v = T_z,$0, $T_v = T_z,$1, $T_v = T_z,$2, $T_v = T_z,$3, IS, IB-score, DeSync, and MAE on azimuth and elevation for coarse localization accuracy (Zhao et al., 18 Aug 2025). FoleyDesigner evaluates audio quality with IS, KL divergence, FAD, and CLAP score; spatio-temporal alignment with GCC-MAE, CRW-MAE, FSAD, and IoU; and cinematic Foley quality with ImageBind Score, AV-Sync, Sonic Richness Score, and Cinematic Clarity Score (Li et al., 7 Apr 2026).

6. Empirical findings and reported performance

StereoFoley reports that StereoFoley-base is competitive with MMAudio and Kling-Foley on standard video-to-audio metrics, despite using a simpler temporal alignment strategy based on matched latent time grids (Karchkhadze et al., 22 Sep 2025). The paper highlights Stereo-Score values of 0.21 for StereoFoley-base and 0.24 for StereoFoley-obj, and states that adding AudioSet yields only modest improvement. It further notes that StereoFoley-obj slightly degrades some standard metrics, which it attributes to partial replacement of real data by synthetic VGG-obj during fine-tuning (Karchkhadze et al., 22 Sep 2025). The principal gain appears in the object-awareness metric BAS, which improves from 0.23 for StereoFoley-base to 0.33 for StereoFoley-obj.

The listening study in StereoFoley comprises 1,341 ratings from 131 raters, with 49 disqualified raters. Mean opinion scores for stereo alignment are reported separately for in-frame only, leave/enter-frame events, and all cases. StereoFoley-obj achieves 3.54 in-frame, 3.37 for enter/exit events, and 3.46 overall; StereoFoley-base achieves about 3.05, 2.82, and 2.93 respectively; MMAudio is around 2.2 overall (Karchkhadze et al., 22 Sep 2025). A Kruskal–Wallis test with Bonferroni-corrected pairwise comparisons shows StereoFoley-obj significantly outperforming all others with $T_v = T_z,$4, MMAudio underperforming with $T_v = T_z,$5, StereoFoley-base and original VGGSound being statistically indistinguishable with $T_v = T_z,$6, and no significant difference between in-frame and off-frame conditions with $T_v = T_z,$7. The paper interprets this as evidence that BAS correlates well with human perception.

FoleySpace reports best or near-best subjective scores overall, especially for perceived spatiality and spatial alignment. The table values given for FoleySpace are PSS 3.72, SA 3.85, TA 3.79, SC 3.81, and AQ 3.69 (Zhao et al., 18 Aug 2025). The paper also describes waveform examples in which left and right channel amplitudes change dynamically to match source motion across the screen, whereas baselines often show static or identical stereo energy. An ablation finds that using offline vocabularies in YOLO-World instead of manual text labels yields only small MAE changes, suggesting the method can function without human-provided labels at inference. Another reported ablation shows HRIR-based binaural synthesis to be far superior to RIR-based synthesis on FD, KL, and IS, supporting the importance of human-ear filtering cues (Zhao et al., 18 Aug 2025).

FoleyDesigner reports best CLAP 0.679 and best FAD 1.88, as well as best GCC-MAE 48.79, best CRW-MAE 34.23, best FSAD 0.138, and best IoU 32.2 (Li et al., 7 Apr 2026). For film Foley quality it reports ImageBind 0.402, SRS 8.27, CCS 6.2, and AV-Sync 0.726. An ablation removing spatio-temporal cues changes GCC from 62.02 to 48.79, CRW from 55.89 to 34.23, FSAD from 0.297 to 0.138, and FAD from 2.14 to 1.88, which the paper attributes to the value of its position-aware injection mechanism (Li et al., 7 Apr 2026).

A concise comparison of representative reported results is given below.

System	Primary spatial mechanism	Representative reported results
StereoFoley (Karchkhadze et al., 22 Sep 2025)	Synthetic object-aware stereo supervision with panning and mass-based loudness	BAS 0.23 $T_v = T_z,$8 0.33; MOS 3.46 overall for StereoFoley-obj
FoleySpace (Zhao et al., 18 Aug 2025)	2D detection + depth $T_v = T_z,$9 3D trajectory + HRIR-trained binaural diffusion	PSS 3.72; SA 3.85; TA 3.79; SC 3.81; AQ 3.69
FoleyDesigner (Li et al., 7 Apr 2026)	Spatio-temporal cue injection into latent diffusion + workflow-aware refinement	CLAP 0.679; FAD 1.88; GCC-MAE 48.79; FSAD 0.138

These results are not directly interchangeable because the tasks, datasets, and evaluation protocols differ. A plausible implication is that the field currently lacks a single unified benchmark for stereo or binaural Foley, which complicates direct cross-paper comparison.

7. Limitations, misconceptions, and research trajectory

A common misconception is that StereoFoley is simply “mono-to-stereo conversion” or a post-processing method. The defining claim of StereoFoley is instead that object-aware stereo generation can be learned end-to-end from video, provided that supervision is spatially informative (Karchkhadze et al., 22 Sep 2025). Similarly, FoleySpace is not described as merely producing stereo output channels; its novelty is the explicit conditioning on estimated 3D sound-source trajectories derived from video (Zhao et al., 18 Aug 2025). FoleyDesigner likewise frames stereo Foley as a broader production-pipeline problem involving analysis, decomposition, controllable synthesis, and refinement (Li et al., 7 Apr 2026).

The limitations reported across these systems are substantial and partly complementary. StereoFoley acknowledges that its synthetic stereo pipeline uses amplitude panning and simple loudness heuristics rather than full physically based acoustics; relies on heuristic object tracking and scene parsing; is tested on 2D video rather than true 3D scenes; uses objective metrics that are not fully suited to stereo evaluation; and may incur slight reductions in some standard audio quality metrics after fine-tuning on synthetic data (Karchkhadze et al., 22 Sep 2025). FoleySpace depends on sound source localization by an open-vocabulary detector and monocular depth estimation, assumes a mostly single dominant sound source in the main evaluation setting, uses a simplified listener-centered 3D coordinate transformation, and trains on synthetically constructed HRIR-convolved mono audio rather than complex real acoustic recordings (Zhao et al., 18 Aug 2025). FoleyDesigner explicitly notes degraded performance in scenes with densely overlapping concurrent sound events, failures in spatial localization when multiple objects interact simultaneously, and a need for improved multi-object tracking and hierarchical reasoning (Li et al., 7 Apr 2026).

These limitations indicate several active directions. One is richer supervision: StereoFoley argues that data is the main bottleneck, FoleySpace demonstrates the utility of HRIR-based dynamic binaural datasets, and FoleyDesigner introduces film-oriented stereo annotations with spatial metadata. Another is improved spatial modeling: StereoFoley uses cinematic panning heuristics, FoleySpace uses explicit 3D listener-centered trajectories, and FoleyDesigner uses depth–azimuth controls injected at selected diffusion layers. This suggests a progression from heuristic stereo imaging toward more structured spatial representations, although the current literature still reflects multiple competing abstractions rather than a settled standard.

At a broader level, StereoFoley research sits at the intersection of video understanding, controllable generative audio, spatial hearing, and post-production practice. The papers considered here collectively support the view that high-quality stereo or binaural Foley requires not only semantic sound generation and synchronization, but also explicit treatment of source location, motion, persistence, and scene structure (Karchkhadze et al., 22 Sep 2025, Zhao et al., 18 Aug 2025, Li et al., 7 Apr 2026).