FoleySpace: Vision-Aligned Binaural Audio

• FoleySpace is a joint representational regime that organizes text, audio, video, timing, and spatial cues into a controllable latent space.
• It generates immersive binaural audio by mapping per-frame visual cues to 3D trajectories and conditioning diffusion models on these signals.
• Comparative evaluations using CLAP embeddings, DDPM loss, and subjective tests highlight its effectiveness in synchronization and spatial consistency.

FoleySpace denotes either a specific framework for vision-aligned binaural spatial audio generation or, in a broader technical sense, a representational regime for Foley synthesis in which text, audio, video, timing, and spatial cues are organized into a controllable latent or embedding space. In the explicit sense, FoleySpace is a three-stage pipeline that estimates per-frame sound-source position and depth, maps them into a 3D trajectory, and conditions a diffusion model to generate spatially consistent binaural audio from video-guided monaural audio (Zhao et al., 18 Aug 2025). In adjacent literature, the term itself is often absent, but closely related systems instantiate analogous structures: CLAP-conditioned latent diffusion for text-driven Foley (Yuan et al., 2023), CLAP-regularized latent alignment (Karchkhadze et al., 2024), audio-conditioned acoustic transfer for video-to-audio synthesis (Fang et al., 16 Mar 2026), and multimodal control spaces based on text, video, reference audio, or time-varying acoustic control signals (Chen et al., 2024, Wang, 14 Oct 2025).

1. Terminology and scope

The literature represented here uses “FoleySpace” in two distinct ways. One is nominal: a concrete model family for binaural spatial audio generation from video. The other is conceptual: a joint control or embedding space in which Foley semantics, timing, timbre, and spatial attributes become navigable through learned representations.

Sense	Representative work	Core representation
Explicit framework	"FoleySpace: Vision-Aligned Binaural Spatial Audio Generation" (Zhao et al., 18 Aug 2025)	3D trajectory + mono audio conditioning for binaural diffusion
Joint text–audio manifold	"Text-Driven Foley Sound Generation With Latent Diffusion Model" (Yuan et al., 2023); "Latent CLAP Loss for Better Foley Sound Synthesis" (Karchkhadze et al., 2024)	CLAP-aligned embeddings and latent diffusion
Multimodal control space	"AC-Foley" (Fang et al., 16 Mar 2026); "MultiFoley" (Chen et al., 2024); "Audio Palette" (Wang, 14 Oct 2025)	Audio/video/text controls or time-varying acoustic signals

The benchmark context is important. The proposed Foley Sound Synthesis Challenge defines progressive levels ranging from categorical clip generation to sequential generation from text descriptions, sequential generation from video, and mixed soundtrack generation, explicitly emphasizing rigorous, unified, and efficient evaluation (Choi et al., 2022). This benchmark framing clarifies why FoleySpace-like systems repeatedly center on controllability, synchronization, and evaluation under constrained Foley taxonomies.

A recurrent distinction across the literature is that Foley generation is not speech or music generation. It targets “background and incidental sounds” such as footsteps, dog barks, gun shots, keyboard typing, rain, cough or sneeze, or moving motor vehicles, often under short-clip or video-synchronous conditions (Yuan et al., 2023). This domain specificity drives the design of its latent spaces: they must encode event category, transient structure, noise texture, timbre, and timing more tightly than generic audio models.

2. CLAP-aligned Foley spaces for text-driven generation

A foundational realization of FoleySpace is the CLAP-conditioned latent diffusion pipeline used for DCASE 2023 Task 7. In this formulation, a CLAP text encoder maps a label-derived prompt such as “Someone using keyboard” to a 512-dim embedding, a CLAP audio encoder maps waveforms into the same 512-dim space, a latent diffusion model denoises VAE latents conditioned on the embedding, and a HiFi-GAN vocoder reconstructs waveform audio (Yuan et al., 2023). The paper adds a trainable linear tuning layer after the text encoder,

$e' = W e + b,$

with $W$ initialized to identity and $b \sim \mathcal{N}(0,\sigma^2 I)$ and no activation, so that text conditioning can be adapted during fine-tuning (Yuan et al., 2023).

This system operationalizes FoleySpace as a shared text–audio manifold. During pretraining, the latent diffusion generator is conditioned on CLAP audio embeddings; during fine-tuning, it is conditioned on tuned text embeddings. At inference, multiple candidates are generated and reranked by cosine similarity between each candidate’s CLAP audio embedding and the tuned text embedding. The same paper reports that the system ranked first among submitted systems in DCASE Challenge 2023 Task 7, and that on its validation set it achieved overall FAD 4.52 compared with a baseline FAD 9.7 (Yuan et al., 2023). A closely related DCASE system report gives an average FAD of 4.765 across the seven classes, again against a 9.7 baseline, with class-specific specialization for transient classes versus spectrally diffuse classes such as rain and moving motor vehicle (Yuan et al., 2023).

The CLAP manifold is effective but computationally inefficient when semantic alignment is enforced only through post-filtering. "Latent CLAP Loss for Better Foley Sound Synthesis" moves that alignment into training by learning an encoder $\phi$ from diffusion latents to CLAP audio embeddings and adding a weighted latent-space alignment term,

$$L_{\mathrm{LCLAP}}(\theta)=\mathbb{E}_n \Big[w(n)\cdot \|E_{\mathrm{audio}}-E_{\mathrm{latent},n}\|^2\Big],$$

with $w(n)=10^{-n/200}$ (Karchkhadze et al., 2024). In that work, the best average FAD is 6.52 for LDM + Tuning + Latent, and removing post-filtering yields approximately an order-of-magnitude increase in generation speed for 100 clips of 10 seconds each on an NVIDIA GeForce RTX 3090 (Karchkhadze et al., 2024). This suggests a FoleySpace interpretation in which denoising trajectories are explicitly regularized toward the embedding distribution of real Foley recordings rather than being filtered against it after sampling.

The same design axis appears in class-conditioned latent models that do not use CLAP. "Exploring Domain-Specific Enhancements for a Neural Foley Synthesizer" constructs a hybrid representation, CEmbed, by concatenating mel-spectrograms and MERT embeddings into a $1152 \times 300$ representation, then learns an MVQVAE and a Zen PixelSNAIL prior with class-conditioning (Pillay et al., 2023). The paper reports that class-conditioning reduces latent diff from 0.2973 to 0.0179 on train and from 0.3669 to 0.0208 on validation, indicating a more structured class-separable latent space (Pillay et al., 2023).

3. Acoustic transfer and controllable conditioning

A second family of FoleySpace constructions emphasizes micro-acoustic control rather than text semantics alone. "Conditional Generation of Audio from Video via Foley Analogies" formulates conditional Foley as a user-guided analogy problem: an input silent video specifies what happens when, while a conditional audio-visual clip specifies how it should sound (Du et al., 2023). The model uses a VQGAN over mel spectrograms, a decoder-only Transformer, and a MelGAN vocoder, and autoregressively predicts target audio tokens conditioned on input-video tokens, conditional-video tokens, and conditional-audio tokens (Du et al., 2023). The paper reports that re-ranking improves human preference over the base model for both material and synchronization, and that the conditional formulation substantially outperforms unconditional SpecVQGAN on material control (Du et al., 2023).

"AC-Foley" makes this acoustic-control view explicit. It argues that text prompts are ill-suited to encode micro-acoustic features such as attack sharpness, resonance color, or decay profile, and therefore conditions generation directly on reference audio rather than CLAP-style semantic embeddings (Fang et al., 16 Mar 2026). Its audio pathway encodes waveforms into 40-D audio latents at 43.07 fps with a pretrained VAE, average-pools them into a compact acoustic embedding, and concatenates that embedding with pooled CLIP video and text features, Synchformer synchronization features, and timestep encoding into a multimodal conditioning vector injected through adaptive LayerNorm (Fang et al., 16 Mar 2026). The model is trained with conditional flow matching, and a two-stage overlap/non-overlap strategy forces the system to adapt reference timbre to new visual timing. On the VGGSound test split with audio conditioning, AC-Foley outperforms MMAudio+CLAP on FD, KL, IB, onset metrics, and MCD, with human evaluation showing an acoustic fidelity win rate of 83.5% and a temporal alignment win rate of 61.1% (Fang et al., 16 Mar 2026).

This control-centric reading of FoleySpace is further generalized by "Audio Palette," which defines a four-dimensional, time-varying control space: loudness, pitch, spectral centroid, and timbre via the first 13 MFCCs (Wang, 14 Oct 2025). The per-frame controls form a 16-D vector, are projected to the DiT latent channel dimension, and are injected at every denoising step, while text remains in cross-attention (Wang, 14 Oct 2025). Audio Palette introduces three independent guidance scales, $s_{\text{text}}$, $s_{\text{ctrls}}$, and $s_{\text{timbre}}$, and reports that LoRA adaptation trains only about 0.85 percent of the original parameters while maintaining FAD and LAION-CLAP scores close to the baseline model (Wang, 14 Oct 2025). In this formulation, FoleySpace is an explicitly navigable control manifold rather than only a semantic embedding space.

A more performance-oriented but similarly interactive interpretation appears in FOLEY-VAE, which compresses a 128-dimensional VAE latent space to an 8-dimensional PCA subspace retaining approximately 95% of the relevant information for Foley manipulation, exposes those variables in a Max 8 interface, and supports real-time timbre transfer from prerecorded audio or microphone-captured speech (Cámara et al., 2023). The report gives ViSQOL ≈ 4.1 ± 0.2, FAD ~ 0, and MSE en Mel ≈ 51 ± 27, framing FoleySpace as a directly manipulable artistic latent space rather than a purely generative conditioning manifold (Cámara et al., 2023).

4. Temporal alignment and video-conditioned Foley spaces

A separate line of work constructs FoleySpace primarily around synchronization. "AutoFoley" maps visual action sequences to spectrogram frames by combining ResNet-50 features, space-time images, Temporal Relationship Networks, and a multi-scale recurrent decoder (Ghose et al., 2020). Its central design choice is frame interpolation to 190 FPS, yielding roughly 950 video frames for approximately 5-second clips, which matches the approximately 950 STFT frames and enables a 1:1 correspondence between video time steps and audio spectrogram frames (Ghose et al., 2020). The model reconstructs audio by predicting per-class spectrogram offsets added to a class-average spectrogram and then applying ISTFT with Griffin–Lim. Human evaluation reports that 73.71% of test subjects selected the generated soundtrack from Model 1 as original (Ghose et al., 2020).

"Diff-Foley" replaces recurrent or autoregressive timing models with latent diffusion conditioned on contrastive audio-visual pretraining. Its CAVP module learns temporally aligned visual features through semantic and temporal InfoNCE losses, and those features condition a Stable Diffusion V1.4 UNet operating on spectrogram latents via cross-attention (Luo et al., 2023). Sampling uses “double guidance,” combining classifier-free guidance and classifier guidance:

$W$0

With DPM-Solver and 25 steps, the paper reports IS 62.37, FID 9.87, KL 6.43, Align Acc 94.05, and 0.38 s/sample on VGGSound (Luo et al., 2023). Here FoleySpace is effectively a temporally aligned cross-modal conditioning domain, built to maximize synchronization and audio-visual relevance.

"MambaFoley" pushes the timing-oriented view into waveform-domain diffusion. It conditions on category labels through FiLM and on an RMS-derived temporal profile through Block-FiLM, while replacing an LSTM bottleneck with a bidirectional Mamba layer (Colombo et al., 2024). On DCASE 2023 Task 7, it reports the best FAD across VGGish and PANNs embeddings among the compared systems, with FAD-V 7.77, FAD-P16 58.65, and FAD-P32 35.43, while maintaining comparable E-L1 alignment to other temporally conditioned baselines (Colombo et al., 2024). This indicates that a FoleySpace can be defined not only by semantic or timbral embeddings but also by the temporal envelope supplied to the generator.

MultiFoley extends this synchronization-first perspective into a multimodal latent diffusion transformer. It uses a DAC-VAE to encode 48 kHz audio into 40 Hz continuous latents, concatenates upsampled visual features from a frozen CAVP visual encoder with noisy audio latents, injects T5-base text through cross-attention, and can prepend reference-audio latents with mask embeddings for in-context conditioning (Chen et al., 2024). On VGGSound, MultiFoley reports ImageBind 28.0, CLAP 34.4, AV-Sync 0.80 s, FAD@VGG 2.92, FAD@AUD 4.62, and KLD 1.43, while human 2AFC against FoleyCrafter yields win rates of 85.8% for semantics, 94.5% for sync, 86.5% for quality, and 90.2% overall (Chen et al., 2024).

5. FoleySpace as vision-aligned binaural spatial audio generation

The explicit FoleySpace framework is "FoleySpace: Vision-Aligned Binaural Spatial Audio Generation" (Zhao et al., 18 Aug 2025). Its purpose is not merely to generate semantically plausible Foley, but to produce immersive and spatially consistent stereo sound guided by visual information. The system has three stages: sound source estimation per frame, coordinate mapping to a 3D trajectory, and diffusion-based binaural generation conditioned on both the trajectory and a monaural audio stream generated by MMAudio (Zhao et al., 18 Aug 2025).

Sound source estimation uses YOLO-World to identify the per-frame 2D pixel coordinates of the source and DepthMaster to estimate monocular depth. The model then applies a resolution-normalized mapping with

$W$1

where $W$2 is set to 1.47 m in implementation, and computes

$W$3

with depth normalized through $W$4 (Zhao et al., 18 Aug 2025). The resulting trajectory is smoothed by removing motion outliers above the 95% percentile and linearly interpolating missing coordinates. Azimuth, elevation, and distance are then derived from the 3D trajectory and interpolated to the audio timeline (Zhao et al., 18 Aug 2025).

The binaural generator is based on DiffWave and receives monaural audio together with per-sample trajectory features. Training data are synthesized from HUTUBS HRIR recordings for subjects pp87–pp96 under fine-grained and coarse-grained motion schemes, with 10,000 mono clips from VGGSound used to construct 10k binaural samples per scheme (Zhao et al., 18 Aug 2025). The explicit training objective is the standard DDPM noise-prediction loss on binaural waveform targets conditioned on mono audio and trajectory. The paper reports that FoleySpace outperforms spatial baselines in user study and objective evaluation. In the user study, FoleySpace achieves PSS 3.72 ± 0.96, SA 3.85 ± 0.97, TA 3.79 ± 0.96, SC 3.81 ± 0.99, and AQ 3.69 ± 1.08, while on VGGSound-Solo it reports FD_PANNs 7.25, FD_VGG 1.73, KL_PANNs 1.76, KL_PaSST 1.45, IS 14.55, IB-score 26.42, and DeSync 0.50 (Zhao et al., 18 Aug 2025).

This explicit formulation also makes the assumptions of FoleySpace unusually transparent. It assumes a single dominant sound source, a listener at the image center facing forward, no calibrated camera intrinsics or extrinsics, and no explicit room acoustics beyond HRIR-based rendering (Zhao et al., 18 Aug 2025). In exchange, it produces a spatial Foley representation in which visual motion is mapped to direction, elevation, and distance, and binaural cues evolve consistently over time.

6. Evaluation regimes, limitations, and future directions

Across the literature, FoleySpace is evaluated by a heterogeneous but increasingly standardized metric set. Distributional quality is commonly measured by Fréchet Audio Distance or related FD variants (Yuan et al., 2023, Karchkhadze et al., 2024, Fang et al., 16 Mar 2026, Zhao et al., 18 Aug 2025). Cross-modal semantics are measured with CLAP or ImageBind similarities (Chen et al., 2024, Fang et al., 16 Mar 2026). Temporal consistency appears as Align Acc in Diff-Foley, Synchformer-derived AV-Sync or DeSync in video-conditioned systems, onset accuracy and onset AP in AC-Foley, and E-L1 in MambaFoley (Luo et al., 2023, Fang et al., 16 Mar 2026, Colombo et al., 2024). Spatial systems add localization or spatial-alignment judgments, while subjective protocols range from MOS to pairwise preference studies (Zhao et al., 18 Aug 2025, Chen et al., 2024).

The dominant limitations are also consistent. Text-conditioned systems inherit data scarcity and possible embedding bias from general-purpose audio-text models; composite or noisy classes such as moving motor vehicle remain difficult because a single text embedding may be weakly discriminative for engine, road, and wind components (Yuan et al., 2023, Karchkhadze et al., 2024). Audio-conditioned systems improve timbral fidelity but still struggle with multi-source scenes and extreme tempo or timing mismatches between reference and target video (Fang et al., 16 Mar 2026). Video-conditioned systems improve synchronization but often rely on frozen feature extractors, latent-codec bottlenecks, or heuristic upsampling schemes that may limit long-range continuity or micro-timing precision (Luo et al., 2023, Chen et al., 2024). Spatial FoleySpace explicitly notes single-source focus, noisy monocular depth, heuristic geometry, missing room acoustics, and absent personalized HRTFs (Zhao et al., 18 Aug 2025).

Several future directions recur across the corpus. One is finer control: hybrid text+audio conditioning, learned attribute axes, multi-reference conditioning, and direct slider control over micro-acoustic properties (Fang et al., 16 Mar 2026, Wang, 14 Oct 2025). Another is longer and more structured generation: overlap-add segment generation, hierarchical latent models, event timing tokens, or explicit event detectors (Yuan et al., 2023, Fang et al., 16 Mar 2026). A third is broader realism: room acoustics, source separation, personalized HRTFs, and multi-source spatial rendering (Zhao et al., 18 Aug 2025). Finally, the benchmark literature suggests that a mature FoleySpace should be tied to progressive challenge design, hidden test sets, mixed objective and subjective evaluation, and clear separation between closed and open data settings (Choi et al., 2022).

Taken together, these works indicate that FoleySpace is best understood not as a single architecture but as a technical pattern. Whether realized through CLAP embeddings, pooled VAE audio latents, time-varying acoustic control signals, temporally aligned video features, or explicit 3D spatial trajectories, it is the learned coordinate system in which Foley generation becomes controllable, synchronizable, and increasingly decomposable into semantics, timing, timbre, and space (Yuan et al., 2023, Fang et al., 16 Mar 2026, Zhao et al., 18 Aug 2025).