Woosh: Open Sound Effects Model

• Woosh is an open sound effects foundation model suite that integrates GAN-based audio encoding, robust text–audio alignment, and latent diffusion for text-to-audio and video-to-audio generation.
• It employs a modular pipeline with components such as Woosh-AE, Woosh-CLAP, Woosh-Flow, and distilled variants to enable efficient, real-time, high-fidelity synthesis.
• It achieves superior performance on benchmarks like MelDist, SI-SDR, and CLAP scores, supporting applications in rapid prototyping for games, films, and creative sound design.

Woosh is a publicly released sound effects foundation model suite developed by Sony AI, designed to provide a fully open, high-fidelity pipeline for professional sound-effect generation under both text and video conditioning. It delivers end-to-end capabilities for encoding/decoding audio, robust text–audio alignment, and state-of-the-art generative diffusion models with both full and distilled variants for efficient, real-time synthesis. Woosh comprises four primary models—an audio encoder/decoder (Woosh-AE), a text–audio alignment model (Woosh-CLAP), a text-to-audio latent diffusion model (Woosh-Flow), and a video-to-audio latent diffusion model (Woosh-VFlow)—supplemented by distilled versions (Woosh-DFlow and Woosh-DVFlow) that enable low-resource deployment without significant performance sacrifice (Hadjeres et al., 2 Apr 2026).

1. System Architecture and Component Models

Woosh organizes its pipeline into four core modules, each addressing a specific sub-task in sound effect synthesis.

1. Woosh-AE (Audio Encoder/Decoder): Based on VOCOS, this module is a GAN-based latent encoder/decoder operating on Short-Time Fourier Transform (STFT) complex coefficients. The model architecture employs cascades of ConvNeXt blocks, predicting real $(x')$, imag $(y')$, and magnitude $m$, with complex STFT reconstructed as:

$$m' = \sqrt{x'^2 + y'^2}, \quad \mathbf{x} = \frac{\mathbf{x}'}{m'}, \quad \mathbf{y} = \frac{\mathbf{y}'}{m'}, \quad \mathrm{STFT} = \mathrm{softplus}(m)\,(\mathbf{x} + j\,\mathbf{y}).$$

No quantization is performed; aliasing is avoided by single-step STFT/iSTFT up- and downsampling.

1. Woosh-CLAP (Contrastive Language–Audio Pretraining): Utilizes a RoBERTa-Large (355 M parameters) text encoder, a PaSST (86 M parameters) audio encoder, and projects both modalities into a shared 1024-d space. Training is supervised by symmetric InfoNCE loss, supporting robust conditioning for downstream generation.
2. Woosh-Flow / Woosh-DFlow (Text-to-Audio Generation): Employs 12 multimodal transformer blocks, alternating between MultiStream (modality-specific attention/FFN) and SingleStream (joint attention across text tokens and noisy audio latents), with rotary position embeddings. Generation occurs in the Woosh-AE latent space, conditioned on CLAP embeddings.
3. Woosh-VFlow / Woosh-DVFlow (Video-to-Audio Generation): Extends Woosh-Flow by integrating SynchFormer-based video conditioning, increasing the multimodal block dimensionality to accommodate text, video, and noise. Video embeddings are projected to 1024-d space, with distinct attention mechanisms per modality.

Distilled models (DFlow/DVFlow) leverage MeanFlow distillation and adversarial objectives to reduce computational steps while maintaining competitive performance, enabling real-time operation.

2. Training Procedure and Datasets

2.1 Training Data Composition

Woosh leverages both public and private datasets:

• Public:
  • Freesound (370k clips, ≥44.1 kHz, CC0/CC-BY/CC-Sampling+)
  • AudioCaps (48k clips @32 kHz)
  • WavCaps (99k AudioSet clips @32 kHz)
  • VCTK speech (44k @48 kHz)
  • Wapy synthetic dataset (100k FM-synthesized @48 kHz)
  • Internal music (78k popular songs @44.1 kHz)
• Private:
  • ≈1M studio-quality SFX (>5500 hours) for the private variant, with large-scale captioning via LLMs

These are uniformly preprocessed to 48 kHz sampling, with model-specific chunking (e.g., 1 s for Woosh-AE, 5–10 s for generative modules).

2.2 Loss Functions and Optimization

Woosh-AE is trained with a composite GAN loss:

$$\mathcal{L} = 15\,\mathcal{L}_{\mathrm{spec}} + \mathcal{L}_{\mathrm{adv}} + 2\,\mathcal{L}_{\mathrm{FM}},$$

where $\mathcal{L}_{\mathrm{spec}}$ is a multi-scale Mel L¹ spectrogram loss, $\mathcal{L}_{\mathrm{adv}}$ is an LSGAN loss with eight discriminators (5 MPD, 3 MBMS-STFT), and $\mathcal{L}_{\mathrm{FM}}$ is the discriminator feature-matching loss. The model size is 221M parameters, with a latent dimension of 128 and a compression ratio of 3.75×.

Generation models (Flow/VFlow) are initially pretrained using flow-matching, which employs straight-line latent diffusion and an L² pathwise loss. The distillation stage applies MeanFlow and classifier-free guidance with latent adversarial losses, drastically reducing the number of function evaluations (NFEs) required for sampling—from ~140 to 4 per step.

3. Evaluation Methodology and Benchmarks

3.1 Objective Metrics

Modules are benchmarked against state-of-the-art open-source baselines, including StableAudio-Open (SAO-VAE) and TangoFlux.

• Woosh-AE (AudioCaps-Test, 950 samples):
  • MelDist: 0.032 (vs. SAO-VAE 0.217)
  • STFTDist: 1.18 (vs. 1.55)
  • SI-SDR: 20.8 dB (vs. –0.08 dB)
• Woosh-CLAP:
  • Recall@10 text-to-audio: 0.546 (public), 0.655 (private); 248% improvement over LAION-CLAP on professional SFX
• Woosh-Flow, Woosh-DFlow (AudioCaps-Test):
  • FD: 109.1 (Woosh-Flow), 132.1 (Woosh-DFlow)
  • KL(PaSST): 1.60, 1.84
  • CLAP score: 0.372, 0.323
  • Tangoflux: 131.9, 1.70, 0.347
  • SAO-VAE: 150.1, 4.07, 0.146
• Woosh-VFlow, Woosh-DVFlow (FoleyBench):
  • FD: 24.1 (Woosh-VFlow) vs. 30.5 (MMAudio-M)
  • KL: 1.77 vs. 1.98
  • CLAP: 0.325 vs. 0.292
  • IB: 0.293 vs. 0.297 (ground truth: 0.300)

3.2 Additional Diagnostics

Qualitative assessments via online demonstrations indicate high prompt compliance, low distortion, and robust video–audio synchronization, especially in complex scenes (e.g., FoleyBench). Limitations in video–audio sync metric reliability necessitate perceptual evaluation for deployment-critical use cases.

4. Inference, Distillation, and Deployment

Woosh supports rapid inference through two-tier modeling: full generative models (Flow/VFlow) and their distilled counterparts (DFlow/DVFlow).

• Inference details:
  • Full Flow/VFlow: DOPRI5 ODE (tolerance $10^{-3}$, ~70 steps $\rightarrow$ 140 NFEs with classifier-free guidance)
  • Distilled DFlow/DVFlow: Euler solver, 4 steps (4 NFEs); single-pass CFG (scale=7)
  • Model sizes: AE 221M, CLAP text 355M, CLAP audio 86M, Flow/VFlow 337M
  • Hardware: Full models require $(y')$024 GB GPU VRAM; distilled can run real-time on $(y')$112 GB

This tiered approach enables both high-fidelity and practical low-resource deployment, with code and weights provided for non-commercial use.

5. Applications, Limitations, and Future Directions

5.1 Target Applications

Woosh is intended for:

• Rapid prototyping of SFX in games and movies
• Creative sound-design with controls (e.g., loudness, timbre) via cross-attention or AdaLN
• Generation of sound variations or morphing (footsteps, ambient environments)
• Inpainting, loop generation, and few-shot personalization (e.g., DreamBooth/TokenVerse methodologies)

5.2 Limitations

Critical observations include:

• Domain shift: Public datasets do not match the quality or diversity of professional studio SFX; private models are necessary for top-tier output.
• Evaluation: Existing video–audio sync metrics (e.g., SynchFormer DeSync) are unreliable; perceptual user studies remain essential.

5.3 Future Research

Areas highlighted for extension:

• Fine-grained, time-wise conditioning for precise control
• Joint diffusion over multiple audio tracks (multichannel SFX synthesis)
• End-to-end audio–video–text co-training for holistic conditional generation
• On-device, real-time deployment through further model compression

6. Qualitative Insights and Access

Demo samples exemplify Woosh's ability to generate crisp transients, minimal distortion, and strong adherence to textual and video prompts. Deployment materials, including code and weights (non-commercial), are accessible at https://github.com/SonyResearch/Woosh, and demos are hosted at https://sonyresearch.github.io/Woosh.

7. Comparative Context and Significance

Woosh delivers a fully open sound-effects foundation model stack optimized for instantaneous, high-fidelity sound-effect synthesis. It consistently outperforms or rivals existing open alternatives in multiple benchmarks, illustrating the impact of integrating tailored domain data, robust alignment models, and effective distillation strategies for scalable deployment (Hadjeres et al., 2 Apr 2026).