WavJEPA-Nat: Dual-Channel Audio Modeling

• WavJEPA-Nat is a dual-channel extension of JEPA that robustly learns semantic audio representations from raw waveforms in noisy and reverberant settings.
• The architecture uses independent waveform encoders with 2D sinusoidal positional embeddings and transformer-based context prediction to achieve superior generalizability on various benchmarks.
• Empirical evaluations show 4–6% improvement in acoustic event recognition and 12.5 ms inference latency, enabling real-time performance on edge devices.

WavJEPA-Nat is a multi-channel extension of the WavJEPA architecture designed to learn robust, semantic audio representations from raw waveforms in naturalistic, noisy, and reverberant environments. Building upon the Joint-Embedding Predictive Architecture (JEPA) paradigm for time-domain foundation audio models, WavJEPA-Nat introduces a two-channel (binaural) waveform encoder and targets improved resilience to adverse acoustic conditions by training with spatialized and reverberant scene data. This approach achieves state-of-the-art robustness and generalizability for downstream audio tasks, while maintaining low computational latency and resource requirements (Yuksel et al., 27 Sep 2025).

1. Architectural Framework

WavJEPA-Nat ingests a two-channel raw waveform $x(t) \in \mathbb{R}^{T\times2}$, where each channel corresponds to a binaural audio signal. Each channel is processed by an independent waveform encoder $W$ (shared architecture, separate weights), yielding $w_1 = W(x_1) \in \mathbb{R}^{N\times d}$ and $w_2 = W(x_2) \in \mathbb{R}^{N\times d}$, with $d=768$ and $N \approx 200$ for 2 s segments sampled at 100 Hz tokenization.

The channel embeddings are concatenated to form $w = [w_1;\,w_2] \in \mathbb{R}^{2N \times d}$. Learnable 2D sinusoidal positional encodings $\mathrm{PE}_\text{time}$ and $\mathrm{PE}_\text{chan}$ are added to each token to encode both temporal and channel (spatial) information.

A distributed context $c \subset \{1, ..., 2N\}$ is sampled alongside $W$0 target blocks $W$1. The context encoder $W$2 operates on the context tokens $W$3 and is implemented as a ViT-B transformer with masked attention. The prediction step inserts learned mask tokens at the target indices and runs a predictor network $W$4 (ViT-S) to output $W$5. Target features are computed using an EMA-updated target encoder $W$6 (ViT-B).

The architecture is depicted textually as:

Module	Details	Output Shape
Waveform encoders	6 conv layers, 512 channels each	$W$7: $W$8
Concatenation + 2D PE	Channel/time sinusoidal embeddings	$W$9
Context encoder (ViT-B)	Masked attention over context tokens	$w_1 = W(x_1) \in \mathbb{R}^{N\times d}$0
Predictor (ViT-S)	Target block masked context	$w_1 = W(x_1) \in \mathbb{R}^{N\times d}$1
Target encoder (ViT-B, EMA)	Full $w_1 = W(x_1) \in \mathbb{R}^{N\times d}$2 embedding	$w_1 = W(x_1) \in \mathbb{R}^{N\times d}$3

This dual-channel formulation enables context–target prediction across a 2D grid of "time × channel" tokens.

2. Training Objective and Loss

WavJEPA-Nat adopts the JEPA $w_1 = W(x_1) \in \mathbb{R}^{N\times d}$4 objective, extended to the 2D token grid. For each batch, the context encoder $w_1 = W(x_1) \in \mathbb{R}^{N\times d}$5 processes $w_1 = W(x_1) \in \mathbb{R}^{N\times d}$6; masked context vectors $w_1 = W(x_1) \in \mathbb{R}^{N\times d}$7 are constructed by inserting learned mask tokens at the target indices $w_1 = W(x_1) \in \mathbb{R}^{N\times d}$8. The predictor $w_1 = W(x_1) \in \mathbb{R}^{N\times d}$9 produces predictions $w_2 = W(x_2) \in \mathbb{R}^{N\times d}$0 for each of $w_2 = W(x_2) \in \mathbb{R}^{N\times d}$1 blocks.

The target encoder $w_2 = W(x_2) \in \mathbb{R}^{N\times d}$2, tracking the context encoder with exponential moving average (EMA) $w_2 = W(x_2) \in \mathbb{R}^{N\times d}$3, produces reference targets $w_2 = W(x_2) \in \mathbb{R}^{N\times d}$4. The per-block loss is

$w_2 = W(x_2) \in \mathbb{R}^{N\times d}$5

and the total batch loss

$w_2 = W(x_2) \in \mathbb{R}^{N\times d}$6

The sampling parameters are $w_2 = W(x_2) \in \mathbb{R}^{N\times d}$7, $w_2 = W(x_2) \in \mathbb{R}^{N\times d}$8, $w_2 = W(x_2) \in \mathbb{R}^{N\times d}$9, and $d=768$0.

3. Simulation of Multi-Channel Data

WavJEPA-Nat is trained on a synthetic dataset of spatialized, multi-channel scenes engineered to reflect realistic acoustic environments. The data pipeline incorporates:

• RIR generation: Utilizing 85 3D house meshes from MatterPort3D, room impulse responses are simulated per house via a Monte-Carlo ray-tracer (SoundSpaces 2.0). These are combined with head-related impulse responses (HRIR) to generate binaural RIRs (BRIR). Reverberation times $d=768$1 span $d=768$2 s.
• Scene composition: Each example blends a 10 s AudioSet clip (resampled to 16 kHz) with a 10 s WHAMR! noise segment. For each, a random listener pose and multiple noise (and source) positions are selected. The signals are convolved with BRIRs to yield binaural clean and noisy components:

$d=768$3

The observed signals are mixed at a random SNR $d=768$4 dB,

$d=768$5

• Scale: 1000 scenes are generated per house, totaling 85,000 BRIR scene sets; 70,000 for training and 15,000 for internal validation.

No additional data augmentation is applied beyond the mixture and instance normalization.

4. Training Regimen and Hyperparameters

WavJEPA-Nat is trained with the following configuration:

• Input: 2 s segments at 16 kHz, downsampled to $d=768$6 time tokens; 2-channel (binaural) input.
• Model sizes: Waveform encoder: 6 convolutional layers, 512 channels per layer, $d=768$7 million params per channel. Context encoder (ViT-B): $d=768$8 million params. Predictor (ViT-S): $d=768$9 million params. Total: $N \approx 200$0 million parameters.
• Optimization: AdamW ($N \approx 200$1, $N \approx 200$2, $N \approx 200$3), learning rate with 100k linear warmup steps to $N \approx 200$4 peak, then cosine decay.
• Batching: Actual batch size of 16 with $N \approx 200$5 in-batch sampling for an effective batch size of 128.
• Sampling: $N \approx 200$6, $N \approx 200$7, $N \approx 200$8 tokens, $N \approx 200$9 for top-K layer averaging.
• Total steps: 375,000.
• Data augmentation: None used beyond initial data composition.

5. Empirical Robustness and Evaluation

WavJEPA-Nat is benchmarked on the HEAR (clean) and NatHEAR (noisy/reverberant) suites, using the generalizability score $w = [w_1;\,w_2] \in \mathbb{R}^{2N \times d}$0 (0–100%). Results show:

Benchmark	WavJEPA-Nat	WavJEPA (single-channel)	WavJEPA (full batch)
HEAR (clean)	60.0 %	58.3 %	66.0 %
NatHEAR	61.2 %	55.8 %	-

On acoustic scene and event recognition tasks (e.g., DCASE, FSD50K), WavJEPA-Nat shows 4–6% absolute improvement over WavJEPA; speech and music see 2–5% gains. Under varying SNR and reverberation ($w = [w_1;\,w_2] \in \mathbb{R}^{2N \times d}$1), WavJEPA-Nat's accuracy degrades less than 5% across +5 to +40 dB SNR, and maintains over 90% of clean-scene performance across $w = [w_1;\,w_2] \in \mathbb{R}^{2N \times d}$2 s.

6. Latency and Computational Profile

WavJEPA-Nat contains approximately 114 million parameters, with an inference cost of ~50 GFLOPs per 2 s, two-channel audio segment. The model achieves an end-to-end processing latency of approximately 12.5 ms (given a 10 ms stride and 12.5 ms receptive field), with measured real-time factor $w = [w_1;\,w_2] \in \mathbb{R}^{2N \times d}$3 (20 times faster than real time) on a V100 GPU.

Resource efficiency is ensured by direct raw waveform processing—avoiding the overhead of spectrogram (STFT) computation—and by using parallel, compact waveform encoders and efficient ViT-based stacks at 100 Hz tokenization. This enables real-time deployment on edge hardware.

7. Significance and Implications

WavJEPA-Nat demonstrates the practicality of foundation audio models that operate in the time domain and are robust to challenging, variable acoustic environments. Its dual channel, JEPA-style context–target prediction—trained on large-scale spatialized, reverberant, and noisy data—enables not only competitive generalization on clean benchmarks but also consistent performance under noise and reverberation, with low latency and modest resource demands. These characteristics make WavJEPA-Nat suited for real-world, real-time audio applications where traditional spectrogram-based or non-spatial models are limited (Yuksel et al., 27 Sep 2025).