Gray-Wyner Network for Audio Compression

• Gray-Wyner network is a multiterminal source coding architecture that decomposes correlated signals into common and private streams for efficient lossy and lossless compression.
• It employs transform coding and neural methods to decorrelate channels and allocate bits optimally, preserving crucial inter-channel spatial cues.
• Implementation techniques using DFT, RVQ, and two-branch neural designs improve rate-distortion trade-offs, enhancing ASR performance and spatial fidelity.

The Gray-Wyner network is a canonical multiterminal source coding architecture for lossy and lossless distributed compression of correlated signals—especially relevant in the context of microphone array audio, spatial audio capture, and multi-channel speech recognition. At its core, the network formalizes the problem of trade-offs between compression rate, distortion, and the preservation of inter-channel dependencies vital for spatial fidelity.

1. Fundamental Architecture and Principles

The Gray-Wyner network divides correlated sources into common and private components at the encoder, transmitting these as separate bitstreams to one or more decoders. This paradigm allows the compression system to allocate bitrate between information that is useful jointly (across all channels) and information that is only valuable when reconstructing each channel individually. In practice, for multi-microphone far-field ASR or spatial audio, this translates into transform or neural approaches that decorrelate channel signals, and then compress the decorrelated signals efficiently while retaining phase relationships necessary for downstream beamforming or audio rendering.

A representative block implementation for three microphones is as follows (Drude et al., 2021):

• Apply a three-point discrete Fourier transform (DFT) across channels, yielding one real-valued “DC” signal ($X_0$) and one complex pair ($X_1$, $X_2$), which are conjugates.
• Encode the DC and conjugate pair with a joint codec such as a mid/side SILK-style Opus configuration.
• Allocate bits among streams using constrained Lagrangian techniques to minimize distortion proxy (e.g., ASR normalized word error rate) at a given bitrate.

2. Transform Coding and Bit Allocation

The channelwise DFT is given by:

$$X_k(t) = \sum_{d=0}^2 x_d(t) e^{-j2\pi d k / 3},\quad k=0,1,2.$$

For real microphone signals, $X_2 = X_1^*$, so energy is compacted mostly into $X_0$, with $X_1$ and $X_2$ encoding spatial (inter-mic phase) information. This decorrelation step is critical: as shown in (Drude et al., 2021), about 90% of energy is typically concentrated in $X_0$, allowing aggressive bit allocation schemes that preserve spatial cues with limited bitrate:

• Distribute total encoded bitrate $B$ among $X_0$, $\mathrm{Re}\{X_1\}$, and $\mathrm{Im}\{X_1\}$ by minimizing front-end distortion $D(b)$ subject to $b_0 + b_1 + b_2 = B$.
• Empirically, allocating proportionally more bits to $X_0$ exploits energy compaction, but sufficient bits must remain for $X_1$/$X_2$ to preserve phase, else spatial fidelity degrades and beamforming fails.

3. Preservation of Spatial Cues and ASR Performance

Preserving inter-channel phase differences through the codec is paramount for beamforming accuracy and, by extension, ASR performance when operating on compressed signals. The transform-based Gray-Wyner-style approach:

• Enables post-decoding recreation of original phase lags, allowing downstream neural MVDR or Delay-and-Sum beamformers to operate as if on uncompressed inputs (Drude et al., 2021).
• At low bitrates (e.g., 24 kb/s/channel for three mics), achieves a <5% normalized WER loss relative to naive per-mic coding at 32 kb/s/channel, or offers a 3–4% relative WER improvement at fixed bitrate.
• Avoids the catastrophic WER increases (up to 15% relative) observed with independent per-channel coding when phase cues collapse.

4. Neural and Learned Extensions

Deep learning architectures have extended the Gray-Wyner principle to more expressive representations:

• In VCNAC (“Variable-Channel Neural Audio Codec”) (Grötschla et al., 21 Jan 2026), three input channels are processed by weight-shared convolutional streams with learned channel embeddings, fused into a single latent vector by summation, then quantized by residual vector quantization (RVQ). Channel outputs are reconstructed with small learned decoder embeddings and transposed convolutions, optionally with cross-channel attention.
• Channel-compatibility objectives add reconstruction losses for mixes down to stereo and mono, enforcing joint representational consistency and graceful quality degradation under channel drops or target-side downmixing.
• Tokenized codebooks for all channel counts allow cross-modal generative modeling with a single LLM for mono, stereo, and three-channel tasks.

5. Two-Branch Neural Designs for Spatial Speech Coding

SpatialCodec (Xu et al., 2023) operationalizes Gray-Wyner-like two-branch compression for spatial speech:

• Branch I compresses the spectral content of a reference channel using a neural sub-band codec.
• Branch II encodes only the spatial relations (relative signal structure) between reference and non-reference channels via spatial covariance statistics, with both branches employing RVQ.
• Non-reference channels are reconstructed using complex ratio filters parameterized by the spatial representation and applied to the (potentially quantized) reference channel’s STFT.
• Training combines time-domain SNR, spectral reconstruction, and adversarial losses.

6. Rate–Distortion Trade-offs and Performance

The Gray-Wyner network and its neural successors optimize channel-rate allocations to balance spatial fidelity and overall distortion:

Method/paper	Channels	Typical Rate (kb/s/ch)	Spatial Similarity	Noted Gains
Opus transform (Drude et al., 2021)	3	24–32	Phase preserved for beamforming	25% bitrate savings or 3–4% WER gain
VCNAC (Grötschla et al., 21 Jan 2026)	3	7.9 total	SI-SDR ≈ 5.7 dB (front chans)	Outperforms ≥14 kb/s neural codecs
SpatialCodec (Xu et al., 2023)	3	12 total	Spatial sim ~0.95	Real-time, >0.95 cosine similarity

Noise and beamformed metrics (SI-SDR, PESQ, cosine beamspace similarity) confirm that transform coding and neural branch separation preserve spatial cues significantly better than per-channel independent codecs at equivalent bitrates.

7. Implementation Details and Best Practices

Key design and deployment considerations include:

• For transform-based systems (Drude et al., 2021), use standard three-point DFT and its inverse around the codec, enable mid/side coding and waveform matching in Opus SILK mode, and use Lagrange-point optimization for bit allocation.
• Neural codecs (Grötschla et al., 21 Jan 2026, Xu et al., 2023) should employ weight-shared convolutional networks with learned embeddings to maintain channel identity, fusion at the bottleneck, and RVQ for codebook efficiency. Auxiliary loss functions on derived mixes enforce compatibility and spatial cue preservation.
• For spatial speech, separating reference and spatial branches aligns bitrate allocation with perceptually and functionally distinct information, attaining real-time feasibility (<0.1× real-time GPU, <1× optimized CPU), and maximizing spatial similarity metrics while maintaining spectral fidelity (Xu et al., 2023).

The Gray-Wyner network thus underpins both classical transform and advanced neural network approaches, crucially enabling bitrate-efficient, spatially coherent, and ASR-compatible multi-channel audio compression.