FocalCodec-Stream: Streaming Neural Speech Codec

• The paper introduces FocalCodec-Stream, a streaming low-bitrate neural speech codec that employs multi-stage causal distillation and a single-codebook framework to preserve both semantic and acoustic information.
• It redesigns the encoder, compressor, and decoder for streamability by using causal convolutions, chunked attention with 80 ms latency, and a lightweight refiner to bridge teacher-student feature gaps.
• Empirical evaluations reveal competitive intelligibility and naturalness in real-time resynthesis and downstream tasks, outperforming several prior streamable codecs despite a high 249M parameter count.

FocalCodec-Stream is a streaming low-bitrate neural speech codec that extends the earlier non-causal FocalCodec into a latency-constrained setting while retaining its central design goal: a single-codebook representation intended to preserve both semantic and acoustic information. The model is described as a hybrid codec based on focal modulation, operating at $0.55$–$0.80$ kbps with a theoretical latency of $80$ ms, and built around multi-stage causal distillation of WavLM together with a lightweight refiner module that compensates for quality loss under streaming constraints (Libera et al., 19 Sep 2025).

1. Origins, problem setting, and design position

FocalCodec-Stream addresses a specific gap in neural speech coding: many low-bitrate codecs are useful for offline reconstruction and downstream representation learning, but are not deployable in real-time systems because they depend on substantial future context. The model is therefore positioned for latency-sensitive applications such as speech assistants, interactive dialogue, low-latency generative speech systems, and speech LLMs that require responsive turn-taking (Libera et al., 19 Sep 2025).

Its immediate antecedent is "FocalCodec" (Libera et al., 6 Feb 2025), which introduced a low-bitrate codec using focal modulation networks, a single binary spherical codebook, and a frozen WavLM-large front end, but explicitly remained non-causal. FocalCodec-Stream preserves the same broad semantic-acoustic objective while reworking the encoder, bottleneck, and decoder for streamability. In this sense it belongs to the same research lineage as other low-rate single-stream codecs, but it also intersects with streamable speech coding work that treats causality and latency as first-class design constraints.

A useful neighboring reference point is "TS3-Codec: Transformer-Based Simple Streaming Single Codec" (Wu et al., 2024), which established that a single-codebook streaming codec can remain competitive without convolution-heavy backbones by using frame blocking, bounded left-context attention, and low token rate. FocalCodec-Stream differs in architectural substrate—WavLM distillation and focal modulation rather than a transformer-only waveform codec—but it shares the same core commitments to streaming operation, low bitrate, and a single discrete token stream.

A common misconception is to treat FocalCodec-Stream as merely a causalized version of FocalCodec. The paper instead describes a broader redesign: the WavLM front end is causally distilled, focal modules are modified for streaming, a new refiner is added, and the decoder is made streamable. The result is not a trivial causal wrapper around the original offline model.

2. Architecture and streaming runtime path

The end-to-end pipeline is:

1. a streamable version of the first six layers of WavLM-large,
2. a focal-modulation-based compressor,
3. binary spherical quantization (BSQ) with a single codebook,
4. a focal-modulation-based decompressor,
5. a lightweight refiner,
6. a streamable decoder based on a Vocos-like design (Libera et al., 19 Sep 2025).

The encoder is the most consequential departure from the original FocalCodec. Standard WavLM convolutions are replaced with causal convolutions, and full-context gated relative attention is replaced with sliding-window gated relative chunked attention. The streaming attention uses a chunk size of 80 ms, corresponding to 4 feature frames, allows look-ahead of up to 3 frames, and caps past context at 10.24 s, or 512 feature frames. Very old context is dropped, so streaming memory remains bounded.

The compressor and decompressor remain focal-modulation-based, but their streaming implementation is also altered. FocalCodec-Stream uses three focal downscaling/upscaling layers, each with hidden size 1024, focal window 14, focal factor 4, and DyT replacing layer normalization. To remove non-streamable global context, standard convolutions are replaced with causal convolutions and global pooling is replaced with a large-kernel causal convolution, described as acting like a learnable moving average over past context.

The quantizer is still a single binary codebook. The reported codebook sizes are 2048, 4096, and 65536, corresponding at approximately 50 Hz token rate to the three main operating points: 0.55 kbps, 0.60 kbps, and 0.80 kbps. The paper presents this single-codebook design as an architectural advantage for downstream sequence modeling because it avoids multi-stream residual token interfaces.

The decoder is based on Vocos, but streamabilized in two ways. First, ConvNeXt blocks are replaced with causal convolutions. Second, the inverse STFT output layer is removed and replaced with linear projection + flattening following WaveNeXt. The decoder is enlarged relative to the earlier recipe, with hidden size 1024, feed-forward dimension 2048, and upsampling factor 480 instead of 320, enabling waveform reconstruction at 24 kHz rather than 16 kHz (Libera et al., 19 Sep 2025).

Between decompressor and decoder, the paper inserts a refiner. For an input $x \in \mathbb{R}^{N \times D}$, chunked into $x_c \in \mathbb{R}^{N/C \times CD}$, the refiner is a residual chunk-wise MLP:

$$\hat{x}_c = x_c + W_{\text{out}} \, \mathrm{GELU}\!\left(W_{\text{in}} x_c + b_{\text{in}}\right) + b_{\text{out}},$$

with

$$W_{\text{in}}, W_{\text{out}} \in \mathbb{R}^{CD \times CD}, \qquad b_{\text{in}}, b_{\text{out}} \in \mathbb{R}^{CD}.$$

The paper characterizes this module as lightweight and reports that it has minimal impact on inference speed while improving perceptual quality (Libera et al., 19 Sep 2025).

3. Multi-stage causal distillation and training procedure

The methodological center of FocalCodec-Stream is not the quantizer alone but a four-stage causal distillation scheme that converts a full-context WavLM teacher into a streamable student encoder. This responds directly to the failure mode observed in the original non-streaming FocalCodec: high-quality WavLM-derived representations are valuable, but naïvely causalizing them degrades both reconstruction and downstream utility (Libera et al., 6 Feb 2025).

In Stage 1, the model distills a causal positional embedding from the original full-context positional embedding. The paper identifies the positional embedding and the full-context gated relative attention as the most problematic components under streaming constraints. The original positional embedding has receptive field 2.56 s, or 128 feature frames, which would violate the target latency budget. The description implies an $L_2$ objective of the form

$$\mathcal{L}_{\text{pos}} = \left\| E^{\text{causal}}_{\text{pos}} - E^{\text{teacher}}_{\text{pos}} \right\|_2^2.$$

This stage involves only a single convolutional layer with 8.4M parameters.

In Stage 2, the remaining encoder layers are distilled. The authors apply an $L_2$ loss at each of the six encoder layers, with a reversed linear schedule that weights deeper layers more heavily: layer 6 receives weight 1.0, layer 5 receives 0.9, layer 4 receives 0.8, and so on. The text implies a loss of the form

$0.80$0

This suggests that the streamable encoder is trained not only to match final teacher outputs but to preserve the internal representational geometry of WavLM across depth.

In Stage 3, the causal compressor–quantizer–decompressor stack is trained using an $0.80$1 loss between the causally distilled encoder features and the decompressed features, while the causal decoder is trained in parallel using the original full-context WavLM representations as conditioning features. The implied bottleneck loss is

$0.80$2

The asymmetry here is deliberate: the bottleneck reconstructs causal features, but the decoder is exposed to a cleaner full-context target distribution.

In Stage 4, the model jointly fine-tunes the encoder, compressor, quantizer, decompressor, and refiner. This stage addresses a distribution shift created in Stage 3: the decoder was trained on full-context WavLM features, but the decompressor produces causally distilled features. The refiner is introduced precisely to bridge this mismatch. The text implies an additional objective

$0.80$3

The training data scale increases across stages. Stage 1 uses LibriTTS, 585 hours, resampled to 16 kHz. Stages 2 and 3 use Libri-Light-medium, 5k hours, with fixed-length chunks of 15 s. Stage 4 uses full Libri-Light, 60k hours, with chunk size increased to 30 s. Decoder training uses the clean-100 split of LibriTTS at original 24 kHz with 3 s chunks. Reported hardware is NVIDIA A100 (80 GB) GPUs, and the learning rate decreases when validation loss stops improving, with early stopping and, in some runs, a learning-rate reset after apparent convergence (Libera et al., 19 Sep 2025).

4. Codebook design, bitrate, and latency accounting

FocalCodec-Stream uses a single codebook at approximately 50 Hz token rate, with bitrate determined by codebook cardinality. The paper states the relation explicitly:

$0.80$4

or, in kilobits per second,

$0.80$5

where $0.80$6 is token rate and $0.80$7 is codebook size (Libera et al., 19 Sep 2025).

Using $0.80$8 Hz, the three reported operating points are:

Variant	Codebook	Reported operating point
FocalCodec-S@50-2k	$0.80$9	$80$0 kbps
FocalCodec-S@50-4k	$80$1	$80$2 kbps
FocalCodec-S@50-65k	$80$3	$80$4 kbps

The paper notes that the 65k row reports token rate 50.5 Hz, but still reports 0.80 kbps. This preserves the central design point: bitrate scaling is achieved by codebook size while keeping a single token stream at approximately one token every 20 ms.

Latency is governed by chunked encoder attention rather than by the mostly causal modules. The paper reports theoretical latency = 80 ms, using a chunk size $80$5 frames and hop $80$6 ms per frame:

$80$7

The same section also notes that, except for chunked attention, most modules are strictly causal and would only incur about 20 ms latency. This means the system is not strictly zero-look-ahead end to end; rather, it is mostly causal with bounded intra-chunk look-ahead of up to 3 feature frames (Libera et al., 19 Sep 2025).

The codec table in the paper places FocalCodec-Stream against several streamable baselines:

Codec	Latency ms	Params M
EnCodec	13	15
AudioDec	13	8
HILCodec	13	11
Mimi5 / Mimi6	80	82
PAST	20	126
FocalCodec-S variants	80	249

This table makes two properties explicit. First, FocalCodec-Stream targets substantially lower bitrate than the 1.5–1.6 kbps acoustic baselines while remaining streamable. Second, it is not a lightweight model in parameter count; 249M parameters is one of its most important practical caveats.

5. Reconstruction, voice conversion, and downstream performance

On LibriSpeech test-clean, the three streamable variants remain close to the non-streaming FocalCodec@50, while outperforming several prior streamable baselines in intelligibility and naturalness. The key resynthesis numbers are:

Variant	English resynthesis
FocalCodec-S@50-2k	UTMOS 3.88, dWER 4.63, Sim 96.1
FocalCodec-S@50-4k	UTMOS 3.87, dWER 4.39, Sim 96.3
FocalCodec-S@50-65k	UTMOS 3.85, dWER 3.68, Sim 97.0
FocalCodec@50	UTMOS 4.05, dWER 2.18, Sim 97.4

Against streamable baselines, the most informative comparisons are Mimi6 and PAST. Mimi6 reports UTMOS 3.44, dWER 4.77, Sim 96.6, while PAST reports UTMOS 2.33, dWER 4.04, Sim 83.8. The paper therefore interprets FocalCodec-S as substantially improving naturalness over acoustic codecs, with the 65k model achieving the best intelligibility among streaming codecs in that table (Libera et al., 19 Sep 2025).

On the MLS subset, all models degrade, but the degradation is structured rather than catastrophic. FocalCodec-S@50-65k reports UTMOS 2.65, dWER 19.88, Sim 98.1, compared with non-streaming FocalCodec@50 at UTMOS 2.96, dWER 12.57, Sim 98.3. The paper describes the streaming model as remaining clearly stronger than acoustic codecs and competitive with Mimi in multilingual evaluation.

On one-shot VCTK voice conversion, FocalCodec-S is distinctive because it combines intelligibility and speaker retention in a way that other streaming codecs in the comparison table do not. FocalCodec-S@50-65k reports UTMOS 3.10, dWER 22.71, Sim 92.5, while Mimi6 reports UTMOS 2.62, dWER 110.00, Sim 91.3, and PAST reports UTMOS 1.42, dWER 18.28, Sim 68.5. The paper therefore characterizes FocalCodec-S as the only streaming codec in that table with strong speaker fidelity and strong content preservation simultaneously (Libera et al., 19 Sep 2025).

The model also reports highly efficient use of its single codebook. For speech resynthesis, code usage is 100.0% for the 2k, 4k, and 65k English settings, with entropy 99.4, 99.1, and 98.7, respectively. This supports the claim that the single-codebook representation does not collapse to a small subset of tokens.

Beyond waveform reconstruction, the paper evaluates the reconstructed features after the quantization bottleneck and before the decoder on downstream tasks. On discriminative tasks, FocalCodec-S remains competitive despite its lower bitrate. For example, ASR WER is 16.87, 17.21, and 17.02 for the three streamable variants, compared with 15.33 for non-streaming FocalCodec@50 and 22.56 for Mimi6. On SER, FocalCodec-S@50-4k achieves 33.18, the best value in the reported table. On SI, the larger codebook helps: FocalCodec-S@50-65k reaches 2.18 error, compared with 3.13 for Mimi6 and 3.43 for PAST.

On generative tasks, the bottleneck representation remains usable. For speech enhancement, FocalCodec-S@50-65k reports DNSMOS 3.56, dWER 19.56, Sim 87.7; for speech separation, FocalCodec-S@50-65k reports DNSMOS 3.68, dWER 75.43, Sim 90.8. These numbers remain close to the non-streaming FocalCodec and exceed several streamable baselines, suggesting that causal distillation preserves more than just codec reconstruction fidelity.

The ablation study on FocalCodec-S@50-4k is especially informative. The proposed system reports UTMOS 3.87, dWER 4.39, Sim 96.3. Removing the refiner degrades these to 3.84, 4.65, 96.1, and removing stage 4 degrades them further to 3.78, 5.05, 95.8. This indicates that the final fine-tuning stage is more consequential than the refiner alone, and that the staged causal distillation is an essential part of the method rather than a training convenience.

6. Efficiency, interpretation, and limitations

FocalCodec-Stream is designed for bounded-memory streaming inference, low-rate tokenization, and real-time speech coding pipelines, but the paper does not present it as a tiny or edge-oriented model. Its parameter count is 249M, larger than non-streaming FocalCodec@50 at 142M, Mimi at 82M, PAST at 126M, EnCodec at 15M, AudioDec at 8M, and HILCodec at 11M (Libera et al., 19 Sep 2025).

The runtime evidence is therefore qualified. On 1/8 NVIDIA H100 (80 GB) MIG partition, the paper reports an RTF “for inference speed” of around 106–108 for the FocalCodec-S variants, 123 for FocalCodec@50, 154 / 216 for Mimi6 depending on set, and 41 for HILCodec. Because larger values are treated as better in the table, a plausible implication is that the reported quantity is being used in an inverse-RTF or throughput-like sense rather than the conventional smaller-is-better real-time factor. The paper nonetheless treats the streamable variants as efficient enough for practical real-time use in the reported server-side setting.

Several limitations are explicit. The gap to non-streaming FocalCodec remains visible, especially in dWER and multilingual performance. The latency budget of 80 ms is responsive for conversational systems, but it is higher than the 13–20 ms latencies reported for some acoustic streaming codecs. The model is also GPU-oriented in evaluation, and the paper does not provide explicit FLOPs or memory figures comparable to lightweight streamable codecs such as StreamCodec or StreamCodec2, which emphasize deployment on much smaller parameter and compute budgets (Jiang et al., 9 Apr 2025).

There is also a conceptual limitation in the streaming formulation itself. FocalCodec-Stream is often described as causal, but the paper’s own architecture includes chunked attention with look-ahead of up to 3 frames. The most accurate characterization is therefore mostly strict causality outside chunked attention, with the total latency dominated by the 4-frame chunk. This is distinct from fully causal designs that prohibit any future dependency within the student network.

From a historical perspective, FocalCodec-Stream occupies a specific point in the design space opened by the original FocalCodec and by streaming single-codebook baselines such as TS3-Codec. It preserves the single-codebook, low-rate, semantics-aware philosophy of FocalCodec (Libera et al., 6 Feb 2025), but replaces its offline assumptions with multi-stage causal distillation, bounded-memory chunked attention, and a refiner calibrated to the mismatch between teacher-style and causal features. The result is a codec that remains unusually close to an offline semantically rich tokenizer while meeting an explicit streaming latency budget (Libera et al., 19 Sep 2025).