DTM-Codec: Dynamic Token Masking for VFR Speech Coding with Efficient Boundary Selection

Abstract: Variable frame rate (VFR) coding has recently emerged in neural speech codecs, allocating fewer frames to redundant regions and more frames to rapidly changing speech. VFR must transmit side information about retained time steps, but prior gains are either not rigorously addressed or often minor once these overhead bits are included in total bitrate. We present Dynamic Token Masking (DTM)-Codec, a neural speech codec that demonstrates clear gains over fixed-frame-rate baselines under a strict matched-total-bitrate protocol. DTM keeps selected encoder tokens, fills masked positions with a learned <MASK> embedding, and transmits a binary keep-mask for position-aware decoding. We further introduce Path Length Equalization (PLE), a linear-time boundary selector for VFR coding that yields well-spread adaptive segments with negligible overhead. Across operating points, DTM-Codec broadly improves reconstruction quality and intelligibility over fixed-frame-rate baselines.

• The paper introduces DTM-Codec with dynamic token masking and a linear-time PLE boundary selector to outperform fixed frame rate codecs under matched bitrate conditions.
• It employs a two-stage transformer architecture with an STFT front-end and a single-codebook VQ layer, ensuring precise bitrate accounting including side-information.
• Empirical results on LibriSpeech show improved PESQ, reduced WER, and enhanced speaker similarity, particularly at lower bitrates such as 800 bps.

DTM-Codec: Dynamic Token Masking for Variable Frame Rate Speech Coding

Introduction and Motivation

Recent advancements in neural audio codecs have increased their utility as a tokenization layer for speech LLMs, TTS, and audio generation systems. While the evolution toward single-codebook tokenizers and semantic-aware designs has improved integration with text and LLMs, a key open challenge remains: temporal allocation of tokens. Conventional fixed-frame-rate (FFR) codecs allocate uniform temporal resolution, irrespective of local information density, leading to inefficiency, particularly at low bitrates.

Variable frame rate (VFR) coding addresses this by allocating more tokens to rapidly changing regions and fewer tokens to stationary segments. However, the literature lacks a rigorous, controlled comparison of VFR and FFR under strictly matched, total bitrate accounting—including the side information needed to signal frame boundaries. This paper introduces DTM-Codec, which proposes Dynamic Token Masking (DTM) and a linear-time Path Length Equalization (PLE) boundary selector, demonstrating robust VFR advantages over FFR at equivalent bitrates.

DTM-Codec Architecture

DTM-Codec builds on a two-stage transformer encoder–decoder architecture. The first stage encodes STFT features at high temporal resolution, while the second stage processes a compressed, adaptively masked token sequence selected via VFR.

Figure 1: DTM-Codec architecture. DTM operates between encoder stages: kept tokens ($m_t=1$), selected by PLE, are quantized via VQ and decoded; masked positions are filled with a learnable <MASK> embedding. The binary mask $\mathbf{m}$ is transmitted as position bits.

The critical architectural features are:

• Front-end/Back-end: An STFT/iSTFT pair replaces learned patch projections for waveform/latent mapping, stabilizing training across frame rates.
• Token Masking: Instead of averaging or merging features, DTM keeps selected encoder tokens and fills masked slots with a learned <MASK> embedding. The binary keep-mask is transmitted as side information.
• Quantization: DTM-Codec uses a single-codebook VQ layer ($|\mathcal{C}|=16,384$; $b=14$ bits/token), which simplifies bitrate management and ensures precise accounting of both content and position bits.
• Adaptive Boundary Selection: Path Length Equalization (PLE) partitions the time axis so each segment covers an equal portion of the encoder's path in feature space, efficiently spreading kept tokens.

This combination enables direct, position-aware decoding, leveraging masked embeddings for robust reconstruction at masked locations.

Dynamic Token Masking and Path Length Equalization

Dynamic Token Masking (DTM) is formulated for variable-rate operation. For a sequence of Stage-1 encoder outputs ($N$ frames), DTM packs the $K$ retained frames (positions $m_t=1$ from mask $\mathbf{m}$) and processes them through the bottleneck and decoder. At reconstruction, missing slots are filled with a learned <MASK> token. This scheme diverges from average pooling or repetition strategies and treats masked positions as explicit unknowns, which improves robustness to boundary misalignment.

Boundary selection is performed by PLE, which computes the cumulative cosine distance trajectory over encoder features and inserts a boundary when this cumulative difference exceeds a threshold $\tau$. The approach guarantees content-adaptive selection in linear time and, when coupled with a running threshold adaptation (Robbins–Monro), reaches target keep ratios reliably.

Bitrate Accounting and Matched Comparison

One of the paper’s main strengths is the systematic, rigorous inclusion of position bits in bitrate accounting, distinguishing it from much prior work. The total bitrate is:

$B_\text{total} = f_\text{tok} \cdot b + p$

where $\mathbf{m}$0 is the effective token rate, $\mathbf{m}$1 is bits per token, and $\mathbf{m}$2 is the side-information (position mask) bitrate (one bit per Stage-1 timestep). FFR baselines use a larger codebook to match total bitrate but do not transmit position side information.

Empirical Evaluation

Main Results: Rate–Distortion and Reconstruction

On LibriSpeech test-clean, DTM-Codec exhibits clear VFR advantages over FFR across bitrate regimes 400–1280 bps. Gains in objective quality and intelligibility metrics (PESQ, STOI, UTMOSv2, Spk-Sim, and WER) are observed, particularly at low–mid bitrates.

Figure 2: Matched-rate comparison on LibriSpeech test-clean. Rate–distortion curves for UTMOS, PESQ, STOI, Spk-Sim, and WER compare DTM-Codec variants (VFR and FFR) against external codecs.

Numerically, at 800 bps:

• PESQ improves by +8.2% (2.46 → 2.66) for VFR vs. FFR
• Speaker similarity rises +10.5%
• WER decreases considerably (3.31 → 2.91)

These trends hold for lower bitrates, though the VFR improvement diminishes at higher bitrates (e.g., 1280 bps) due to reduced temporal redundancy. Subjective evaluations via MUSHRA-inspired listening test also confirm these findings.

Masking, Boundary, and Ablation Analysis

Ablations show that mask-guided packing with <MASK> fill achieves the best overall reconstruction. Unlike prior VFR codec strategies that aggregate or repeat tokens, DTM's approach directly leverages explicit missingness, allowing the decoder to more accurately fill in uncertain segments and properly leverage context.

Among boundary selectors, PLE substantially outperforms alternative low-complexity methods (e.g., similarity-threshold merging) and is highly competitive with dynamic programming segmentation while being orders of magnitude faster.

Out-of-domain Generalization and Semantic Content

OOD evaluation on the MLS non-English set shows VFR gains persist in unseen languages (e.g., +8.3% PESQ at 800 bps). However, global semantic classification metrics (from the ARCH benchmark) do not strongly differentiate between VFR and FFR when the codebook size is controlled, indicating VFR's primary advantage is in reconstruction over temporally adaptive allocation, not necessarily in the preservation of global semantic structure after temporal pooling.

Theoretical and Practical Implications

This work demonstrates that, under strict matched-total-bitrate conditions (including side information), well-designed VFR codecs like DTM-Codec can reliably outperform FFR baselines, particularly in the low-mid bitrate regime. The masking-based approach avoids the brittleness of token repetition, and PLE provides a high-quality, efficient boundary selection strategy, making high granularity frame allocation practical for deployment and further downstream integration (e.g., speech generation, compression, efficient speech language modeling).

Practically, DTM-Codec achieves rate–distortion points competitive with or superior to larger or more complex codecs despite being trained only on LibriSpeech-960 with 127M parameters, as compared with external baselines using larger architectures or multi-domain data.

Theoretically, the findings support the notion that adaptive allocation of coding resources—when properly regularized and with efficient boundary signaling—is fundamental to neural representation efficiency. This opens avenues for more broadly applicable adaptive token masking/selection strategies in other domains (e.g., general audio, music, or streaming contexts).

Future work may leverage DTM-style VFR tokenizers for streaming applications and for generative speech models, where the cost of transmitting timing side information is manageable, and content-adaptive representations are directly beneficial.

Conclusion

DTM-Codec establishes that explicit, masking-based VFR coding with efficient, content-adaptive boundary selection outperforms fixed-rate approaches at matched total bitrate, both empirically and practically. The architecture's methodological rigor in bitrate accounting and its efficient algorithmic components provide a robust basis for future work in low-bitrate speech coding and token-based speech generation.