Elastic Time: Dynamic Frame Rate Bottlenecks for Neural Audio Coding

Abstract: Neural audio autoencoders have become a core component of compression, feature extraction, and generation. However, while existing systems support variable bitrate, the vast majority of models still operate at a fixed latent frame-rate, allocating equal temporal budget to regions with very different information density, which can result in unnecessarily long sequences. We introduce Elastic Time, a dynamic frame-rate bottleneck that converts fixed-frame-rate autoencoders to dynamic ones. Our method learns a lightweight latent predictor used to decide which frames can be skipped and later reconstructed, enabling efficient greedy boundary selection at inference. Experiments show our method enables deployment-time rate control while improving efficiency-quality tradeoffs relative to baselines. Overall, we provide a flexible mechanism for adjusting temporal resolution in audio autoencoders, potentially facilitating more efficient downstream modeling for generation and long-context tasks.

• The paper presents Elastic Time, a plug-in Re-Bottleneck module that adaptively reduces frame-rate in neural audio autoencoders to minimize redundancies while preserving audio quality.
• It employs a learned latent predictor with greedy or dynamic programming algorithms to partition latent sequences, balancing resource allocation and reconstruction error.
• Experimental results demonstrate significant improvements in SI-SDR, mel-spectrogram distance, and FAD scores across diverse audio domains, underscoring its practical efficiency.

Elastic Time: Dynamic Frame Rate Bottlenecks for Neural Audio Coding

Introduction

The paper "Elastic Time: Dynamic Frame Rate Bottlenecks for Neural Audio Coding" (2606.27320) introduces Elastic Time (ET), a framework for enabling dynamic frame-rate control in neural audio autoencoders. Neural audio codecs have advanced in compressing audio into latent codes, typically leveraging fixed frame-rate models. However, fixed temporal allocation results in inefficiencies, especially in regions of the signal with variable information density, leading to unnecessarily long latent sequences and increased computational demands. The Elastic Time approach tackles this by introducing a content-adaptive, scalable frame-rate reduction module—allowing for flexible bitrate control during deployment while maintaining high reconstruction quality.

Methodology

ET operates as a plug-in Re-Bottleneck module, augmenting any pretrained neural audio autoencoder. The method centers on a learned lightweight latent predictor $\mathcal{P}$, trained to model short-term latent dynamics. ET applies a chunking procedure to partition the latent sequence based on predictability, retaining only anchor frames, and synthesizes omitted frames during dechunking via autoregressive rollouts of $\mathcal{P}$. The frame selection process is governed by either an efficient greedy algorithm ($O(T\log T)$ complexity) or an exact dynamic programming scheme ($O(T^2 K_{\max})$), optimizing the trade-off between resource allocation and reconstruction error without the need for external semantic supervision.

The chunk boundaries are selected by minimizing multi-step prediction errors; segments where future latents can be predicted from an anchor are deemed redundant and are collapsed. The core pipeline preserves audio fidelity by ensuring compatibility with the frozen decoder of the base autoencoder, thus greatly simplifying training and leveraging the abundance of public pretrained models. Training recipes combine multi-step rollout prediction losses and adversarial feature matching objectives to robustify the Re-Bottleneck layer under variable frame-rate transformations.

Experimental Evaluation

ET was evaluated on diverse audio domains, including instrumental music, sound effects, vocal music, and speech, using unseen datasets (SongDescriber, AudioCaps, MuChin, DAPS). Key baselines included Conv-Downsample, CodecSlime, H-Net, and H-Net-YOTO, all instantiating comparable architectures for fair benchmarking.

Reconstruction metrics encompassed SI-SDR, mel-spectrogram distance, STFT distance, and Fréchet Audio Distance (FAD), quantifying both per-segment fidelity and distribution-level perceptual quality.

Figure 1: Reconstruction quality as a function of latent frame-rate ($\rho$), reporting mel-d (top row) and FAD (bottom row); lower is better.

Results reveal that ET achieves strong efficiency-quality tradeoffs, outperforming prior dynamic chunking baselines and Conv-Downsample in most scenarios. Notably, ET variants (both greedy and DP) were competitive or superior in mel-d and FAD, particularly in music and speech domains. The benefit of ET was most pronounced in FAD scores on SongDescriber and DAPS, underscoring its capacity for preserving perceptual continuity across aggressive rate reductions. Greedy and DP chunking performed similarly, validating the practicality of the faster greedy algorithm for deployment-time control. Rate-scalable ET models maintained performance across a wide range of latent frame-rates, demonstrating robustness and flexibility.

Numerical Results and Claims

• ET consistently achieves lower FAD and mel-spectrogram distance across music and speech domains compared to dynamic chunking baselines (e.g., CodecSlime, H-Net), with clear numerical gains in distribution-level perceptual quality.
• The greedy algorithm offers nearly identical performance to DP-based selection, suggesting that most of the optimization benefit can be realized with much lower computational overhead.
• Single-rate ET models outperform scalable versions at their trained operating point ($\rho=0.5$), highlighting a specialization-versus-scalability tradeoff.
• ET delivers competitive or superior SI-SDR, STFT, and mel-d scores compared to fixed-rate Conv-Downsample, despite using discrete boundary-based compression.

Implications and Future Directions

Elastic Time enables deployment-time adaptive bitrate control, facilitating efficient modeling for audio generation and long-context tasks without sacrificing quality. The content-adaptive frame-rate reduction directly addresses computational bottlenecks in generative models, diffusion backbones, and memory-intensive language-modeling approaches.

The method’s independence from external semantic supervision and compatibility with existing pretrained autoencoders broadens its applicability. The ET framework could pioneer new approaches in conditional audio generation, variable-rate streaming, and resource-aware synthesis. Future research may extend ET to online settings, explore integration with discrete quantization schemes, and optimize predictors for more heterogeneous audio domains.

Theoretically, ET underscores the utility of dynamic bottlenecks in sequence modeling, shifting the paradigm from fixed-rate constraints to content-driven temporal allocation. This opens avenues for more intelligent, adaptive representation learning in both speech and general audio tasks.

Conclusion

Elastic Time advances neural audio coding by introducing scalable, content-adaptive dynamic frame-rate control, resulting in improved efficiency-quality tradeoffs and enabling practical rate control at inference. The framework is robust, efficient, and integrates seamlessly with existing pretrained autoencoders, obviating the need for retraining and external semantic features. This approach positions dynamic frame-rate bottlenecks as a promising blueprint for future developments in generative audio modeling, compression, and downstream long-context tasks.