Neural Audio Codec with Adjustable Token Temporal Resolution Using Sampling-Frequency-Independent Convolutional Layers

Abstract: Discrete tokens obtained from neural audio codecs (NACs) have been used as compact representations in audio generation and understanding models. In such token-based systems, token temporal resolution (TTR), defined as the time interval between adjacent token frames, is important because it controls the trade-off between representing rapid acoustic events and reducing token-sequence length. However, most NACs are trained at a single TTR and require separate training for each TTR. This paper proposes a mechanism that enables a single NAC to operate at multiple TTRs using sampling-frequency-independent convolutional layers. The mechanism regards TTR as the sampling period of the token sequence and generates TTR-dependent convolutional kernels from a shared parameter set, while adjusting the kernel size and stride for each TTR. We incorporate the mechanism into Descript Audio Codec, leaving the quantizer unchanged. Experiments on environmental sound reconstruction show that the proposed model outperforms a single-model baseline that switches TTR-specific layers for each TTR.

• The paper presents a method using SFI convolutional layers to dynamically adjust token temporal resolution in a single, parameter-efficient neural audio codec.
• It leverages continuous frequency domain filter design to generate TTR-conditioned kernel weights, matching per-TTR trained models with far fewer parameters.
• Experiments demonstrate superior performance over naive baselines and near-reference quality at coarse TTRs, offering practical benefits for deployment.

Neural Audio Codec with Adjustable Token Temporal Resolution Using Sampling-Frequency-Independent Convolutional Layers

Introduction and Motivation

The paper introduces a novel mechanism for neural audio codecs (NACs) to flexibly adjust token temporal resolution (TTR) using sampling-frequency-independent (SFI) convolutional layers, targeting applications where token sequences serve as compact, discrete representations for audio generation and understanding. TTR is a central parameter, balancing representation of rapid acoustic events against sequence length and associated modeling complexity. Conventional NACs handle TTR by training a distinct model for each required resolution, resulting in redundancy and limited flexibility.

The new approach leverages SFI convolutional architectures to enable a single model to dynamically generate TTR-conditioned kernel weights, extending support for variable tokenization rates while maintaining a fixed set of shared parameters. This addresses the need for unified, flexible, and parameter-efficient NACs, especially relevant for downstream models where memory and computation associated with token sequence length are critical bottlenecks.

Technical Approach

The core contribution is the integration of SFI convolutional methodology for TTR control in NACs. Traditional convolutional layers learn discrete-time filters at a specific sampling period. In contrast, SFI layers define trainable filters in a continuous frequency domain, converting these into discrete filter weights corresponding to any target sampling period before application.

The architecture builds upon the Descript Audio Codec (DAC) framework but replaces only the convolutional layers directly adjacent to the quantizer with SFI variants. This allows seamless adaptation to different TTRs by dynamically generating the kernel size, stride, and weights based on the target temporal resolution (sampling interval) of tokens.

Figure 1: Overview of the SFI convolutional layer based on frequency-domain filter design.

The SFI layer mechanism interprets TTR as equivalent to the sampling period of the generated token sequence. For each TTR, the parameter generator evaluates the learned continuous frequency response at frequencies corresponding to the Nyquist rate of the intended sampling period, and then fits a kernel in discrete time. The accompanying transposed SFI layers symmetrically handle upsampling in the decoder.

This modified architecture retains the original DAC quantizer and overall structure, requiring only the SFI layers to incorporate the additional input corresponding to the desired output TTR. All other training and loss mechanics from DAC, including RVQ, adversarial, and feature-matching objectives, are preserved.

Figure 2: DAC architecture equipped with the proposed TTR-adjustment mechanism. The top shows the full codec, the lower half illustrates TTR-conditioned behavior for two example TTRs.

The adjustment algorithm ensures that for every TTR, the SFI layers operate with kernel size and stride chosen to span a consistent duration in virtual "token time," maintaining coverage and receptive field behavior as TTR varies. Selection is parameterized by defining, for all TTRs, a virtual minimum temporal resolution, with the actual TTR implemented via stride and kernel adaptation. This is critical for maintaining stable and comparable signal coverage as the frame rate changes.

Experimental Evaluation

Experiments were performed on the CochlScene dataset, encompassing diverse environmental scenes. The proposed method was evaluated against:

• A Reference: TTR-specific DAC models trained separately for each TTR ("oracle" per-TTR upper bound)
• A Naive baseline: A single model with TTR-specific (but not shared) standard convolutional layers for each TTR (parameter-inefficient sharing)
• The Proposed model: A single model utilizing SFI layers with a shared parameterization for all TTRs

Objective evaluation was conducted using mel distance, STFT distance, and Zimtohrli score, measuring spectral and perceptual reconstruction quality.

Figure 3: Mel distance achieved across a range of TTRs for Reference, Naive, and the proposed SFI method (lower is better).

Quantitative results demonstrated that:

• The SFI-based model outperformed the Naive baseline at every TTR in all metrics, despite using dramatically fewer trainable parameters adjacent to the quantizer (3 per input-output channel pair vs. 231 for Naive).
• The SFI-based approach matched or closely approached the Reference (independently-trained per-TTR) performance for large TTRs (i.e., lower token frame rates).
• Performance gap between the proposed method and Reference increases for fine TTRs (higher frame rates), likely due to the use of a shared codebook in the quantizer that may not be optimal for high-resolution token sequences.

Thus, the central technical claim—parameter-efficient, high-quality TTR adjustability via SFI parameterization—is substantiated across all results.

Broader Implications and Future Directions

The symbolically parameterized SFI convolution mechanism permits a single NAC instantiation to flexibly adjust TTR for deployment across application settings with disparate temporal modeling needs. This confers practical benefits in system deployment (reduced storage, simpler model management) and in downstream modeling where TTR adjustment might be selected adaptively for context-dependent tradeoffs between signal fidelity and sequence tractability.

The implementation is broadly applicable to other convolutional NAC architectures, with minimal modifications—only the layers surrounding the quantizer are altered. Notably, this approach decouples TTR adjustment from quantizer specifics, opening avenues for exploration in non-convolutional backbone codecs and for SFI-based architectures beyond audio, e.g., video, biosignal, or event sequence compression.

Remaining challenges reside primarily in shared quantization for small TTRs, where the representational capacity per token is limited, and a shared codebook constrains adaptation to rapid signal changes. Extending to TTR-conditional quantization or hybrid adaptive codebooks is an immediate area for research, as is full bandwidth adaptation (spanning both temporal and spectral tokenization scales).

Conclusion

This paper presents a method for realizing adjustable token temporal resolution in neural audio codecs via SFI convolutional layers, allowing dynamic tradeoffs between sequence length and acoustic temporal granularity without requiring multiple distinct models. The solution is implemented efficiently within the DAC baseline by modifying only local layers around the quantizer. Experimental results show superior parameter efficiency and reconstruction quality compared to conventional parameter-sharing baselines, as well as near reference-level performance at coarse TTRs. This approach represents a principled advance in flexible, deployable NAC architectures and informs future work on adaptive and application-conditional audio representation learning.

Reference: "Neural Audio Codec with Adjustable Token Temporal Resolution Using Sampling-Frequency-Independent Convolutional Layers" (2607.01865)