NAC: Neural Action Codec for Vision-Language-Action Models

Abstract: Vision-language-action (VLA) models rely on discrete action tokenizers to bridge continuous robot control and autoregressive sequence modeling, yet existing tokenizers often trade off between compression, latency, and downstream performance. We revisit this design through the lens of neural audio codecs-convolutional encoder-decoder architectures with residual vector quantization that serve as the standard front end for audio foundation models. Motivated by their success, we introduce the Neural Action Codec (NAC), which treats short robot action trajectories as multi-channel 1D signals and compresses them using a multi-scale RVQGAN architecture. We observe that audio-specific mel-spectrogram objectives are ill-suited for kinematic signals; however, by replacing them with simple time-domain and non-mel spectral reconstruction losses, audio-codec-style models can autoencode actions with high fidelity without substantial architectural changes. NAC provides a compact, ordered token space via offset codebooks, enabling standard autoregressive policies to operate over short, structured sequences. Meanwhile, a Vocos-style decoder with an ISTFT head and adversarial discriminators recovers smooth, detailed trajectories. Across LIBERO-10, RoboMimic, and a suite of real-world manipulation tasks, NAC achieves lower reconstruction error and higher success rates than binning, FAST, and prior VQ-based tokenizers at comparable or better compression rates. These results demonstrate that repurposed neural audio codecs offer a strong, practical backbone for learned action tokenization in modern VLAs.

• The paper introduces NAC, a neural action codec that adapts audio architectures to efficiently compress continuous robot control trajectories.
• It employs multi-scale residual vector quantization and adversarial training to create robust, structured token spaces for vision-language-action models.
• Empirical evaluations on LIBERO-10, RoboMimic, and real-world tasks demonstrate significant improvements in task success rates and compression ratios.

NAC: Neural Action Codec for Vision-Language-Action Models

Motivation and Background

Autoregressive vision-language-action (VLA) models critically depend on discrete action tokenizers that compress continuous robot control trajectories into compact, learnable sequences. Existing tokenizers exhibit tradeoffs in compression, latency, and downstream performance. Drawing inspiration from neural audio codecs, which have solved analogous sequence compression challenges in the audio domain by leveraging convolutional encoder-decoder architectures with residual vector quantization (RVQ), this paper presents Neural Action Codec (NAC), a practical adaptation of audio codec architectures to action tokenization.

By representing short robot action trajectories as multi-channel 1D signals, NAC enables high-fidelity compression using a multi-scale RVQGAN. Key audio-specific elements, notably mel-spectrogram losses, are removed in favor of simple time-domain and non-mel spectral reconstruction objectives, reflecting the mismatched perceptual priorities of acoustic and kinematic signals.

Figure 1: Neural audio codecs compress waveforms via discrete codes, preserving temporal details; NAC adapts this for action tokenization, treating robot actions as multi-channel 1D signals.

Architecture and Methodological Advances

Signal Representation and Encoder

NAC flattens action chunks into 1D pseudo-waveforms, subsequently processed by a SEANet-style encoder composed of cascaded 1D convolutions and residual blocks. This fully convolutional backbone efficiently downscales temporal resolution, yielding compact continuous latent representations. The encoder architecture is directly borrowed from state-of-the-art neural audio codecs, preserving their scalability and temporal structuring properties.

Figure 2: NAC overview: Action trajectories are flattened and encoded by SEANet; multi-scale RVQ compresses the latent into structured discrete tokens; ISTFT decoder reconstructs smooth actions.

Multi-Scale Residual Vector Quantization

Multi-scale RVQ discretizes the encoder latent across multiple hierarchical codebooks with temporal pooling, enforced by aggressive commitment losses to prevent latent space collapse. This architectural choice guarantees ordered, coarse-to-fine compression, enabling structured discrete token spaces that facilitate causal, autoregressive modeling.

Decoder and Adversarial Training

A Vocos-style decoder, augmented by an ISTFT head, reconstructs the continuous action chunk from quantized latent codes. Adversarial losses delivered via DAC discriminators are critical for maintaining high-frequency fidelity in reconstructed trajectories, as evidenced by ablation studies. The decoder head aligns with phase-aware audio architectures but is tuned for kinematic, not perceptual, reconstruction quality.

Loss Function and Frequency-Domain Shift

Mel-spectrogram losses, fundamental to audio generation, are inappropriate for action signals. NAC exclusively employs time-domain (MSE/L1) and non-mel spectral losses, with empirical evidence showing that mel-losses degrade action fidelity and downstream policy performance to near-zero. Adversarial losses and strict commitment loss scaling further regularize encoder and decoder behavior.

Autoregressive Behavioral Policy

The NACPolicy learns to autoregressively generate structured token sequences from vision and language contexts, leveraging offset codebooks for scale-wise token ordering. The resulting token sequences are partitioned and detokenized for downstream execution, allowing for receding-horizon planning and low-latency policy rollout.

Experimental Evaluation

NAC was benchmarked across LIBERO-10, RoboMimic, and real-world manipulation tasks, compared against Bin, Diffusion Policy, FAST, VQ-VLA, and OAT tokenization baselines.

Figure 3: Simulation environments for benchmarking tokenizers: LIBERO-10 and RoboMimic validate downstream policy performance.

Ablation Studies

Ablations confirmed the necessity of adversarial discriminators, ISTFT-head decoders, and time-domain losses. For example, removing the discriminator collapses policy performance to zero, while replacing ISTFT with a linear head with simple upsampling reduces both reconstruction quality and task success rates.

Numerical Results

NAC achieves task success rates of 49.73% on LIBERO-10, surpassing FAST by 11.71 points and OAT by 5.56 points. On RoboMimic, NAC attains 33.94%, outperforming all baselines. In real-world evaluations spanning eight physical manipulation tasks, NAC demonstrates robustness, logging a 50% overall success rate, outperforming both OAT and FAST (each at 40%). The most robust transfer advantage surfaces in precision tasks like grapes grasping and block stacking.

Figure 4: Real-world evaluation tasks, illustrating high success rate transitions from initial (red) to completed (green) states for NAC.

Figure 5: Complete set of real-world tasks with start and end states, encompassing fine-grained and gross motor control scenarios.

Compression and Latency Analysis

NAC matches learned tokenizers in action chunk compression (12 tokens per chunk), achieves a 19$\times$ reduction over Bin, and is three times more compressed than FAST. Tokenizer encoding/decoding times are practical for real-time closed-loop policies, outperforming diffusion- and VQ-based approaches in deployment latency.

Implications and Theoretical Considerations

NAC substantiates that the core architecture and training protocol underlying neural audio codecs, when properly refactored for action signals, offer a strong backbone for learned action tokenization in VLAs. The findings accentuate the importance of domain-appropriate loss functions, the value of adversarial training, and structured discrete token spaces in autoregressive policy learning. The practical gains in compression and latency suggest broad adoption potential for high-frequency control, with foundational implications for scalable, cross-embodiment token spaces and large-corpus training.

On the theoretical front, NAC exemplifies how cross-domain architectural convergence, paired with careful adaptation of training objectives, yields substantial improvements in representational efficiency and downstream performance. The structured, ordered token space buttresses causal sequence modeling, and the adversarial kinematic fidelity fosters robust execution under real-world physical constraints.

Conclusion

The Neural Action Codec demonstrates that neural audio codec architectures, judiciously adapted via domain-specific objective shifts, deliver superior action tokenization for vision-language-action models. NAC's contributions comprise robust empirical improvement in reconstruction and policy performance, favorable compression-latency tradeoffs, and high real-world transfer reliability. Limitations include constraints on sequence length divisibility and explicit dimensionality requirements for detokenization. Future directions entail scaling NAC to larger, more diverse action corpora, cross-embodiment applications, and high-frequency control domains.