Neural Action Codec (NAC)

• Neural Action Codec (NAC) is a learned action tokenizer that converts continuous robot trajectories into compact discrete tokens for efficient VLA model integration.
• It employs a convolutional residual vector quantization autoencoder with a GAN decoder to maintain fine temporal details and achieve high-fidelity reconstruction.
• NAC produces a structured, causally ordered token stream that reduces sequence length while improving downstream manipulation accuracy in robotic tasks.

to=arxiv_search 山大发cript code: {"query":"(Jawaid et al., 19 Jun 2026) NAC: Neural Action Codec for Vision-Language-Action Models"} Neural Action Codec (NAC) is a learned action tokenizer for vision-language-action (VLA) models that adapts the “codec-front end + autoregressive LLM” recipe of modern neural audio codecs to short robot action trajectories. In NAC, an action chunk is treated as a multi-channel 1-D signal, compressed by a convolutional residual vector quantization autoencoder with a GAN decoder, and exposed as a discrete vocabulary for standard transformer-based sequence models. The design is motivated by the observation that existing action tokenizers often trade off compression, latency, and downstream performance, whereas repurposed audio-codec components can provide compact, ordered tokens together with high-fidelity reconstruction when mel-specific objectives are removed and kinematic reconstruction losses are emphasized (Jawaid et al., 19 Jun 2026).

1. Conceptual framing

NAC addresses the interface problem between continuous robot control and autoregressive sequence modeling. VLA policies typically require a discrete token space, but robot actions are continuous and temporally structured. NAC therefore tokenizes short action trajectories rather than individual scalar commands, with the explicit goal of preserving structure while reducing sequence length (Jawaid et al., 19 Jun 2026).

The method takes an action chunk $a \in \mathbb{R}^{B \times H_a \times D_a}$, where $B$ is batch size, $H_a$ is action horizon, and $D_a$ is action dimension, and recasts it as a “pseudo-waveform.” After per-dimension normalization, for example min–max scaling to $[-1,1]$, NAC flattens the time and feature axes into

$$w \in \mathbb{R}^{B \times 1 \times L}, \qquad L = H_a \cdot D_a.$$

This representation departs from tokenizers that treat each time step or each joint dimension independently. The intended consequence is to let a 1-D codec model local and multi-scale temporal structure across the entire action chunk rather than over isolated coordinates (Jawaid et al., 19 Jun 2026).

The paper situates this choice against the broader success of neural audio codecs. It specifically revisits action tokenization through the lens of convolutional encoder-decoder architectures with residual vector quantization, which in audio serve as the standard front end for audio foundation models. A plausible implication is that NAC imports not only compression machinery but also inductive biases associated with multi-resolution sequence modeling.

2. Architecture and signal representation

The encoder is SEANet-style: a stack of 1-D convolutions with strided downsampling factors $R = [r_1,\dots,r_K]$, residual blocks, ELU activations, weight normalization, and reflection padding. If the total downsampling factor is $\prod_i r_i$, the encoder produces

$$z \in \mathbb{R}^{B \times D_{\rm enc} \times T'}, \qquad T' = \frac{L}{\prod_i r_i}.$$

NAC discretizes $z$ with multi-scale residual vector quantization (MRVQ). Let $B$0. At quantization stage $B$1, NAC may average-pool $B$2 in time, with coarser stages using more pooling, and then maps each pooled latent vector $B$3 to the nearest entry in a learnable codebook $B$4:

$B$5

The pooled embeddings are upsampled back to encoder resolution and subtracted from the residual to form the next stage,

$B$6

After $B$7 stages, the final quantized latent is the sum of the upsampled stagewise embeddings. Codebooks are initialized by k-means and updated by exponential moving averages. A commitment loss,

$B$8

pre-scaled by $B$9, is used to prevent encoder drift and collapse (Jawaid et al., 19 Jun 2026).

The decoder follows a Vocos-style vocoder design. Quantized latents pass through Conv1D embedding layers, attention-augmented ResNet blocks, and ConvNeXt blocks. The output head is not a naive linear regressor; instead, NAC predicts magnitude and phase of an STFT and synthesizes the action signal through an inverse short-time Fourier transform (ISTFT) with matching hop length. The paper states that this choice helps restore fine temporal detail, including sharp velocity changes and corrective “spikes,” that a purely linear decoder would smooth away. Adversarial discriminators adapted from the DAC architecture in audio GANs are used to improve high-frequency fidelity; without adversarial loss, reconstructed trajectories become overly smooth and lose the precise corrections essential for manipulation tasks (Jawaid et al., 19 Jun 2026).

3. Objectives, reconstruction criteria, and discrete vocabulary

The generator, consisting of encoder, quantizer, and decoder, is trained with three terms: reconstruction, commitment, and adversarial loss. NAC explicitly rejects mel-spectrogram objectives as ill-suited for kinematic signals and instead uses time-domain and non-mel spectral criteria (Jawaid et al., 19 Jun 2026).

The two primary reconstruction terms are:

$H_a$0

and

$H_a$1

Empirically, the paper reports that MSE or L1 in the time domain gave the best downstream policy performance, whereas the straight spectrogram loss minimized MSE best but yielded slightly weaker policies. The full generator objective is

$H_a$2

with $H_a$3. The discriminator is trained with the usual hinge or least-squares GAN objective on real versus reconstructed 1-D signals (Jawaid et al., 19 Jun 2026).

After training, each action chunk of length $H_a$4 is converted into a compact discrete sequence. In the main experiments, NAC uses $H_a$5 codebooks, each of size $H_a$6, and each chunk yields 2 sequences of 6 tokens each, for a total of 12 tokens. This compresses $H_a$7 continuous values into 12 discrete symbols, which the paper describes as roughly a $H_a$8 reduction relative to FAST and $H_a$9 relative to per-dimensional binning (Jawaid et al., 19 Jun 2026).

The policy-visible vocabulary is

$D_a$0

where the additional symbol is a BOS token. Because each scale’s codebook occupies an offset range in $D_a$1, the resulting token stream remains causally ordered by scale. This ordered offset-codebook construction is central to NAC’s claim that standard autoregressive policies can operate over short, structured sequences (Jawaid et al., 19 Jun 2026).

4. Policy integration in VLA systems

Once trained and frozen, the tokenizer $D_a$2 and detokenizer $D_a$3 are wrapped around an autoregressive transformer policy $D_a$4. During policy training, continuous demonstration chunks $D_a$5 are flattened, encoded, quantized into code indices, offset by scale, and concatenated into a single causal sequence of the form

$D_a$6

The policy is then trained with causal cross-entropy on these discrete targets (Jawaid et al., 19 Jun 2026).

At inference time, the policy samples tokens until the required length $D_a$7 is produced, partitions and de-offsets them per quantization scale, and passes them through $D_a$8, the ISTFT decoder, to recover $D_a$9. Execution is performed with a receding-horizon controller that applies the first few actions before re-tokenizing using new observations (Jawaid et al., 19 Jun 2026).

This arrangement makes NAC a front-end module rather than a complete policy architecture. Its role is to provide a compact and structured action vocabulary compatible with existing transformer-based VLAs. The paper’s broader framing is that NAC supplies a learned tokenizer in place of hand-crafted discretization such as binning or DCT-style approaches. This suggests that NAC is best understood as infrastructure for action representation learning rather than as a standalone control algorithm.

5. Empirical behavior

The reported evaluation covers LIBERO-10, RoboMimic, and eight real-world manipulation tasks. In simulation, results are averaged over 50 trials per task and 8 seeds. NAC yields higher downstream success than FAST and OAT on both benchmark suites (Jawaid et al., 19 Jun 2026).

Setting	NAC	Comparison
LIBERO-10 success	$[-1,1]$0	FAST $[-1,1]$1, OAT $[-1,1]$2
RoboMimic success	$[-1,1]$3	FAST $[-1,1]$4, OAT $[-1,1]$5
Real-world total success	$[-1,1]$6	FAST $[-1,1]$7, OAT $[-1,1]$8

Reconstruction-only metrics are also reported. With L1 or time-domain loss, spectrogram L1 is approximately $[-1,1]$9 versus MSE approximately $$w \in \mathbb{R}^{B \times 1 \times L}, \qquad L = H_a \cdot D_a.$$0. The paper further states that removing adversarial loss causes mean squared error to increase by two orders of magnitude and reduces policy success to zero. This is presented as evidence that high-frequency detail in reconstructed action trajectories materially affects downstream manipulation performance (Jawaid et al., 19 Jun 2026).

Latency measurements on an NVIDIA 4090 place NAC between manually optimized and earlier learned tokenizers. The encoder plus decoder requires approximately $$w \in \mathbb{R}^{B \times 1 \times L}, \qquad L = H_a \cdot D_a.$$1 ms per chunk, compared with approximately $$w \in \mathbb{R}^{B \times 1 \times L}, \qquad L = H_a \cdot D_a.$$2 ms for FAST and $$w \in \mathbb{R}^{B \times 1 \times L}, \qquad L = H_a \cdot D_a.$$3 ms for VQ-VLA. The reported trade-off is therefore not minimum latency but improved reconstruction and policy performance at comparable or better compression rates (Jawaid et al., 19 Jun 2026).

6. Relation to neural audio codecs, limitations, and scope

NAC is explicitly derived from the design vocabulary of neural audio codecs: convolutional encoder-decoder backbones, multi-scale RVQ, ISTFT heads, and adversarial discriminators. In the audio literature, “NAC” more commonly abbreviates neural audio codec; for example, SUNAC extends a standard neural audio codec into a source-aware codec for mixed audio signals (Aihara et al., 20 Nov 2025). The robotics NAC should therefore be distinguished from the generic audio usage of the same acronym.

The central adaptation in the action-setting is not a wholesale architectural replacement but the removal of mel-spectrogram bias and the re-weighting of reconstruction objectives toward kinematic fidelity. The paper argues that audio-codec-style models can autoencode actions with high fidelity without substantial architectural changes once those objective mismatches are corrected (Jawaid et al., 19 Jun 2026).

The stated strengths are threefold: a principled end-to-end learned tokenizer rather than hand-crafted binning or DCT, fine temporal detail from adversarial and ISTFT-based decoding, and a configurable compression–fidelity trade-off through the number of RVQ stages and codebook sizes. The stated limitations are also specific: the flattening approach requires fixed divisibility of $$w \in \mathbb{R}^{B \times 1 \times L}, \qquad L = H_a \cdot D_a.$$4 by the product of downsampling ratios, and detokenization must know the exact action dimensionality a priori (Jawaid et al., 19 Jun 2026).

Future directions identified in the paper include variable-length tokenization, cross-embodiment codebooks, shared open-vocabulary action spaces, and larger-scale pretraining. These are presented as possible extensions rather than demonstrated capabilities. A plausible implication is that the authors view action tokenization as evolving toward a foundation-model interface analogous to the role of neural codecs in audio, but the reported results are confined to short, fixed-horizon action chunks in the evaluated VLA settings (Jawaid et al., 19 Jun 2026).