NanoCodec: Ultra-Low Latency Neural Codecs

• NanoCodec is a neural codec paradigm that achieves ultra-low model size and FLOP counts through frame rate optimization and efficient quantization.
• The approach employs specialized convolutional encoders, FSQ quantization, and tailored decoders to balance rate–distortion trade-offs.
• NanoCodec models support diverse modalities such as speech, 3D scenes, and images, offering real-time inference on low-power hardware.

NanoCodec denotes a class of neural codec architectures that deliver high compression efficiency, ultra-low-latency encoding/decoding, and minimal model size ("nano-scale" or "nano-codec" footprint) for diverse signal processing tasks, typically by combining judicious trade-offs in frame rate, parametric efficiency, and quantization. NanoCodec approaches are characterized by their ability to support high-fidelity reconstruction at previously unattainable bitrates and inference speeds, making them foundational for real-time machine perception, speech LLM inference, and compact neural representation of 3D scenes.

1. Core Concept and Definition

NanoCodec refers to a paradigm in learned neural codecs emphasizing extreme computational and storage efficiency, typically via frame rate and architecture engineering, while maintaining competitive rate–distortion and perceptual metrics. This concept has been explicitly instantiated in the context of speech/audio tokenization for LLMs "NanoCodec: Towards High-Quality Ultra Fast Speech LLM Inference" (NanoCodec) and as a goal for high-compression 3D scene codecs "CodecNeRF: Toward Fast Encoding and Decoding, Compact, and High-quality Novel-view Synthesis". The term is also used descriptively in generalized neural codec design paradigms "LiVeAction: a Lightweight, Versatile, and Asymmetric Neural Codec Design for Real-time Operation".

NanoCodecs systematically target:

• Ultra-low model size and FLOP counts (≤ 1 MB, sub-GFLOP/s workloads)
• Aggressive temporal and/or spatial reduction (e.g., 12.5 FPS, >64× bottlenecks)
• Application-agnostic workflows for broad data modalities
• Real-time operation on constrained hardware

2. Architectural Principles

NanoCodec designs employ encoder–quantizer–decoder pipelines with modality-specific architecture tailoring and quantization. Notable strategies include:

• Fully-convolutional Encoders: In NanoCodec (Casanova et al., 7 Aug 2025), the encoder consists of five strided residual blocks (Multi-Receptive Field Fusion) combining three 1D residual layers with dilations {1, 3, 5}. Stride scheduling [2, 3, 6, 7, 7] ensures compression to 12.5 FPS from 22.05 kHz input.
• Quantization via FSQ: Finite Scalar Quantization (FSQ) with multiple codebooks (4–16), each of dimension 4 and codebook levels [8, 7, 6, 6], yields a spectrum of bitrates (0.6–1.78 kbps).
• Decoder Design: Decoding leverages HiFi-GAN-inspired upsampling, with modifications (e.g., Snake activations) for periodic signals. Causality is tunable: fully non-causal, partially causal (default), or fully causal.
• Discriminator Setup: Adversarial training uses WavLM-based "pseudo-perceptual" discriminator, multi-period, and multi-band STFT discriminators. Speaker Consistency Loss (SCL) regularizes embedding similarity.

In CodecNeRF (Kang et al., 2024), the encoder projects multi-view images to vector quantized indices via downsample CNN + VQ (codebook K=4096, low-res grid V'=16). The decoder reconstructs tri-plane NeRF representations in a single pass. Optional scene-specific deltas refine feature planes and MLP weights using low-rank LoRA adapters and tensor decomposition, transmitted compactly after entropy coding.

LiVeAction (Jacobellis et al., 7 May 2026) extends the NanoCodec blueprint to multidimensional signals via:

• FFT-inspired, group-conv, block-diagonal encoders (factorized to minimize FLOPs)
• Asymmetric depth (lightweight encoder, heavy decoder)
• Modality-agnostic EfficientViT-style synthesis transforms

3. Training Objectives and Rate–Distortion

NanoCodec objectives encode a balance between distortion, perceptual/consistency realism, and bitrate:

• Rate–distortion formulation: $L = D(x, \hat{x}) + \lambda R(q)$, with $D$ being typically $\ell_1$ or $\ell_2$ losses between log-mel spectrograms or waveforms, and $R$ the entropy-regularized bitrate of quantized codes (Casanova et al., 7 Aug 2025).
• Additional losses:
  • Adversarial realism ($L_{\text{GAN}}$)
  • Feature-matching ($L_{\text{FM}}$)
  • Speaker Consistency ($$L_{\text{SCL}} = -(\alpha/n) \sum_i \text{cos\_sim}(\varphi(g_i), \varphi(h_i))$$ with $\alpha=0.1$)
• LiVeAction (Jacobellis et al., 7 May 2026) replaces adversarial/perceptual regularization with a variance-based rate penalty: $$\mathcal{L}(x, \hat{y}) = \log_{10} \|x - \hat{y}\|_2^2 + \lambda \log_2(\operatorname{Var}[z])$$, offering robust, modality-invariant entropy control.

For CodecNeRF (Kang et al., 2024), the joint objective at test-time is:

$D$0

balancing reconstruction, total-variation, and entropy-regularized code sizes.

4. Quantitative Performance

NanoCodec architectures have established new benchmarks in several domains:

Modality	Compression Ratio / Bottleneck	Bitrate (kbps)	Key Metrics	Latency
Speech (NanoCodec, MLS)	1.1–1.78 kbps, 12.5 FPS	0.6–1.78	SQMOS: 4.441, PESQ: 2.760, CER: 2.423	RTF: 1.06, TTFA: 1.01, >2× speedup (Casanova et al., 7 Aug 2025)
3D NeRF (CodecNeRF)	>100× (Objaverse)	—	PSNR: 30.12 dB, SSIM: 0.951	5 ms/view decode, ~0.27 MB code (Kang et al., 2024)
Image (LiVeAction)	64× size/195× compress	—	BD-rate vs JPEG2000 (SSIM): –70.3%	9.95 Mpix/s mobile CPU encode (Jacobellis et al., 7 May 2026)

NanoCodec (Casanova et al., 7 Aug 2025) achieves SQMOS of 4.441, PESQ 2.760, and CER of 2.423 for 1.78 kbps at 12.5 FPS on MLS, and demonstrates 2–2.5× latency reduction compared to SOTA LFSC. CodecNeRF attains compression ratios up to 171× with code size under 0.3 MB and PSNR improvement over baseline multi-plane methods, while dramatically reducing encoding/decoding time. LiVeAction matches or exceeds JPEG2000 and prior tokenizers in image/audio/video rate–distortion at sub-milliwatt encoding and <1 MB model size (Jacobellis et al., 7 May 2026).

5. Modality Agnosticism, Deployment, and Efficiency

A defining property of NanoCodec frameworks is adaptability to broad signal classes:

• Audio, image, hyperspectral, and volumetric signals: LiVeAction (Jacobellis et al., 7 May 2026) is configurable via convolution/wavelet "dimension" flags, latent channel/bottleneck sizing, and fixed layer counts.
• Encoder–decoder asymmetry: Light encoders (e.g., 50–200k params, real-time mobile encode for LiVeAction) and heavier decoders (EfficientViT, HiFi-GAN, or tri-plane NeRF MLP) localize resource usage.
• Test-time fine-tuning in 3D: CodecNeRF (Kang et al., 2024) applies parameter-efficient fine-tuning with delta coding for per-instance adaptation.

Practical settings for deployment include:

• Batch size, context, and dataset composition for robust cross-language and cross-speaker generalization (NanoCodec (Casanova et al., 7 Aug 2025))
• Hardware: Modern low-power CPUs (LiVeAction), high-throughput decoding (CodecNeRF single forward pass)
• Software: Integration with LLMs (Koel-TTS with NanoCodec tokens) for rapid Speech LLM inference.

6. Limitations and Directions for Further Research

NanoCodecs, while establishing new benchmarks, face specific challenges:

• Ultra-low bitrate collapse: For speech, intelligibility (as measured by CER) becomes untenable below 0.8 kbps at 12.5 FPS (Casanova et al., 7 Aug 2025).
• Latency–quality trade-off: While partially causal decoders remove look-ahead and maintain quality, fully causal architectures still lag in perceptual and intelligibility scores.
• Contextual requirements: Low-frame-rate operation in speech codecs necessitates extended token context for speaker consistency.

Research directions include:

• Novel LLM decoding strategies aligned to low-frame-rate tokenizers (alignment, context augmentation)
• Hybridized quantization–language modeling objectives to directly couple rate–distortion trade-offs with generative modeling
• Further optimization of codebook partitioning and entropy models for tighter bitrate–fidelity operating points, especially below 1.1 kbps in speech and sub-MB footprints in 3D scenes

A plausible implication is that the NanoCodec design pattern—aggressively minimizing encoder complexity while retaining flexibility in decoders and quantization—will drive broader adoption in embedded, streaming, and cloud-to-edge machine perception pipelines. The consistent use of frame rate, quantization, and efficient architectures as levers for rate–distortion–latency trade-offs is likely to propagate into future generations of neural codecs.