RepCodec: Neural Representation Compression

• RepCodec is a representation-centric compression method that targets latent neural activations or speech features rather than raw signal fidelity.
• The transformer inference variant uses rate–distortion optimization with learned hyper-priors to significantly reduce inter-device communication bandwidth while preserving task performance.
• The speech tokenization variant reconstructs latent speech representations into discrete semantic tokens, outperforming standard k-means and VQ methods in low-bitrate ASR tasks.

Searching arXiv for the specified RepCodec papers and closely related context. RepCodec is the name of two distinct neural codec formulations in recent arXiv literature: a learnable, rate–distortion optimized representation codec for compressing intermediate transformer representations during partitioned multi-device inference (Andrade et al., 29 Jan 2026), and a speech representation codec for low-bitrate semantic tokenization that reconstructs pretrained speech encoder representations rather than raw waveforms (Huang et al., 2023). In both cases, the central design principle is representation-centric compression: the codec is optimized for preserving task-relevant latent information under explicit communication constraints, rather than for faithful reconstruction of the original signal domain. This shared orientation places RepCodec at the intersection of neural compression, representation learning, and systems-oriented model deployment, while the two lines of work address different modalities, objectives, and downstream interfaces.

1. Naming and conceptual scope

The term RepCodec denotes two separate architectures with different operational targets. In "Rate-Distortion Optimization for Transformer Inference" (Andrade et al., 29 Jan 2026), RepCodec is a representation codec for intermediate hidden states in transformers, designed for the case where a model is split across devices and activations must be transmitted between modules. The codec learns compact encodings of hidden representations and a learned hyper-prior, with training governed by a rate–distortion objective that trades bitrate against task accuracy.

In "RepCodec: A Speech Representation Codec for Speech Tokenization" (Huang et al., 2023), RepCodec refers to an end-to-end neural codec for speech tokenization. Instead of reconstructing audio waveforms as in audio codecs, it reconstructs internal speech representations extracted from pretrained encoders such as HuBERT, data2vec, or Whisper. The output is a sequence of discrete semantic tokens intended for downstream language-model-style processing.

The shared nomenclature reflects a common methodological stance: both systems compress representations rather than raw signals. However, they should not be conflated. The transformer-inference RepCodec is a lossy inter-device communication mechanism embedded inside a partitioned model pipeline (Andrade et al., 29 Jan 2026), whereas the speech RepCodec is a semantic tokenizer that maps audio to discrete units for speech understanding and generation (Huang et al., 2023).

2. Transformer inference RepCodec

RepCodec for transformer inference is defined as a learnable, rate–distortion optimized representation codec designed to compress intermediate transformer representations when the model is partitioned across devices for inference (Andrade et al., 29 Jan 2026). The motivating setting is edge–cloud or similar multi-device execution, where transmitting raw activations between an upstream and a downstream module is prohibitively expensive in bandwidth and latency.

The compressed object is the intermediate activation at a split point $S$. For decoder-only LLMs, the network is partitioned into two contiguous modules: $f_{1\ldots s}$ produces $Y$, which is compressed, and $f_{s+1\ldots L}$ consumes $Y$ to produce predictions $Z$ (Andrade et al., 29 Jan 2026). A learned hyper-prior $W$ derived from $Y$ is also compressed and transmitted as side information. The main experiments do not compress attention KV caches, although the formulation is described as applicable to KV caches for prefill–decode disaggregation.

Training uses the unconstrained Lagrangian form

$$L(Y, W, \hat{Z}, Z) = \mathbb{E}[d(\hat{Z}, Z) + \lambda (r_y(Y;W) + r_w(W))],$$

where $d(\hat{Z}, Z)$ is a bounded, non-negative task loss and $f_{1\ldots s}$0 controls the rate–distortion trade-off (Andrade et al., 29 Jan 2026). For language modeling, distortion is measured by cross-entropy loss and reported in perplexity as $f_{1\ldots s}$1; LAMBADA accuracy is also reported. Rate is measured in bits per token for language tasks and bits per pixel for image tasks.

The rate estimators are defined by negative log-likelihood over unit intervals centered at quantized symbols:

$f_{1\ldots s}$2

and

$f_{1\ldots s}$3

Arithmetic coding can be used at inference (Andrade et al., 29 Jan 2026).

Architecturally, the upstream device computes a continuous pre-quantized representation $f_{1\ldots s}$4, quantizes it as $f_{1\ldots s}$5 using integer rounding with a straight-through estimator, computes a hyper-prior $f_{1\ldots s}$6, quantizes $f_{1\ldots s}$7, estimates a fully factorized CDF for $f_{1\ldots s}$8 with $f_{1\ldots s}$9, predicts $Y$0 for a fully factorized normal on $Y$1 with $Y$2, arithmetic-encodes $Y$3 and $Y$4, and transmits the resulting bitstreams (Andrade et al., 29 Jan 2026). The receiver arithmetic-decodes $Y$5 and $Y$6 and continues inference through the downstream module.

The entropy model for $Y$7 is an MLP with 9 dense layers, monotonicity enforced via reparameterization, and 118 parameters per dimension; it is fully factorized over $Y$8 dimensions. The entropy model for $Y$9 consists of 4 transformer blocks with causal masks over time and uses only the hyper-prior $f_{s+1\ldots L}$0 as context, not previously coded $f_{s+1\ldots L}$1 symbols (Andrade et al., 29 Jan 2026). This design induces conditional independence of $f_{s+1\ldots L}$2 elements given $f_{s+1\ldots L}$3, allowing parallel coding within a time frame. The hyper-prior analysis transform $f_{s+1\ldots L}$4 contains 4 transformer blocks that reduce the embedding dimension through $f_{s+1\ldots L}$5.

The paper studies split points $f_{s+1\ldots L}$6 in GPT-2 Small and reports that rate–distortion performance varies systematically with depth (Andrade et al., 29 Jan 2026). This depth dependence becomes central in the paper’s theoretical analysis.

3. Speech tokenization RepCodec

The speech RepCodec is an end-to-end neural codec that converts speech waveforms into discrete semantic tokens optimized to preserve the information in pretrained speech encoder representations (Huang et al., 2023). Rather than reconstructing audio, it reconstructs framewise latent representations $f_{s+1\ldots L}$7 extracted by a speech encoder such as HuBERT, data2vec, or Whisper.

The forward path begins with speech waveform $f_{s+1\ldots L}$8 passed through an encoder $f_{s+1\ldots L}$9 to obtain $Y$0, followed by a RepCodec encoder $Y$1, implemented as a 1D ConvNet with residual units over time, to produce latent $Y$2 (Huang et al., 2023). A vector quantizer then maps each $Y$3 to codebook entries. In the regular VQ case, each frame yields one code index; in residual vector quantization (RVQ), residuals are quantized layer-by-layer, so each frame yields a tuple of indices. The RepCodec decoder $Y$4 consumes the quantized vectors and reconstructs $Y$5 at the same frame rate and dimensionality.

The primary loss is reconstruction in representation space:

$Y$6

For standard VQ, the quantization loss is

$Y$7

and for RVQ with $Y$8 layers,

$Y$9

The total loss is

$Z$0

(Huang et al., 2023).

Codebook entries are updated by EMA:

$Z$1

$Z$2

The paper states that this smooth update avoids instability and sharp cluster jumps associated with EM/k-means, enabling end-to-end training (Huang et al., 2023).

The encoder and decoder are non-causal 1D convolutional networks with residual units, no downsampling or upsampling, stride 1, and kernel size 3 (Huang et al., 2023). The regular model uses $Z$3 and $Z$4; an RVQ variant uses $Z$5 for higher fidelity. Because RepCodec preserves the speech encoder’s frame rate, often around 50 frames/s, and with $Z$6 corresponding to 10 bits per token, the bitrate is approximately $Z$7 bits/s, or 0.5 kbps (Huang et al., 2023).

The intended downstream interface is decoder-only conditional modeling of text from audio tokens:

$Z$8

The training sequence is given as "[audio tokens] <|transcribe|> [text tokens]" (Huang et al., 2023).

4. Rate–distortion theory and information retention

The transformer-inference RepCodec is accompanied by an explicit theoretical analysis based on the V-entropy framework under computational constraints (Andrade et al., 29 Jan 2026). The key object is the V-entropy gap, which measures how far the achievable rate under a constrained predictive family $Z$9 is from entropy. With predictive family

$W$0

the conditional V-entropy is defined as

$W$1

The V-entropy gap is

$W$2

and equivalently

$W$3

Under RepCodec’s design, minimizing the expected rate for $W$4 is stated to be equivalent to directly minimizing the V-entropy gap (Andrade et al., 29 Jan 2026).

The paper provides covariance determinant bounds and PAC-style generalization bounds. For continuous $W$5 with covariance $W$6 and $W$7,

$W$8

For quantized $W$9,

$Y$0

With typical $Y$1, the paper notes that the bound is approximate but empirically informative (Andrade et al., 29 Jan 2026).

The generalization analysis defines

$Y$2

and gives the bound

$Y$3

when $Y$4 with probability at least $Y$5 (Andrade et al., 29 Jan 2026). A more general bound with side information is also given. The stated interpretation is that increasing model complexity, reflected in larger Lipschitz constants, can worsen generalization error, so empirical negative log-likelihoods and achievable rates are tied jointly to target representation complexity and entropy-model smoothness.

The speech RepCodec develops a different form of information-retention analysis. Its empirical claim is that lower reconstruction loss $Y$6 correlates with lower WER, whereas higher 1-gram PNMI does not necessarily imply better ASR (Huang et al., 2023). To capture sequence-level structure, the paper defines

$Y$7

For HuBERT large 18th-layer features, k-means has $Y$8, $Y$9, $$L(Y, W, \hat{Z}, Z) = \mathbb{E}[d(\hat{Z}, Z) + \lambda (r_y(Y;W) + r_w(W))],$$0, $$L(Y, W, \hat{Z}, Z) = \mathbb{E}[d(\hat{Z}, Z) + \lambda (r_y(Y;W) + r_w(W))],$$1, whereas RepCodec has $$L(Y, W, \hat{Z}, Z) = \mathbb{E}[d(\hat{Z}, Z) + \lambda (r_y(Y;W) + r_w(W))],$$2, $$L(Y, W, \hat{Z}, Z) = \mathbb{E}[d(\hat{Z}, Z) + \lambda (r_y(Y;W) + r_w(W))],$$3, $$L(Y, W, \hat{Z}, Z) = \mathbb{E}[d(\hat{Z}, Z) + \lambda (r_y(Y;W) + r_w(W))],$$4, $$L(Y, W, \hat{Z}, Z) = \mathbb{E}[d(\hat{Z}, Z) + \lambda (r_y(Y;W) + r_w(W))],$$5 (Huang et al., 2023). The paper interprets this as evidence that RepCodec aligns better with phoneme sequences for $$L(Y, W, \hat{Z}, Z) = \mathbb{E}[d(\hat{Z}, Z) + \lambda (r_y(Y;W) + r_w(W))],$$6, which matters for downstream sequence modeling.

These two theoretical lenses are distinct. One formalizes rate under constrained probabilistic predictors for intermediate activations (Andrade et al., 29 Jan 2026); the other evaluates the sequence-level informational adequacy of discrete speech tokens relative to phonemic structure (Huang et al., 2023). A plausible implication is that the shared term "representation codec" conceals substantially different operational notions of information preservation.

5. Empirical performance

The transformer-inference RepCodec is evaluated on GPT-like LLMs and additional image architectures (Andrade et al., 29 Jan 2026). GPT-2 Small has 12 layers, 12 heads, embedding dimension $$L(Y, W, \hat{Z}, Z) = \mathbb{E}[d(\hat{Z}, Z) + \lambda (r_y(Y;W) + r_w(W))],$$7, and 124M parameters; experiments use OpenWebText with tokens per sample $$L(Y, W, \hat{Z}, Z) = \mathbb{E}[d(\hat{Z}, Z) + \lambda (r_y(Y;W) + r_w(W))],$$8. At split point $$L(Y, W, \hat{Z}, Z) = \mathbb{E}[d(\hat{Z}, Z) + \lambda (r_y(Y;W) + r_w(W))],$$9, RepCodec is compared against a Fourier basis density model for $d(\hat{Z}, Z)$0 and a direct-access entropy model for $d(\hat{Z}, Z)$1. Relative to RepCodec, BD-rate is reported as $d(\hat{Z}, Z)$2 for the Fourier basis model and $d(\hat{Z}, Z)$3 for the direct-access model, both worse than RepCodec (Andrade et al., 29 Jan 2026).

The paper reports selected quantitative examples for GPT-2: at $d(\hat{Z}, Z)$4 and $d(\hat{Z}, Z)$5, total rate is 148.68 BPT with perplexity 19.10; at $d(\hat{Z}, Z)$6 and $d(\hat{Z}, Z)$7, total rate is 355.93 BPT with perplexity 20.89; at $d(\hat{Z}, Z)$8 and $d(\hat{Z}, Z)$9, total rate is 492.08 BPT with perplexity 21.61 (Andrade et al., 29 Jan 2026). At $f_{1\ldots s}$00, adding a rate penalty yields up to 17.8% improvement in perplexity over the rate-unconstrained setting, which the paper describes as consistent with information bottleneck effects that discard irrelevant information. LAMBADA accuracy can decline slightly under compression, but at some $f_{1\ldots s}$01 operating points RepCodec achieves the best overall accuracy among compressed variants.

A major empirical finding is that rate–distortion performance degrades as the split point moves deeper into transformers, even though data processing inequalities imply entropy cannot increase (Andrade et al., 29 Jan 2026). The reported explanation is that deeper layers exhibit larger covariance determinants and Rademacher complexities, which tighten theoretical bounds and increase generalization error. Correlation analysis shows strong Pearson correlation between covariance determinant and bitrate, with average approximately 0.96, and between Rademacher complexity and bitrate, with average approximately 0.965; Lipschitz constants of trained entropy models show mild negative correlation with bitrate, average approximately $f_{1\ldots s}$02 (Andrade et al., 29 Jan 2026). In ResNet-34, the opposite trend is observed: deeper split points reduce covariance determinant and Rademacher complexity, consistent with lower rates and improved rate–distortion performance.

The speech RepCodec is evaluated on decoder-only ASR, multilingual ASR, speech resynthesis, and voice conversion (Huang et al., 2023). On LibriSpeech test-clean with 960h ASR training, single-layer data2vec large 18th-layer features yield 2.18% WER using continuous representations, 2.87% with RepCodec, 4.55% with k-means, and 4.68% with VQ-only. For HuBERT large 18th-layer features the corresponding numbers are 2.81%, 4.02%, 5.00%, and 5.19%. For Whisper medium 24th-layer features they are 3.94%, 5.04%, 9.52%, and 24.35%; for Whisper large 32nd-layer features, 3.96%, 5.01%, 9.97%, and 44.43% (Huang et al., 2023).

In linear-combination feature settings, RepCodec remains stronger than k-means and VQ-only, while Whisper-medium and Whisper-large combinations cause both k-means and VQ to fail with WER reported as "100+" (Huang et al., 2023). EnCodec baselines perform much worse for semantics: 35.44% WER at 0.75 kbps and 16.53% at 1.5 kbps. Scaling RepCodec tokenizer training from 100h to 960h improves HuBERT large 18th-layer WER from 4.03% to 3.72% and data2vec large 18th-layer WER from 2.87% to 2.65%. Using two-layer RVQ improves fidelity further, reducing HuBERT large 18th-layer WER from 4.03% to 3.85% and data2vec large 18th-layer WER from 2.87% to 2.48% (Huang et al., 2023).

The following table condenses representative quantitative comparisons already stated in the papers.

Setting	RepCodec	Comparator(s)
GPT-2, $f_{1\ldots s}$03, BD-rate vs RepCodec	baseline	Fourier basis: +99.46%; Direct-access: +10.7%
GPT-2, $f_{1\ldots s}$04, $f_{1\ldots s}$05	148.68 BPT, perplexity 19.10	—
GPT-2, $f_{1\ldots s}$06, $f_{1\ldots s}$07	355.93 BPT, perplexity 20.89	—
LibriSpeech test-clean, data2vec large 18th	2.87% WER	k-means: 4.55%; VQ: 4.68%
LibriSpeech test-clean, HuBERT large 18th	4.02% WER	k-means: 5.00%; VQ: 5.19%
EnCodec ASR baseline	—	0.75 kbps: 35.44%; 1.5 kbps: 16.53%

These results support a narrow but important commonality across the two RepCodec lines: learned representation-aware compression can outperform generic or less structured discretization schemes at comparable or lower communication cost (Andrade et al., 29 Jan 2026, Huang et al., 2023).

6. Systems characteristics and deployment considerations

For the transformer-inference RepCodec, the systems objective is explicit: reducing inter-device bandwidth and latency during partitioned inference (Andrade et al., 29 Jan 2026). Raw transmission would require 16 bits per activation times $f_{1\ldots s}$08, approximately 12,288 bits/token. At the best perplexity point for $f_{1\ldots s}$09, RepCodec achieves 230.16 BPT, which corresponds to approximately 1.87% of raw, or about 98.1% memory/bandwidth saving for the inter-device link (Andrade et al., 29 Jan 2026).

The paper also reports comparisons with lossless compressors. DEFLATE on RepCodec-induced $f_{1\ldots s}$10 gives 1,053.88 BPT and Zstandard gives 1,172.16 BPT, whereas RepCodec uses 230.16 BPT. On raw uncompressed tensors, DEFLATE gives 11,415.13 BPT and Zstd 11,383.07 BPT; RepCodec uses 2.02% of those bitrates (Andrade et al., 29 Jan 2026). Codec latency is reported as 0.348 ms/token for RepCodec and 0.672 ms/token for DEFLATE, while Zstd is faster at 0.022 ms/token but with much higher bitrate. With a 9% protocol overhead, RepCodec is more efficient than sending raw tensors if effective link speed is below approximately 37.77 Mbps (Andrade et al., 29 Jan 2026).

The implementation requires deploying the hyper-prior analysis transform and entropy models on both sender and receiver; arithmetic coding provides bitstream compatibility (Andrade et al., 29 Jan 2026). Causal masking ensures streaming behavior, so only new hyper-prior and representation segments need to be transmitted for new time frames. Entropy models were benchmarked on GPU using an NVIDIA 2080 Ti, while arithmetic coding ran on CPU using an Intel i9-9900K; training used a single NVIDIA A40 with bfloat16 precision and up to 20 GB VRAM.

The speech RepCodec is optimized for low-bitrate semantic tokenization rather than real-time coding throughput. It reports no explicit inference speed or throughput figures (Huang et al., 2023). Its deployment guidance instead concerns token-rate planning and LLM integration: with approximately 50 Hz frame rate and $f_{1\ldots s}$11, token bitrate is about 0.5 kbps; RVQ with $f_{1\ldots s}$12 raises bitrate and fidelity; long audio can be chunked by frames; and standard positional encodings should follow audio frame order (Huang et al., 2023). The practical downstream interface is conditional modeling of text from tokens, not joint modeling of audio and text, since full language modeling $f_{1\ldots s}$13 is reported as markedly worse than conditional $f_{1\ldots s}$14.

A common misconception is that all codecs named RepCodec are intended to reconstruct human-perceptual signals. That is incorrect in both cases. The transformer-inference RepCodec never targets waveform or pixel-domain fidelity; it targets task loss under bitrate constraints (Andrade et al., 29 Jan 2026). The speech RepCodec also does not optimize waveform reconstruction in its tokenization stage; it reconstructs latent speech representations, and waveform generation is deferred to a unit-conditioned vocoder if needed (Huang et al., 2023).

7. Limitations, open questions, and relation to adjacent methods

The transformer-inference RepCodec identifies several open problems (Andrade et al., 29 Jan 2026). In transformers, rate–distortion performance worsens at deeper layers, so adaptive per-layer rate–distortion control remains an open direction. Theoretical bounds depend on Lipschitz and quantization assumptions, and the $f_{1\ldots s}$15 bound is described as approximate. The current implementation is not fully optimized; arithmetic coding on CPU is a present bottleneck, and GPU implementations or hardware-aware entropy models may reduce latency. Proposed extensions include KV cache compression for module replication across devices, adaptive $f_{1\ldots s}$16 schedules, streaming-aware hyper-priors, and richer entropy models for $f_{1\ldots s}$17 such as state-space models while keeping $f_{1\ldots s}$18 simple.

The speech RepCodec also leaves an acknowledged gap between continuous representations and discretized tokens (Huang et al., 2023). The paper suggests stronger codec architectures, including Transformers, and adversarial objectives as possible future directions. Training coverage was limited to LibriSpeech 100h and 960h plus MLS subsets for French and Spanish, and non-causal 1D convolutions were used, so streaming and low-latency scenarios were not addressed.

Relative to adjacent methods, the speech RepCodec is positioned against k-means tokenization, VQ-only systems, and audio codecs such as SoundStream and EnCodec (Huang et al., 2023). The paper’s evidence indicates that k-means can have stronger 1-gram phoneme alignment while still losing more task-relevant information, and that audio codecs preserve acoustic detail at the cost of higher bitrate and substantially worse semantic-task performance. Relative to general-purpose lossless compressors, the transformer-inference RepCodec demonstrates that learned entropy modeling of intermediate activations can dominate hand-engineered codecs when the object being compressed is a task-conditioned neural representation (Andrade et al., 29 Jan 2026).

Taken together, the two RepCodec formulations exemplify a broader shift from signal reconstruction to representation-aware compression. In one case, this is formalized as end-to-end rate–distortion optimization for partitioned transformer inference (Andrade et al., 29 Jan 2026); in the other, as low-bitrate semantic tokenization of speech representations for LLM-compatible downstream processing (Huang et al., 2023). This suggests that "RepCodec" is best understood not as a single architecture but as a recurring design pattern: learned compression targeted at the informational geometry of internal representations rather than the original data domain.