Generative Latent Coding (GLC)

• Generative Latent Coding is a framework that shifts representation from pixel space to a learned latent space with generative priors, enabling ultra-low bitrate compression while preserving perceptual quality.
• It leverages techniques like VQ-VAE, continuous tokenizers, and conditional coding to maintain semantic consistency and achieve state-of-the-art performance in image and video compression.
• Beyond compression, GLC underpins applications in GAN inversion, latent matching, and content-style factorization, unifying diverse generative tasks under a single design principle.

Searching arXiv for papers on Generative Latent Coding and closely related formulations. Generative Latent Coding (GLC) denotes a family of methods that relocate representation, compression, or control from pixel or observation space into a learned latent space endowed with a generative prior. In the 2025 compression literature, the term is used most explicitly for ultra-low-bitrate image and video compression schemes that perform transform coding in the latent space of a generative autoencoder or tokenizer rather than in pixel space, with the stated aim of reconciling realism, fidelity, and bitrate efficiency (Jia et al., 23 Dec 2025, Qi et al., 22 May 2025, Guo et al., 11 Oct 2025). In adjacent literatures, closely related uses of GLC include analytically matching an encoder’s aggregate latent distribution to a known target distribution, recovering latent codes of pretrained generators, factorizing latent content and style variables, and replacing continuous latent priors with structured discrete compositional priors (Saha et al., 2020, Egan et al., 2018, Guo et al., 2024, Ntavelis et al., 2023).

1. Scope and definitional variants

Across the cited literature, GLC is not a single architecture but a unifying design principle: encode data into a generative latent space whose geometry is intended to be semantically meaningful, perceptually aligned, or distributionally tractable, and then perform compression, inversion, sampling, or controllable generation in that space rather than directly in the observation domain (Jia et al., 23 Dec 2025, Saha et al., 2020).

Formulation	Core latent object	Representative papers
Ultra-low-bitrate compression	VQ-VAE or continuous tokenizer latents	(Jia et al., 23 Dec 2025, Qi et al., 22 May 2025, Guo et al., 11 Oct 2025)
Analytical latent matching	Aggregate latent distribution matched to $p_t(z)$	(Saha et al., 2020)
GAN inversion	Recovered latent $z$ for a pretrained generator	(Egan et al., 2018)
Style/content factorization	Deterministic content code and probabilistic style code	(Guo et al., 2024)
Discrete structured prior	Gene-wise discrete latent composition	(Ntavelis et al., 2023)

The compression-centric usage is the most explicit. "Generative Latent Coding for Ultra-Low Bitrate Image Compression" defines GLC as transform coding in the latent space of a generative VQ-VAE rather than in pixel space, emphasizing sparse, semantically meaningful, and perceptually aligned latent representations (Jia et al., 23 Dec 2025). "Generative Latent Coding for Ultra-Low Bitrate Image and Video Compression" extends the same formulation to both images and videos via GLC-image and GLC-video (Qi et al., 22 May 2025). "Generative Latent Video Compression" uses the closely related term Generative Latent Coding to describe a compression paradigm that relocates coding from pixel space into a perceptually aligned latent space learned by a generative tokenizer, with Generative Latent Video Compression (GLVC) as a concrete instantiation (Guo et al., 11 Oct 2025).

The broader usage is more heterogeneous. GENs frames GLC as learning an encoder-decoder whose aggregate latent distribution matches a known target such as a Gaussian or Gaussian mixture, thereby enabling probabilistic reasoning and direct sampling in latent space (Saha et al., 2020). The GAN inversion literature described in "Generalized Latent Variable Recovery for Generative Adversarial Networks" treats recovered generator latents as compact, structured embeddings for real images, which the supplied technical summary identifies as a GLC problem (Egan et al., 2018). Motion stylization and discrete GAN priors further extend the term toward latent factorization and latent prior design rather than compression per se (Guo et al., 2024, Ntavelis et al., 2023).

2. Compression as the dominant modern formulation

In image compression, GLC factorizes the pipeline into two parts: learning a perception-aligned generative latent space with a VQ-VAE and performing transform coding of the latent representation with a rate-variable entropy model. The canonical chain is

$$l = E(x), \quad y = g_a(l), \quad \hat{y} = Q(y), \quad \hat{l} = g_s(\hat{y}), \quad \hat{x} = D(\hat{l}),$$

where $E$ and $D$ are the VQ-VAE encoder and decoder, $g_a$ and $g_s$ are analysis and synthesis transforms, and entropy coding is driven by a categorical hyper module and spatial context model (Jia et al., 23 Dec 2025). The stated motivation is that pixel-space transform coding struggles at ultra-low bitrates because pixel-level distortions do not align well with human perception, whereas generative VQ-VAE latents are described as sparse, semantically meaningful, and perceptually aligned (Jia et al., 23 Dec 2025).

GLC-image augments this latent transform codec with two notable components. The first is a categorical hyper module, introduced to reduce the bit cost of hyper-information at ultra-low bitrates by modeling semantic priors categorically rather than with factorized Gaussian hyper-latents. The second is code-prediction-based supervision, in which an auxiliary predictor is trained to predict the VQ indices of the original latent from the reconstructed latent, thereby tightening semantic consistency without adding an inference bottleneck because the predictor is used only during training (Jia et al., 23 Dec 2025). Reported results include high visual quality with less than $0.04$ bpp on natural images and less than $0.01$ bpp on facial images, and on the CLIC2020 test set the same FID as MS-ILLM with $45\%$ fewer bits (Jia et al., 23 Dec 2025).

GLC-video extends the same basic principle to temporal data. It retains VQ-VAE latent-space coding but adds conditional coding on previously decoded latents and introduces a spatio-temporal categorical hyper module that captures global semantic dynamics via sparse tokens (Qi et al., 22 May 2025). The first frame is coded without temporal context, while subsequent frames are conditionally coded using temporal context extracted from the previously decoded latent. The paper reports $z$0 average bitrate saving over PLVC in terms of DISTS on HEVC Class B, MCL-JCV, and UVG (Qi et al., 22 May 2025).

GLVC differs from VQ-based GLC by using a pretrained continuous tokenizer rather than a vector-quantized tokenizer. Given an RGB video $z$1 of size $z$2 with $z$3, the tokenizer maps $z$4 to latents $z$5 of size $z$6 (Guo et al., 11 Oct 2025). The tokenizer is pretrained with adversarial and perceptual losses so that perceptual detail synthesis is front-loaded into the latent representation, and the codec then performs rate-distortion optimization only on semantic content. GLVC redesigns the codec architecture for the latent domain, including unified intra/inter coding, a parametric Gaussian hyperprior, context modeling, and recurrent memory (Guo et al., 11 Oct 2025). Reported findings include state-of-the-art perceptual quality in DISTS and LPIPS, $z$7 BD-rate savings versus DCVC-RT on UVG in DISTS, and a user study in which DCVC-RT at double bitrate wins only $z$8 of pairwise comparisons against GLVC (Guo et al., 11 Oct 2025).

3. Latent-space structure, objectives, and entropy models

A central organizing theme in compression-oriented GLC is the rate-distortion-perception decomposition. The generic objective is written as

$z$9

or, in GLVC’s notation,

$$l = E(x), \quad y = g_a(l), \quad \hat{y} = Q(y), \quad \hat{l} = g_s(\hat{y}), \quad \hat{x} = D(\hat{l}),$$0

where $$l = E(x), \quad y = g_a(l), \quad \hat{y} = Q(y), \quad \hat{l} = g_s(\hat{y}), \quad \hat{x} = D(\hat{l}),$$1 is expected code length, $$l = E(x), \quad y = g_a(l), \quad \hat{y} = Q(y), \quad \hat{l} = g_s(\hat{y}), \quad \hat{x} = D(\hat{l}),$$2 is distortion, and $$l = E(x), \quad y = g_a(l), \quad \hat{y} = Q(y), \quad \hat{l} = g_s(\hat{y}), \quad \hat{x} = D(\hat{l}),$$3 quantifies perceptual divergence (Jia et al., 23 Dec 2025, Guo et al., 11 Oct 2025). The distinctive claim of GLVC is that $$l = E(x), \quad y = g_a(l), \quad \hat{y} = Q(y), \quad \hat{l} = g_s(\hat{y}), \quad \hat{x} = D(\hat{l}),$$4 can be offloaded to tokenizer pretraining: the pretrained continuous tokenizer enforces perceptual realism and temporal smoothness, while the latent codec optimizes only $$l = E(x), \quad y = g_a(l), \quad \hat{y} = Q(y), \quad \hat{l} = g_s(\hat{y}), \quad \hat{x} = D(\hat{l}),$$5 and $$l = E(x), \quad y = g_a(l), \quad \hat{y} = Q(y), \quad \hat{l} = g_s(\hat{y}), \quad \hat{x} = D(\hat{l}),$$6 in the semantic regime (Guo et al., 11 Oct 2025).

In GLVC, latent-domain training uses

$$l = E(x), \quad y = g_a(l), \quad \hat{y} = Q(y), \quad \hat{l} = g_s(\hat{y}), \quad \hat{x} = D(\hat{l}),$$7

with $$l = E(x), \quad y = g_a(l), \quad \hat{y} = Q(y), \quad \hat{l} = g_s(\hat{y}), \quad \hat{x} = D(\hat{l}),$$8 implemented as MSE on latents because tokenizer latent variances are extremely small, making likelihood-based objectives ineffective (Guo et al., 11 Oct 2025). Perception-oriented finetuning is then performed only on the codec’s reconstruction head, with tokenizer and codec core frozen:

$$l = E(x), \quad y = g_a(l), \quad \hat{y} = Q(y), \quad \hat{l} = g_s(\hat{y}), \quad \hat{x} = D(\hat{l}),$$9

This training schedule is explicitly intended to preserve rate and semantic alignment while adding perceptual refinement (Guo et al., 11 Oct 2025).

In VQ-based GLC-image and GLC-video, the training is stagewise. Stage I learns the generative VQ-VAE with reconstruction, LPIPS, adversarial, and codebook losses; Stage II fixes the autoencoder and learns latent transform coding with a code-prediction-based distortion; Stage III jointly fine-tunes with pixel-space supervision and an additional code-prediction term (Jia et al., 23 Dec 2025, Qi et al., 22 May 2025). The Stage II image objective is

$E$0

with

$E$1

and the default $E$2 (Jia et al., 23 Dec 2025).

Entropy modeling is correspondingly shifted into the latent domain. In GLC-image, the hyper path computes

$E$3

and the probability $E$4 is estimated by a categorical hyper module backed by the hyper codebook and a quadtree-partition spatial context model (Jia et al., 23 Dec 2025). In GLVC, the rate is directly

$E$5

with $E$6 under a parametric Gaussian hyperprior, plus context modeling adapted from prior neural video compression work (Guo et al., 11 Oct 2025). This distinction between discrete latent hyper-information and continuous latent Gaussian hyperpriors reflects two different latent geometries: discrete VQ latents in GLC-image and continuous tokenizer latents in GLVC (Jia et al., 23 Dec 2025, Guo et al., 11 Oct 2025).

4. Broader latent-coding formulations beyond compression

Outside compression, one important formulation of GLC is explicit latent distribution design. GENs imposes a known target distribution on the aggregate encoded distribution by minimizing a kernel-density-estimated Jensen-Shannon divergence in latent space. With encoder $E$7 and decoder $E$8, the total objective is

$E$9

where $D$0 is the aggregate encoded distribution and $D$1 is a known target such as $D$2 or a Gaussian mixture (Saha et al., 2020). The method is distinctive in avoiding a discriminator: the JSD is estimated nonparametrically via KDE in latent space. The paper further derives that, under a standard-normal target and at steady state, the encoded distribution becomes Gaussian with variance $D$3, where $D$4 is the KDE bandwidth (Saha et al., 2020). Reported results include entropy close to the theoretical $D$5 entropy on MNIST and better FID than AAE and VAE on SVHN and CelebA (Saha et al., 2020).

A second formulation is latent variable recovery for GANs. Given a pretrained generator $D$6 and a target image $D$7, the inversion problem solves

$D$8

by gradient-based optimization in $D$9, with probabilistic resampling criteria designed for Gaussian priors (Egan et al., 2018). The paper generalizes stochastic clipping from uniform priors to Gaussian priors through hard, logistic, and truncated-normal resampling functions. In the reported recovery experiments on a DCGAN trained on Yelp food images, disabled resampling gives average latent error $g_a$0, whereas logistic$g_a$1 gives $g_a$2, truncated normal$g_a$3 gives $g_a$4, and logistic$g_a$5 gives $g_a$6 over $g_a$7 runs (Egan et al., 2018). In this usage, the recovered latent is treated as a generative latent code for analysis, interpolation, and downstream representation learning.

A third formulation is latent factorization for controllable generation. In generative human motion stylization, a pretrained motion autoencoder yields a latent motion code $g_a$8, which is decomposed into deterministic content $g_a$9 and probabilistic style $g_s$0 with prior $g_s$1 (Guo et al., 2024). A temporal CNN generator then produces a stylized code $g_s$2, which is decoded back to motion. The framework supports stylization from a reference motion, a label, or an unconditional style prior sample, and is trained by reconstruction, homo-style alignment, cycle consistency, and KL regularization (Guo et al., 2024).

A fourth formulation is structured discrete priors for GANs. StyleGenes replaces a continuous Gaussian prior with a gene-wise discrete latent distribution. If gene $g_s$3 has embeddings $g_s$4, the latent is

$g_s$5

with training-time sampling uniform per gene (Ntavelis et al., 2023). Under fixed total latent dimension $g_s$6, the total parameter count is $g_s$7 when all genes share $g_s$8 variants, while the number of unique codes is $g_s$9 (Ntavelis et al., 2023). The paper reports mean attribute-prediction accuracy from latents of $0.04$0 for StyleGenes versus $0.04$1 for StyleMapping, while maintaining competitive FID (Ntavelis et al., 2023). This suggests a different but closely related interpretation of GLC: the latent code itself is the main object of structural design.

5. Empirical findings, ablations, and applications

The strongest empirical case for GLC presently comes from ultra-low-bitrate compression. For images, GLC maintains high visual quality with less than $0.04$2 bpp on natural images and less than $0.04$3 bpp on facial images, achieves the same FID as MS-ILLM with $0.04$4 fewer bits on CLIC2020, and outperforms baselines such as EVC, TCM, HiFiC, and MS-ILLM on CelebAHQ across all metrics (Jia et al., 23 Dec 2025). For joint image-and-video GLC, GLC-image achieves FID parity with MS-ILLM on CLIC 2020 at less than $0.04$5 bpp with $0.04$6 fewer bits, while GLC-video achieves $0.04$7 bitrate saving over PLVC in DISTS (Qi et al., 22 May 2025). For GLVC, the reported broad BD-rate table indicates roughly $0.04$8 average savings versus VTM in DISTS and $0.04$9 in LPIPS on selected benchmarks while remaining competitive in PSNR at low rates (Guo et al., 11 Oct 2025).

Ablation studies identify several recurring mechanisms. In GLC-image, naive indices-map coding underperforms latent transform coding, with a reported $0.01$0 BD-Rate increase relative to the proposed transform coding. Replacing the categorical hyper module with a factorized hyperprior gives a $0.01$1 worse BD-Rate, and removing code-prediction supervision degrades performance by $0.01$2 BD-Rate (Jia et al., 23 Dec 2025). In the broader GLC image/video paper, inserting code prediction into the inference path harms performance by $0.01$3 BD-Rate, whereas using it only as supervision is best; for video, removing conditional coding increases BD-Rate by $0.01$4, and the proposed spatio-temporal categorical hyper module improves BD-Rate by $0.01$5 over the spatial hyper module baseline (Qi et al., 22 May 2025). In GLVC, removing recurrent memory increases BD-rate by $0.01$6 in DISTS, parametric Gaussian hyperpriors reduce hyperprior bit cost at ultra-low rates, and continuous tokenizers are reported to stabilize temporal behavior relative to VQ tokenizers, which exhibit strong temporal inconsistency and flicker (Guo et al., 11 Oct 2025).

Several papers emphasize that GLC’s latent organization enables secondary applications. The image GLC pipeline supports image restoration by training a restoration encoder to map distorted images directly to clean latents, and supports style transfer by replacing the latent decoder with a stylization decoder supervised by content and style losses (Jia et al., 23 Dec 2025). The motion-stylization formulation uses the same compressed or learned latent content representation as a stable substrate for diverse style transfer, including reference-based, label-based, and prior-sampled stylization (Guo et al., 2024). StyleGenes uses gene-conditioned latent probabilities for post-hoc conditional sampling from an unconditionally trained model, with reported average attribute classification accuracy rising from $0.01$7 at temperature $0.01$8 to $0.01$9 at $45\%$0, alongside a corresponding FID increase from approximately $45\%$1 to approximately $45\%$2 (Ntavelis et al., 2023).

Complexity measurements indicate that current high-performing GLC systems are not uniformly lightweight. GLC-image is reported at approximately $45\%$3M parameters, with Kodak $45\%$4 latency of $45\%$5 ms encode and $45\%$6 ms decode on an NVIDIA A100 (Qi et al., 22 May 2025). The natural-image GLC model in the standalone image paper is also approximately $45\%$7M parameters, and the facial model approximately $45\%$8M parameters (Jia et al., 23 Dec 2025). GLVC is reported at approximately $45\%$9M parameters, with single-A100 throughput of about $z$00 ms/frame encode and $z$01 ms/frame decode at $z$02 (Guo et al., 11 Oct 2025).

6. Limitations, misconceptions, and open directions

A common misconception is that GLC names one settled technique. The cited literature indicates otherwise: the term covers at least latent transform coding for compression, analytical latent distribution matching, GAN inversion, probabilistic content-style factorization, and discrete structured priors (Jia et al., 23 Dec 2025, Saha et al., 2020, Egan et al., 2018, Guo et al., 2024, Ntavelis et al., 2023). A plausible implication is that GLC is better understood as a research program centered on generative latent-space design rather than as a single algorithmic family.

Another misconception is that moving to latent space automatically resolves perceptual quality problems. The compression papers are more specific. They attribute gains to particular latent properties and architectural choices: sparse or semantically rich VQ-VAE latents, continuous temporally smooth tokenizers, categorical or parametric hyperpriors, conditional coding, recurrent memory, and carefully staged optimization (Jia et al., 23 Dec 2025, Qi et al., 22 May 2025, Guo et al., 11 Oct 2025). The ablations show that latent-space coding alone is insufficient without these surrounding mechanisms.

The limitations are correspondingly heterogeneous. Compression-oriented GLC currently struggles with real-time high-resolution operation, tokenizer-induced latency, dependence on tokenizer generalization, possible mismatch between automated perceptual metrics and human judgment, and engineering complexity of large models (Guo et al., 11 Oct 2025). The image-compression formulations report challenges on screen content and structured patterns such as straight grid lines, where semantic generators may prefer natural textures (Jia et al., 23 Dec 2025). GENs is limited by KDE scalability in high-dimensional latent spaces, bandwidth sensitivity, and the curse of dimensionality when latent dimension grows beyond moderate size (Saha et al., 2020). GAN inversion remains computationally expensive and sensitive to prior mismatch and hyperparameter settings, with reported recovery experiments using $z$03 iterations per code (Egan et al., 2018). StyleGenes inherits biases from the attribute classifiers used for post-hoc conditioning and exhibits an explicit accuracy-diversity trade-off through temperature (Ntavelis et al., 2023). Motion stylization depends strongly on the quality of the pretrained motion autoencoder and can fail on rare or out-of-distribution actions (Guo et al., 2024).

Future directions in the compression literature are comparatively concrete. Proposed avenues include learned variable-rate control with finer granularity, stronger temporal priors such as state-space models or transformers with sparse attention, motion-aware latent prediction, diffusion or flow models in latent space, domain-adaptive tokenizers, better distribution matching for latent distortion terms, joint spatial-temporal latent tokenizers, and hardware-friendly real-time designs (Guo et al., 11 Oct 2025, Qi et al., 22 May 2025). More broadly, the literature suggests that the long-term significance of GLC will depend on whether generative latent spaces can remain both semantically structured and entropy-efficient as tasks scale in spatial resolution, temporal extent, and controllability requirements.