Causally Regularized Tokenization (CRT)

• CRT is a methodology that integrates causal regularization into two-stage visual tokenization, aligning autoencoder latent representations with downstream autoregressive models.
• It deliberately trades off reconstruction fidelity to lower token conditional entropy, making autoregressive modeling more efficient.
• Empirical benchmarks show CRT achieves state-of-the-art generation with fewer parameters and tokens, significantly reducing compute requirements.

Causally Regularized Tokenization (CRT) is a methodology for optimizing the compression–generation trade-off in two-stage visual tokenization pipelines. CRT utilizes knowledge of the downstream causal modeling procedure to regularize the latent representation produced by an autoencoder, intentionally sacrificing reconstruction fidelity in exchange for making the latent tokens significantly easier to model autoregressively. This procedure achieves substantial improvements in compute efficiency and parameter efficiency for autoregressive visual generation, yielding state-of-the-art results with fewer parameters and tokens than previous approaches (Ramanujan et al., 2024).

1. Two-Stage Generative Paradigm and Motivation

State-of-the-art image generation pipelines employ a two-stage approach. In Stage 1, an autoencoder (e.g., VQGAN) is trained to compress images $x$ into discrete latent tokens $z = (z_1, \dots, z_N)$, optimizing for minimal reconstruction loss (“distortion”) and low entropy (“rate”). In Stage 2, a causal autoregressive transformer is trained to model the distribution $p(z) \approx \prod_i p(z_i \mid z_{<i})$, with its irreducible cross-entropy loss fundamentally lower-bounded by the entropy of the Stage 1 tokens.

A core observation is the existence of a trade-off between the objectives of the two stages. Improved Stage 1 reconstruction (lower distortion) incurs higher token entropy, increasing the difficulty for Stage 2 (higher irreducible loss). Conversely, more aggressive compression (lower rate) degrades reconstruction but produces latents with lower conditional entropy, allowing smaller or compute-constrained autoregressive models to achieve better generation performance. CRT is motivated by the insight that, since Stage 2 is strictly causal, aligning Stage 1's latent space to directly accommodate causal modeling yields a more tractable distribution for the downstream model [(Ramanujan et al., 2024), Figs. 3–5].

2. Formalization of Causally Regularized Tokenization

CRT augments the base VQGAN autoencoder loss with a causal next-token prediction term, explicitly encouraging the latent tokens to be easy to model with an autoregressive transformer. The VQGAN loss function is: $$\mathcal{L}_{\mathrm{VQGAN}}(x) = \lambda_{\mathrm{VQ}}\,\mathcal{L}_{\mathrm{VQ}}(x, \hat x) + \lambda_{\mathrm{GAN}}\,\mathcal{L}_{\mathrm{GAN}}(\hat x) + \lambda_{\mathrm{Percep}}\,\mathcal{L}_{\mathrm{LPIPS}}(x, \hat x) + \lambda_{2}\,\|x-\hat x\|_2^2$$ where $\lambda_{\mathrm{VQ}}=1.0$, $\lambda_{\mathrm{GAN}}=0.5$, $\lambda_{\mathrm{Percep}}=1.0$, $\lambda_{2}=1.0$.

The CRT regularizer is an $\ell_2$ loss operating on pre-quantized latents $\hat z_i$, with a two-layer causal transformer $z = (z_1, \dots, z_N)$0: $z = (z_1, \dots, z_N)$1 The full CRT Stage 1 objective is: $z = (z_1, \dots, z_N)$2 $z = (z_1, \dots, z_N)$3 is set to 4.0 and linearly annealed from 0 to 4 over the first 1,000 optimization steps.

3. Influence on the Latent Distribution

The central goal of CRT is to minimize the conditional entropy $z = (z_1, \dots, z_N)$4, directly corresponding to the dependency structure learned by an autoregressive transformer in Stage 2. Empirically, CRT reduces per-position cross-entropy for a fixed-size generative model (notably at later token positions), with no reduction in codebook utilization or marginal entropy [Fig. 8].

This is achieved by propagating the gradient of $z = (z_1, \dots, z_N)$5 back to the encoder, which leads to latents that are inherently more sequentially predictable:

• Causal graph: $z = (z_1, \dots, z_N)$6, with $z = (z_1, \dots, z_N)$7 for next-token prediction.
• CRT regularization shapes $z = (z_1, \dots, z_N)$8 such that it exhibits lower conditional entropy, facilitating efficient learning for Stage 2 models.

A plausible implication is that CRT introduces a form of task-aware inductive bias into the representation space, bridging compression and generation objectives more holistically than optimizing for distortion alone.

4. Implementation Details and Algorithmic Specification

The CRT methodology incorporates the following practical elements:

• Stage 1 architecture: VQGAN base (codebook size 16k, downsampling to $z = (z_1, \dots, z_N)$9 feature map, $p(z) \approx \prod_i p(z_i \mid z_{<i})$0 tokens/image) with an added two-layer causal transformer $p(z) \approx \prod_i p(z_i \mid z_{<i})$1.
• Stage 1 hyperparameters: $p(z) \approx \prod_i p(z_i \mid z_{<i})$2; AdamW optimizer (lr $p(z) \approx \prod_i p(z_i \mid z_{<i})$3, $p(z) \approx \prod_i p(z_i \mid z_{<i})$4, weight decay $p(z) \approx \prod_i p(z_i \mid z_{<i})$5); annealing of $p(z) \approx \prod_i p(z_i \mid z_{<i})$6 from $p(z) \approx \prod_i p(z_i \mid z_{<i})$7 in 1k iterations; $p(z) \approx \prod_i p(z_i \mid z_{<i})$8k training iterations (up to $p(z) \approx \prod_i p(z_i \mid z_{<i})$9k for CRT$$\mathcal{L}_{\mathrm{VQGAN}}(x) = \lambda_{\mathrm{VQ}}\,\mathcal{L}_{\mathrm{VQ}}(x, \hat x) + \lambda_{\mathrm{GAN}}\,\mathcal{L}_{\mathrm{GAN}}(\hat x) + \lambda_{\mathrm{Percep}}\,\mathcal{L}_{\mathrm{LPIPS}}(x, \hat x) + \lambda_{2}\,\|x-\hat x\|_2^2$$0).
• Stage 2 modeling: Llama-2 transformer variants (50M–775M parameters), causal attention, class token augmentation, classifier-free guidance, batch size $$\mathcal{L}_{\mathrm{VQGAN}}(x) = \lambda_{\mathrm{VQ}}\,\mathcal{L}_{\mathrm{VQ}}(x, \hat x) + \lambda_{\mathrm{GAN}}\,\mathcal{L}_{\mathrm{GAN}}(\hat x) + \lambda_{\mathrm{Percep}}\,\mathcal{L}_{\mathrm{LPIPS}}(x, \hat x) + \lambda_{2}\,\|x-\hat x\|_2^2$$1, cosine learning rate to $$\mathcal{L}_{\mathrm{VQGAN}}(x) = \lambda_{\mathrm{VQ}}\,\mathcal{L}_{\mathrm{VQ}}(x, \hat x) + \lambda_{\mathrm{GAN}}\,\mathcal{L}_{\mathrm{GAN}}(\hat x) + \lambda_{\mathrm{Percep}}\,\mathcal{L}_{\mathrm{LPIPS}}(x, \hat x) + \lambda_{2}\,\|x-\hat x\|_2^2$$2 with $$\mathcal{L}_{\mathrm{VQGAN}}(x) = \lambda_{\mathrm{VQ}}\,\mathcal{L}_{\mathrm{VQ}}(x, \hat x) + \lambda_{\mathrm{GAN}}\,\mathcal{L}_{\mathrm{GAN}}(\hat x) + \lambda_{\mathrm{Percep}}\,\mathcal{L}_{\mathrm{LPIPS}}(x, \hat x) + \lambda_{2}\,\|x-\hat x\|_2^2$$3k warmup, trained for $$\mathcal{L}_{\mathrm{VQGAN}}(x) = \lambda_{\mathrm{VQ}}\,\mathcal{L}_{\mathrm{VQ}}(x, \hat x) + \lambda_{\mathrm{GAN}}\,\mathcal{L}_{\mathrm{GAN}}(\hat x) + \lambda_{\mathrm{Percep}}\,\mathcal{L}_{\mathrm{LPIPS}}(x, \hat x) + \lambda_{2}\,\|x-\hat x\|_2^2$$4k steps.

This structure aligns encoder representations with causal predictability, embedding inductive bias for Stage 2 efficiency [§3.5, Appendix A.1].

5. Empirical Evaluation and Quantitative Benchmarks

CRT exhibits substantial gains both in compute and parameter efficiency compared to baseline tokenizations.

Compute-controlled gFID results (Table 4, Fig. 6):

Model Size (M)	Baseline FID	CRT FID
111	4.90	4.34
211	3.32	2.94
340	2.89	2.75
550	2.77	2.55
740	2.55	2.35

At 256 tokens/image, CRT tokens reach the same gFID with 1.5–3$$\mathcal{L}_{\mathrm{VQGAN}}(x) = \lambda_{\mathrm{VQ}}\,\mathcal{L}_{\mathrm{VQ}}(x, \hat x) + \lambda_{\mathrm{GAN}}\,\mathcal{L}_{\mathrm{GAN}}(\hat x) + \lambda_{\mathrm{Percep}}\,\mathcal{L}_{\mathrm{LPIPS}}(x, \hat x) + \lambda_{2}\,\|x-\hat x\|_2^2$$5 fewer training FLOPs across model scales.

System-level comparison (Table 1): CRT$$\mathcal{L}_{\mathrm{VQGAN}}(x) = \lambda_{\mathrm{VQ}}\,\mathcal{L}_{\mathrm{VQ}}(x, \hat x) + \lambda_{\mathrm{GAN}}\,\mathcal{L}_{\mathrm{GAN}}(\hat x) + \lambda_{\mathrm{Percep}}\,\mathcal{L}_{\mathrm{LPIPS}}(x, \hat x) + \lambda_{2}\,\|x-\hat x\|_2^2$$6–AR–775M achieves 2.18 FID at ImageNet scale, matching the prior 3.1B parameter model (LlamaGen) but utilizing just one-fourth the parameters and half the tokens (256 vs. 576).

CRT demonstrates improved scaling laws versus baseline methods over codebook and token count sweeps (Figs. 4–5), and generalizes across diverse datasets, yielding better gFID$$\mathcal{L}_{\mathrm{VQGAN}}(x) = \lambda_{\mathrm{VQ}}\,\mathcal{L}_{\mathrm{VQ}}(x, \hat x) + \lambda_{\mathrm{GAN}}\,\mathcal{L}_{\mathrm{GAN}}(\hat x) + \lambda_{\mathrm{Percep}}\,\mathcal{L}_{\mathrm{LPIPS}}(x, \hat x) + \lambda_{2}\,\|x-\hat x\|_2^2$$7 on LSUN categories (Table 5).

6. Ablation Studies and Hyperparameter Sensitivity

Ablative analysis indicates:

• Causal transformer depth: Two layers in $$\mathcal{L}_{\mathrm{VQGAN}}(x) = \lambda_{\mathrm{VQ}}\,\mathcal{L}_{\mathrm{VQ}}(x, \hat x) + \lambda_{\mathrm{GAN}}\,\mathcal{L}_{\mathrm{GAN}}(\hat x) + \lambda_{\mathrm{Percep}}\,\mathcal{L}_{\mathrm{LPIPS}}(x, \hat x) + \lambda_{2}\,\|x-\hat x\|_2^2$$8 achieves optimal trade-off; larger depths degrade both reconstruction (rFID) and generative FID, with ≥4 layers impairing performance (Fig. 9, top).
• CRT weight $$\mathcal{L}_{\mathrm{VQGAN}}(x) = \lambda_{\mathrm{VQ}}\,\mathcal{L}_{\mathrm{VQ}}(x, \hat x) + \lambda_{\mathrm{GAN}}\,\mathcal{L}_{\mathrm{GAN}}(\hat x) + \lambda_{\mathrm{Percep}}\,\mathcal{L}_{\mathrm{LPIPS}}(x, \hat x) + \lambda_{2}\,\|x-\hat x\|_2^2$$9: Increasing $\lambda_{\mathrm{VQ}}=1.0$0 monotonically sacrifices reconstruction in favor of generation ease; $\lambda_{\mathrm{VQ}}=1.0$1 provides the best generative FID without severe degradation in reconstruction (Fig. 9, bottom).

These findings constrain the design space and highlight the necessity of balancing causal regularization intensity and model complexity.

7. Context and Implications

CRT reexamines the fundamental assumption underlying two-stage generative modeling: that maximizing Stage 1 reconstruction yields optimal downstream generativity. The methodology instead exploits the causal nature of autoregressive transformers, sacrificing distortion for tractability in generative modeling. This paradigm achieves 1.5–3$\lambda_{\mathrm{VQ}}=1.0$2 compute efficiency improvements and parameter-lean state-of-the-art performance on discrete autoregressive image generation, establishing CRT as an effective approach for compute-constrained and scaling-sensitive generative modeling (Ramanujan et al., 2024).

A plausible implication is the extensibility of CRT to other modalities and modeling paradigms in which downstream modeling assumptions can be integrated into the upstream compression process, potentially generalizing the causal regularization principle to a broad range of latent variable modeling tasks.