---
title: Autoregressive Transformers with Continuous Tokens
url: https://www.emergentmind.com/topics/autoregressive-transformers-with-continuous-tokens
type: topic
---

# Autoregressive Transformers with Continuous Tokens

Autoregressive Transformers with Continuous Tokens are a paradigm for generative modeling in which sequence models eschew discretization bottlenecks and instead operate directly over vector-valued, high-dimensional continuous token representations. Originating as a response to both the information loss and scalability limits inherent to vector quantization and discrete tokenization, this approach leverages continuous latent spaces produced by advanced tokenizers (e.g., VAEs or diffusion autoencoders) and adapts the Transformer architecture—traditionally designed for categorical token prediction—to model the conditional distributions and generation dynamics of continuous-valued sequences. Applications span visual, audio, language, motion, and hybrid domains, with strong evidence for improved fidelity, richer spectrum of editability, and superior scaling compared to both discrete and diffusion-dominated baselines.

## 1. Continuous Tokenization and Sequence Construction

The core departure from discrete-token transformers is the tokenizer, which maps data into a sequence of continuous vectors. In vision, for example, a VAE or diffusion autoencoder encodes each 256×256 RGB image into a 32×32 grid with 4 or more floating-point channels per location, yielding T = 256 tokens, each a vector in ℝ^d (d typically 4–16), e.g., cₜ∈ℝ¹⁶ [2410.13863]. Patch aggregation (such as 2×2) is commonly employed to reduce sequence length while maintaining spatial information. For text, continuous embedding spaces or learned latent codes support related formations; for audio and motion, analogous latent-space encoders produce frame- or window-wise continuous tokens [2602.12370, 2507.09834].

These continuous codes are projected via a learnable linear transformation into the model’s hidden space:
\[
x_t = W_e\,c_t + b_e,\quad W_e \in \mathbb{R}^{d_{\text{model}} \times d},\;b_e \in \mathbb{R}^{d_{\text{model}}}
\]
The resulting sequence of vectors serves as inputs for the autoregressive Transformer, preserving finer structure and information compared to discrete tokens.

## 2. Transformer Architecture Adaptations

The base model is typically a decoder-only Transformer but with important modifications to accommodate continuous tokens:

- **Input Embedding**: All continuous tokens are linearly projected (sometimes with normalization) into the Transformer’s hidden space.
- **Attention Mechanisms**: Variants include causal (raster order), flexible bidirectional (random order or masked), block-wise, or frequency/progressive ordering. The random order configuration uses bidirectional attention, allowing unmasked (observed) tokens to mutually attend and masked (to-be-predicted) tokens to attend only to the unmasked set at every iteration [2410.13863, 2503.05305].
- **Cross-Modal Conditioners**: For text-conditional tasks, cross-attention to language embeddings, often from frozen LLMs (e.g., T5-XXL in Fluid), is standard.
- **Efficiency and Locality**: Innovations such as linear attention, depthwise convolution, gating mechanisms (KV gates), and block-wise context are incorporated for scalability and spatial locality, notably in LINA [2601.22630].

The architecture remains structurally similar to language transformers but with modality-specific heads (e.g., per-token diffusion or flow-matching MLPs) for continuous value prediction.

## 3. Training Objectives for Continuous Token Prediction

For continuous tokens, traditional categorical prediction is replaced by objectives suited for vector-valued targets:

- **Denoising Objectives**: Diffusion-based token prediction is prevalent. At training, each continuous token is noised by a schedule (e.g., √α(t)xₜ + √(1−α(t))ε), and the model predicts the added noise, minimizing the L₂ loss:
  \[
  \mathcal{L} = \mathbb{E}_{c,\,t,\epsilon} \left\| \epsilon_\theta(x_t^\text{noised}, t, \text{context}) - \epsilon \right\|_2^2
  \]
  [2410.13863, 2503.05305, 2507.09834]
- **Flow-Matching**: Predicting the instant “velocity” of denoising trajectories for each token, particularly in audio/motion and high-fidelity image generation [2602.12370, 2508.10711].
- **Proper Scoring Rules**: Energy score minimization (EAR), Hyvärinen score (diffusion), and log-likelihood (GIVT/flow-based) enable likelihood-free or implicit sampling with theoretical guarantees of distributional consistency [2505.07812].
- **Hybrid Losses**: For multimodal or hybrid architectures (e.g., UniFluid, AGDC), both cross-entropy (for discrete sub-sequences) and per-token continuous losses are combined, often with a balancing coefficient.

Regularization (e.g., weight decay, EMA) is employed to stabilize dense, continuous-space training. For domain-specific settings, auxiliary heads (exit or length regularizers) enable precise sequence termination, especially in variable-length generation [2601.05680].

## 4. Generation Strategies and Ordering

Autoregressive factorization over continuous tokens can follow several orderings:

- **Raster/Causal Order**: Standard left-to-right generation, where each token depends only on its predecessors. Limitation: previously generated tokens are irrevocable, leading to issues in spatial coherence for images.
- **Random/BERT-style Order**: At each iteration, a random permutation determines the prediction order; masked tokens are predicted based on all available context, providing bidirectional context within each generation step. This bidirectionality allows revision and improved global consistency, especially for vision and multimodal generation [2410.13863, 2503.13436].
- **Blockwise/Hierarchical**: Tokens are grouped into blocks or predicted at coarse-to-fine (e.g., low- to high-frequency) granularity. FAR [2503.05305] employs frequency bands as the AR axis, and blockwise schemes improve computational parallelism without sacrificing causality [2507.00425].
- **Fusion with Discrete Tokens**: Hybrid approaches condition continuous generation on pre-selected discrete “modes” (e.g., object categories), as in D2C, DisCon, and AGDC, which ensures global structure is determined before fine-grained detail [2503.17155, 2507.01756, 2601.05680].

## 5. Empirical Performance and Comparative Analysis

Autoregressive Transformers with continuous tokens substantially close the gap to, and in some scenarios surpass, diffusion models and discrete AR systems in several metrics:

| Model           | Modality      | Params    | Inference FID↓ (ImageNet 256) | GenEval (COCO) | Highlights                                  |
|-----------------|--------------|-----------|-------------------------------|----------------|---------------------------------------------|
| Fluid (rand, cont.)    | Image         | 10.5B     | 6.16                         | 0.69           | State-of-the-art among AR T2I [2410.13863]  |
| NextStep-1             | Image         | 14B       | ≈6.2 (COCO 30k)              | 0.73           | Unified, strong editing [2508.10711]        |
| LINA-H                 | Image         | 1.4B      | 2.18                         | 0.66           | Linear attention, efficient [2601.22630]    |
| D2C-L (q-former)       | Image         | 633M      | 3.14                         | –              | Outperforms discrete/cont. alone [2503.17155] |
| DisCon-L               | Image         | 558M      | 1.38 (gFID)                  | –              | Conditional continuous AR [2507.01756]      |
| EAR-H                  | Image         | 937M      | 1.97                         | –              | Single-pass, scoring rule AR [2505.07812]   |
| LLaMo-3B               | Motion        | 3B        | FID 22.5 (text→motion)       | –              | Streaming, no quantization artifact [2602.12370] |
| ARDiT (B=4, cont.)     | Audio         | –         | –                            | –              | 170ms/step, near-perfect speech editing [2406.05551] |

Larger models and random-order generation further improve results, with validation loss scaling roughly as a power law with model size for continuous-token AR [2410.13863]. For hybrid and fusion models, conditioning on discrete modes dramatically improves stability and sample quality [2507.01756, 2503.17155]. In multimodal settings, unified models such as UniFluid and AGDC can simultaneously handle text, vision, and vector data, maintaining or improving performance on both discrete and continuous subspaces [2503.13436, 2601.05680].

## 6. Theoretical and Practical Challenges

Modeling continuous tokens poses density estimation and OOD risk challenges: the support is unbounded, and a naive AR chain over continuous space can result in mode collapse or artifact generation [2507.01756, 2503.17155]. Several architectural and objective innovations address these issues:

- **Diffusion/Flow Heads**: Implicitly parameterize token distributions to avoid explicit pdf estimation, as explicit normalizing flows or Gaussian heads [2410.13863, 2602.12370, 2508.10711].
- **Conditional Hybridization**: Conditioning on discrete, high-level structure stabilizes continuous AR and enables efficient, accurate sample reconstruction [2507.01756].
- **Attention and Memory Management**: Linear attention, locality augmentation, and gating ensure tractable scaling for high-resolution generations [2601.22630].
- **Training Schemes**: Random-order generation, blockwise AR, and curriculum learning all play roles in stabilizing learning and optimizing convergence [2410.13863, 2503.05305, 2507.00425].
- **Objective Formulation**: Proper scoring rule objectives (EAR, Hyvärinen, log-likelihood) offer likelihood-free or amortized viable training [2505.07812, 2507.00425].

## 7. Extensions, Limitations, and Future Directions

Autoregressive Transformers with continuous tokens are being extended to new domains:

- **Video and 3D**: Fluid, LINA, and related frameworks explicitly propose extending to spatiotemporal continuous tokens and higher-dimensional shapes, though challenges in sequence length and context modeling remain [2410.13863, 2601.22630].
- **Hybrid Discrete-Continuous and Multimodal**: Hybrid models (DisCon, D2C, AGDC) showcase improved sample quality, stability, and inference efficiency by leveraging both discrete and continuous token spaces [2503.17155, 2507.01756, 2601.05680].
- **Language, Motion, and Audio**: Continuous latent representations have been successfully exploited in language modeling (TarFlowLM, Token Maturation), motion-language integration (LLaMo), and audio (ARDiT, AudioNTP), with benefits for editability, streaming, and fidelity [2602.12370, 2507.09834, 2406.05551, 2507.00425, 2601.04854].
- **Scaling and Compute Efficiency**: Linear attention, blockwise AR, gating, and learned permutation schedules are critical to scaling continuous-token AR to extreme sequence lengths and large models [2601.22630, 2507.00425, 2410.13863].
- **Sampling and Inference**: Speedups via distillation (e.g., distilling multi-step flow-matching teachers into single-step students), blockwise prediction, and hybrid decoding improve practical applicability for real-time use cases [2406.05551, 2602.12370].

Persisting challenges include stability of unbounded density modeling, efficient long-sequence generation, and further acceleration of diffusion/flow-based sampling in AR contexts. There is active research into learned orderings, blockwise generation, and integration with strong pre-trained LLM or visual backbones [2410.13863, 2503.05305, 2503.13436].

---

References:
- "Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens" [2410.13863]
- "NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scale" [2508.10711]
- "LINA: Linear Autoregressive Image Generative Models with Continuous Tokens" [2601.22630]
- "Continuous Visual Autoregressive Generation via Score Maximization" [2505.07812]
- "Frequency Autoregressive Image Generation with Continuous Tokens" [2503.05305]
- "Rethinking Discrete Tokens: Treating Them as Conditions for Continuous Autoregressive Image Synthesis" [2507.01756]
- "D2C: Unlocking the Potential of Continuous Autoregressive Image Generation with Discrete Tokens" [2503.17155]
- "Autoregressive Diffusion Transformer for Text-to-Speech Synthesis" [2406.05551]
- "Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction" [2507.09834]
- "LLaMo: Scaling Pretrained Language Models for Unified Motion Understanding and Generation with Continuous Autoregressive Tokens" [2602.12370]
- "Token Maturation: Autoregressive Language Generation via Continuous Token Dynamics" [2601.04854]
- "Unified Autoregressive Visual Generation and Understanding with Continuous Tokens" [2503.13436]
- "AGDC: Autoregressive Generation of Variable-Length Sequences with Joint Discrete and Continuous Spaces" [2601.05680]
- "Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows" [2507.00425]
- "Mixture of Tokens: Continuous MoE through Cross-Example Aggregation" [2310.15961]

Source: https://www.emergentmind.com/topics/autoregressive-transformers-with-continuous-tokens