---
title: Continuous-Token Autoregressive Transformers
url: https://www.emergentmind.com/topics/autoregressive-transformer-with-continuous-tokens
type: topic
---

# Continuous-Token Autoregressive Transformers

Autoregressive Transformers with continuous tokens are a generalization of the standard Transformer architecture, extending its autoregressive next-token prediction paradigm—historically defined for discrete token spaces—into high-dimensional, continuous domains. This transition supports finer-grained modeling and avoids fundamental limitations of discretization, which is particularly relevant for applications in vision, audio, robotics, and time series data.

## 1. Fundamental Concepts and Rationale

Traditional autoregressive Transformers operate on sequences of discrete tokens, modeling the joint distribution $p(x_1, ..., x_n)$ in a factorized manner, typically as $p(x) = \prod_{i=1}^n p(x_i | x_{<i})$, where each $x_i$ is a categorical variable. However, many modalities—images, audio, time series, actions—are naturally continuous and lose fidelity when quantized. Continuous token modeling overcomes quantization-induced information loss and improves representation capacity. Methods for continuous tokens include:

- **Diffusion loss**: Supervising next-token prediction by denoising noisy versions of the token, as in text-to-image or audio generation [2410.13863, 2507.09834].
- **Flow matching loss**: Teaching shortcut MLP heads to match stochastic flows in latent space; efficient for image generation [2504.18391, 2508.10711].
- **Mixture-based flows**: Using normalizing flows over continuous latent codes, enabling bi-directional context and block-wise generation [2507.00425].
- **Direct regression**: Employing MSE for time series forecasting [2503.09791].

## 2. Model Architectures and Continuous Tokenization

The architectural shift centers on three domains:

**A. Tokenization:**
- **Continuous tokenizers**: VAEs or autoencoders produce continuous latent patches/tokens (e.g., 16 channels per visual patch) for subsequent modeling [2410.13863, 2503.13436, 2503.05305, 2504.18391].
- **Hybrid tokenizers**: Models like HART decompose the image encoder output as $z_\text{continuous} = z_\text{discrete} + z_\text{residual}$ so that discrete tokens model coarse structure, while continuous residuals capture fine detail [2410.10812].
- **Post-training quantization**: TokenBridge discretizes continuous VAE latents dimension-wise using data-driven Gaussian bins, then sequences discrete indices efficiently for AR modeling [2503.16430].

**B. Autoregressive Modeling:**
- **Causal Transformer**: Autoregressively predicts the next continuous token given the previous tokens and/or multimodal inputs [2410.13863, 2508.10711].
- **Bidirectional/random-order Transformer**: Permutes prediction order to enhance global coherence and avoid raster artifacts [2410.13863, 2503.13436, 2507.00425].

**C. Output Heads and Losses:**
- **Diffusion heads**: Model per-token conditional distributions and supervise with denoising loss [2410.13863, 2507.09834].
- **Shortcut heads**: MLPs trained under flow matching and consistency losses enable efficient few-step sampling [2504.18391].
- **Mixture-based flows**: Enable invertible mapping between latent and standard normal distributions [2507.00425].
- **Cross-entropy over discretized tokens**: When using TokenBridge-type post-quantization [2503.16430].

## 3. Domain-Specific Implementations

Continuous token autoregressive modeling has seen concrete deployment across:

| Domain      | Method Highlights                                                                             | Key Metrics                       |
|-------------|----------------------------------------------------------------------------------------------|-----------------------------------|
| Vision      | Random-order AR with continuous tokens [Fluid, 2410.13863]; FAR with frequency AR [2503.05305]; Hybrid tokenization [HART, 2410.10812]; Flow matching [NextStep-1, 2508.10711] | FID, GenEval, PSNR                |
| Audio       | AudioNTP and AudioMNTP: Continuous-token AR with token-wise diffusion and masked next-token tasks [2507.09834] | FAD, FD, KL, CLAP, IS             |
| Speech      | DiTAR: Patch-based AR with LM and diffusion transformer [2502.03930]                         | Speaker similarity, TTS metrics   |
| Video       | VideoMAR: AR with continuous tokens, next-frame diffusion loss, KV caching [2506.14168]; GPDiT: AR diffusion transformer with rotation-based time conditioning [2505.07344] | VBench-I2V; throughput, diversity |
| Language    | TarFlowLM: AR normalizing flows for sentence-level continuous latent codes [2507.00425]; SONAR-LLM: AR over continuous sentence embeddings with cross-entropy supervision [2508.05305] | BPC, perplexity, NLG metrics      |
| Time series | Minimal adaptations: linearly mapping continuous tokens, expanded positional encoding [2503.09791] | Forecasting accuracy              |
| Robotics    | FreqPolicy: AR in DCT frequency space with continuous tokens, hierarchical generation [2506.01583] | Task success, efficiency          |

## 4. Optimization and Efficiency Techniques

Several efficiency-oriented developments have been introduced:

- **Shortcut heads and flow matching**: FAR achieves up to 2.3× faster inference versus MAR (diffusion-based AR), with comparable FID [2504.18391].
- **Hierarchical frequency progression**: FAR for vision [2503.05305] and FreqPolicy for robotics [2506.01583] autoregressively build solutions in the frequency domain, stabilizing low-frequency structure before refining detail.
- **Temporal and spatial curriculum**: VideoMAR employs short-to-long training and progressive resolution to manage long video sequences efficiently [2506.14168].
- **Parallel generation**: Within frames, VideoMAR uses bidirectional attention for parallel masked prediction; spatial/temporal extrapolation is afforded by 3D relative position embeddings [2506.14168].
- **Token mixing and transition tuning**: MoT models aggregate token mixtures cross-example for continuous, scalable MoE, with a softmax temperature for privacy and interpretable routing [2310.15961].
- **Cache management**: Efficient key-value (KV) caching is retained, especially for causal decoding, enhancing throughput and memory use [2410.10812, 2508.10711].

## 5. Evaluation, Scaling Trends, and Trade-Offs

- **Scaling in vision**: As model size increases, continuous token models (like Fluid) maintain or improve FID and GenEval, while discrete-token counterparts saturate or degrade due to quantization bottlenecks [2410.13863].
- **Trade-offs**: Unified models (UniFluid [2503.13436]) show a loss-balance hyperparameter $\lambda$ affects both image generation (FID, GenEval) and understanding (CIDEr, QA scores). Careful selection of $\lambda$ enables competitive multitask performance.
- **Comparative quality**: DisCon [2507.01756] and TokenBridge [2503.16430] show that continuous tokens—either directly or as post-quantized discrete proxies—yield reconstruction and generation quality on par with or superior to prior discrete AR approaches. DisCon achieves gFID 1.38 on ImageNet 256×256; TokenBridge achieves similar results with nearly 6× speedup in token prediction.

## 6. Extensions and Open Research Directions

- **Hybrid paradigms**: Models like HART [2410.10812] and DisCon [2507.01756] combine discrete- and continuous-token AR, using discrete tokens as high-level priors or conditioning signals to stabilize dense generation.
- **Normalizing flows for flexible language modeling**: TarFlowLM [2507.00425] demonstrates exact invertible modeling for continuous latent texts, enabling bi-directional context and multi-pass hierarchical editing.
- **Multimodal architectures**: NextStep-1 [2508.10711], UniFluid [2503.13436], and GPDiT [2505.07344] open new approaches for joint text-image, video, or image editing and question answering tasks using AR with continuous tokens.
- **Efficient continuous token discretization**: Post-training quantization as in TokenBridge may inspire future architectures to merge generation efficiency of categorical loss with the expressivity of continuous latent spaces.
- **Future refinement**: Potential exists for improving residual token modeling (e.g., lighter-weight diffusion or shortcut heads), alternate positional embeddings, and scalable patch-wise prediction.

## 7. Limitations and Controversies

- **Computational overhead**: Early continuous token AR models (MAR [2504.18391]; lookahead attention [2305.12272]) incur significant inference cost due to iterative denoising or bidirectional attention. Shortcut heads and efficient discretization have mitigated but not fully solved these issues.
- **Robustness/stability**: Continuous distributions impose density estimation challenges and risk out-of-distribution artifacts. DisCon [2507.01756] circumvents this via discrete conditioning, and post-training quantization (TokenBridge) avoids codebook instability.
- **Interpretability**: Continuous token outputs can be less interpretable compared to discrete symbol sequences; hybrid or conditional paradigms partially address this.
- **Multitask trade-offs**: Simultaneously optimizing for generation and understanding may reduce performance in either task if loss balance is not tuned correctly [2503.13436].

---

Autoregressive Transformers with continuous tokens encompass a rapidly expanding family of models that generalize AR generation and understanding across modalities. By moving beyond quantization, these models achieve gains in fidelity, scaling, and flexibility, while ongoing research addresses computational efficiency and stability in high-dimensional continuous spaces. This paradigm shift is evidenced across image, audio, video, language, and robotics domains, with a variety of architectures and optimization strategies now demonstrating state-of-the-art performance.

Source: https://www.emergentmind.com/topics/autoregressive-transformer-with-continuous-tokens