Continuous Tokenization Explained

Updated 5 April 2026

Continuous tokenization is a method that converts continuous data streams (e.g., text, speech, images) into adaptive, learnable representations without hard quantization.
It employs learnable compression and adaptive structures—using techniques like RL, variational encodings, and spline representations—to preserve detailed information.
This approach minimizes information loss, simplifies model pipelines, and enhances performance in applications ranging from robotics to scientific computation.

Continuous tokenization refers to methods that convert continuous data streams (e.g., text, speech, images, actions, time series, or numbers) into compact, learnable, often non-quantized representations—"continuous tokens"—for use within sequence models. These approaches either replace or generalize conventional discrete tokenization (such as byte-pair-encoding, codebook quantization, or binning), preserve more information, and facilitate end-to-end training or adaptive runtime behavior across a broad range of modalities. Techniques span stochastic boundary policies for text segmentation, variational encodings for images, continuous spline representations for time series, and domain-specific architectures for scientific numbers and robot behaviors.

1. Core Principles and Theoretical Motivation

Continuous tokenization emerges from the limitations of conventional, discrete tokenization, which is a deterministic, hard-coded input compression step. Discrete methods—such as BPE, VQ-VAE, or per-dimension binning—impose fixed boundaries and quantization bottlenecks that lead to loss of fine-grained information, domain inflexibility, and hindered performance as the field moves toward fully end-to-end trainable architectures. In response, continuous tokenization enables:

Learnable Compression: Instead of static codebooks or boundary rules, token boundaries, chunk pooling, or latent mappings are directly optimized as part of the model’s overall objective (Dauncey et al., 15 Feb 2026, Zeng et al., 31 Mar 2026).
Information Preservation: Continuous tokens, by avoiding hard quantization, retain both high-frequency and context-conditional details, critical for high-fidelity modeling in speech (Li et al., 2024), images (Zeng et al., 31 Mar 2026), and robotics (Yashima et al., 28 Mar 2026).
Adaptive Structure: Tokenization can flexibly adjust the number, granularity, or content of tokens, often conditioned on data domain, complexity, or task-specific requirements (Chen et al., 3 Nov 2025, Dauncey et al., 15 Feb 2026).
Gradient Flow and Variance Reduction: Integrating the tokenization step inside the learning pipeline introduces discrete (boundary or segmentation) stochasticity, necessitating advanced training strategies—score function estimators, RL-style variance reduction, or surrogate gradients (Dauncey et al., 15 Feb 2026).

A schematic contrast:

Aspect	Discrete Tokenization	Continuous Tokenization
Boundary/Quantization	Fixed codebooks/bins/BPE	Learnable boundaries, real-valued latent tokens
Information Flow	Quantization bottleneck	Minimal information loss, preserves high frequency/context
Training	Pipeline, staged/offline	Joint, end-to-end, often with RL/ST/Splines/VAEs
Application Suitability	Text, heavily preprocessed	Speech, image, action, time series, numerics (esp. for dense/continuous data)

2. Methodological Implementations Across Modalities

Text: End-to-End Joint Tokenization

Continuous tokenization for text reformulates token boundary prediction as a stochastic, learnable policy within the network. Each byte position is assigned a boundary decision sampled from a Bernoulli distribution, with policy parameters optimized via a score-function gradient (REINFORCE) (Dauncey et al., 15 Feb 2026). The model balances tokens' compression (downsampling) and downstream language-modeling loss. Standard RL techniques—variance reduction via baselines, time discounting, and batch centering—make training feasible and stable.

Speech: Continuous Speech Tokenizers and Joint Semantic-Acoustic Representations

Speech continuous tokenizers (e.g., Cont-SPT) encode raw waveforms through convolution + attention encoders, producing a sequence of real-valued vectors per frame (Li et al., 2024). Unlike RVQ-based tokenizers, these models maintain spectral fidelity, especially in high-frequency bands, and achieve better downstream TTS quality (lower WER, higher MOS). Architectures can further blend semantic and acoustic latent spaces via multi-stage residual quantization and teacher-student distillation (Jung et al., 9 Jul 2025).

Images: Variational Latent Tokens and Posterior Collapse Mitigation

Continuous image tokenizers map images to low-dimensional sequences of real-valued latent tokens via VAE frameworks, regularized by KL divergence (Zeng et al., 31 Mar 2026). Posterior collapse—where the encoder outputs become uninformative—is actively avoided through masking (random and semantic), local/global representation alignment (to pretrained feature spaces, e.g., DINOv2), and reconstruction objectives. Compact 1D tokenizations (64–128 tokens) retain high-fidelity (gFID 1.52 at 512x512), competitive with discrete token count methods.

Actions and Time Series: Action Chunk Pooling, DCT, or Splines

For sequenced actions, continuous tokenization strategies replace per-dimension/binning with approaches such as:

Temporal Pooling: HiFlow deterministically pools action trajectories into multiscale means and processes all scales jointly in continuous space, leveraging a flow-matching objective (Yashima et al., 28 Mar 2026).
Discrete Cosine Transform (DCT): FAST compresses multi-dimensional action chunks by projecting to the frequency domain, quantizing coefficients, and applying BPE, improving both efficiency and task performance (Pertsch et al., 16 Jan 2025).
Spline-Based Kinematic Tokens: SplineGPT parameterizes noisy time series (e.g., finance) as explicit cubic splines, extracting coefficients (position, velocity, acceleration, jerk) as physically interpretable continuous tokens. This outperforms patch/difference-based baselines in risk-averse regimes (Kearney, 15 Jan 2026).

Numbers: Continuous Numerical Embeddings

xVal ensures numbers in scientific text are represented as real-valued scalings of a dedicated numeric embedding, introduced via a mask vector during tokenization and handled via a parallel regression head for output generation, leading to substantial decreases in token count and improved out-of-distribution generalization (Golkar et al., 2023).

3. Key Losses, Objectives, and Training Algorithms

Continuous tokenization pipelines replace or augment hard assignments with stochastic, RL-based, or differentiable objectives:

Score-Function Estimator: For text, boundary positions are sampled, and the expected negative log-likelihood is minimized directly via combined cross-entropy and REINFORCE-style gradients, with baseline subtraction, time-discounted return, and batch-advantage centering to control variance (Dauncey et al., 15 Feb 2026).
VAE/ELBO-Based: For images, encoders and decoders are jointly trained on the ELBO, optionally extended with perceptual, adversarial, and representational alignment losses (Zeng et al., 31 Mar 2026).
Spline Optimization: Noisy time series are fit by minimizing data-fidelity plus smoothness functionals to recover basis coefficients (Kearney, 15 Jan 2026).
Action Policies: Policy tokenizations employ flow-matching or prefix-masked decoding, with loss functions reflecting information-theoretic, causal, or ordinal structure, as seen in OAT and HiFlow (Liu et al., 4 Feb 2026, Yashima et al., 28 Mar 2026).

4. Comparative Performance and Modality-Specific Benefits

Empirical evidence reveals:

Text: RL-trained continuous tokenization outperforms fixed or straight-through estimators at 100M parameter scale, reducing bits-per-byte and improving zero-shot NLU on PIQA, HellaSwag, ARC-Easy, and LAMBADA (Dauncey et al., 15 Feb 2026).
Speech: Continuous tokenizers achieve better spectral information retention and higher speech quality (lower WER 6.59%, higher speaker similarity 0.73, improved MoS 1.32) compared to discrete baselines (Li et al., 2024). They are robust across domains and frame rates.
Images: MacTok delivers low generation FID (1.52 at 512x512) with only 128 tokens, avoiding collapse and capturing both local and global semantics (Zeng et al., 31 Mar 2026).
Actions/Time Series: HiFlow achieves state-of-the-art success rates across diverse robotic benchmarks, outperforming both discrete-tokenizer autoregressive and diffusion policies, and producing globally coherent trajectories at lower inference cost (Yashima et al., 28 Mar 2026).
Numbers: xVal achieves higher out-of-distribution regression accuracy and much higher token efficiency (one token per number) than digit-wise or codebook-based schemes, enhancing the applicability of LLMs to scientific domains (Golkar et al., 2023).

5. Limitations, Open Questions, and Future Directions

Despite empirical success, continuous tokenization poses several open challenges:

Variance and Tuning: Score-function or RL-style objectives are sensitive to hyperparameter tuning (e.g., λ_π, γ) and require careful baseline choices for tractable variance (Dauncey et al., 15 Feb 2026).
Posterior Collapse: VAEs used for image tokenization may collapse unless actively regularized; masking, semantics, and alignment are necessary but do not guarantee perfect information preservation (Zeng et al., 31 Mar 2026).
Expressivity vs. Efficiency Trade-Offs: Adapting the number or structure of continuous tokens to content complexity (cf. CDD-VT) points toward future advances in allocative mechanisms, but also raises questions about optimal scheduling and mutual informativeness (Chen et al., 3 Nov 2025).
Numerical Range Saturation: LayerNorm dynamics in continuous numerical encoding can limit the effective dynamic range; broader extensions (Fourier-feature, log-scale) are proposed but not yet standard (Golkar et al., 2023).
Generalization Across Domains: Methods like speech tokenizers may need different teacher architectures depending on the downstream paralinguistic information of interest (e.g., emotion vs. speaker ID) (Jung et al., 9 Jul 2025).
Integration with Multimodal Models: The optimal interface for continuous tokens in large multimodal LLMs remains active research; token matching, cross-modal alignment, and joint training pipelines vary by task (Chen et al., 3 Nov 2025, Li et al., 2024).

6. Practical Implications and Engineering Considerations

Compute Overhead: Continuous tokenization implementations (e.g., Bernoulli boundary sampling for text, 1D latent compression for images) introduce minimal extra computational cost (<0.1% FLOPs for some tasks (Dauncey et al., 15 Feb 2026)) while delivering measurable performance gains.
Pipeline Simplification: Emerging tokenization-free strategies (e.g., HiFlow) further simplify model design, eliminating the need for codebook management, lookup tables, or multi-stage training (Yashima et al., 28 Mar 2026).
Flexibility and Adaptivity: Learnable, differentiable tokenization is highly adaptable, allowing sequence length or compression rate to adjust dynamically to data domain (e.g., code vs. natural text) and room for future, decentralized or conditionally scheduled tokenization modules.
Compatibility: Continuous tokens are directly compatible with standard Transformer architectures, often requiring only minimal changes to encoder/embedding layers and final output heads (Golkar et al., 2023, Li et al., 2024).
Domain-Specific Encoding: Many proposals, such as continuous splines for time series or xVal for numerics, demonstrate performance gains especially pronounced in their respective domains, where traditional discrete representations are brittle or inefficient.

7. Representative Results Across Modalities

Modality	Continuous Tokenization Method	Key Performance or Finding	Reference
Text	RL boundary learning + U-Net	Lower bits-per-byte, higher zero-shot accuracy	(Dauncey et al., 15 Feb 2026)
Speech	Cont-SPT (continuous speech tokens)	WER: 6.59%, SIM: 0.73, EMoS: 1.32 (LibriTTS test-clean)	(Li et al., 2024)
Image	MacTok (masked KL-VAE)	gFID: 1.52 @ 512x512, 128 tokens; avoids collapse	(Zeng et al., 31 Mar 2026)
Actions	HiFlow (multiscale pooling + flow)	88% avg success (MimicGen), smoother global trajectory	(Yashima et al., 28 Mar 2026)
Numbers	xVal (continuous embedding)	1 token/number, best OoD MSE, smallest vocab, ↑efficiency	(Golkar et al., 2023)

Continuous tokenization constitutes a paradigm shift away from static, out-of-band compression towards dynamic, learnable, information-preserving representation, now proving its worth across language, vision, speech, scientific computation, and embodied policy learning.