MingTok: Unified Continuous Tokenization

• MingTok is a continuous-space tokenizer that eliminates quantization errors to reconcile the differing needs of autoregressive image/speech generation and multimodal understanding.
• It uses a three-stage architecture for vision and a VAE-based design for speech, ensuring efficient encoding, semantic expansion, and high-fidelity reconstruction.
• Empirical evaluations show state-of-the-art performance across vision-language benchmarks, text-to-image generation, and speech processing tasks.

MingTok is a continuous-space tokenizer introduced for unified autoregressive generation and understanding in vision, and extended in speech as a unified continuous tokenizer for understanding, generation, and editing. In the visual setting, MingTok is defined as a new family of visual tokenizers with a continuous latent space, designed to eliminate quantization error and reconcile the competing requirements of vision–language understanding and autoregressive image generation. Built on top of it, Ming-UniVision formulates both understanding and generation as next-token prediction in a shared continuous space and supports multi-round, in-context tasks such as iterative understanding, generation and editing (Huang et al., 8 Oct 2025). In the speech setting, MingTok-Audio is the VAE-based continuous speech tokenizer at the heart of Ming-UniAudio, where the principal design goal is to produce a single, continuous representation that simultaneously carries high-level semantic cues for understanding tasks and rich acoustic detail for generation tasks (Yan et al., 26 Oct 2025).

1. Motivation and problem setting

MingTok is motivated by the claim that visual tokenization remains a core challenge in unifying visual understanding and generation within the autoregressive paradigm. Existing methods typically employ tokenizers in discrete latent spaces to align with the tokens from LLMs, and the associated quantization errors can limit semantic expressiveness and degrade the capability of vision-language understanding. The detailed description further states that most existing unified multimodal models use vector quantization to align image tokens with the discrete vocabularies of LLMs, and that quantization introduces reconstruction error, limits the capacity to represent fine-grained semantics and textures, and creates a hard bottleneck that degrades downstream understanding and image fidelity during generation (Huang et al., 8 Oct 2025).

The central alternative is a continuous latent space. In MingTok, each image patch is represented as a continuous vector $\mathbf{z}_t \in \mathbb{R}^d$, which removes quantization artifacts. The same continuous space is intended to carry both compact, generation-friendly codes at the low level and rich, discriminative, text-aligned features at the high level. This design directly addresses the stated tension that understanding tasks favor discriminative high-dimensional features, whereas generation tasks prefer compact low-level codes (Huang et al., 8 Oct 2025).

A closely related formulation appears in speech. MingTok-Audio is introduced in response to the claim that existing speech models suffer from competing requirements on token representations by understanding and generation tasks, and that this discrepancy in representation prevents speech LLMs from performing instruction-based free-form editing. In that setting, the tokenizer remains fully continuous, foregoing a codebook, and is explicitly presented as a solution to the semantic–acoustic trade-off (Yan et al., 26 Oct 2025).

2. Three-stage visual tokenizer architecture

MingTok tokenizes an image via a sequential pipeline of three transformer-based modules. The output of each stage feeds into the next, and all stages are trained end-to-end under masked modeling objectives. The three stages are low-level encoding, semantic expansion, and visual reconstruction (Huang et al., 8 Oct 2025).

The Low-Level Encoder takes raw image patches as input, with the summary giving the example of $512 \times 512$ inputs with $32 \times 32$ patches. Its core consists of vision transformer blocks with full self-attention to capture spatial structure. The output is a sequence of compact continuous latents $\{z_t\}$ with dimension, for example, $32$ per token. Compression is performed by a linear output layer with channel-averaging shortcuts that reduces each patch embedding to a small generation-friendly vector.

The Semantic Decoder consumes the compact latents, with some tokens randomly masked. Its core is causal-mask transformer layers that autoregressively expand each latent into a higher-dimensional semantic feature, for example $1{,}024$ dimensions, suitable for vision–language tasks. The output is a set of rich, text-aligned semantic vectors.

The Pixel Decoder receives the high-dimensional semantic features with no masking. Its preprocessing stage uses pixel-unshuffle to increase token count and reduce patch size, with the example of moving from $32 \times 32$ to $16 \times 16$. The core again uses transformer blocks, here with full attention, to reconstruct raw pixels. The stated output is a high-fidelity reconstruction of the original image (Huang et al., 8 Oct 2025).

This staged design is important because it separates the compactness required for generation efficiency from the richer semantic expansion required for understanding, while retaining a path back to pixels for reconstruction. A plausible implication is that the tokenizer is not merely a compression device; it is also an interface that reshapes visual information across representational levels without requiring a discrete codebook.

3. Mathematical formulation and autoregressive unification

Let $z_1,\ldots,z_T \in \mathbb{R}^d$ denote the continuous token sequence output by the low-level encoder for an image, where $T$ is the total number of tokens. The continuous token definition is

$512 \times 512$0

MingTok is trained under the multi-task objective

$512 \times 512$1

The autoregressive generation objective, used when the LLM predicts $512 \times 512$2 one by one, is

$512 \times 512$3

where $512 \times 512$4 is parameterized by a small rectified flow vision head attached to the LLM. Masked feature-prediction for low-level latents is

$512 \times 512$5

where $512 \times 512$6 is the set of masked token positions and $512 \times 512$7 is the pretrained DINOv2 feature at that patch. Masked semantic-feature distillation is

$512 \times 512$8

where $512 \times 512$9 are text-aligned CLIP features. Pixel reconstruction is

$32 \times 32$0

which encourages the pixel decoder to recover full images even when some semantic tokens are missing (Huang et al., 8 Oct 2025).

Within Ming-UniVision, the input side and output side are organized as a single loop. For understanding, real images pass through the low-level encoder and semantic decoder, and the semantic tokens are fed in parallel to the LLM as visual context. For generation, the LLM autoregressively outputs continuous latents $32 \times 32$1; each $32 \times 32$2 is immediately expanded by the semantic decoder into a semantic feature, which conditions the prediction of the next token. On the output side, text tokens are predicted using the pretrained LLM head, while image tokens $32 \times 32$3 are predicted via the rectified-flow vision head, enabling seamless interleaving of text and image generation or editing in a single autoregressive sequence (Huang et al., 8 Oct 2025).

A common misconception is that aligning image tokens with the token stream of a LLM necessarily requires discrete visual indices. MingTok is introduced precisely to avoid that assumption: both understanding and generation are formulated as next-token prediction in a shared continuous space.

4. Empirical profile in vision

The reported empirical results cover vision–language understanding, text-to-image generation, image reconstruction, image editing, and efficiency. On vision–language understanding benchmarks including MMBench, MMStar, MMMU, MathVista, HallusionBench, AI2D, MM-Vet, OCRBench, and MME, a 16B-parameter Ming-UniVision model (“16B-A3B”) achieves an average understanding score competitive with specialized VL understanding models and surpasses previous unified discrete-token approaches by several points on key benchmarks (Huang et al., 8 Oct 2025).

On text-to-image generation, Ming-UniVision reports an overall GenEval score of $32 \times 32$4 versus $32 \times 32$5 for the best prior unified model, BAGEL, and DPG-Bench $32 \times 32$6, setting a new state of the art among unified frameworks. The strongest performance is reported on spatial reasoning and attribute control sub-tasks, with Position $32 \times 32$7, Colors $32 \times 32$8, and Color Attribute $32 \times 32$9.

On image reconstruction over ImageNet validation, the tokenizer reports, at $\{z_t\}$0 compression, $\{z_t\}$1 and $\{z_t\}$2. After joint pretraining, $\{z_t\}$3 drops to $\{z_t\}$4 and $\{z_t\}$5 to $\{z_t\}$6, which is described as demonstrating that end-to-end unified training further refines the semantic decoder’s ability to preserve fine detail.

On image editing with GEdit-Bench, multi-round editing reports Semantic Consistency $\{z_t\}$7, Perceptual Quality $\{z_t\}$8, and Overall $\{z_t\}$9 on a $32$0–$32$1 scale. Qualitative examples are stated to show clean subject matting, progressive super-resolution, and old-photo restoration without repeated encode-decode cycles.

The efficiency gains are also explicit: $32$2 faster text-to-image generation versus discrete SD-VAE, approximately $32$3 fewer visual tokens than hybrid AR-diffusion models, and approximately $32$4 fewer than prior unified autoregressive models (Huang et al., 8 Oct 2025).

These results are presented as evidence that a unified continuous visual representation can reconcile the competing requirements imposed by understanding and generation tasks. This suggests that the tokenizer is evaluated not only by reconstruction fidelity, but also by how effectively it functions as a shared interface across heterogeneous downstream tasks.

5. Design insights, trade-offs, and open limitations

The principal design insight is framed as continuous vs. discrete latents. Removing quantization is reported to greatly improve semantic expressiveness, with no token ceiling, and to preserve texture fidelity. Continuous codes can be smoothly shaped by feature-prediction losses at multiple stages, rather than forcing a hard assignment to a finite codebook (Huang et al., 8 Oct 2025).

A second insight is reconciling competing demands. Stage 1 latents are compressed for generation efficiency; Stage 2 expands semantics for understanding; Stage 3 reconstructs pixels for visual fidelity. The shared continuous representation is stated to avoid costly detours between pixel, latent, and semantic spaces during multi-round editing or mixed-modality tasks. In this formulation, MingTok is not a single-level representation; it is a staged representation in which compactness, semantics, and fidelity are deliberately separated and then reconnected.

A third insight is unified training synergies. Knowledge from generation, specifically fine-grained texture synthesis, is stated to benefit understanding via shared weights in the semantic decoder. Semantic supervision through CLIP features shapes the latent space to be more aligned with text, improving controllability during generation. This suggests that the tokenizer is optimized not only for reconstruction or compression, but also for alignment properties that matter in instruction-following generation (Huang et al., 8 Oct 2025).

The limitations are equally explicit. Extremely high information density per token can make very fine-grained editing difficult. Multi-round editing beyond training length and fully free-form interleaved tasks such as describe→edit→compare→regenerate remain areas for future work. These constraints temper any interpretation that a unified continuous tokenizer fully resolves all multimodal sequencing problems; the paper instead presents a design that improves the balance while leaving difficult long-horizon and highly localized manipulations open.

6. Extension to speech: MingTok-Audio

MingTok-Audio extends the same continuous-tokenization logic into speech. It is described as the VAE-based continuous speech tokenizer at the heart of Ming-UniAudio, with the principal design goal of producing a single, continuous representation that simultaneously carries high-level semantic cues for understanding tasks such as ASR and rich acoustic detail for generation tasks such as TTS and flow matching (Yan et al., 26 Oct 2025).

Its core architecture again has three transformer-only components. The Encoder is a causal transformer that ingests $32$5 waveform frames, with $32$6 samples per frame corresponding to a $32$7 frame rate. Each frame is linearly projected to dimension $32$8, passed through several transformer layers, and finally projected to two vectors of size $32$9, which parameterize a diagonal Gaussian

$1{,}024$0

Sampling follows

$1{,}024$1

The Semantic module is a lightweight projector initialized from Whisper-large v3’s encoder, with convolutions stripped. It maps the low-dimensional $1{,}024$2 into a high-dimensional unified feature sequence

$1{,}024$3

The Decoder is a causal transformer that turns $1{,}024$4 back into a time-frequency representation, followed by a small synthesis head and inverse STFT. By design, $1{,}024$5 is compact, $1{,}024$6–$1{,}024$7 dim, for efficient diffusion modeling in generation, and $1{,}024$8 is rich, $1{,}024$9–$32 \times 32$0 dim, for semantic alignment with an LLM.

Training proceeds in three stages. Stage S0 uses an acoustic reconstruction objective based on a hybrid VAE + GAN loss:

$32 \times 32$1

Stage S1 performs semantic feature distillation with

$32 \times 32$2

Stage S2 blends semantic alignment with reconstruction through

$32 \times 32$3

and

$32 \times 32$4

The empirical profile is correspondingly broad. In speech reconstruction, MingTok-Audio achieves PESQ $32 \times 32$5 / SIM $32 \times 32$6 / STOI $32 \times 32$7 on Mandarin and PESQ $32 \times 32$8 / SIM $32 \times 32$9 / STOI $16 \times 16$0 on English, outperforming all competing tokenizers at $16 \times 16$1. Downstream TTS yields Seed-zh WER $16 \times 16$2 versus $16 \times 16$3 for seed-TTS, with SIM $16 \times 16$4. In large-scale pretraining on ContextASR Bench, Ming-UniAudio records SOTA on $16 \times 16$5 subtasks, and for Chinese voice cloning it achieves a competitive Seed-TTS-WER of $16 \times 16$6. The applications listed include free-form speech editing without timestamp annotation, voice cloning, instruction-based manipulation of pitch, emotion, or accent by natural-language commands, and multimodal extensions via shared continuous spaces (Yan et al., 26 Oct 2025).

Taken together, MingTok and MingTok-Audio define a modality-specific but conceptually unified program: replace discrete codebooks with continuous latent representations that can be compressed for generation, expanded for semantics, and integrated into a single autoregressive or speech-language loop. The visual work emphasizes low-level encoding, semantic expansion, and visual reconstruction; the speech work emphasizes VAE inference, semantic projection, and waveform reconstruction. In both cases, the underlying claim is that a unified continuous representation can better reconcile the competing demands of understanding and generation than discrete latent tokenization.