---
title: 'MingTok: Unified Continuous Tokenization'
url: https://www.emergentmind.com/topics/mingtok
type: topic
---

# MingTok: Unified Continuous Tokenization

MingTok is a continuous-space tokenizer introduced for unified autoregressive generation and understanding in vision, and extended in speech as a unified continuous tokenizer for understanding, generation, and editing. In the visual setting, MingTok is defined as a new family of visual tokenizers with a continuous latent space, designed to eliminate quantization error and reconcile the competing requirements of vision–language understanding and autoregressive image generation. Built on top of it, Ming-UniVision formulates both understanding and generation as next-token prediction in a shared continuous space and supports multi-round, in-context tasks such as iterative understanding, generation and editing [2510.06590]. In the speech setting, MingTok-Audio is the VAE-based continuous speech tokenizer at the heart of Ming-UniAudio, where the principal design goal is to produce a single, continuous representation that simultaneously carries high-level semantic cues for understanding tasks and rich acoustic detail for generation tasks [2511.05516].

## 1. Motivation and problem setting

MingTok is motivated by the claim that visual tokenization remains a core challenge in unifying visual understanding and generation within the autoregressive paradigm. Existing methods typically employ tokenizers in discrete latent spaces to align with the tokens from large language models, and the associated quantization errors can limit semantic expressiveness and degrade the capability of vision-language understanding. The detailed description further states that most existing unified multimodal models use vector quantization to align image tokens with the discrete vocabularies of large language models, and that quantization introduces reconstruction error, limits the capacity to represent fine-grained semantics and textures, and creates a hard bottleneck that degrades downstream understanding and image fidelity during generation [2510.06590].

The central alternative is a continuous latent space. In MingTok, each image patch is represented as a continuous vector $\mathbf{z}_t \in \mathbb{R}^d$, which removes quantization artifacts. The same continuous space is intended to carry both compact, generation-friendly codes at the low level and rich, discriminative, text-aligned features at the high level. This design directly addresses the stated tension that understanding tasks favor discriminative high-dimensional features, whereas generation tasks prefer compact low-level codes [2510.06590].

A closely related formulation appears in speech. MingTok-Audio is introduced in response to the claim that existing speech models suffer from competing requirements on token representations by understanding and generation tasks, and that this discrepancy in representation prevents speech language models from performing instruction-based free-form editing. In that setting, the tokenizer remains fully continuous, foregoing a codebook, and is explicitly presented as a solution to the semantic–acoustic trade-off [2511.05516].

## 2. Three-stage visual tokenizer architecture

MingTok tokenizes an image via a sequential pipeline of three transformer-based modules. The output of each stage feeds into the next, and all stages are trained end-to-end under masked modeling objectives. The three stages are low-level encoding, semantic expansion, and visual reconstruction [2510.06590].

The **Low-Level Encoder** takes raw image patches as input, with the summary giving the example of $512 \times 512$ inputs with $32 \times 32$ patches. Its core consists of vision transformer blocks with full self-attention to capture spatial structure. The output is a sequence of compact continuous latents $\{z_t\}$ with dimension, for example, $32$ per token. Compression is performed by a linear output layer with channel-averaging shortcuts that reduces each patch embedding to a small generation-friendly vector.

The **Semantic Decoder** consumes the compact latents, with some tokens randomly masked. Its core is causal-mask transformer layers that autoregressively expand each latent into a higher-dimensional semantic feature, for example $1{,}024$ dimensions, suitable for vision–language tasks. The output is a set of rich, text-aligned semantic vectors.

The **Pixel Decoder** receives the high-dimensional semantic features with no masking. Its preprocessing stage uses pixel-unshuffle to increase token count and reduce patch size, with the example of moving from $32 \times 32$ to $16 \times 16$. The core again uses transformer blocks, here with full attention, to reconstruct raw pixels. The stated output is a high-fidelity reconstruction of the original image [2510.06590].

This staged design is important because it separates the compactness required for generation efficiency from the richer semantic expansion required for understanding, while retaining a path back to pixels for reconstruction. A plausible implication is that the tokenizer is not merely a compression device; it is also an interface that reshapes visual information across representational levels without requiring a discrete codebook.

## 3. Mathematical formulation and autoregressive unification

Let $z_1,\ldots,z_T \in \mathbb{R}^d$ denote the continuous token sequence output by the low-level encoder for an image, where $T$ is the total number of tokens. The continuous token definition is

$$
z_t = \mathrm{Encoder}_{\mathrm{low\_level}}(\mathrm{patch}_t; \theta_{\mathrm{enc}}), \qquad z_t \in \mathbb{R}^d.
$$

MingTok is trained under the multi-task objective

$$
L_{\mathrm{total}} = \lambda_{\mathrm{feat}} L_{\mathrm{feat}} + \lambda_{\mathrm{sem}} L_{\mathrm{sem}} + \lambda_{\mathrm{pix}} L_{\mathrm{pix}} + \lambda_{\mathrm{AR}} L_{\mathrm{AR}}.
$$

The autoregressive generation objective, used when the language model predicts $z_t$ one by one, is

$$
L_{\mathrm{AR}} = - \sum_{t=1}^T \log p(z_t \mid z_{<t}, \mathrm{text}),
$$

where $p(\cdot)$ is parameterized by a small rectified flow vision head attached to the language model. Masked feature-prediction for low-level latents is

$$
L_{\mathrm{feat}} = \sum_{i \in \mathcal{M}} \| \hat{f}_i - f_i^{(\mathrm{DINO})} \|_2^2,
$$

where $\mathcal{M}$ is the set of masked token positions and $f_i^{(\mathrm{DINO})}$ is the pretrained DINOv2 feature at that patch. Masked semantic-feature distillation is

$$
L_{\mathrm{sem}} = \sum_{i \in \mathcal{M}} \| \hat{s}_i - s_i^{(\mathrm{CLIP})} \|_2^2,
$$

where $s_i^{(\mathrm{CLIP})}$ are text-aligned CLIP features. Pixel reconstruction is

$$
L_{\mathrm{pix}} = \| \hat{I} - I \|_2^2,
$$

which encourages the pixel decoder to recover full images even when some semantic tokens are missing [2510.06590].

Within Ming-UniVision, the input side and output side are organized as a single loop. For understanding, real images pass through the low-level encoder and semantic decoder, and the semantic tokens are fed in parallel to the language model as visual context. For generation, the language model autoregressively outputs continuous latents $z_t$; each $z_t$ is immediately expanded by the semantic decoder into a semantic feature, which conditions the prediction of the next token. On the output side, text tokens are predicted using the pretrained LLM head, while image tokens $z_t$ are predicted via the rectified-flow vision head, enabling seamless interleaving of text and image generation or editing in a single autoregressive sequence [2510.06590].

A common misconception is that aligning image tokens with the token stream of a large language model necessarily requires discrete visual indices. MingTok is introduced precisely to avoid that assumption: both understanding and generation are formulated as next-token prediction in a shared continuous space.

## 4. Empirical profile in vision

The reported empirical results cover vision–language understanding, text-to-image generation, image reconstruction, image editing, and efficiency. On vision–language understanding benchmarks including MMBench, MMStar, MMMU, MathVista, HallusionBench, AI2D, MM-Vet, OCRBench, and MME, a 16B-parameter Ming-UniVision model (“16B-A3B”) achieves an average understanding score competitive with specialized VL understanding models and surpasses previous unified discrete-token approaches by several points on key benchmarks [2510.06590].

On text-to-image generation, Ming-UniVision reports an overall GenEval score of $0.85$ versus $0.82$ for the best prior unified model, BAGEL, and DPG-Bench $\approx 82.1$, setting a new state of the art among unified frameworks. The strongest performance is reported on spatial reasoning and attribute control sub-tasks, with Position $= 0.92$, Colors $= 0.93$, and Color Attribute $= 0.70$.

On image reconstruction over ImageNet validation, the tokenizer reports, at $32 \times$ compression, $\mathrm{rFID} = 0.54$ and $\mathrm{PSNR} = 30.77\,\mathrm{dB}$. After joint pretraining, $\mathrm{rFID}$ drops to $0.38$ and $\mathrm{LPIPS}$ to $0.12$, which is described as demonstrating that end-to-end unified training further refines the semantic decoder’s ability to preserve fine detail.

On image editing with GEdit-Bench, multi-round editing reports Semantic Consistency $= 6.60$, Perceptual Quality $= 6.25$, and Overall $= 5.78$ on a $0$–$10$ scale. Qualitative examples are stated to show clean subject matting, progressive super-resolution, and old-photo restoration without repeated encode-decode cycles.

The efficiency gains are also explicit: $3.5\times$ faster text-to-image generation versus discrete SD-VAE, approximately $66\%$ fewer visual tokens than hybrid AR-diffusion models, and approximately $50\%$ fewer than prior unified autoregressive models [2510.06590].

These results are presented as evidence that a unified continuous visual representation can reconcile the competing requirements imposed by understanding and generation tasks. This suggests that the tokenizer is evaluated not only by reconstruction fidelity, but also by how effectively it functions as a shared interface across heterogeneous downstream tasks.

## 5. Design insights, trade-offs, and open limitations

The principal design insight is framed as **continuous vs. discrete latents**. Removing quantization is reported to greatly improve semantic expressiveness, with no token ceiling, and to preserve texture fidelity. Continuous codes can be smoothly shaped by feature-prediction losses at multiple stages, rather than forcing a hard assignment to a finite codebook [2510.06590].

A second insight is **reconciling competing demands**. Stage 1 latents are compressed for generation efficiency; Stage 2 expands semantics for understanding; Stage 3 reconstructs pixels for visual fidelity. The shared continuous representation is stated to avoid costly detours between pixel, latent, and semantic spaces during multi-round editing or mixed-modality tasks. In this formulation, MingTok is not a single-level representation; it is a staged representation in which compactness, semantics, and fidelity are deliberately separated and then reconnected.

A third insight is **unified training synergies**. Knowledge from generation, specifically fine-grained texture synthesis, is stated to benefit understanding via shared weights in the semantic decoder. Semantic supervision through CLIP features shapes the latent space to be more aligned with text, improving controllability during generation. This suggests that the tokenizer is optimized not only for reconstruction or compression, but also for alignment properties that matter in instruction-following generation [2510.06590].

The limitations are equally explicit. Extremely high information density per token can make very fine-grained editing difficult. Multi-round editing beyond training length and fully free-form interleaved tasks such as describe→edit→compare→regenerate remain areas for future work. These constraints temper any interpretation that a unified continuous tokenizer fully resolves all multimodal sequencing problems; the paper instead presents a design that improves the balance while leaving difficult long-horizon and highly localized manipulations open.

## 6. Extension to speech: MingTok-Audio

MingTok-Audio extends the same continuous-tokenization logic into speech. It is described as the VAE-based continuous speech tokenizer at the heart of Ming-UniAudio, with the principal design goal of producing a single, continuous representation that simultaneously carries high-level semantic cues for understanding tasks such as ASR and rich acoustic detail for generation tasks such as TTS and flow matching [2511.05516].

Its core architecture again has three transformer-only components. The **Encoder** is a causal transformer that ingests $16\,\mathrm{kHz}$ waveform frames, with $320$ samples per frame corresponding to a $50\,\mathrm{Hz}$ frame rate. Each frame is linearly projected to dimension $d_{\text{model}}$, passed through several transformer layers, and finally projected to two vectors of size $d_{\text{latent}}$, which parameterize a diagonal Gaussian
$$
q(z|x)=\mathcal{N}\bigl(\mu(x),\,\mathrm{diag}(\sigma^2(x))\bigr).
$$
Sampling follows
$$
Z_{\text{latent}}=\mu(x)+\sigma(x)\odot\epsilon,\quad \epsilon\sim\mathcal{N}(0,I).
$$

The **Semantic module** is a lightweight projector initialized from Whisper-large v3’s encoder, with convolutions stripped. It maps the low-dimensional $Z_{\text{latent}} \in \mathbb{R}^{T \times d_{\text{latent}}}$ into a high-dimensional unified feature sequence
$$
Z_{\text{uni}} = f_{\text{sem}}(Z_{\text{latent}})\in\mathbb{R}^{T\times d_{\text{model}}}.
$$

The **Decoder** is a causal transformer that turns $Z_{\text{uni}}$ back into a time-frequency representation, followed by a small synthesis head and inverse STFT. By design, $Z_{\text{latent}}$ is compact, $32$–$64$ dim, for efficient diffusion modeling in generation, and $Z_{\text{uni}}$ is rich, $512$–$1024$ dim, for semantic alignment with an LLM.

Training proceeds in three stages. Stage S0 uses an acoustic reconstruction objective based on a hybrid VAE + GAN loss:
$$
\mathcal{L}_G
= \lambda_{\rm rec}\,\mathcal{L}_{\rm rec}
+ \lambda_{\rm adv}\,\mathcal{L}_{\rm adv}
+ \lambda_{\rm fm}\,\mathcal{L}_{\rm fm}
+ \lambda_{\rm KL}\,\mathcal{L}_{\rm KL}.
$$
Stage S1 performs semantic feature distillation with
$$
\mathcal{L}_{\rm distill}
= \|\,Z_\text{uni} - Z_\text{semantic}\|_2^2.
$$
Stage S2 blends semantic alignment with reconstruction through
$$
\mathcal{L}_{\rm align}
= -\sum_{t=1}^T \log P\bigl(y_t\mid Z_\text{uni},\,y_{<t}\bigr),
$$
and
$$
\mathcal{L}
= \lambda_{\rm align}\,\mathcal{L}_{\rm align}
+ \lambda_{\rm rec}\,\mathcal{L}_{\rm rec}.
$$

The empirical profile is correspondingly broad. In speech reconstruction, MingTok-Audio achieves PESQ $4.21$ / SIM $0.96$ / STOI $0.98$ on Mandarin and PESQ $4.04$ / SIM $0.96$ / STOI $0.98$ on English, outperforming all competing tokenizers at $50\,\mathrm{Hz}$. Downstream TTS yields Seed-zh WER $1.04\%$ versus $1.12\%$ for seed-TTS, with SIM $0.75$. In large-scale pretraining on ContextASR Bench, Ming-UniAudio records SOTA on $8/12$ subtasks, and for Chinese voice cloning it achieves a competitive Seed-TTS-WER of $0.95\%$. The applications listed include free-form speech editing without timestamp annotation, voice cloning, instruction-based manipulation of pitch, emotion, or accent by natural-language commands, and multimodal extensions via shared continuous spaces [2511.05516].

Taken together, MingTok and MingTok-Audio define a modality-specific but conceptually unified program: replace discrete codebooks with continuous latent representations that can be compressed for generation, expanded for semantics, and integrated into a single autoregressive or speech-language loop. The visual work emphasizes low-level encoding, semantic expansion, and visual reconstruction; the speech work emphasizes VAE inference, semantic projection, and waveform reconstruction. In both cases, the underlying claim is that a unified continuous representation can better reconcile the competing demands of understanding and generation than discrete latent tokenization.

Source: https://www.emergentmind.com/topics/mingtok