---
title: 'VibeToken: Resolution-Agnostic Image Tokenizer'
url: https://www.emergentmind.com/topics/vibetoken
type: topic
---

# VibeToken: Resolution-Agnostic Image Tokenizer

Searching arXiv for the specified VibeToken paper to ground the article in the cited source.
VibeToken is a resolution-agnostic 1D image tokenizer introduced for autoregressive image synthesis across arbitrary resolutions and aspect ratios [2604.24885]. In the same work, VibeToken-Gen denotes the class-conditioned autoregressive generator trained on top of its discrete latents. The central design goal is to decouple token length from image resolution: instead of inheriting a 2D latent grid whose token count scales with image size, VibeToken compresses an image into a dynamic, user-controllable sequence of 32–256 tokens, with headline experiments emphasizing 64-token generation at \(1024\times1024\) [2604.24885]. The paper positions this as a way to narrow the gap between autoregressive models and diffusion models at scale while preserving out-of-the-box support for arbitrary resolutions.

## 1. Problem setting and definition

VibeToken is motivated by a specific bottleneck in visual autoregressive modeling: standard visual AR pipelines usually inherit tokens from 2D grid tokenizers, so the token count grows with spatial resolution as
\[
T = \frac{H}{f}\cdot \frac{W}{f},
\]
which makes transformer cost rise rapidly with image size [2604.24885]. The autoregressive factorization used in the paper is the standard next-token form
\[
p(x_{1:T}) = \prod_{t=1}^{T} p_{\theta}(x_t \mid x_{<t}),
\]
or, with end-of-sequence handling,
\[
p(x_{1:T}, \langle\mathrm{eos}\rangle) = \prod_{t=1}^{T+1} p_{\theta}(x_t \mid x_{<t}), \qquad x_{T+1}=\langle\mathrm{eos}\rangle.
\]
Because transformer self-attention scales quadratically in sequence length, fixed-grid tokenizers make high-resolution AR generation computationally expensive [2604.24885].

The paper defines VibeToken as a 1D tokenizer intended to break that coupling. Rather than preserving a latent 2D lattice, it maps an image of arbitrary resolution and aspect ratio into a short 1D discrete sequence, then decodes that sequence back to a target canvas [2604.24885]. In the paper’s framing, “resolution-agnostic” means that the tokenizer can encode images at different resolutions and aspect ratios, decode to arbitrary target resolutions, and support native super-resolution without retraining a separate model per canvas [2604.24885].

A common point of confusion is nomenclature. In the VibeVoice technical report, “VibeToken” is not the paper’s formal term; that work instead describes a continuous VAE-style acoustic tokenizer for speech [2508.19205]. In [2604.24885], by contrast, VibeToken refers specifically to a 1D image tokenizer for dynamic-resolution image generation.

## 2. Tokenizer architecture

VibeToken is a Transformer-based 1D tokenizer built around four components named explicitly in the paper: dynamic grid positional embedding, adaptive patch embedding, adaptive decoder resolution, and dynamic-length latent tokenization [2604.24885]. The starting point is a TiTok-style encoder-decoder formulation. For an image
\[
v \in \mathbb{R}^{3\times H \times W},
\]
with patch size \(k\), the image is patchified into
\[
N = \frac{HW}{k^2}
\]
patches, projected to \(\mathbb{R}^d\), concatenated with \(L\) learned latent tokens, and passed through the encoder:
\[
x^{\mathrm{enc}} = \mathcal{E}_{\theta}(x_0)\in\mathbb{R}^{(N+L)\times d}, \qquad h=x^{\mathrm{enc}}_{N+1:N+L}\in\mathbb{R}^{L\times d}.
\]
A codebook
\[
C=[c_1 \cdots c_m]\in\mathbb{R}^{m\times d}
\]
then quantizes the latent sequence as
\[
z = Q(h)\in[m]^L.
\]
The decoder consumes quantized token embeddings and masked output tokens:
\[
u=\mathcal{D}_{\phi}\!\bigl([\,C(z)\,\Vert\,y_{\text{mask}}\,]\bigr)\in\mathbb{R}^{(L+N)\times d}.
\]
This is the baseline 1D-tokenizer formulation that VibeToken generalizes [2604.24885].

The dynamic grid positional embedding addresses arbitrary-resolution inputs. The paper learns a base positional grid
\[
\mathbf{G}\in\mathbb{R}^{d \times T_H^{\max} \times T_W^{\max}},
\]
with \(T_H^{\max}=T_W^{\max}=32\), and resizes it to the current patch lattice:
\[
T_H=\Bigl\lceil\frac{H}{k_h}\Bigr\rceil, \qquad T_W=\Bigl\lceil\frac{W}{k_w}\Bigr\rceil,
\]
\[
\widehat{\mathbf{G}} = \operatorname{resize}\!\bigl(\mathbf{G};\,T_H,\,T_W\bigr) \in\mathbb{R}^{d\times T_H\times T_W}.
\]
The paper reports that this dynamic grid embedding yields about 33% FLOPs reduction relative to learnable axial RoPE, without quality loss [2604.24885].

Adaptive patch embedding makes patch size variable rather than fixed. Let \(W_{k_{\max}}\) be the learned patch projection for the maximum patch size \(k_{\max}\), and \(R_{k\leftarrow k_{\max}}\) a resizing operator. The paper defines
\[
W_k = W_{k_{\max}}\, R_{k\leftarrow k_{\max}}, \qquad e_{\text{patch}}(p_k) = W_k\, p_k + b.
\]
In practice, the model uses \(k_{\max}=32\) and trains over patch sizes \(\{8,12,16,32\}\) [2604.24885]. This allows the encoder to handle variable patchification schemes without learning separate projections for each case.

Adaptive decoder resolution decouples patch layout from output canvas. The decoder first predicts an intermediate image at
\[
H' = T_H\,k_{\max}, \qquad W' = T_W\,k_{\max},
\]
producing \(\tilde v\), then applies a learned 2D convolutional downscaling layer to obtain the target output
\[
\hat v = \mathcal{D}_{H,W}(\tilde v)\in\mathbb{R}^{3\times H\times W}.
\]
This is the mechanism that the paper uses to support arbitrary target resolutions and native super-resolution [2604.24885].

Dynamic-length latent tokenization is the final architectural ingredient. Instead of fixing one sequence length, VibeToken samples a latent length
\[
L \sim \mathcal{P}(L), \qquad L\in[L_{\min},L_{\max}],
\]
with the reported operating range \(L\in[32,256]\) [2604.24885]. The encoder produces exactly \(L\) latent tokens and the decoder consumes exactly \(L\) latent tokens, without padding. This gives a direct quality–efficiency trade-off under user control.

## 3. Quantization, objectives, and training regime

The final tokenizer uses multi-vector quantization. The implementation details summarized in the paper specify 8 codebooks, each of size 4096, for an effective vocabulary size of 32,768; the token latent dimension is 256, factorized into 8 sub-codes of 32 dimensions each [2604.24885]. The paper does not reproduce the full MVQ update equations in the visible main text, but it does state that quantization uses
\[
z = Q(h)\in[m]^L
\]
and that a reparameterization trick is applied so gradients flow to \(h\) and \(C\) [2604.24885].

The full tokenizer loss is not written out explicitly in the main text excerpt, but the supplement reports the component weights. The loss includes reconstruction, perceptual, discriminator, quantizer, and commitment terms, with reconstruction weight \(1\), perceptual weight \(1.1\), discriminator weight \(1\), quantizer weight \(1\), commitment cost \(0.25\), and discriminator start at 300,000 steps [2604.24885]. A structured summary consistent with the reported components is
\[
\mathcal{L}_{\text{tok}} = \lambda_{\text{rec}}\mathcal{L}_{\text{rec}} + \lambda_{\text{perc}}\mathcal{L}_{\text{perc}} + \lambda_{\text{adv}}\mathcal{L}_{\text{adv}} + \lambda_{\text{q}}\mathcal{L}_{\text{q}} + \lambda_{\text{commit}}\mathcal{L}_{\text{commit}},
\]
with
\[
\lambda_{\text{rec}}=1,\quad \lambda_{\text{perc}}=1.1,\quad \lambda_{\text{adv}}=1,\quad \lambda_{\text{q}}=1,\quad \lambda_{\text{commit}}=0.25.
\]
The paper itself does not spell out the exact closed forms of each component in the excerpted text [2604.24885].

Tokenizer training is carried out on ImageNet-1k under mixed resolutions between \(256\times256\) and \(512\times512\), with aspect ratios sampled from \(\{1\!:\!1,\,1\!:\!2,\,2\!:\!1,\,2\!:\!3,\,3\!:\!2\}\) [2604.24885]. Patch sizes are sampled from \(\{8,12,16,32\}\), subject to the condition \(N\le 1024\), and the input resolution \((H_{\text{in}},W_{\text{in}})\) and target resolution \((H_{\text{out}},W_{\text{out}})\) are sampled independently [2604.24885]. This suggests that the model is trained not merely for same-resolution reconstruction, but for a genuinely flexible encode–decode mapping across canvases.

The paper trains two tokenizer variants: VibeToken-SL, described as a small encoder with large decoder, and VibeToken-LL, described as a large encoder with large decoder [2604.24885]. The default downstream choice is VibeToken-LL.

## 4. VibeToken-Gen and autoregressive generation

VibeToken-Gen is the class-conditioned autoregressive generator trained over VibeToken latents [2604.24885]. The AR training objective is the standard next-token likelihood, which the paper describes as maximizing log-likelihood or minimizing next-token cross-entropy. In structured form,
\[
\mathcal{L}_{\text{AR}} = -\sum_{t=1}^{T+1}\log p_\theta(x_t\mid x_{<t}, c),
\]
where \(c\) denotes conditioning. The paper’s conditioning includes both class and target resolution:
\[
c = \bigl[\,\operatorname{emb}(y) + \operatorname{MLP}((H,W)/\beta)\,\bigr],
\]
with \(\beta = 1536\) [2604.24885]. This resolution-conditioning is used to prevent stretching artifacts and to make the generator compatible with arbitrary output canvases.

Architecturally, the paper keeps the LlamaGen-style stack largely unchanged, while replacing the tokenizer with VibeToken-MVQ-LL, adding Query-Key LayerNorm for stability, and using a 4-layer residual transformer prediction head after UniTok to predict the 8 MVQ sub-codes per token [2604.24885]. Reported model scales include GPT-B at about 90M parameters and GPT-XXL at about 1.4B parameters, with another table reporting 1.5B for GPT-XXL [2604.24885]. The paper also notes that AR training is performed in fp32 because bf16 was unstable [2604.24885].

VibeToken-Gen is trained after the tokenizer, not jointly with it [2604.24885]. The training data are again ImageNet-1k under the same mixed-resolution, mixed-aspect-ratio regime, using latent lengths sampled from \(\{64,128,256\}\), with class dropout probability 0.1 for classifier-free guidance [2604.24885]. Training duration is reported as 300 epochs for GPT-B and 150 epochs for GPT-XXL [2604.24885].

At inference, generation proceeds by choosing a class \(y\), choosing a target resolution \((H,W)\), sampling a short latent sequence autoregressively, and decoding it with the VibeToken decoder [2604.24885]. Quantitative results generally use temperature \(1.0\), top-\(k=0\), and top-\(p=1.0\), while qualitative settings include CFG \(=4.0\), temperature \(0.9\), top-\(k=500\), and top-\(p=1.0\) [2604.24885]. A notable empirical result is that 64 tokens are often sufficient for generation, and in one GPT-B ablation at \(256^2\), 64 tokens outperform 128 and 256 on gFID [2604.24885].

## 5. Efficiency, reconstruction, and generation results

The paper’s principal systems claim is that VibeToken shifts AR compute from being resolution-dependent to being token-budget-dependent [2604.24885]. For conventional 2D tokenizers, token count scales as
\[
T=(H/f)\cdot(W/f),
\]
so attention cost is approximately \(\mathcal{O}(T^2)\). For VibeToken-Gen with fixed latent length \(L\), the AR cost is instead governed by \(\mathcal{O}(L^2)\), which is effectively constant across resolutions when \(L\) is held fixed [2604.24885].

The tokenizer-side scaling argument is analogous. The paper reports that a 2D tokenizer baseline, IBQ, grows from about 0.64T to 10.30T FLOPs from \(256^2\) to \(1024^2\), whereas VibeToken remains around 1.04T FLOPs maximum, effectively constant [2604.24885]. On the generator side, the abstract contrasts fixed-resolution AR and VibeToken-Gen by stating that LlamaGen reaches about 11T FLOPs at \(1024\times1024\), while VibeToken-Gen maintains 179G FLOPs, or 63.4× greater efficiency, independent of resolution [2604.24885].

The main reported high-resolution comparison is summarized below.

| Model / comparison | Tokens at \(1024^2\) | Reported result |
|---|---:|---|
| VibeToken-Gen | 64 | 179G FLOPs |
| Diffusion alternative (NiT) | 1,024 | 5.87 gFID |
| LlamaGen | not stated here as token count | 11T FLOPs |

The abstract reports that VibeToken-Gen synthesizes \(1024\times1024\) images using only 64 tokens and achieves 3.94 gFID, whereas the diffusion-based alternative requires 1,024 tokens and attains 5.87 gFID [2604.24885]. A later high-resolution results table, however, reports 3.54 gFID for VibeToken-Gen-XXL at \(1024\times1024\) against 5.87 gFID for NiT-XL [2604.24885]. This discrepancy is present in the paper summary itself.

Latency results show the same scaling pattern. For tokenizer encode/decode, LlamaGen-Tok rises from 0.005 s at \(256^2\) to 0.082 s at \(1024^2\), while VibeToken-LL remains at 0.017 s at both resolutions [2604.24885]. For end-to-end generation, LlamaGen-XXL rises from 0.20 s at \(256^2\) to 32.79 s at \(1024^2\), whereas VibeToken-Gen-XXL remains at 0.46 s at both \(256^2\) and \(1024^2\) [2604.24885].

On tokenizer reconstruction, the reported VibeToken-LL rFID values are 0.40 at \(256^2\), 0.51 at \(512^2\), 2.40 at \(1024^2\), and 3.60 on arbitrary-resolution stress tests [2604.24885]. On ImageNet, VibeToken-LL at \(256^2\) reports PSNR 25.04, SSIM 0.8194, LPIPS 0.1048, and rFID 0.40; at \(512^2\), PSNR 23.37, SSIM 0.7649, LPIPS 0.1867, and rFID 0.51 [2604.24885]. The paper interprets this as strong generalization beyond the tokenizer’s \(512^2\) training ceiling.

The paper also reports native super-resolution. On FFHQ \(1024^2\), VibeToken-LL gives, for \(2\times\) super-resolution, PSNR 24.98, SSIM 0.838, and LPIPS 0.261; for \(4\times\), PSNR 24.11, SSIM 0.805, and LPIPS 0.310 [2604.24885]. The paper notes that diffusion upscalers may have higher PSNR, while VibeToken can be stronger on SSIM or LPIPS in some cases [2604.24885].

For arbitrary-resolution generation, VibeToken-Gen-XXL reports an average gFID of 5.63 on lower-resolution arbitrary-canvas settings, compared with 4.22 for NiT-XL, and an average gFID of 5.53 on higher resolutions from 512 to 1024, compared with 6.05 for NiT-XL [2604.24885]. This suggests that the model is less dominant at low resolutions than at high resolutions.

## 6. Interpretation, trade-offs, and limitations

The paper’s central technical claim is not merely that VibeToken is a compact tokenizer, but that it makes a generalist AR image model practical across multiple resolutions and aspect ratios [2604.24885]. This is distinct from fixed-resolution specialist systems such as LlamaGen, which may achieve lower FID at exactly \(256^2\) but incur rapidly increasing cost when extended to \(1024^2\) [2604.24885]. A plausible implication is that VibeToken shifts the main burden of scaling from the AR backbone to the tokenizer interface, making token budget rather than canvas size the dominant control variable.

The trade-off is explicit in the ablations. For reconstruction, 128 or 256 tokens improve fidelity, but for generation the paper finds that 64 tokens can be best [2604.24885]. This suggests that later tokens encode detail valuable for reconstruction yet not necessarily beneficial for the distribution learned by the generator. The paper also notes that VibeToken-Gen, as a generalist trained under a mixed-resolution regime and limited compute budget, can trail specialist fixed-resolution models at exactly \(256^2\) or \(512^2\) [2604.24885].

Several limitations are stated directly. Experiments are limited to ImageNet-1k and class-conditional generation [2604.24885]. The paper does not present text-to-image generation or open-vocabulary conditioning [2604.24885]. It suggests that longer training, stronger augmentation, larger pretraining corpora, larger models, randomized orderings, scale-wise AR training, and alternative quantization schemes are natural future directions [2604.24885]. The paper also notes that without target-resolution conditioning the decoder can exhibit stretching artifacts, and qualitative generations may show crops or truncations attributed to randomized cropping during AR training [2604.24885].

Within the broader literature supplied here, VibeToken belongs to a wider trend toward flexible token budgets and token-efficient multimodal modeling. TokenFLEX, for example, addresses variable visual token counts in VLMs through stochastic multi-budget training [2504.03154]. VibeToken differs in focusing on discrete 1D image tokenization and autoregressive image generation rather than vision-language understanding [2604.24885]. This suggests that “flexible token count” and “resolution agnosticism” are related but not identical research programs.

Taken as a whole, VibeToken is best understood as a tokenizer-centered reconfiguration of autoregressive image generation [2604.24885]. Its significance lies in demonstrating that a short, dynamic, resolution-agnostic 1D latent sequence can support arbitrary-resolution generation, native super-resolution, and substantially flatter compute scaling than fixed-grid AR baselines.

Source: https://www.emergentmind.com/topics/vibetoken