SFTok: Self-Forcing Tokenizer for Images

• The paper introduces SFTok, a discrete image tokenizer that employs self-forcing reconstruction to align training and inference, reducing error cascades.
• It uses a ViT-based encoder and autoregressive transformer decoder with multi-step prediction to achieve high reconstruction fidelity at high compression rates.
• Empirical benchmarks and ablations demonstrate SFTok’s superior performance over traditional VQ-style and masked autoregressive tokenizers on high-resolution images.

SFTok refers to a discrete image tokenizer architecture designed to bridge the performance gap between discrete and continuous tokenizers for high-resolution image generation. Specifically, SFTok (“Self-Forcing Tokenizer”) introduces a multi-step, self-forcing reconstruction strategy and a debias-and-fitting training protocol, enhancing both reconstruction fidelity and class-conditional generative quality at high compression rates. The contributions address key limitations of previous discrete tokenization schemes—primarily, their reliance on one-step VQ-style codebook lookups and their vulnerability to training-inference discrepancies in iterative, masked token prediction settings. SFTok enables autoregressive, transformer-based models to operate more efficiently and effectively in the discrete latent space, making the approach suitable for contemporary multimodal and vision-LLMs (Rao et al., 18 Dec 2025).

1. Motivation and Context

Discrete image tokenizers compress high-dimensional images into compact sequences of discrete codes (tokens), enabling transformer-based generative models to process long-range dependencies efficiently. Continuous tokenizers, such as diffusion-based VAEs, achieve high-quality reconstructions via multi-step denoising, but discrete tokenizers—typically based on vector quantization (VQ) with single-step lookup—exhibit inferior fidelity and downstream performance. Masked autoregressive models (e.g., MaskGIT) improve discrete token prediction via iterative unmasking, but suffer from training-inference inconsistency due to the substitution of ground-truth codes during training versus model predictions at inference. SFTok addresses these deficiencies by incorporating multi-step, self-forcing guided visual reconstruction and debias-and-fitting curriculum learning, yielding competitive or superior performance even at high compression ratios (e.g., 64 tokens per image) (Rao et al., 18 Dec 2025).

2. Architecture and Algorithms

SFTok employs a ViT-based encoder to partition an input image $x\in\mathbb{R}^{H\times W\times 3}$ into $L_1$ non-overlapping patches. These are embedded, along with $K$ learnable query tokens, into latent codes $\{e_i\}_{i=1}^K$. Each feature vector is quantized via a codebook $\mathcal{C}=\{c_j\}_{j=1}^N$: $z_q = q(e) = \arg\min_{c\in\mathcal{C}} \|e-c\|_2.$ The decoder $f_d$ then ingests the quantized codes $z_q$, along with masked code-position tokens, and autoregressively predicts the distribution over all latent positions.

Unlike traditional, single-step decoding, SFTok leverages a multi-step iterative mechanism: at each step $i$, a subset of $N_i$ masked tokens $\mathcal{M}_i$ is predicted conditioned on $z_q$ and de-masked context from prior steps, $$p_\theta(m_{\mathcal{M}_i} | z_q, \hat{m}_{\overline{\mathcal{M}_i}})$$. The multi-step conditional factorization provably reduces the optimal cross-entropy relative to the single-step regime: $$L_{\min}^m = \sum_{i=1}^{L_2} H(m_i | z_q, m_{\backslash i}^{\mathrm{pred}}) = L_{\min}^s - \sum_i I(m_i; m_{\backslash i}^{\mathrm{pred}} | z_q).$$ [(Rao et al., 18 Dec 2025), Sec. 2.2, 4]

3. Self-Forcing Guided Visual Reconstruction (SFVR)

SFTok’s SFVR strategy directly tackles the training-inference inconsistency of prior multi-step masked modeling. Standard MaskGIT-style training replaces masked positions with ground-truth codes $m^g$ during each step, whereas during inference, only model predictions $\hat m$ are available, resulting in error cascades and performance degradation. SFVR remedies this by first performing a forward pass to obtain step-1 predictions $\hat m^{(1)}$ (with no gradient), then replacing all masked positions in subsequent steps with these predictions rather than the ground-truth codes. This procedure tightly aligns the conditional statistics of training and inference: $$\mathcal{L}_\mathrm{SFVR} = -\sum_{i=1}^{L_2} \log p_\theta(m^g_i | z_q, \hat m^{(1)}_{\backslash i}).$$ Empirical evidence demonstrates that the Kullback–Leibler divergence between step-1 and final predictions is minimal, justifying the use of $\hat m^{(1)}$ throughout training. This setting (replacement ratio $r=1.0$) yields optimal rFID performance [(Rao et al., 18 Dec 2025), Sec. 2.3, 7.2].

4. Debias-and-Fitting Training Protocol

SFTok training proceeds in three explicit stages:

1. Warm-Up: Single-step cross-entropy optimization with no SFVR, training the decoder to perform all-token prediction based only on $z_q$.
2. Distribution Alignment: Multi-step SFVR training using a frozen MaskGIT decoder as the pixel reconstruction head. Objective: minimize cross-entropy under fully self-forced token context.
3. Fine-Tuning: Joint optimization of SFTok decoder and pixel head (MaskGIT), using a VQGAN-style loss comprising cross-entropy, pixel-wise $\ell_2$, perceptual (LPIPS), and adversarial (GAN) components:

$$\mathcal{L}_3 = \lambda_\mathrm{CE} \mathcal{L}_\mathrm{CE} + \lambda_{\ell_2} \|x-\hat x\|_2^2 + \lambda_\mathrm{LPIPS} \mathcal{L}_\mathrm{LPIPS} + \lambda_\mathrm{GAN} \mathcal{L}_\mathrm{GAN}$$

[(Rao et al., 18 Dec 2025), Sec. 2.4]

5. Empirical Results and Benchmarks

SFTok demonstrates superior performance under stringent compression compared to previous discrete and continuous methods. On ImageNet 256×256 validation with only 64 tokens:

Method	Tokens	Codebook	rFID
SFTok-B	64	8K	1.44
SFTok-L	64	8K	1.21
TiTok-B (1D)	64	4K	1.70
One-D-Piece-B	64	4K	2.39
MaskGIT (discrete)	256	1K	2.28
ViT-VQGAN (cont.)	1024	8K	1.28

For class-to-image generation using the 8-step MaskGIT sampler:

Method	Type	gFID
SFTok-B+MaskGIT	transformer	2.32
SFTok-L+MaskGIT	transformer	2.29
TiTok-B+MaskGIT	transformer	2.48
MaskGIT	transformer	6.18
DC-AE	diffusion	1.88

Ablations show >2× rFID improvement of SFVR over vanilla masking (e.g., SFVR yields 4.33 vs. 6.47), with further reduction from multi-step inference (most gains saturating after 8 steps). Full self-forced replacement (ratio $r=1.0$) outperforms partial ($r<1$) or vanilla masking [(Rao et al., 18 Dec 2025), Sec. 6, 7].

6. Theoretical and Practical Insights

Multistep, self-forcing reconstruction provably achieves lower minimal cross-entropy than single-step approaches, as conditional dependence between predicted tokens is exploited. Training with SFVR conditions the network on its own prediction distribution, aligning training and inference statistics and mitigating error accumulation across iterative steps. Empirical results confirm that SFVR reduces divergence between training and test-time model behavior.

SFTok scales efficiently with standard hardware (e.g., 8×RTX4090), requires no additional inference-time overhead beyond its autoregressive transformer decoder, and is compatible with large-scale multimodal settings [(Rao et al., 18 Dec 2025), Sec. 3, 4].

7. Limitations and Future Directions

Current SFTok results are restricted to $256\times 256$ images and 64-token compression; extension to higher resolutions and finer codebooks remains an open direction. Potential applications include large-scale multimodal pretraining and video tokenization. Further research may incorporate adaptive mask scheduling, cross-modal token alignment, and architectural modifications tailored to specific data modalities.

SFTok establishes that discrete tokenizers, appropriately regularized and trained with self-forcing and multi-step methods, can approach or match the fidelity of continuous tokenization in image generation, supporting powerful, efficient, and scalable multimodal generative systems (Rao et al., 18 Dec 2025).