RecTok: Visual Tokenizer for Diffusion Models

• RecTok is a visual tokenizer that shifts semantic enrichment from the clean latent to the entire flow trajectory in diffusion models.
• It introduces Flow Semantic Distillation (FSD) and Reconstruction–Alignment Distillation (RAD) to balance reconstruction fidelity, semantic richness, and generative stability.
• Empirical results on ImageNet demonstrate state-of-the-art generation (gFID) and reconstruction performance while effectively utilizing high-dimensional latent spaces.

Searching arXiv for the specified paper and adjacent tokenizer work to ground the article in current literature. RecTok is a visual tokenizer for diffusion models that relocates semantic enrichment from the clean latent $x_0$ to the forward flow trajectories $\{x_t \mid t \in [0,1]\}$ used to train diffusion transformers. Introduced in "RecTok: Reconstruction Distillation along Rectified Flow" (Shi et al., 15 Dec 2025), it is motivated by a longstanding trade-off in latent generative modeling: higher latent dimensionality improves reconstruction fidelity and can preserve more discriminative structure, but it has historically degraded generation quality and training stability. RecTok addresses this bottleneck through Flow Semantic Distillation (FSD) and Reconstruction–Alignment Distillation (RAD), and reports state-of-the-art gFID-50K under both with and without guidance on class-conditional ImageNet 256$\times$256 while maintaining a semantically rich latent space structure.

1. Problem formulation and motivation

Visual tokenizers in diffusion models are encoder–decoder networks that map images from pixel space into a compact latent space, in which downstream diffusion transformers are trained (Shi et al., 15 Dec 2025). In this setting, latent dimensionality directly governs the balance among reconstruction fidelity, semantic richness, and generative stability or quality. Higher dimensions preserve more pixel and detail information and can retain more discriminative features, but they also introduce optimization challenges and redundant latent channels that can confuse the training of velocity fields and diffusion transformers.

Historically, tokenizers have therefore been kept low-dimensional, such as 32 channels, to simplify and stabilize diffusion training. This improves tractability but constrains both reconstruction and semantics. Prior approaches in the paper’s framing fall into three broad groups. VAE and VQ-VAE tokenizers improve reconstruction with KL, perceptual, and adversarial losses, but still restrict latent channels and struggle with semantic expressiveness. DiT-side adaptations such as RAE widen the diffusion transformer to handle higher-dimensional latents, but do not systemically resolve the dimensionality–semantics–generation trade-off. Methods that leverage vision foundation models, including CLIP and DINO variants, enrich latent semantics and accelerate convergence in low dimensions, but typically inject semantics only into the clean latent $x_0$ and observe degraded generation when latent dimensionality is scaled.

RecTok is built around a more specific diagnosis: diffusion transformers are trained on intermediate forward-flow states $x_t$, not solely on $x_0$. The paper’s central claim is that improving only the clean latent leaves the actual diffusion training space insufficiently semantic. This shifts the design target from latent-space semantics in isolation to flow-trajectory semantics throughout rectified flow training.

2. Rectified-flow formulation and the two distillation mechanisms

RecTok uses a rectified-flow or flow-matching setup in which the forward trajectory is a linear interpolation between clean latent and endpoint noise (Shi et al., 15 Dec 2025):

$$x_t = (1 - t)\, x_0 + t\, x_1, \qquad v_t = \frac{d x_t}{d t} = x_1 - x_0.$$

The velocity predictor is trained with

$$\mathcal{L}_{\text{RF}} = \mathbb{E}_{t,\,x_0,\,x_1} \left[ \left\| v_\theta(x_t,t) - (x_1 - x_0) \right\|_2^2 \right].$$

Generation then uses an Euler ODE solver with $v_\theta(x,t)$ to transform noise toward clean data along the rectified flow. The paper emphasizes that this differs from standard diffusion by learning a deterministic velocity field for linear interpolant trajectories.

The first RecTok component, Flow Semantic Distillation, distills VFM semantic information into every forward-flow state $x_t$ rather than only the un-noised latent. A lightweight semantic decoder $\{x_t \mid t \in [0,1]\}$0 predicts VFM-aligned features from $\{x_t \mid t \in [0,1]\}$1, with semantic supervision defined by

$\{x_t \mid t \in [0,1]\}$2

Here $\{x_t \mid t \in [0,1]\}$3 is a fixed VFM feature extractor; the experiments use DINOv2 and DINOv3. FSD is applied along the trajectory $\{x_t \mid t \in [0,1]\}$4, with $\{x_t \mid t \in [0,1]\}$5 Gaussian noise, so that the entire training space becomes semantically discriminative. Timestep sampling is also modified by a dimension-dependent shift,

$\{x_t \mid t \in [0,1]\}$6

where $\{x_t \mid t \in [0,1]\}$7 and $\{x_t \mid t \in [0,1]\}$8 are latent spatial resolution and channel dimension. The paper states that this shift counteracts redundancy at high $\{x_t \mid t \in [0,1]\}$9.

The second component, Reconstruction–Alignment Distillation, complements FSD by masked feature reconstruction and alignment to VFM features. Random masks are applied to the input image with mask ratio in $\times$0, where a negative value means no mask. Visible regions are encoded into $\times$1, propagated along rectified flow to

$\times$2

and then fed into the semantic decoder to reconstruct VFM features. The reconstructed features are aligned to full-image VFM features using the same semantic loss on both masked and unmasked regions. The paper characterizes RAD as pairing semantic alignment with a reconstruction target, thereby encouraging robust, semantically aligned latents under masking while preserving reconstruction quality.

Together, FSD and RAD define RecTok’s core idea: the forward flow trajectories themselves become the semantic training substrate of the downstream diffusion transformer.

3. Architecture, training regime, and inference path

The tokenizer uses a ViT-based encoder–decoder with ViT-B backbones, ROPE, SwiGLU, and RMSNorm for both encoder and decoder (Shi et al., 15 Dec 2025). Latent dimensionality is explored over $\times$3. The encoder–decoder is trained as a VAE with KL regularization, which the paper describes as yielding a smooth latent manifold for generative training. The semantic decoder $\times$4 is a lightweight transformer of approximately 1.5M parameters; the paper reports that larger semantic decoders degrade performance, because they reduce the pressure on the encoder to carry semantics.

Tokenizer pretraining is performed on ImageNet-1K at 256$\times$5256 resolution for 200 epochs with batch size 1024, Adam with $\times$6 and $\times$7, learning rate $\times$8, linear warmup for 50 epochs, cosine decay for the remaining 150 epochs, and EMA 0.999. The loss weights are set to $\times$9, $x_0$0, $x_0$1, $x_0$2, and $x_0$3. FSD and RAD are active during tokenizer pretraining. A subsequent decoder finetuning stage freezes the encoder and finetunes only the pixel decoder, while disabling FSD and RAD and removing KL and semantic loss to maximize reconstruction quality.

The downstream generator is DiT$x_0$4-XL, a RAE-inspired architecture trained with rectified flow and timestep shift. Main experiments train it for 600 epochs; ablations use 80 epochs. The learning rate is $x_0$5, linearly decayed to $x_0$6 after 40 epochs, with batch size 1024, gradient clipping 1.0, no weight decay, and EMA 0.995. Evaluations use EMA weights. Sampling uses Euler ODE integration, with 150 steps in the main results and 50 in ablations. Guidance is implemented through AutoGuidance, using a bad version DiT$x_0$7-S trained 30 epochs.

The paper reports 32 H100 GPUs, approximately 19 hours for RecTok tokenizer training, approximately 10 hours for 80 epochs of DiT$x_0$8 training, and approximately 3 days for 600 epochs. At inference, the semantic decoder and the VFMs are discarded. The deployed path is therefore: encode an image with $x_0$9 to $x_t$0, generate in latent space by integrating the rectified flow with the DiT, and reconstruct the image with $x_t$1.

4. Reported performance on ImageNet

On class-conditional ImageNet 256$x_t$2256, RecTok with DiT$x_t$3-XL reports, without guidance, gFID $x_t$4, IS $x_t$5, Precision $x_t$6, and Recall $x_t$7; with AutoGuidance, it reports gFID $x_t$8, IS $x_t$9, Precision $x_0$0, and Recall $x_0$1 after 600 epochs (Shi et al., 15 Dec 2025). The paper identifies these as state-of-the-art gFID results without guidance and best overall with AutoGuidance among latent diffusion baselines.

The tokenizer itself is reported to achieve rFID $x_0$2, PSNR $x_0$3, and gFID $x_0$4 with 176M parameters and 44.4 GFLOPs. In the tokenizer comparison summarized in the paper, this outperforms ViT-based tokenizers such as MAETok and DeTok and offers better generation than SD-VAE while keeping strong reconstruction. RAE is reported at gFID $x_0$5 but PSNR $x_0$6, and the paper presents RecTok as balancing reconstruction and generation more effectively.

The convergence analysis is also part of the claimed contribution. gFID is reported to converge in approximately 600 epochs, and the paper describes this as 7.75$x_0$7 faster than certain prior approaches in convergence comparisons. gFID-10K stabilizes near 600 epochs, whereas IS continues to rise. Strong performance is already achieved with approximately 60 sampling steps, although the final evaluations use 150 Euler steps and guidance scale 1.29.

Qualitative analyses reported in the paper align with the numerical claims. Reconstructions preserve structure, color, and fine details; generations demonstrate class-conditional diversity and high quality; cosine-similarity heatmaps, PCA, and t-SNE of $x_0$8 at different $x_0$9 indicate semantically localized latent features and strong semantic consistency along the flow.

5. Scaling behavior and ablation evidence

A central empirical claim of RecTok is that increasing latent dimensionality improves semantics, reconstruction, and generation simultaneously, rather than forcing the usual trade-off (Shi et al., 15 Dec 2025). The paper presents the following scaling results.

Dimension	Latent and reconstruction metrics	Generation metric
16	L.P. Acc. 24.1; rFID 0.74; PSNR 22.75	gFID 2.75
32	L.P. Acc. 38.8; rFID 0.71; PSNR 24.08	gFID 2.64
64	L.P. Acc. 47.2; rFID 0.66; PSNR 24.93	gFID 2.57
128	L.P. Acc. 55.4; rFID 0.65; PSNR 25.28	gFID 2.27

The paper explicitly describes this as the first demonstration that higher-dimensional latents can simultaneously improve reconstruction, semantics measured by linear probing, and generation in a diffusion framework when semantics are enriched along the flow.

The ablations are organized to isolate the roles of FSD, RAD, semantic weighting, timestep schedule, VFM choice, decoder capacity, and initialization. For FSD, the comparison is direct: without FSD using cosine similarity, L.P. Acc. is 44.35, rFID 0.69, gFID 3.35, and IS 157.3; without FSD using VF loss, L.P. is 37.52, rFID 0.72, gFID 3.91, and IS 142.1; with FSD using cosine similarity, L.P. rises to 55.40, rFID improves to 0.65, gFID to 2.27, and IS to 196.4. The paper therefore attributes substantial gains across metrics to distilling semantics into the full flow trajectory.

For the semantic loss weight, $$x_t = (1 - t)\, x_0 + t\, x_1, \qquad v_t = \frac{d x_t}{d t} = x_1 - x_0.$$0 is reported as best overall, with gFID 2.27 and IS 196.4. Lowering it to 0.5 degrades generation, while raising it to 2 worsens both reconstruction and gFID. For timestep schedules, Uniform gives the best reconstruction, with rFID 0.53 and PSNR 26.71, but the worst generation, with gFID 2.50; Shift gives the best generation, with gFID 2.27 and IS 196.4, and is adopted in the final system.

VFM choice is dimension-sensitive. The paper reports that DINOv2 is best for low-dimensional latents such as $$x_t = (1 - t)\, x_0 + t\, x_1, \qquad v_t = \frac{d x_t}{d t} = x_1 - x_0.$$1 and $$x_t = (1 - t)\, x_0 + t\, x_1, \qquad v_t = \frac{d x_t}{d t} = x_1 - x_0.$$2, whereas DINOv3 is best at higher dimensions, including $$x_t = (1 - t)\, x_0 + t\, x_1, \qquad v_t = \frac{d x_t}{d t} = x_1 - x_0.$$3, where it yields gFID 2.27 and IS 196.4. Attempts to use two VFMs simultaneously degrade generation. For RAD, alignment-only with an MLP decoder performs worse than a transformer decoder, reconstruction-only is inferior, and the joint reconstruction-plus-alignment objective with a transformer gives the best reported combination of rFID 0.65, gFID 2.27, and IS 196.4. The lightweight semantic decoder is also an ablation target: a larger 10M transformer degrades linear probing and generation relative to the approximately 1.5M design.

The cumulative ablation stack is especially revealing. A baseline system reports L.P. 7.1, rFID 0.22, gFID 12.07, and IS 57.5. Adding FSD reduces gFID to 2.52 and raises IS to 184.5. Adding RAD further improves to gFID 2.27 and IS 196.4. Adding decoder finetuning then improves reconstruction to rFID 0.48 and PSNR 26.16, while slightly improving generation to gFID 2.23 and IS 198.2. This sequencing supports the paper’s claim that semantic alignment during tokenizer pretraining and reconstruction-oriented finetuning afterward serve distinct roles.

6. Limitations, interpretation, and relation to adjacent tokenizer research

RecTok’s main interpretive contribution is to redefine where semantic supervision should enter a diffusion tokenizer. Rather than treating the latent $$x_t = (1 - t)\, x_0 + t\, x_1, \qquad v_t = \frac{d x_t}{d t} = x_1 - x_0.$$4 as the only semantically meaningful object, it treats the entire forward flow trajectory as the relevant training space for the diffusion transformer (Shi et al., 15 Dec 2025). This challenges the assumption that higher-dimensional latents necessarily worsen generation. A plausible implication is that at least part of the historical dimensional bottleneck arose from a mismatch between where semantics were injected and where the generator was actually trained.

The paper nevertheless identifies several limitations. The semantic discriminative capability of RecTok still lags behind top VFMs. KL regularization improves generative performance but weakens reconstruction relative to deterministic autoencoders. The method is also sensitive to VFM choice, and domain generalization beyond ImageNet remains an open question. Future directions named in the paper include dynamic VFM selection per dimension, advanced time-weighting along trajectories, and alternative endpoint distributions for $$x_t = (1 - t)\, x_0 + t\, x_1, \qquad v_t = \frac{d x_t}{d t} = x_1 - x_0.$$5.

Within the broader tokenizer literature, RecTok occupies a distinct position. It differs from WeTok, which is a discrete visual tokenizer centered on group-wise lookup-free quantization and generative decoding for high-fidelity reconstruction (Zhuang et al., 7 Aug 2025). It also differs from RepTok, which represents an image with a single continuous latent token adapted from a self-supervised ViT and uses flow matching to model that token efficiently (Gui et al., 16 Oct 2025). It is likewise separate from ReToK, a flexible image tokenizer for autoregressive image generation that uses Redundant Token Padding and Hierarchical Semantic Regularization in variable-length 1D token sequences (Fu et al., 4 Jan 2026). These comparisons suggest that contemporary tokenizer research is diverging along at least three axes: discrete versus continuous latents, fixed-grid versus variable-length or single-token interfaces, and reconstruction-centric versus generation-centric semantic shaping.

In that landscape, RecTok is most precisely understood as a high-dimensional VAE tokenizer for rectified-flow diffusion whose distinctive move is to distill semantics along the trajectory actually consumed by the downstream generator. Its reported results make the forward flow—not merely the clean latent—the central object of tokenizer design.