Level-of-Token Diffusion

Abstract: Image and video diffusion models allocate equal computation to every region, even when the intended scene calls for varying levels of detail. The spatial distribution of detail can often be anticipated before generation, indicating where computation can be reduced. We introduce Level-of-Token (LoT) Diffusion, a framework that turns this knowledge into an explicit multiresolution token layout (Level-of-Token layout) for adaptive and efficient generation. Tokens represent rectangular patches of varying sizes and shapes, allocating finer tokens where detail is needed and coarser tokens elsewhere. We adapt pretrained diffusion transformers to LoT layouts through a patch-wise asymmetric flow parametrization and embeddings for multiresolution tokens, preserving full-resolution flow prediction at every denoising step while processing only a reduced token sequence. LoT Diffusion enables layout-adaptive generation while preserving pretrained generative priors. We demonstrate LoT with layouts derived from semantic masks, bounding boxes, texture variance, and depth-of-field cues, as well as agentic plans. Across image and video generation, LoT offers favorable quality-efficiency tradeoffs, with significant speedups determined by the layout's token budget. Our project website is at https://georgenakayama.github.io/lotdiffusion/.

• The paper introduces a Level-of-Token (LoT) Diffusion framework that adaptively allocates computational resources to different regions of an image or video based on spatial priors, allowing for heterogeneous token extents that result in reduced sequence length while retaining full-resolution latent fields at every denoising step.
• The researchers employ patch-wise asymmetric flow parameterization, which produces dense velocity predictions for each latent site in a patch, allowing for high resolution predictions from compressed tokens achieving an HPSv3 of 10.3477, and FID of 13.800.
• Video generation experiments using the Wan2.1 video model demonstrate that LoT-fine tuning can maintain a higher resolution in detail-rich areas while reducing overall token count by up to 3.53 times, resulting in notable improvements in aesthetic and imaging metrics, and preserving high temporal consistency.

Problem formulation and contribution

“Level-of-Token Diffusion” addresses an inefficiency in diffusion transformers (DiTs): standard image and video generators apply the same spatial token resolution to every region, despite substantial variation in the amount of detail required. Faces, typography, object boundaries, and high-frequency textures generally require fine spatial representation, whereas skies, defocused backgrounds, smooth surfaces, and other low-frequency regions can often be synthesized with coarser representations. The paper’s central claim is that spatial priors available before generation can determine not only what appears in a region, but also how much computation should be allocated to it (2610.05816).

The proposed Level-of-Token (LoT) Diffusion framework represents an image or video with a spatial partition of rectangular tokens having heterogeneous extents. Fine tokens are assigned to regions judged important or detail-rich, while coarse tokens cover regions for which lower spatial resolution is acceptable. Unlike token merging or pruning methods that adaptively reduce an already-computed representation, LoT defines the computational allocation before and during denoising. Unlike conventional layout-conditioning methods, which use masks or bounding boxes only to control spatial content, LoT uses the same information to redistribute computation.

The framework is designed to preserve the generative prior of a pretrained DiT. It reduces the transformer sequence length from the dense token count to the number of variable-size regions while still producing a full-resolution latent flow field at every denoising step. The method is evaluated on FLUX.2 for image generation and Wan2.1 for video generation, with layouts derived from semantic masks, bounding boxes, texture variance, depth-of-field cues, and manually or agentically specified detail maps.

Level-of-token representation

The method begins with a dense latent grid corresponding to the pretrained model’s native patchification. A LoT layout partitions this grid into disjoint, axis-aligned rectangular regions. Each region is represented by one token, but its spatial support can vary in height and width. If the dense grid contains $HW$ tokens and the layout contains $L$ regions, the nominal image token compression is $HW/L$; for videos, the corresponding quantity includes the temporal dimension.

This representation supports more than a binary fine/coarse distinction. In the image model, token extents include combinations of $1$, $2$, $4$, and $8$ along each spatial axis. The video model uses temporal extent one and spatial extents up to $4 \times 4$. Consequently, the layouts can be anisotropic as well as multiresolution: a region may be coarsened more strongly along one spatial direction than another. This is important for representing elongated or directionally homogeneous structures without imposing a square-grid restriction.

The paper derives layouts from several sources. Semantic masks and bounding boxes assign finer levels to selected objects and coarser levels to the background. Texture-variance layouts adapt the variable-rate-shading principle to estimated luminance variation. Depth maps are converted into blur-radius fields so that regions near a focal plane receive finer tokens while defocused regions are coarsened. A custom detail map allows users or agents to specify spatial importance directly. These mechanisms separate layout construction from the generative model: the model receives the resulting token partition and text condition, rather than necessarily receiving the source RGB image or detail map as an additional appearance condition.

This separation also enables explicit budget control. Increasing the requested token budget refines selected regions while maintaining coarse representations elsewhere. The resulting budget is therefore spatially interpretable, unlike a uniform reduction in output resolution.

Patch-wise asymmetric flow parameterization

The main technical difficulty is that a pretrained DiT expects a fixed-dimensional token embedding and normally predicts a flow vector for each input token. Replacing a dense patch with a single token would appear to make full-resolution prediction impossible. LoT resolves this through a patch-wise adaptation of asymmetric flow matching.

For each supported token extent $e$, the authors fit a semi-orthonormal basis $A_e$ using orthogonal Procrustes regression between dense latent patches and extent-matched multiscale VAE tokens. A dense patch is projected into an extent-specific lower-dimensional subspace, producing one compressed token with the same channel dimension expected by the pretrained transformer. The transformer therefore processes a sequence whose length depends on the layout, while its per-token feature dimension remains compatible with the original architecture.

The model predicts an asymmetric velocity in the extent-specific subspace. A projector $L$0 supplies the component represented by the compressed token, while the orthogonal complement is analytically reconstructed using the current noisy state and the asymmetric-flow correction. This yields a dense velocity for every latent site in the patch. The reconstruction is not merely a decoder applied after denoising; it is part of the flow parameterization and training objective. Thus, the model performs reduced-sequence computation while maintaining a full-resolution flow state.

This design is consequential because direct high-resolution prediction from compressed tokens performs substantially worse in the ablations. At approximately $L$1 token compression, the direct high-resolution variant obtains HPSv3 of 7.0595 and FID of 26.487, compared with 10.3477 and 13.800 for the complete model. At approximately $L$2 compression, the corresponding direct-prediction variant collapses to HPSv3 of 1.3967 and FID of 68.508, whereas the proposed parameterization retains HPSv3 of 9.5522 and FID of 13.345. The result indicates that the analytic asymmetric-flow recovery is not an implementation detail: it is central to preserving a usable dense generative field under aggressive spatial token reduction.

The authors additionally fit an extent-dependent scale between projected and reference tokens. Rather than using extent-specific internal timesteps, which would introduce heterogeneous timestep conditioning within one sample, they scale the clean data before adding noise and invert the scale before decoding. This preserves a shared diffusion timestep across all tokens and avoids a mismatch with the pretrained DiT’s conditioning structure.

LoT DiT architecture

The transformer modifications are deliberately limited. Extent-specific input projections map compressed tokens into the pretrained hidden space, and extent-specific output heads predict asymmetric velocities for all dense positions represented by a token. The heads are initialized from the pretrained output projection through the fitted extent bases, allowing the adaptation to begin near the original model’s parameterization.

Because conventional RoPE encodes token positions but not their spatial support, LoT introduces two forms of layout conditioning. First, a zero-initialized MLP embeds each token’s height, width, area, and aspect ratio using logarithmic features. This patch-shape embedding is added to the token representation. Second, the existing axial RoPE is evaluated at the geometric center of each token on the finest uniform grid. The center encodes location, while the learned shape embedding encodes extent. This retains query-independent dense attention and recovers the pretrained positional encoding for unit-sized tokens.

The ablation results establish the importance of both aspects of the design. Removing size embeddings reduces HPSv3 from 10.3477 to 10.0284, increases FID from 13.800 to 14.584, and lowers MUSIQ from 70.410 to 69.433 at the principal operating point. Restricting the layout to a binary grid of only $L$3 and $L$4 tokens yields HPSv3 of 8.1955 and FID of 20.603. The complete multiscale, shape-conditioned model therefore benefits from representing token extent continuously through multiple supported shapes rather than treating spatial reduction as a binary decision.

Image-generation results

The image experiments fine-tune FLUX.2-klein-base-4B at $L$5 using mixed-source layouts. Pretrained text encoders and VAEs remain frozen; LoRA adapters modify selected transformer projections, while extent-dependent heads and shape embeddings are fully trained. The principal comparison includes ToMe-SD, DDiT, and Foveated Diffusion at matched token budgets.

LoT-Flux2-4B achieves speedups from $L$6 to $L$7 across compression levels of approximately $L$8, $L$9, and $HW/L$0. At the most aggressive reported setting, the model achieves $HW/L$1 speedup with HPSv3 of 9.5522, FID of 13.345, pFID of 17.822, TOPIQ of 0.5873, and MUSIQ of 68.760. The same setting gives ToMe-SD a speedup of only $HW/L$2, HPSv3 of 7.8433, FID of 16.124, and MUSIQ of 63.105; DDiT gives $HW/L$3 speedup, HPSv3 of 7.4075, FID of 28.156, and MUSIQ of 63.435. Foveated Diffusion reaches $HW/L$4 speedup, HPSv3 of 7.8433, FID of 16.124, and MUSIQ of 63.105 under the corresponding comparison.

The results support the paper’s claim that nonuniform spatial allocation is more effective than uniform or post hoc reduction at a matched token budget. In an additional comparison against a token-matched dense FLUX.2 baseline, LoT generates directly at $HW/L$5 with spatially varying tokens, whereas the baseline generates at a uniformly reduced resolution and upsamples. LoT obtains lower FID and pFID and higher MUSIQ and TOPIQ over the evaluated compression range. The implication is that preserving high-resolution computation in selected regions is more valuable than distributing a lower resolution uniformly, particularly when local detail determines perceptual or semantic fidelity.

The qualitative experiments show that the model responds to the specified allocation rather than merely accepting it as an inert computational mask. Fine-token regions tend to retain intricate textures, object boundaries, and salient subject details; coarse-token regions exhibit simplified structure and smoother appearance. This behavior is also measurable as layout controllability. On 2,096 COCO prompts with paired original and randomly relocated fine-token regions, target-region subject coverage increases from 45.38% for full-resolution FLUX.2-4B to 61.17% for LoT-Flux2-4B. For the 9B models, coverage increases from 44.94% to 66.83%. The improvement is 15.79 percentage points for the 4B model and 21.89 percentage points for the 9B model. Subject detection remains similar: 98.12% for LoT-4B versus 98.57% for dense 4B, and 99.07% for LoT-9B versus 98.95% for dense 9B. Thus, the finer region appears to influence subject placement without materially reducing the probability that the subject is generated.

Video-generation results

The video experiments adapt Wan2.1-T2V-14B to 81-frame, 720P clips. The layouts include semantic masks, bounding boxes, texture variance, and depth-of-field cues, with temporal extent fixed to one and spatial token extents up to $HW/L$6. Evaluation uses six VBench dimensions over 200 held-out prompts.

LoT-Wan2.1 provides speedups from $HW/L$7 to $HW/L$8 at compression ratios from approximately $HW/L$9 to $1$0 in the principal comparisons. At the highest budget reduction, LoT obtains $1$1 speedup, aesthetic quality of 0.5104, imaging quality of 0.5650, dynamic degree of 0.9500, background consistency of 0.9232, subject consistency of 0.8687, and motion smoothness of 0.9829. Under the same nominal budget, ToMe-SD reaches only $1$2 speedup and aesthetic quality of 0.5080, while Foveated Diffusion reaches $1$3 speedup and aesthetic quality of 0.5080 in the reported comparison. LoT achieves the highest aesthetic and imaging scores among the compressed methods at each evaluated budget and remains competitive on temporal consistency and motion metrics.

These results suggest that spatially adaptive tokenization remains useful when the generated object and background vary over time. However, the evaluation protocol relies on layouts derived from reference videos generated by MiniMax-H3 rather than from the original held-out clips, because the source clips were described as predominantly static. The reported video results therefore measure performance under a specific reference-layout construction pipeline and do not fully isolate the effect of independently predicted layouts.

Relationship to existing efficiency methods

LoT operates along a spatial allocation axis that is complementary to timestep reduction, feature caching, sparse attention, token merging, pruning, and progressive upsampling. Its distinction from token-reduction methods is that the token layout is not solely inferred from intermediate redundancy. It can be prescribed from scene semantics, user intent, depth, texture, or an external plan before synthesis. Its distinction from layout-control methods is that the layout affects both content placement and computation.

This distinction produces a potentially important interface between planning and generation. An agentic plan can designate an object, region, or trajectory as detail-critical, and the resulting allocation can be edited independently of the textual prompt. The paper’s experiments demonstrate this principle but do not establish that automatically generated layouts are optimal. In the quantitative baseline comparisons, image layouts are derived from held-out reference images, and video layouts are derived from generated reference videos. These protocols are useful for controlled evaluation but provide stronger spatial-detail information than would generally be available in an unconstrained text-to-image or text-to-video deployment setting.

Limitations and open questions

The paper explicitly identifies a scene-dependent limit on efficiency. If fine structures or textures occupy most of the image, there is little spatial redundancy to exploit. Dense musical notation, close-up fabric, grass, and similar content may require fine tokenization nearly everywhere. Enforcing a low token budget in such cases degrades structural fidelity; the reported musical-score example shows more fragmented staff lines and distorted notation under $1$4 compression than under $1$5 compression. Consequently, LoT does not guarantee a fixed speedup independent of scene complexity.

The method also depends on the quality and semantics of the supplied layout. A layout that incorrectly marks a region as low-detail can induce irreversible information loss during generation, while an overly conservative layout reduces the efficiency benefit. The training and evaluation distributions contain layouts derived from masks, boxes, VRS cues, and depth, so robustness to systematically erroneous, ambiguous, or adversarial layouts remains open. Similarly, the reported speed measurements exclude layout construction. For automatically extracted masks, depth maps, texture statistics, or agentic plans, end-to-end latency may be higher than the reported generation-time speedups.

A further question concerns scaling the supported extent set. The extent-specific input and output heads, Procrustes bases, and calibration factors are trained for a finite collection of token shapes. Extending the method to arbitrary continuous extents, substantially larger temporal supports, or irregular nonrectangular regions would require either additional parameter sharing or a different representation. The current rectangular partition also limits how efficiently the method can represent thin contours and topologically complex regions.

Conclusion

Level-of-Token Diffusion introduces a pretrained-compatible mechanism for spatially adaptive diffusion computation. Its principal technical contribution is patch-wise asymmetric flow recovery, which permits a reduced heterogeneous token sequence to produce a full-resolution flow field. Shape-conditioned embeddings and center-aligned RoPE provide the transformer with the spatial support and location of each token without replacing its core attention mechanism.

Across image and video generation, the method reports speedups up to $1$6 and $1$7, respectively, while generally outperforming token-reduction baselines at matched budgets. The experiments further show that token layouts affect subject placement and detail distribution, making them both computational controls and generation controls. The principal qualification is that savings depend on spatially varying detail requirements and on the accuracy of the supplied layout; scenes requiring fine structure throughout remain poorly suited to aggressive compression.