Mixed-Resolution Tokens in Transformers

• Mixed-resolution tokens are discrete representations that allocate variable-sized patches based on content saliency to optimize computational resources.
• They enable hierarchical or coarse-to-fine modeling by dynamically adjusting token granularity, thus reducing redundancy in uniformly partitioned data.
• Practical implementations address challenges like phase alignment in positional encoding and dynamic token management to maintain information fidelity.

Mixed-resolution tokens are discrete representations used within Transformer-based architectures that encode data (images, video, or time series) at multiple spatial, spectral, or semantic resolutions. Unlike uniform fixed-size tokenization, mixed-resolution approaches dynamically allocate variable patch or feature sizes according to content saliency, signal structure, or computational budget. This results in more efficient, adaptive, and often more semantically meaningful modeling, especially in domains with highly inhomogeneous information density.

1. Core Principles of Mixed-Resolution Tokenization

Mixed-resolution tokenization generalizes the fixed-grid patch paradigm by allowing tokens to represent data segments of arbitrary size or frequency band, determined dynamically or via predefined hierarchies. Key motivations include:

• Reducing redundancy: Uniform patch grids waste computation in homogeneous areas and under-represent detailed regions.
• Adaptive resource allocation: More tokens are devoted to semantically salient or information-rich regions, enhancing expressivity for a fixed computational budget.
• Hierarchical or coarse-to-fine modeling: Tokenization may proceed in a top-down (e.g., quadtree (Ronen et al., 2023), DWT (Esteves et al., 2024)) or content-driven manner (e.g., saliency scoring, dynamic allocation (Hagerman et al., 27 Mar 2026)).
• Resolution flexibility: Some frameworks enable inference/generation at arbitrary resolutions without retraining (“resolution-agnostic” modeling (Patel et al., 27 Apr 2026), spectral or hierarchical codebooks (Chen et al., 9 Mar 2025)).

Implementation approaches vary from spatially adaptive grids to spectral or hierarchical codebooks, but all share the central property: the model’s tokenization reflects non-uniform, task-driven choices over input structure.

2. Mixed-Resolution Tokenization Algorithms and Formulations

Several paradigms for constructing mixed-resolution tokens have been introduced:

2.1 Quadtree and Patch-Based Partitioning

• Quadformer (Ronen et al., 2023) partitions an image using a quadtree, guided by a saliency scorer (pixel MSE, semantic features, or class activation maps). Patches in low-saliency areas remain coarse; high-saliency patches are recursively subdivided until a target token count is reached. All resulting patches are resized and embedded into a joint token space, preserving grid independence via 2D positional encodings.
• ARTA (Hagerman et al., 27 Mar 2026) employs a coarse-to-fine strategy in visual segmentation: initial 32×32 coarse tokens are iteratively subdivided (up to 4×4) near semantic boundaries, predicted via lightweight MLP allocators.

2.2 Spectral and Hierarchical Tokenization

• Spectral Image Tokenizer (SIT) (Esteves et al., 2024) utilizes multi-level discrete wavelet transforms, patchifying both the lowpass and high-frequency detail coefficients at progressively finer scales. Discrete tokens are obtained per band via learned vector quantizers, ordered for autoregressive transformers in a coarse-to-fine sequence. This enables partial decoding, multiscale upsampling, and resolution-invariant inference without retraining.
• SemHiTok (Chen et al., 9 Mar 2025) constructs a hierarchical codebook: first, patches are quantized into a high-level semantic code (e.g., CLIP-signature); then, for each semantic bin, a subcodebook encodes pixel-level features, yielding tokens as concatenations of (semantic, pixel) indices.

2.3 Dynamic, Resolution-Agnostic, or Mixed-Grain Approaches

• VibeToken (Patel et al., 27 Apr 2026) introduces a 1D tokenizer that linearly interpolates patch and position embeddings to support patchification at any desired granularity. The token sequence length is decoupled from input resolution, enabling constant-FLOPs autoregressive generation for arbitrary (H, W).
• FlexAttention (Li et al., 2024) encodes images at two scales, but allows only a small, dynamically-selected subset of high-resolution tokens (guided by prior attention) to participate in each transformer block, combining coarse coverage with fine-grained focus for efficient VL inference.
• Token-Shuffle (Ma et al., 24 Apr 2025) merges spatially local tokens along the channel dimension, sharply reducing token counts “seen” by the AR model, then inverts this operation at the output via token-unshuffle, maintaining fidelity at extremely high resolutions.

2.4 Multi-Resolution Tokenization in Time Series

• MRT (Peršak et al., 2024) generalizes the patchification notion to temporal tokens, embedding multi-scale segments across the lookback window. Variable-length compressor modules and mixer blocks support efficient, information-rich sequence modeling for forecasting.

3. Mixed-Resolution Attention, Embedding, and Decoding

Mixed-resolution tokens require architectural adaptations for embedding, attention, and decoding:

• Embedding: Variable-size patches (spatial or temporal) are resized (as in Quadformer (Ronen et al., 2023)) or parametrically embedded (e.g., dynamic patch/positional weights in VibeToken (Patel et al., 27 Apr 2026)). Spectral tokens (SIT) encode subbands of differing spatial coarseness and channel width.
• Positional Encoding: 2D physical position embeddings replace learned index-based PEs, supporting arbitrary layouts. Rotary embeddings (RoPE) need phase-aligned adjustments to avoid cross-scale aliasing when mixing grid resolutions in transformers, as detailed by CRPA (Wu et al., 24 Nov 2025).
• Attention: Specialized schemes mix tokens of heterogeneous size. Cross-attention/cascade (LookupViT (Koner et al., 2024), FlexAttention), cluster/block-sparse attention (ARTA), and scale-causal masking (SIT (Esteves et al., 2024)) all enable joint inference across scales. CRPA enforces phase consistency within RoPE wherever tokens originate from different spatial grids (Wu et al., 24 Nov 2025).
• Decoding/Output: In generative models, the decoder must invert the mixed-resolution embedding to reconstruct the full-resolution signal (wavelet-style IDWT in SIT, patch unshuffle/token-unshuffle in Token-Shuffle, adaptive convolutional heads in VibeToken). In time series, dedicated reverse split/combiner heads are adopted (Peršak et al., 2024).

4. Empirical Comparisons and Performance Characteristics

Mixed-resolution tokenization strategies have demonstrated substantial gains in efficiency and/or accuracy across domains:

Model/Domain	Main Advantage	Compute Reduction	Performance
Quadformer (Ronen et al., 2023)	Adaptive ViT token budget	~Same GMACs as ViT, less overhead in large models	+0.8–2pp Top-1 (ViT Base/Small), superior oracle limit
ARTA (Hagerman et al., 27 Mar 2026)	Segmentation boundary focus	Down to ~44G vs 79G FLOPs ADE20K	+2–5pp mIoU (Tiny–Base), ≥20% faster, ≤40% memory
FlexAttention (Li et al., 2024)	Hierarchical selective attn	~31–37% speedup (V100)	+2.7–6.9pp V*Bench, +3.3pp TextVQA
SIT (Esteves et al., 2024)	Coarse-to-fine wavelet tokens	2–5× faster at low-res	+5dB PSNR at 32×32, partial/upsampling features
Token-Shuffle (Ma et al., 24 Apr 2025)	Channel-fused high-res AR	4–16× less tokens	2048² AR generation, surpasses diffusion on GenEval
VibeToken (Patel et al., 27 Apr 2026)	Constant-FLOPs, any resolution	63× less FLOPs 1024²	3.54 gFID (beating NiT-XL at 5.87)
LookupViT (Koner et al., 2024)	Two-stream, compressive blocks	2–3× less FLOPs	Similar or better accuracy, improved robustness
MRT (time series) (Peršak et al., 2024)	Temporal multi-scale patching	Lower output head param count	Outperforms deep learning/industry baselines

These metrics are all present in the cited works; task-specific improvements are evident in classification, segmentation, text-to-image, and sequence forecasting settings.

5. Architectural and Theoretical Considerations

A central architectural challenge is maintaining information flow and attention consistency across tokens of heterogeneous granularity or position:

• Attention pitfalls: Naive mixing of tokens at different resolutions disrupts rotary and relative positional encoding due to phase aliasing, as rigorously demonstrated by (Wu et al., 24 Nov 2025). Cross-Resolution Phase-Aligned Attention (CRPA) solves this by aligning phases to the stride of the query stream, restoring consistent positional increments.
• Token management: Dynamic or hierarchical allocation (e.g., ARTA (Hagerman et al., 27 Mar 2026), VibeToken (Patel et al., 27 Apr 2026)) must budget token counts under varying content distributions to prevent pathological worst-case compute (e.g., if the entire image is highly detailed).
• Codebook structure: Hierarchical codebook designs (SemHiTok, SIT) support disentangling high-level semantic and low-level detailed generation, sidestepping joint-optimization pitfalls in multi-modal models (Chen et al., 9 Mar 2025).

6. Applications and Practical Implications

Mixed-resolution tokenization finds utility across modalities and tasks:

• Vision: Enhanced image classification, robust segmentation, text-to-image generation with reduced computational cost and improved detail handling (Ronen et al., 2023, Esteves et al., 2024, Hagerman et al., 27 Mar 2026, Ma et al., 24 Apr 2025).
• Multimodal and VLMs: Efficient vision-language modeling for high-resolution inputs, selective fine-scale reasoning (FlexAttention) (Li et al., 2024), and unified discrete vocabularies suitable for both understanding and generation (SemHiTok) (Chen et al., 9 Mar 2025).
• Time Series: Multi-scale patching for complex forecasting tasks, yielding better coverage of both global trends and fine detail (Peršak et al., 2024).
• Diffusion and AR models: Resolution-agnostic autoregressive image synthesis at constant compute (VibeToken), and efficient, phase-consistent mixed-resolution diffusion (Patel et al., 27 Apr 2026, Wu et al., 24 Nov 2025).

7. Limitations and Future Directions

Mixed-resolution schemes introduce challenges:

• Token management complexity: Allocation logic and scheduler design (ARTA, VibeToken) add pipeline complexity.
• Worst-case compute: Unbounded splits in highly textured regions reduce efficiency gains.
• Positional encoding brittleness: Pretrained models without cross-scale alignment (e.g., RoPE) can fail catastrophically if misapplied.
• Loss of micro-structure: Excessive compression or aggressive token merging can degrade fine detail for image or sequence generation tasks (Token-Shuffle, LookupViT).
• Extensions: Future research includes dynamic learnable allocation thresholds, generalization to 3D/temporal tokens, and hybrid fusion with learned upsampling (ARTA, LookupViT).

Overall, the use of mixed-resolution tokens enables both targeted capacity allocation and a new class of flexible, efficient, and semantically-aligned Transformer architectures across domains (Ronen et al., 2023, Esteves et al., 2024, Chen et al., 9 Mar 2025, Wu et al., 24 Nov 2025, Li et al., 2024, Patel et al., 27 Apr 2026, Hagerman et al., 27 Mar 2026, Koner et al., 2024, Ma et al., 24 Apr 2025, Peršak et al., 2024).