Multi-Scale Token Space in ML

• Multi-Scale Token Space is a paradigm that constructs tokens at varying granularities to capture both global context and fine details.
• It employs techniques such as multi-resolution patching, quantization, and cross-scale attention to efficiently represent complex signals.
• This approach improves model scalability and resource allocation, supporting applications in vision, language, time series, 3D, graphs, and beyond.

A multi-scale token space is a formal paradigm in which token representations are constructed, organized, or aggregated to capture information at multiple, typically hierarchical, resolutions or granularities. This design principle has emerged independently and recurrently across vision, language, time series, 3D, and graph domains, enabling models to efficiently represent, reason about, and process signals with both fine and coarse structure. Its instantiations range from explicit multi-resolution patching, quantization, or wavelet decompositions at the tokenization stage to hierarchical aggregation, cross-scale fusion, and scale-conditioned attention within deep model layers. The multi-scale token space concept addresses both the representation capacity and computational resource allocation bottlenecks of traditional flat token interfaces.

1. Mathematical Foundations of Multi-Scale Token Spaces

Multi-scale token spaces are typically organized by constructing sets of tokens that encode different levels of abstraction, locality, or granularity, either through explicit multi-resolution feature extraction, quantization hierarchies, or learned transformations over raw input or latent representations.

Formal Definitions:

• For time series, token sets are produced by non-overlapping or overlapping patching at a range of resolutions $K=\{k_1,\dots,k_r\}$, with each token summarizing a segment of the signal whose length depends on the scale parameter $k$ (Peršak et al., 2024).
• In images, hierarchical tokenization schemes, such as those using multi-level Residual Quantization (RQ) or wavelet transforms, generate discrete or continuous token groups at increasing spatial resolutions, where each subsequent level only encodes finer-scale details not captured by previous ones (Hadji et al., 14 May 2026, Ding, 30 May 2026).
• For graphs, multi-scale token spaces are constructed by extracting node representations at different GNN layers (local, meso, global), each quantized by a dedicated codebook and aggregated via learned gating (Xiang et al., 14 Oct 2025).
• In 3D, tokens are built from spatial partitions at multiple volumetric scales (e.g., grid cells or voxels of progressive sizes), with features such as means, covariances, and colors characterizing each cell (Tang et al., 26 Nov 2025).
• In LLMs, hierarchical latent space folding applies a series of linear and geometric smoothing operations to produce token embeddings at progressive abstraction levels, allowing both short-range context and global theme organization (Harcourt et al., 13 Feb 2025).

General Formulation:

If $Z^{(\ell)}$ denotes token embeddings at scale $\ell$, then

$$Z^{(\ell+1)} = \mathcal{F}_\ell\left(Z^{(\ell)}\right)$$

where $\mathcal{F}_\ell$ may include linear projections, pooling/downsampling, residual quantization, or geometric smoothing. The final token space is the union or aggregation over all scales.

2. Multi-Scale Tokenization Schemes and Architectures

Patch and Pool-Based Constructions:

In transformers for vision and time series, parallel or hierarchical patchification at multiple spatial or temporal strides produces streams of tokens representing the same signal at varying granularity (Bian et al., 2023, Peršak et al., 2024, Zhong et al., 22 Sep 2025). MeshTok implements adaptive mesh refinement to allocate more tokens to physically informative subregions (Zhao et al., 3 Jun 2026).

Quantization and Residual Hierarchies:

DTMs for time series (MSDformer) and hierarchical quantization frameworks for graphs construct token spaces using cascades of vector quantizers. Each scale encodes the residual unexplained by previous coarser scales, enabling the composite token space to span the signal’s multi-scale structure (Chen et al., 20 May 2025, Xiang et al., 14 Oct 2025).

Wavelet/DWT and Frequency-Space Tokenization:

Wavelet as Tokenizer (WAT) uses a fixed wavelet transform to map input signals (audio, image, video) to a flattened set of multi-scale coefficient tokens, with each token indexed by subband (frequency), spatial location, and optional modality metadata. This construction is agnostic to modality and preserves both coarse and fine structure (Ding, 30 May 2026).

Scale-Conditioned Transformers and Fusion Modules:

Transformers may operate directly over multi-scale tokens using cross-scale fusion (via concatenation, cross-attention, or FiLM-style modulation) (Tang et al., 26 Nov 2025, Zhao et al., 3 Jun 2026). In vision, multi-scale features are often upsampled or fused before pruning or merging to enrich token viewpoint (Bian et al., 2023).

Progressive/Adaptive Token Schedules:

In segmentation and detection, encoders may process only coarse-scale tokens in early layers and progressively introduce finer scales, as in PRO-SCALE, yielding major computational efficiency gains while maintaining accuracy (Aich et al., 2024).

3. Integration of Multi-Scale Token Spaces in Model Workflows

Token Recycling, Merging, and Pruning:

Tokens at less informative positions or scales can be pruned or merged, with their information merged into surviving tokens by soft assignment based on similarity or importance scores (Bian et al., 2023), or adaptively recycled as in Multi-Scale Token Recycling modules (MSTR) (Jiang et al., 2024).

Cross-Scale Attention and Aggregation:

Cross-attention mechanisms condition fine-scale tokens on coarse-scale tokens or vice versa, sometimes using learned compatibility gates to dynamically reweigh information from each scale (Sun et al., 2 May 2026, Tang et al., 26 Nov 2025). FiLM modulations encode both absolute scale and relative position (Zhao et al., 3 Jun 2026).

Autoregressive Multi-Scale Generation:

In discrete autoregressive models (e.g., MSDformer for time series, HIT for image SR), scale-indexed tokens are decoded sequentially or in blocks, enabling single-pass multi-resolution output and partial reconstructions at intermediate scales (Chen et al., 20 May 2025, Hadji et al., 14 May 2026).

Energy-Based Token Selection for Sparsity:

In fixed-resource or streaming settings, token selection by energy thresholds (global or per-subband) enables sparse allocation while retaining critical multi-scale information, exploiting the inherent compressibility of wavelet or quantized representations (Ding, 30 May 2026).

4. Empirical Performance and Comparative Evaluations

Retrieval, Segmentation, and Localization:

Spectral Retrieval demonstrates dramatic recall and ranking improvements for document retrieval by aggregating over multi-scale convolved token embeddings, especially for localized relevance signals (Morandi, 23 May 2026). Multi-scale transformers for time series classification and generation (MTM, MSDformer) outperform state-of-the-art single-scale and diffusion baselines both in sample efficiency and predictive fidelity (Zhong et al., 22 Sep 2025, Chen et al., 20 May 2025). PRO-SCALE and shunted self-attention architectures show that progressive and heterogeneous scale allocation yields large reductions in computational requirements with minimal or no accuracy loss in panoptic/semantic segmentation and image classification (Aich et al., 2024, Ren et al., 2021).

3D and Graph Domains:

Hierarchical multi-scale tokenization of 3D scenes (NDTokenizer3D) and open-vocabulary affordance detection (VoxAfford) utilize cross-scale scene and voxel tokens for strong generalization and spatial precision, each outperforming prior VLM and segmentation architectures in benchmarks and real robotic tasks (Tang et al., 26 Nov 2025, Sun et al., 2 May 2026). In graphs, dynamic gating across quantized hierarchy levels yields state-of-the-art node classification and link prediction results on Cora, Citeseer, Pubmed, and ogbn-proteins (Xiang et al., 14 Oct 2025).

Compression and Sparsity:

Wavelet tokenization with fixed-rate energy selection achieves substantial perceptual quality gains for audio, image, and video signals compared to uniform or random masking, with masked sparse training further improving PSNR in the limited-token regime (Ding, 30 May 2026).

5. Theoretical Underpinnings and Resource Allocation

Rate-Distortion Analysis:

Multi-scale token spaces are grounded in rate-distortion theory: partitioning the representational budget across multiple hierarchies or resolutions enables higher total coding rate and lower distortion for fixed token count compared to expanding a single-scale codebook (Chen et al., 20 May 2025).

Structured Latent-Space Folding:

Hierarchical latent-space folding regularizes token embeddings via progressive linear/geometric smoothing, resulting in lower intra-layer variance, reallocation of attention heads to deeper layers, and increased activation sparsity. This controlled redistribution supports both compactness and expressive abstraction (Harcourt et al., 13 Feb 2025).

Efficient FLOP Scaling:

Progressive scaling of token length (PRO-SCALE) and down-sampling rates (shunted self-attention) reduce per-layer and total FLOPs by O(1/s) to O(1/s²) for s-fold token count reduction while preserving multi-scale representation capabilities (Aich et al., 2024, Ren et al., 2021).

6. Domain-Specific Methodological Choices

Domain	Multi-Scale Tokenization Mechanism	Reference
Vision	Parallel patch embeddings, hierarchical RQ, SSA	(Bian et al., 2023, Ren et al., 2021, Hadji et al., 14 May 2026)
Time series	Multi-resolution patching, residual VQ-VAEs	(Peršak et al., 2024, Chen et al., 20 May 2025, Zhong et al., 22 Sep 2025)
3D vision	Multi-scale voxel/NDT partitioning, cross-attn	(Tang et al., 26 Nov 2025, Sun et al., 2 May 2026)
Graphs	Quantized hierarchy, task-driven gating	(Xiang et al., 14 Oct 2025)
Audio/image/video	Wavelet/token flattening + metadata	(Ding, 30 May 2026)
Language modeling	Hierarchical embedding folding, spectral scoring	(Harcourt et al., 13 Feb 2025, Morandi, 23 May 2026)

Domain-adaptive choices include quantization vs. convolutional feature extraction, explicit vs. learned scale mixing, and whether multi-scale fusion occurs pre- or post-attention.

7. Open Challenges and Future Directions

Current limitations include the use of fixed or manually-designed scale grids, limited adaptability to data-dependent or task-specific scale mixing, and open theoretical questions on the optimal allocation of tokens to hierarchies for various computational and modeling budgets. Ongoing work investigates adaptive scale depth, integration with long-context memory modules, cross-lingual scale alignment in NLP, and learning universal token grammars that generalize across modalities and domains (Harcourt et al., 13 Feb 2025, Ding, 30 May 2026). The multi-scale token space remains a central organizing principle for scalable, efficient, and contextually rich representation in next-generation machine learning systems.