Asymmetric Multi-Scale Tokenizer

• Asymmetric Multi-Scale Tokenizer is a design that applies scale-dependent and domain-specific processing to heterogeneous inputs, improving representation fidelity.
• It employs distinct treatment for different components—such as temporal versus frequency or image versus text—thereby optimizing token granularity based on structural variation.
• This approach has been successfully used in EEG modeling, radiology report generation, image spectrum tokenization, and language tokenization to enhance downstream performance.

An asymmetric multi-scale tokenizer can be understood as a class of tokenization architectures that does not impose a single uniform segmentation or compression rule across all positions, modalities, scripts, or frequency bands, and that represents inputs at more than one spatial, temporal, spectral, or semantic scale. Recent work instantiates this design pattern in EEG foundation models, radiology report generation, long-context language modeling, script-aware multilingual tokenization, image generation, and 3D vision-language systems. Across these settings, asymmetry usually refers to differential treatment of heterogeneous components—such as temporal versus frequency content, image versus text, high-entropy versus low-entropy spans, or Abugida versus non-Abugida scripts—while multi-scale processing refers to explicit modeling across coarse and fine resolutions or granularities (Ma et al., 10 Jun 2025, Li et al., 30 Jun 2025, Liu et al., 21 Aug 2025, Darshana, 26 Mar 2026, Esteves et al., 2024, Tang et al., 26 Nov 2025).

1. Conceptual lineage

The immediate antecedents of asymmetric multi-scale tokenization were not tokenizers but architectural modules that challenged the assumption of symmetric processing. "Multi-Scale Spatially-Asymmetric Recalibration for Image Classification" introduced multi-scale spatially-asymmetric recalibration (MS-SAR) to address the fact that convolution is spatially-symmetric, meaning that the response at each spatial position is computed in the same way regardless of absolute location. MS-SAR computes position-dependent importance values from surrounding regions at multiple scales and multiplies them with the original neural response, thereby making contextual modulation both spatially asymmetric and multi-scale (Wang et al., 2018).

A parallel line appeared in EEG modeling. "Simultaneously exploring multi-scale and asymmetric EEG features for emotion recognition" proposed the Multi-Scales Bi-hemispheric Asymmetric Model (MSBAM), motivated by the observation that the left and right hemispheres demonstrate activity differences under different emotional activities and that EEG is non-stationary enough that single kernel sizes may be insufficient. MSBAM therefore combined bi-hemispheric asymmetry with multi-branch temporal kernels (Wu et al., 2021).

These works are not tokenizers, but they established a recurring architectural thesis: symmetric treatment of all regions or scales can suppress structurally important variation. This suggests why later tokenization work repeatedly separates domains, routes inputs differently, or allocates variable token granularity rather than using a single homogeneous rule.

2. Structural principles

Across the recent literature, asymmetry does not denote one fixed operation. In CodeBrain, it means independent temporal and frequency codebooks with different objectives; in $\mu^2$Tokenizer, it means aggressive compression and reweighting of visual tokens while question tokens are used to condition selection; in SemToken, it means heterogeneous token granularity based on semantic density; in WWHO, it means script-specific routing in which Abugida scripts undergo rule-based syllabification and SGPE while non-Abugida text is passed to BPE; and in SIT, it means per-scale tokenization in a coarse-to-fine spectral order rather than raster-scan order (Ma et al., 10 Jun 2025, Li et al., 30 Jun 2025, Liu et al., 21 Aug 2025, Darshana, 26 Mar 2026, Esteves et al., 2024).

Multi-scale processing is equally heterogeneous. Some systems use explicit parallel scales, as in dynamic pooling over $s \in \{1,2,4\}$ in $\mu^2$Tokenizer; some use hierarchical discrete representations, as in SIT’s sequence of $S \times N$ tokens ordered from coarse to fine; some use separate scale-specific codebooks, as in SIT and NeuroRVQ; and some assign fine or coarse spans adaptively according to semantic entropy, as in SemToken (Li et al., 30 Jun 2025, Esteves et al., 2024, Barmpas et al., 15 Oct 2025, Liu et al., 21 Aug 2025).

A concise comparison is given below.

System	Asymmetry	Multi-scale mechanism
TFDual-Tokenizer	Separate temporal and frequency codebooks; different optimization paths	Independent paired tokens with combinatorial code space
$\mu^2$Tokenizer	Visual compression differs from text processing; text-conditioned fusion	Dynamic pooling at scales $\{1,2,4\}$
SemToken	Fine/coarse granularity by semantic density	Variable-length spans from local clustering and entropy
WWHO + SGPE	Abugida routed to LinguisTrie + SGPE; others to BPE	Syllable-level atomicization followed by higher-level merges
SIT	Per-scale tokenizers in spectral order	DWT-based coarse-to-fine token sequence

Two formulas capture recurrent design motifs. In CodeBrain, each EEG patch embedding $\tilde e_i$ is discretized independently by temporal and frequency codebooks,

$$\hat z^{(t)}_i = \arg\min_j \|\tilde e_i - v^{(t)}_j\|^2,\qquad \hat z^{(f)}_i = \arg\min_j \|\tilde e_i - v^{(f)}_j\|^2,$$

so that two codebooks of size $K$ yield a representation space of size $K^2$ (Ma et al., 10 Jun 2025). In SemToken, granularity is assigned from semantic entropy,

$s \in \{1,2,4\}$0

with fine or coarse tokenization determined by whether a local window exceeds a threshold $s \in \{1,2,4\}$1 (Liu et al., 21 Aug 2025).

3. Biosignal and spectral tokenizers

EEG has become a major testbed for asymmetric multi-scale tokenization because the signal is simultaneously temporal, spectral, multichannel, and non-stationary. CodeBrain addresses this with the TFDual-Tokenizer, which independently tokenizes heterogeneous temporal and frequency components and maps each EEG patch to two discrete indices, one from the temporal codebook and one from the frequency codebook. The temporal codebook is trained with reconstruction plus SimCLR-style contrastive loss, whereas the frequency codebook reconstructs DFT-derived amplitude and phase. Because any pair $s \in \{1,2,4\}$2 is valid, the discrete space expands from $s \in \{1,2,4\}$3 to $s \in \{1,2,4\}$4; with $s \in \{1,2,4\}$5, the number of possible token combinations is approximately $s \in \{1,2,4\}$6 million. The tokenizer feeds EEGSSM, which combines Structured Global Convolution for long-range dependencies with Sliding Window Attention for local dependencies, and the full system reports generalizability with linear probing on 10 public EEG datasets (Ma et al., 10 Jun 2025).

NeuroRVQ pursues a different EEG strategy. It segments multivariate EEG into temporal patches, applies a multi-scale temporal encoder inspired by Inception modules, enriches the resulting features with trainable temporal and spatial embeddings, and then quantizes each temporal scale with hierarchical residual vector quantization. For each scale, RVQ uses $s \in \{1,2,4\}$7 codebook layers,

$s \in \{1,2,4\}$8

with reconstruction $s \in \{1,2,4\}$9. The tokenizer decoder reconstructs Fourier spectral amplitude and phase rather than only the time-domain waveform, and training uses log-amplitude, unit-circle-aware phase, time-domain, and quantization losses. The paper attributes lower reconstruction error and improved downstream performance to this full-spectrum, hierarchical tokenization scheme (Barmpas et al., 15 Oct 2025).

For images, the Spectral Image Tokenizer (SIT) replaces spatial raster tokenization with tokenization in the image spectrum obtained from a discrete wavelet transform. DWT decomposes an image into coarse approximation coefficients and scale-specific detail coefficients,

$\mu^2$0

and SIT patchifies each scale into tokens so that the full image becomes a sequence of $\mu^2$1 tokens ordered from coarse to fine. The design is explicitly multi-scale, uses per-scale codebooks, and introduces scale-causal attention in which a token at scale $\mu^2$2 attends only to tokens at scales $\mu^2$3. The paper emphasizes five consequences: better use of the compressibility of high frequencies, reconstruction of different resolutions without retraining, improved next-token conditioning through a coarse reconstruction of the full image, partial decoding in which early tokens reconstruct a coarse image, and the use of autoregressive models for image upsampling (Esteves et al., 2024).

4. Multimodal and 3D formulations

In radiology report generation, $\mu^2$4Tokenizer is presented as a differentiable multi-scale multi-modal middleware between 3D CT features and a downstream LLM. Visual features come from a 3D ViT applied to CT frames, and textual question tokens come from a text tokenizer. The tokenizer then applies four components: Relative Positional Encoding, Differentiable Token Selection, Dynamic Multi-scale Pooling, and multi-modal text-conditioned attention. DTS forms up to $\mu^2$5 soft visual tokens by learned weighted sums, while DMTP pools over scales $\mu^2$6 and uses attention-like dynamic weights to combine the pooled outputs. The system is asymmetric because image tokens are compressed, reweighted, and pooled while question tokens act as conditioning signals with finer semantic granularity. On AMOS-MM, the reported GREEN score progresses from $\mu^2$7 for the baseline to $\mu^2$8 with RPE, $\mu^2$9 with DTS, $S \times N$0 with DMTP, $S \times N$1 for the full $S \times N$2Tokenizer, and $S \times N$3 for the full system with DPO (Li et al., 30 Jun 2025).

NDTokenizer3D extends the same general pattern to 3D scenes. It first constructs a multi-scale NDT representation from raw high-resolution point clouds without downsampling. At each scale, a grid cell is summarized by a mean position $S \times N$4, covariance $S \times N$5, and RGB color $S \times N$6,

$S \times N$7

A transformer-based encoder extracts features at each scale, and the Multi-Scale NDT Decoder (MSDec) then performs progressive cross-scale fusion from coarse to fine. Decoder queries are updated by cross-attention, self-attention, and FFN blocks at each scale,

$S \times N$8

producing holistic scene tokens consumable by LLM endpoints. MSDec is also reused for prompting with points, boxes, or masks and for segmentation-mask decoding, which yields a unified interface across 3D Referring Segmentation, 3D Visual Question Answering, and 3D Dense Captioning (Tang et al., 26 Nov 2025).

5. Language-centric tokenization

In language modeling, asymmetry and multi-scale tokenization depart most sharply from BPE-style uniformity. SemToken begins with contextual semantic embeddings from a lightweight encoder, then performs local semantic clustering to merge semantically equivalent adjacent tokens according to a cosine-similarity threshold. It next computes semantic entropy for each span and allocates fine-grained tokenization in content-rich regions and coarse compression in repetitive or low-entropy regions. The framework reports up to $S \times N$9 reduction in token count and $\mu^2$0 speedup on long-context benchmarks, with negligible or no degradation in perplexity and downstream accuracy. The detailed results include a reduction to $\mu^2$1 of the original token count on WikiText-103, latency dropping from $\mu^2$2 ms to $\mu^2$3 ms, KV cache reductions up to $\mu^2$4, perplexity of $\mu^2$5 versus $\mu^2$6, and additive gains up to $\mu^2$7 speedup when combined with accelerators such as FlashAttention and H2O (Liu et al., 21 Aug 2025).

SupraTok approaches asymmetry through cross-boundary pattern learning. It extends BPE by allowing merges across whitespace and punctuation boundaries so that coherent multi-word expressions become single “superword” tokens. Candidate sequences are filtered by Pointwise Mutual Information and branching entropy, and the vocabulary is learned through a three-phase curriculum: standard within-word BPE merges from $\mu^2$8 to $\mu^2$9k merges, cross-boundary merges from $\{1,2,4\}$0k to $\{1,2,4\}$1k, and more complex or domain-specific expressions from $\{1,2,4\}$2k to $\{1,2,4\}$3k merges. The paper reports a $\{1,2,4\}$4 improvement in English tokenization efficiency, $\{1,2,4\}$5 versus $\{1,2,4\}$6 characters per token compared to OpenAI’s o200k tokenizer, and $\{1,2,4\}$7 improvement over Gemma 3’s tokenizer, while maintaining competitive performance across 38 languages. In a GPT-2 scale model with 124M parameters trained on 10 billion tokens from FineWeb-Edu, SupraTok yields an $\{1,2,4\}$8 improvement on HellaSWAG and $\{1,2,4\}$9 on MMLU. The authors also note that further validation at larger model scales is needed (Tănase et al., 16 Aug 2025).

WWHO and SGPE address a different asymmetry: script complexity. WWHO separates tokenization into Router, LinguisTrie, and SGPE layers. The Router performs $\tilde e_i$0 script segmentation and sends Abugida segments to LinguisTrie while routing others to base BPE. LinguisTrie uses a DFA and external schemas to parse text into atomic orthographic syllables. SGPE then performs statistical merging at the syllable level rather than the byte, character, or grapheme level. For Sinhala, SGPE achieves a Token to Word Ratio of $\tilde e_i$1 with $\tilde e_i$2 characters per token, representing a $\tilde e_i$3 percent reduction in tokens compared to OpenAI’s o200k base; for Hindi, it achieves a TWR of $\tilde e_i$4, a $\tilde e_i$5 percent reduction. On mixed-script Sinhala–Devanagari–English data, the overall TWR is $\tilde e_i$6, with token reductions of $\tilde e_i$7 percent, $\tilde e_i$8 percent, and $\tilde e_i$9 percent relative to o200k base, Llama 4 Scout, and DeepSeek V3, respectively. The paper states that this extends the usable context window by up to $$\hat z^{(t)}_i = \arg\min_j \|\tilde e_i - v^{(t)}_j\|^2,\qquad \hat z^{(f)}_i = \arg\min_j \|\tilde e_i - v^{(f)}_j\|^2,$$0 times for these Abugida languages and enforces a Linguistic Zero-Breakage Guarantee, under which no valid syllable is split across multiple tokens (Darshana, 26 Mar 2026).

6. Evaluation profile, interpretive distinctions, and limitations

The empirical literature shows that asymmetric multi-scale tokenization is not tied to a single representation type. Some systems produce discrete code indices with codebooks, as in CodeBrain, NeuroRVQ, and SIT; some produce differentiable soft visual tokens, as in $$\hat z^{(t)}_i = \arg\min_j \|\tilde e_i - v^{(t)}_j\|^2,\qquad \hat z^{(f)}_i = \arg\min_j \|\tilde e_i - v^{(f)}_j\|^2,$$1Tokenizer; some output variable-length text spans, as in SemToken and SupraTok; and some generate holistic scene tokens from continuous geometric cells, as in NDTokenizer3D (Ma et al., 10 Jun 2025, Barmpas et al., 15 Oct 2025, Esteves et al., 2024, Li et al., 30 Jun 2025, Liu et al., 21 Aug 2025, Tang et al., 26 Nov 2025).

The same point applies to asymmetry. It is not restricted to left-right or spatial asymmetry, although that earlier usage remains important in MS-SAR and MSBAM. In the tokenizer literature, asymmetry may instead mean domain decoupling, as in temporal versus frequency EEG codes; modality-specific processing, as in CT vision tokens versus textual queries; semantic-density-dependent granularity, as in SemToken; cross-boundary phrase formation, as in SupraTok; or script-specific routing and syllable preservation, as in WWHO (Wang et al., 2018, Wu et al., 2021, Ma et al., 10 Jun 2025, Li et al., 30 Jun 2025, Liu et al., 21 Aug 2025, Tănase et al., 16 Aug 2025, Darshana, 26 Mar 2026).

A second interpretive distinction concerns where the gains arise. Several papers study tokenization jointly with a consuming architecture rather than as an isolated preprocessing module: TFDual-Tokenizer is paired with EEGSSM, $$\hat z^{(t)}_i = \arg\min_j \|\tilde e_i - v^{(t)}_j\|^2,\qquad \hat z^{(f)}_i = \arg\min_j \|\tilde e_i - v^{(f)}_j\|^2,$$2Tokenizer with a radiology LLM and DPO, SIT with autoregressive image generation, and NDTokenizer3D with MSDec and LLM endpoints. A plausible implication is that tokenizer quality is often entangled with the inductive biases of the downstream model rather than separable from it (Ma et al., 10 Jun 2025, Li et al., 30 Jun 2025, Esteves et al., 2024, Tang et al., 26 Nov 2025).

Finally, the scope of reported benefits is domain-bounded. WWHO’s strongest gains are specific to complex Abugida scripts; SupraTok explicitly states that larger-scale validation remains to be done; and SIT’s advantages are articulated for autoregressive image generation, partial decoding, and upsampling rather than as a universal replacement for all image tokenizers (Darshana, 26 Mar 2026, Tănase et al., 16 Aug 2025, Esteves et al., 2024). Even so, the literature consistently converges on one general principle: when inputs contain heterogeneous structures across scales, a tokenizer that separates those structures before compression can improve efficiency, interpretability, or downstream modeling fidelity.