Bi-Level Adaptive Token Refinement

• The paper presents a novel two-stage few-shot transformer pipeline that employs Masked Image Modeling pre-training followed by meta-finetuning for token refinement.
• It integrates token clustering, uncertainty-aware weighting, bi-level attention, and graph propagation to enhance both local and global token interactions.
• Evaluations on mini-ImageNet, tiered-ImageNet, and CIFAR-FS demonstrate significant accuracy improvements by using refined token processing and class separation penalties.

Searching arXiv for the cited papers and topic variants to ground the article in the current literature. Bi-Level Adaptive Token Refinement denotes a class of adaptive modeling strategies in which token representations are not processed uniformly, but are improved through two coupled levels of computation, supervision, or allocation. In the explicit formulation titled "Bi-Level Adaptive Token Refinement for Few-Shot Transformers," the approach is a two-stage pipeline: Masked Image Modeling pre-training establishes transferable patch-level representations, and meta-fine-tuning applies Token Clustering, Uncertainty-Aware Token Weighting, a Bi-Level Attention mechanism, Graph Token Propagation, and a Class Separation Penalty to refine support and query tokens for few-shot classification (Al-Habib et al., 16 Sep 2025). Closely related work applies analogous bi-level or two-stage adaptation to reasoning, sparse attention, diffusion decoding, super-resolution, latent computation, and token budgeting, suggesting that the term describes a broader methodological pattern rather than a single architecture (Luo et al., 30 Apr 2025, Zhu et al., 2023, Tian et al., 6 Oct 2025, Chen et al., 2023, Lu et al., 5 Jun 2026).

1. Formal structure of the concept

In BATR-FST, the bi-level aspect appears twice. First, the overall system is divided into a pre-training phase and a meta-fine-tuning phase. Second, within the refinement module itself, token interactions are decomposed into intra-cluster attention and inter-cluster attention, so localized refinement and global contextual exchange are treated as distinct but coupled operations (Al-Habib et al., 16 Sep 2025).

The pre-training stage uses Masked Image Modeling. For an input image split into $L$ non-overlapping patches plus a class token, random masking of 75% of patches produces a masked input, and the Vision Transformer encoder is optimized with the masked MSE objective

$$\mathcal{L}_{\mathrm{MIM}} = \frac{1}{|\mathcal{M}|} \sum_{p \in \mathcal{M}} \left\| \mathbf{x}_p - \hat{\mathbf{x}}_p \right\|^2 .$$

After convergence, the encoder outputs token sequences $\mathbf{z}_i^0 \in \mathbb{R}^{(L+1) \times D}$, which serve as the substrate for later refinement (Al-Habib et al., 16 Sep 2025).

The meta-fine-tuning stage concatenates token embeddings from support and query images, then subjects them to clustered, uncertainty-aware, and graph-mediated refinement. This structure is specifically motivated by few-shot learning constraints: limited training data, weak inductive bias in vanilla ViTs, and the difficulty of learning robust token-level interactions from sparse supervision (Al-Habib et al., 16 Sep 2025).

2. BATR-FST architecture and refinement operators

BATR-FST begins refinement with Token Clustering. All patch tokens from support and query sets are treated as nodes in a fully connected graph with similarity edges $e_{ij} = \text{sim}(z_i, z_j)$, and the normalized attention matrix is

$A_{ij} = \frac{\exp(e_{ij})}{\sum_k \exp(e_{ik})}.$

Tokens are then partitioned into $K$ clusters using the Metis algorithm, which concentrates computation within local contexts while preserving a global graph view (Al-Habib et al., 16 Sep 2025).

Uncertainty-Aware Token Weighting then estimates token reliability with Monte Carlo Dropout. For token $p$, the feature variance is

$$\mathrm{Var}(\mathbf{z}_p) = \frac{1}{T} \sum_{t=1}^T \left\| \mathbf{z}_p^{(t)} - \overline{\mathbf{z}}_p \right\|^2,$$

with normalized uncertainty

$$\tilde{u}_p = \frac{\mathrm{Var}(\mathbf{z}_p)}{\max_q \mathrm{Var}(\mathbf{z}_q)}.$$

Token importance is modulated by both uncertainty and a synergy score $\eta_p = \gamma_p \times \psi_p$, where $$\mathcal{L}_{\mathrm{MIM}} = \frac{1}{|\mathcal{M}|} \sum_{p \in \mathcal{M}} \left\| \mathbf{x}_p - \hat{\mathbf{x}}_p \right\|^2 .$$0 captures attention received and $$\mathcal{L}_{\mathrm{MIM}} = \frac{1}{|\mathcal{M}|} \sum_{p \in \mathcal{M}} \left\| \mathbf{x}_p - \hat{\mathbf{x}}_p \right\|^2 .$$1 captures attention distributed. High-uncertainty tokens are down-weighted through the factor $$\mathcal{L}_{\mathrm{MIM}} = \frac{1}{|\mathcal{M}|} \sum_{p \in \mathcal{M}} \left\| \mathbf{x}_p - \hat{\mathbf{x}}_p \right\|^2 .$$2 (Al-Habib et al., 16 Sep 2025).

The Bi-Level Attention mechanism is the central refinement operator. Intra-cluster attention models local token interactions: $$\mathcal{L}_{\mathrm{MIM}} = \frac{1}{|\mathcal{M}|} \sum_{p \in \mathcal{M}} \left\| \mathbf{x}_p - \hat{\mathbf{x}}_p \right\|^2 .$$3 Cluster summaries are then aggregated and processed with inter-cluster attention: $$\mathcal{L}_{\mathrm{MIM}} = \frac{1}{|\mathcal{M}|} \sum_{p \in \mathcal{M}} \left\| \mathbf{x}_p - \hat{\mathbf{x}}_p \right\|^2 .$$4 The resulting global signal is fused back into token states by

$$\mathcal{L}_{\mathrm{MIM}} = \frac{1}{|\mathcal{M}|} \sum_{p \in \mathcal{M}} \left\| \mathbf{x}_p - \hat{\mathbf{x}}_p \right\|^2 .$$5

This yields token representations influenced by both local cluster structure and broader cluster-to-cluster relations (Al-Habib et al., 16 Sep 2025).

Graph Token Propagation extends the refinement beyond cluster boundaries by applying global attention over all refined tokens: $$\mathcal{L}_{\mathrm{MIM}} = \frac{1}{|\mathcal{M}|} \sum_{p \in \mathcal{M}} \left\| \mathbf{x}_p - \hat{\mathbf{x}}_p \right\|^2 .$$6 The stated role of this module is to ensure semantic consistency between support and query instances, which is especially important when few labeled examples must define the class manifold (Al-Habib et al., 16 Sep 2025).

3. Objectives, discrimination, and reported few-shot performance

BATR-FST supplements the usual classification loss with an explicit class-margin regularizer. The Class Separation Penalty is

$$\mathcal{L}_{\mathrm{MIM}} = \frac{1}{|\mathcal{M}|} \sum_{p \in \mathcal{M}} \left\| \mathbf{x}_p - \hat{\mathbf{x}}_p \right\|^2 .$$7

where $$\mathcal{L}_{\mathrm{MIM}} = \frac{1}{|\mathcal{M}|} \sum_{p \in \mathcal{M}} \left\| \mathbf{x}_p - \hat{\mathbf{x}}_p \right\|^2 .$$8 and $$\mathcal{L}_{\mathrm{MIM}} = \frac{1}{|\mathcal{M}|} \sum_{p \in \mathcal{M}} \left\| \mathbf{x}_p - \hat{\mathbf{x}}_p \right\|^2 .$$9 denote same-class and different-class pairs. The full meta-learning objective is

$\mathbf{z}_i^0 \in \mathbb{R}^{(L+1) \times D}$0

This design directly encodes the requirement that few-shot token refinement remain discriminative, not merely contextually smooth (Al-Habib et al., 16 Sep 2025).

On the reported 5-way few-shot benchmarks, BATR-FST with a ViT-S backbone and 22M parameters attains $\mathbf{z}_i^0 \in \mathbb{R}^{(L+1) \times D}$1 in 1-shot and $\mathbf{z}_i^0 \in \mathbb{R}^{(L+1) \times D}$2 in 5-shot on mini-ImageNet, $\mathbf{z}_i^0 \in \mathbb{R}^{(L+1) \times D}$3 and $\mathbf{z}_i^0 \in \mathbb{R}^{(L+1) \times D}$4 on tiered-ImageNet, and $\mathbf{z}_i^0 \in \mathbb{R}^{(L+1) \times D}$5 and $\mathbf{z}_i^0 \in \mathbb{R}^{(L+1) \times D}$6 on CIFAR-FS (Al-Habib et al., 16 Sep 2025). The same source states that it outperforms ProtoNets, Meta-Baseline, FEAT, QSFormer, FewTURE, and TATM on all datasets and both 1-shot and 5-shot tasks. Its ablation summary further attributes gains to meta-finetuning, bi-level attention, Graph Token Propagation, and uncertainty weighting, with the full model improving from $\mathbf{z}_i^0 \in \mathbb{R}^{(L+1) \times D}$7 to $\mathbf{z}_i^0 \in \mathbb{R}^{(L+1) \times D}$8 in 1-shot and from $\mathbf{z}_i^0 \in \mathbb{R}^{(L+1) \times D}$9 to $e_{ij} = \text{sim}(z_i, z_j)$0 in 5-shot over pre-training only on mini-ImageNet (Al-Habib et al., 16 Sep 2025).

A plausible implication is that, in this formulation, “refinement” is not a single operator but a coordinated stack of locality control, reliability weighting, hierarchical attention, and cross-instance propagation. The data do not claim that any one component alone defines the method.

4. Related bi-level patterns across adjacent literatures

The literature suggests that bi-level adaptive token refinement is better understood as a recurring design pattern than as a domain-specific recipe. The first level typically performs coarse selection, grouping, or budgeting; the second level performs finer correction, supervision, or interaction within the selected substrate.

Method	First level	Second level
BATR-FST	Token clustering / intra-cluster processing	Inter-cluster refinement and propagation
Ada-R1	Group-level reasoning style selection	Instance-level concise-correct preference
BiFormer	Coarse region routing	Token-to-token attention in routed regions
Tolerator	Sequence fill-up	Iterative remask-and-refine decoding
ITER	Distortion removal	Texture generation via token refinement
AdaTok	Prioritized representation	Adaptive token allocation

Ada-R1 constructs a hybrid reasoning model by linearly merging long and short CoT models, then performs bi-level preference training: group-level adaptation chooses long or short reasoning based on sampled expected accuracy with a margin $e_{ij} = \text{sim}(z_i, z_j)$1, and instance-level adaptation prefers the shortest correct reasoning within the chosen group (Luo et al., 30 Apr 2025). Its reported effect is a 51% average inference length reduction for 7B and 43% for 1.5B relative to Long-CoT across standard math datasets, with minimal accuracy drops of $e_{ij} = \text{sim}(z_i, z_j)$2 and $e_{ij} = \text{sim}(z_i, z_j)$3, and more than 50% average reasoning-length reduction on five mathematical datasets (Luo et al., 30 Apr 2025).

BiFormer’s bi-level routing attention first filters irrelevant key-value pairs at a coarse region level by top-$e_{ij} = \text{sim}(z_i, z_j)$4 routing on a region affinity matrix $e_{ij} = \text{sim}(z_i, z_j)$5, then performs dense token attention only inside the routed regions (Zhu et al., 2023). This is not framed as few-shot refinement, but it is an explicit two-level token selection mechanism with query-adaptive computation.

Tolerator in diffusion LLMs separates decoding into sequence fill-up and cross-validation refinement. After an initial fill-up stage, it repeatedly remasks subsets of tokens and decodes them again with an annealed refinement rate

$e_{ij} = \text{sim}(z_i, z_j)$6

so previously accepted tokens can be reconsidered (Tian et al., 6 Oct 2025). ITER in real-world super-resolution similarly decomposes processing into distortion removal and texture generation, with a token evaluation network deciding which tokens are already satisfactory and how many refinement steps are needed, performing well within just 8 iterative steps (Chen et al., 2023).

AdaTok presents a closely related, though not sequential, interpretation. Prioritized Representation Learning orders tokens so that prefixes remain decodable under Nested Tail Masking, while Adaptive Token Allocation learns a deterministic-group GRPO policy over candidate budgets (Lu et al., 5 Jun 2026). The same source explicitly notes that AdaTok occupies an “intermediate position”: representation is multi-level or coarse-to-fine, whereas allocation is one-pass and image-level rather than per-token dynamic refinement.

5. Adaptive signals for deciding what to refine

A major theme across this literature is that refinement is governed by a token- or instance-level signal rather than by a fixed schedule. In BATR-FST this signal is uncertainty estimated by Monte Carlo Dropout; in Ada-R1 it is group-wise expected accuracy and within-group reasoning length; in BiFormer it is content-aware region affinity; in Tolerator it is the subset sampled for remasking under a fixed total budget (Al-Habib et al., 16 Sep 2025, Luo et al., 30 Apr 2025, Zhu et al., 2023, Tian et al., 6 Oct 2025).

Other work makes this control signal even more explicit. EGAD uses teacher output entropy $e_{ij} = \text{sim}(z_i, z_j)$7 to drive a token-level curriculum, token-specific temperature

$e_{ij} = \text{sim}(z_i, z_j)$8

and a dual-branch distillation path in which easy tokens receive logits-only distillation while difficult tokens additionally receive feature and attention alignment (Zhang et al., 3 May 2026). AlignDistil constructs a token-adaptive teacher with logit extrapolation based on the total variation distance between DPO and reverse DPO distributions, using

$e_{ij} = \text{sim}(z_i, z_j)$9

with $A_{ij} = \frac{\exp(e_{ij})}{\sum_k \exp(e_{ik})}.$0 (Zhang et al., 4 Mar 2025).

Latent-compute methods move the same logic inside hidden computation. Adaptive latent CoT introduces per-token probabilistic halting through a continuation probability $A_{ij} = \frac{\exp(e_{ij})}{\sum_k \exp(e_{ik})}.$1, truncating when reach probability falls below $A_{ij} = \frac{\exp(e_{ij})}{\sum_k \exp(e_{ik})}.$2 and mixing outputs by the exit distribution over latent steps (Zeng et al., 9 Feb 2026). AdaAnchor instead monitors anchor stability through cosine dissimilarity

$A_{ij} = \frac{\exp(e_{ij})}{\sum_k \exp(e_{ik})}.$3

and halts when $A_{ij} = \frac{\exp(e_{ij})}{\sum_k \exp(e_{ik})}.$4 for $A_{ij} = \frac{\exp(e_{ij})}{\sum_k \exp(e_{ik})}.$5 consecutive steps (Sheshanarayana et al., 16 Mar 2026). CoRefine uses full-trace confidence as a control signal for a 211k-parameter Conv1D controller that decides HALT, RETHINK, or ALTERNATIVE, averaging 2.7 refinement steps per problem and roughly 190-fold token reduction relative to 512-sample baselines, with 92.6 percent precision when it confidently halts (Jin et al., 9 Feb 2026).

This suggests that bi-level refinement is increasingly tied to learned or derived measures of token difficulty, semantic reliability, or convergence, rather than to fixed-depth processing.

6. Scope, limitations, and terminological ambiguities

The literature does not use “bi-level” in a single uniform sense. In BATR-FST it refers to a two-stage few-shot transformer pipeline and to a local-global attention hierarchy (Al-Habib et al., 16 Sep 2025). In Ada-R1 it denotes group-level versus instance-level preference optimization (Luo et al., 30 Apr 2025). In BiFormer it refers to coarse region filtering followed by fine-grained token attention (Zhu et al., 2023). In Tolerator and ITER it denotes a fill-up or restoration stage followed by iterative refinement (Tian et al., 6 Oct 2025, Chen et al., 2023). In AdaTok it describes the coupling between hierarchical representation and adaptive budget allocation rather than a multi-pass token editor (Lu et al., 5 Jun 2026).

A common misconception is that adaptive refinement necessarily increases computation. The reported results do not support that view. Ada-R1 reduces reasoning length by more than 50% on five mathematical datasets while maintaining performance (Luo et al., 30 Apr 2025). AdaAnchor with adaptive halting reduces average latent refinement steps by 48–60% under the same maximum-step budget and reduces generated tokens by 92–93% relative to standard reasoning baselines (Sheshanarayana et al., 16 Mar 2026). AdaTok-Adaptive attains rFID 1.50 using only $A_{ij} = \frac{\exp(e_{ij})}{\sum_k \exp(e_{ik})}.$6 tokens on average, while AdaTok-Full reaches rFID 1.31 at 256 tokens, and the shorter adaptive representation yields $A_{ij} = \frac{\exp(e_{ij})}{\sum_k \exp(e_{ik})}.$7 throughput over a fixed 256-token decode (Lu et al., 5 Jun 2026). CoRefine also reports 63% wall-clock time reduction over parallel sampling, with negligible controller overhead (Jin et al., 9 Feb 2026).

A second misconception is that bi-level refinement always operates on explicit text tokens. Several methods refine regions, codebook tokens, latent anchors, or adaptive image-token prefixes instead of ordinary autoregressive outputs (Zhu et al., 2023, Chen et al., 2023, Sheshanarayana et al., 16 Mar 2026, Lu et al., 5 Jun 2026). The literature therefore suggests that “token” is often best interpreted operationally: it denotes the discrete or quasi-discrete unit on which adaptive computation is allocated.

Taken together, these works position Bi-Level Adaptive Token Refinement as a general strategy for allocating representational effort unevenly but systematically. In its narrow sense, BATR-FST is a few-shot ViT method that couples MIM pre-training with clustered, uncertainty-aware, and graph-propagated token refinement (Al-Habib et al., 16 Sep 2025). In the broader sense suggested by adjacent research, it is part of a wider movement toward hierarchical, content-conditioned, and budget-aware computation, in which coarse decisions determine where fine refinement is worth performing.