Token Freezing and Reusing (ToFe)

• Token Freezing and Reusing (ToFe) is a strategy that temporarily freezes less important tokens in transformers to save computational cost while allowing their later reuse.
• In Vision Transformers, ToFe employs a lightweight token selector and bottleneck approximator with budget-aware training to reduce self-attention operations by nearly 50% with minimal accuracy loss.
• In diffusion language models, DyLLM uses cosine similarity to identify salient tokens and reuses cached activations, achieving up to 9.60× speedup during inference.

Searching arXiv for the cited ToFe and DyLLM papers to ground the article in current preprints. Token Freezing and Reusing (ToFe) denotes a class of efficiency methods for transformer inference in which tokens deemed temporarily unimportant are not irreversibly discarded, but instead frozen, skipped through expensive computation for some interval, and later reused when their information may again become relevant. In the Vision Transformer setting, ToFe is introduced as “lagged Token Freezing and Reusing,” a framework that couples token selection with token approximation and budget-aware end-to-end training in order to reduce self-attention and MLP cost without permanently losing token identity (Zhang et al., 22 Jul 2025). In masked diffusion LLMs (MDLMs), the same general idea appears as a training-free inference strategy that treats diffusion decoding as a token-freezing / token-reusing problem: stable tokens are frozen and their cached activations reused, while only a salient subset is recomputed during denoising (Lee et al., 9 Mar 2026). Across these formulations, the unifying principle is that token importance and token dynamics vary across depth or time, so temporary inactivity can be exploited for computational savings while preserving the possibility of later reintegration.

1. Conceptual basis and scope

ToFe originates from the observation that token reduction need not be irreversible. In Vision Transformers, existing token reduction methods are described as mostly implementing irreversible token reduction: pruning directly discards tokens, and re-organization methods merge or fuse them. The central critique is that transformers focus on different information among blocks, so tokens reduced in early blocks might be useful later. ToFe addresses this by freezing such tokens temporarily and allowing their lagged reusing at a later stage (Zhang et al., 22 Jul 2025).

This framing shifts the objective from permanent elimination to deferred processing. In a ViT, image inputs are represented as patch tokens plus a [CLS] token, and the self-attention cost scales quadratically with token count. The per-block computational burden is given as

$$\mathrm{FLOPs}_l = 4N_lD^2 + 2N_l^2D + 2N_lD \times D_{\rm hidden},$$

where $N_l$ is the number of tokens entering block $l$ (Zhang et al., 22 Jul 2025). This makes token-count control a primary lever for efficiency, but ToFe differs from standard pruning by insisting that low-importance status is provisional.

A closely related interpretation appears in DyLLM for MDLM inference. There, the system starts from a fully masked response and iteratively denoises it. Because the model uses bidirectional attention, every denoising step globally changes token interactions, so the full sequence is repeatedly reprocessed. DyLLM recasts this expense as a token-freezing / token-reusing problem: most token representations remain stable across adjacent denoising steps, so only a small subset of salient tokens needs full recomputation, while the rest can reuse cached activations (Lee et al., 9 Mar 2026).

Taken together, these formulations define ToFe not as a single architecture, but as a computational pattern. In one instantiation, the axis is depth across transformer blocks; in another, it is time across diffusion denoising steps. A plausible implication is that ToFe is best understood as a general strategy for exploiting token-level redundancy when token relevance is non-monotonic rather than strictly decreasing.

2. Lagged freezing and reuse in Vision Transformers

In the ViT formulation, ToFe introduces a two-part mechanism at each reduction stage: a token selector and a token approximator (Zhang et al., 22 Jul 2025). The backbone may be any standard ViT-style model such as ViT, DeiT, or LV-ViT. If token-reduction stages are placed before blocks $l_1,\dots,l_S$, then at stage $s<S$ a lightweight prediction module decides which patch tokens are kept and which are frozen.

The selector is implemented as an MLP followed by Gumbel-Softmax:

$$\mathbf{z}^s = \mathrm{Softmax}(\mathrm{MLP}(\mathbf{X})) \in \mathbb{R}^{N \times 2},$$

$$\mathbf{M}^{s} = \mathrm{GumbleSoftmax}(\mathbf{z^s}) \in \{0,1\}^{N}.$$

Here $\mathbf{M}^s$ indicates whether each token is kept ($1$) or frozen ($0$); the [CLS] token is always retained (Zhang et al., 22 Jul 2025). Kept tokens continue through subsequent ViT blocks, while frozen tokens skip those blocks entirely.

To prevent frozen tokens from becoming stale, ToFe adds a token approximator, described as a bottleneck MLP that estimates the update a frozen token would have received had it traversed the skipped blocks. The approximation is expressed through the residual change:

$N_l$0

The approximated frozen tokens are then rearranged back into their original spatial order,

$N_l$1

which preserves token identity and permits later reuse (Zhang et al., 22 Jul 2025).

The significance of the “lagged” qualifier is that freezing is temporary rather than terminal. The stated intuition is that transformer representations evolve smoothly for many tokens across consecutive blocks, and attention and token features are often highly similar across adjacent blocks. A token that is not useful at block 4 may still encode details that become relevant by block 7 or 10. ToFe therefore postpones expensive processing instead of erasing the token. This directly distinguishes it from pruning and merge-based schemes, where removed information cannot be recovered later (Zhang et al., 22 Jul 2025).

3. Training objective, differentiable masking, and deployment modes

ToFe is trained end-to-end with classification supervision, approximation supervision, and a compute-budget objective. The approximation loss encourages the approximated frozen tokens to match the backbone’s token outputs:

$N_l$2

Classification is optimized with standard cross-entropy:

$N_l$3

These are combined as

$N_l$4

To adapt token usage to a target inference budget, the method imposes a computation budget-aware constraint,

$N_l$5

which is converted into

$N_l$6

and then incorporated into the full objective

$N_l$7

This budget-aware training is explicitly presented as a way to learn stage-wise token usage rather than manually fixing keep ratios (Zhang et al., 22 Jul 2025).

Because token selection is binary, training is made differentiable and parallelizable by converting freezing into masking:

$N_l$8

The attention mask is modified so frozen tokens do not interact normally with all tokens during this training-time relaxation (Zhang et al., 22 Jul 2025).

At inference, ToFe supports two modes. In instance-adaptive ToFe, the predictor outputs a sample-specific mask at each stage for a single input. In batch-adaptive ToFe, for parallel inference, tokens are sorted by importance scores and the top-$N_l$9 tokens are kept, where $l$0 is the average token count observed during training (Zhang et al., 22 Jul 2025). This separation reflects two distinct deployment constraints: per-instance adaptivity versus batched throughput.

A common misconception is to treat ToFe as ordinary token pruning with a predictor. The architectural and objective design show otherwise: the approximator, rearrangement step, and FLOPs-constrained optimization are all directed toward deferred reuse rather than permanent removal.

4. Diffusion-language-model instantiation: saliency, freezing, and partial attention

DyLLM provides a concrete MDLM instantiation of the token-freezing / token-reusing idea (Lee et al., 9 Mar 2026). In an MDLM, generation begins from a fully masked response, and at timestep $l$1 the model processes the concatenated sequence

$l$2

where $l$3 is the prompt and $l$4 is the partially unmasked response. Unlike autoregressive LLMs, MDLMs predict all positions in parallel and then unmask a subset chosen by confidence. The use of bidirectional attention implies that each denoising step globally changes token interactions, so the system behaves like a repeated prefill pass with dominant cost in the repeated attention/FFN stack (Lee et al., 9 Mar 2026).

DyLLM’s key empirical claim is temporal sparsity: across adjacent denoising steps, most tokens’ attention contexts remain almost unchanged. For token $l$5 at layer $l$6 and diffusion step $l$7, the attention context vector is denoted $l$8, and temporal cosine similarity is defined as

$l$9

The observed distribution is described as heavily concentrated near $l_1,\dots,l_S$0 for most tokens, especially in early layers. This motivates freezing: if a token’s context is nearly stationary, its cached activations can be reused rather than recomputed (Lee et al., 9 Mar 2026).

Salient tokens are selected by thresholding cosine similarity:

$l_1,\dots,l_S$1

Thus, if $l_1,\dots,l_S$2, token $l_1,\dots,l_S$3 is salient and recomputed; if $l_1,\dots,l_S$4, token $l_1,\dots,l_S$5 is treated as stable and its cached result is reused (Lee et al., 9 Mar 2026).

The paper provides two theoretical motivations. First, scale invariance after output projection plus RMSNorm:

$l_1,\dots,l_S$6

Second, an error bound via directional alignment:

$l_1,\dots,l_S$7

where

$l_1,\dots,l_S$8

These expressions are used to justify cosine similarity as a proxy for whether a token can be safely frozen (Lee et al., 9 Mar 2026).

DyLLM caches not only keys and values but also attention context and FFN outputs:

$l_1,\dots,l_S$9

Its execution is split into FullStep and SparseStep. For the first $s<S$0 steps, set to 4, the model performs a normal forward pass and populates caches. After warmup, SparseStep recomputes only salient tokens. The procedure includes computing queries for all tokens,

$s<S$1

reusing cached $s<S$2 for all tokens, recomputing $s<S$3 only for salient indices, and forming

$s<S$4

Exact attention is computed only for salient-token rows,

$s<S$5

while non-salient updates are approximated by

$s<S$6

with

$s<S$7

$s<S$8

and then

$s<S$9

Output projection and FFN are applied only to salient tokens,

$$\mathbf{z}^s = \mathrm{Softmax}(\mathrm{MLP}(\mathbf{X})) \in \mathbb{R}^{N \times 2},$$0

$$\mathbf{z}^s = \mathrm{Softmax}(\mathrm{MLP}(\mathbf{X})) \in \mathbb{R}^{N \times 2},$$1

while non-salient tokens reuse

$$\mathbf{z}^s = \mathrm{Softmax}(\mathrm{MLP}(\mathbf{X})) \in \mathbb{R}^{N \times 2},$$2

The stated big idea is that only the salient subset is recomputed through attention and FFN, while the remainder of the sequence is effectively frozen and pulled from cache (Lee et al., 9 Mar 2026).

5. Mathematical interpretation and unifying abstractions

Both ToFe and DyLLM rest on a shared structural premise: many token updates are small enough that exact recomputation is unnecessary at every layer or step. In the ViT case, this is expressed through the approximation of residual change $$\mathbf{z}^s = \mathrm{Softmax}(\mathrm{MLP}(\mathbf{X})) \in \mathbb{R}^{N \times 2},$$3 for frozen tokens. In the diffusion-language-model case, it is expressed through the temporal stability of attention context vectors and the approximation of context updates using sparse value changes (Zhang et al., 22 Jul 2025, Lee et al., 9 Mar 2026).

DyLLM makes the decomposition explicit. It writes

$$\mathbf{z}^s = \mathrm{Softmax}(\mathrm{MLP}(\mathbf{X})) \in \mathbb{R}^{N \times 2},$$4

and expands the attention context change as

$$\mathbf{z}^s = \mathrm{Softmax}(\mathrm{MLP}(\mathbf{X})) \in \mathbb{R}^{N \times 2},$$5

This yields two update channels: $$\mathbf{z}^s = \mathrm{Softmax}(\mathrm{MLP}(\mathbf{X})) \in \mathbb{R}^{N \times 2},$$6, representing content changes in updated tokens, and $$\mathbf{z}^s = \mathrm{Softmax}(\mathrm{MLP}(\mathbf{X})) \in \mathbb{R}^{N \times 2},$$7, representing rerouting of attention weights. DyLLM assumes that for non-salient tokens, $$\mathbf{z}^s = \mathrm{Softmax}(\mathrm{MLP}(\mathbf{X})) \in \mathbb{R}^{N \times 2},$$8 is negligible, so the context update can be approximated by propagating only the sparse $$\mathbf{z}^s = \mathrm{Softmax}(\mathrm{MLP}(\mathbf{X})) \in \mathbb{R}^{N \times 2},$$9 through cached attention weights (Lee et al., 9 Mar 2026).

This suggests a useful unifying interpretation. In ToFe-style methods, freezing is justified when either token content evolves smoothly enough to be approximated, or interaction patterns remain stable enough that cached representations remain valid. The ViT version foregrounds depth-wise smoothness across blocks; the MDLM version foregrounds temporal stability across denoising iterations. A plausible implication is that token freezing and reusing is most effective when representational drift is localized to a sparse subset, whether along the depth dimension or the iterative-time dimension.

Another important distinction concerns reversibility. ToFe for vision preserves token identity explicitly through rearrangement and later reuse (Zhang et al., 22 Jul 2025). DyLLM preserves token state implicitly through cached activations and selective recomputation (Lee et al., 9 Mar 2026). In both cases, the system avoids the information loss associated with hard pruning, but the mechanism differs: one reconstructs or refreshes skipped tokens, the other retains prior exact states and updates only where saliency demands it.

6. Empirical behavior, trade-offs, and limitations

The vision-transformer ToFe paper reports that ToFe reduces the computational cost of the LV-ViT model by 50% with less than 2% drop in Top-1 accuracy (Zhang et al., 22 Jul 2025). More specifically, on LV-ViT-S, compute is reduced from 6.6 GFLOPs to 3.3 GFLOPs while Top-1 accuracy is 81.9%, only 1.4 points below the original 83.3%. On LV-ViT-M, compute drops from 12.7 GFLOPs to 6.1 GFLOPs with 82.8% Top-1, only 1.3 points below 84.1%. On DeiT-S, the model reaches 2.0 GFLOPs with 77.7% Top-1, a 2.2 point drop from 79.9% (Zhang et al., 22 Jul 2025).

The same paper emphasizes comparative strength at similar compute. On LV-ViT-S, EViT drops to 75.1%, DynamicViT to 77.9%, AS-ViT to 63.5%, and IdleViT to 79.5%, all below ToFe’s 81.9% (Zhang et al., 22 Jul 2025). Ablations further show that removing token reuse drops LV-ViT-S accuracy from 81.9% to 79.2% at the same FLOPs, which directly supports the claim that reusing frozen tokens matters. Among tested token approximators, a simple bottleneck MLP is reported as best; direct identity reuse and depth-wise convolution are cheaper but less accurate, while using a transformer block is too expensive and reduces the number of tokens that can be retained (Zhang et al., 22 Jul 2025).

DyLLM reports large speedups with little accuracy loss on reasoning and code-generation benchmarks, including GSM8K, MBPP, MATH, and MMLU-pro (Lee et al., 9 Mar 2026). For LLaDA 8B, the reported throughput speedup reaches up to 7.60× on GSM8K, with other tasks around 5.12×–7.60×. For Dream 7B, speedup reaches up to 9.60× and is generally 6.98×–9.60× across benchmarks. Accuracy is described as mostly preserved, and in some settings slightly improved over the original model (Lee et al., 9 Mar 2026).

The trade-off in DyLLM is controlled by the threshold $$\mathbf{M}^{s} = \mathrm{GumbleSoftmax}(\mathbf{z^s}) \in \{0,1\}^{N}.$$0. Lower $$\mathbf{M}^{s} = \mathrm{GumbleSoftmax}(\mathbf{z^s}) \in \{0,1\}^{N}.$$1 yields fewer salient tokens and thus higher speed, but increases the risk of accuracy drop; higher $$\mathbf{M}^{s} = \mathrm{GumbleSoftmax}(\mathbf{z^s}) \in \{0,1\}^{N}.$$2 yields more recomputation and greater safety, but lower speed. The reported typical thresholds are $$\mathbf{M}^{s} = \mathrm{GumbleSoftmax}(\mathbf{z^s}) \in \{0,1\}^{N}.$$3 for LLaDA and $$\mathbf{M}^{s} = \mathrm{GumbleSoftmax}(\mathbf{z^s}) \in \{0,1\}^{N}.$$4 for Dream (Lee et al., 9 Mar 2026). The paper also states that DyLLM scales better with larger parallel decoding degree $$\mathbf{M}^{s} = \mathrm{GumbleSoftmax}(\mathbf{z^s}) \in \{0,1\}^{N}.$$5 than methods that depend on periodic full refreshes.

A concise comparison is useful.

Setting	Mechanism	Reported outcome
ViT / LV-ViT ToFe	Selector + approximator + budget-aware training	50% cost reduction on LV-ViT with less than 2% drop in Top-1 accuracy (Zhang et al., 22 Jul 2025)
MDLM DyLLM	Saliency thresholding + cache reuse + partial attention	Up to 9.60× higher throughput while largely preserving baseline accuracy (Lee et al., 9 Mar 2026)

The limitations are formulation-specific. ToFe adds extra components—the selector, approximator, and budget-aware training—so it is more complex than plain pruning, and gains depend on how well approximation matches the backbone (Zhang et al., 22 Jul 2025). DyLLM is explicitly training-free, but its approximation assumes that for non-salient tokens the $$\mathbf{M}^{s} = \mathrm{GumbleSoftmax}(\mathbf{z^s}) \in \{0,1\}^{N}.$$6 channel is negligible; this suggests that performance depends on the validity of that temporal-sparsity assumption in the target MDLM regime (Lee et al., 9 Mar 2026).

7. Relation to pruning, reuse, and broader efficient-transformer research

ToFe is most clearly distinguished from irreversible token reduction. In the vision setting, the paper explicitly contrasts it with pruning, which directly discards tokens, and with re-organization methods that merge or fuse tokens. Its central claim is that because attention patterns change across depth, early “unimportant” tokens can later become important, especially when shallow [CLS] attention is scattered and not yet a reliable importance signal (Zhang et al., 22 Jul 2025).

DyLLM extends this logic into a different modality and generation paradigm. The paper states directly that DyLLM is essentially a ToFe-style method: both identify tokens that do not meaningfully change, both avoid recomputing those tokens, and both rely on cache reuse to accelerate inference (Lee et al., 9 Mar 2026). The DyLLM-specific elements are that saliency is based on cosine similarity of attention contexts across adjacent denoising steps, recomputation is layer-adaptive, caches include not just KV but also attention context and FFN outputs, and the method includes a partial-attention approximate update designed specifically for masked diffusion LLM inference (Lee et al., 9 Mar 2026).

This relationship helps clarify what ToFe is and is not. It is not synonymous with token pruning, because frozen tokens remain available for later reintegration. It is not merely caching, because token activity is selected adaptively rather than uniformly. It is not restricted to vision, because the same freezing/reusing logic can be instantiated in diffusion LLMs when token updates are temporally sparse. A plausible implication is that ToFe names a reusable systems-and-modeling pattern for conditional computation at token granularity, with different implementations depending on whether the relevant redundancy is depth-local, time-local, or both.

Within efficient-transformer research, ToFe therefore occupies an intermediate position between hard token elimination and full dense recomputation. Its defining feature is reversible omission: tokens can be excluded from expensive processing for a limited interval without being permanently lost. That reversible omission is the common thread linking lagged token reuse in Vision Transformers (Zhang et al., 22 Jul 2025) and saliency-based token freezing in masked diffusion LLMs (Lee et al., 9 Mar 2026).