---
title: Token-Selective Propagation in Deep Models
url: https://www.emergentmind.com/topics/token-selective-propagation-tsp
type: topic
---

# Token-Selective Propagation in Deep Models

Searching arXiv for the cited TSP-related papers to ground the article in current literature.
Token-Selective Propagation (TSP) denotes a family of token-selection and token-routing mechanisms that reduce sequence-level computation by allowing only a subset of tokens, or token-derived states, to participate in expensive downstream operations while attempting to preserve task-relevant information. In the recent literature, the term does not identify a single canonical algorithm. In FastKV, TSP is a late-layer decoder mechanism that propagates only salient context tokens beyond a designated layer while decoupling propagated hidden states from KV-cache retention [2502.01068]. In MambaMixer, the role corresponds to a Selective Token Mixer that gates tokenwise signals before a data-dependent state-space scan [2403.19888]. In vision transformers, related formulations include token pausing and restarting in TPC-ViT [2401.01470], graph-based propagation from pruned to retained tokens in GTP-ViT [2311.03035], and top-\(K\) token selection with layer bypass and re-concatenation in ToSA [2406.08816].

## 1. Scope, terminology, and recurring design pattern

Across these works, TSP addresses a common systems problem: full token participation imposes quadratic attention cost in transformers or otherwise forces all tokens through a uniform computation path. The proposed remedies differ in where the selection signal is computed, whether dropped tokens can re-enter, and whether information is discarded, cached, summarized, or bypassed.

| Paper | Selection signal | Propagation rule |
|---|---|---|
| FastKV | attention from the last \(N_{\mathrm{obs}}\) “window” tokens | top \(\lfloor \rho N_I \rfloor\) tokens plus window tokens are forwarded after \(\ell_{\mathrm{tsp}}\) |
| MambaMixer | \(G=\sigma(\mathrm{Conv}_{1D}(\mathrm{Linear}_1(x)))\) | gated token stream is processed by a selective SSM |
| TPC-ViT | pause probability \(p_i^k\), restart probability \(r_i^k\) | tokens may pause in layer \(i\) and restart later |
| GTP-ViT | score \(\mathrm{Score}_i=\gamma_i\psi_i\) | pruned-token features are propagated to kept neighbors |
| ToSA | predicted next-layer attention maps \(\hat A_{i+1}^h\) | top-\(K\) tokens attend; the rest bypass the layer |

A central empirical premise recurs across the papers: token relevance is depth-dependent rather than static. FastKV states that early decoder layers have highly unstable attention focus, whereas deeper layers exhibit stabilization of critical context [2502.01068]. TPC-ViT explicitly argues that “tokens that are redundant in one layer can be useful in later layers” [2401.01470]. ToSA and GTP-ViT similarly avoid equating non-attended or pruned tokens with useless tokens: ToSA keeps skipped tokens in the representation via bypass, and GTP-ViT injects pruned-token information into retained tokens through graph propagation [2406.08816][2311.03035].

## 2. FastKV: late-layer Token-Selective Propagation in decoder LLMs

In FastKV, TSP is the mechanism that reduces prefill computation by forwarding only salient tokens after a dedicated TSP layer \(\ell_{\mathrm{tsp}}\), while KV cache compression is performed independently at retention rate \(\gamma\) [2502.01068]. The saliency score is computed from the attention received by each token from the most recent \(N_{\mathrm{obs}}\) query positions. For head \(h\) and token \(i\),
\[
S^{\ell_{\mathrm{tsp}},h}_i
=
\mathrm{Pooling}\Biggl(\sum_{n=0}^{N_{\mathrm{obs}}-1}
\mathrm{Att}_{\ell_{\mathrm{tsp}}}[\,h,\;N_I-1-n,\;i+m\,]\Biggr),
\]
and the head-averaged saliency is
\[
S^{\ell_{\mathrm{tsp}}}_i
=
\frac{1}{H}\sum_{h=0}^{H-1} S^{\ell_{\mathrm{tsp}},h}_i.
\]
Tokens are ranked by \(S^{\ell_{\mathrm{tsp}}}_i\), and the top \(\lfloor \rho N_I \rfloor\) tokens are selected, with all \(N_{\mathrm{obs}}\) window tokens always included in the propagation set.

The TSP layer index is chosen as the earliest layer at which TSP remains close to the full-context baseline in the final hidden states. With \(H_i\) as the full-context final hidden state for calibration input \(i\), and \(H'_{\ell,i}\) the counterpart obtained when TSP is applied at layer \(\ell\), FastKV chooses
\[
\ell_{\mathrm{tsp}}
=
\arg\min_{\ell \le L_{\max}}
\frac{1}{N}\sum_{i=1}^N \|H_i - H'_{\ell,i}\|_2^2,
\]
subject to \(\ell\) being as small as possible. In practice, the sweep is performed in middle layers, for example \(\ell \approx 15\) in a 32-layer model.

Algorithmically, the model processes the full prompt of length \(N_I\) up to \(\ell_{\mathrm{tsp}}\). At every layer, KVCompress is applied to \((K^\ell,V^\ell)\) at retention rate \(\gamma\). When \(\ell=\ell_{\mathrm{tsp}}\), TSP computes \(S_i\), selects the propagation set, and HiddenCompress forms a reduced hidden state of length approximately \(\rho N_I\). All later layers then operate only on that reduced hidden state. The resulting prefill complexity drops from
\[
O(L\cdot N_I^2)
\]
to
\[
O\bigl(\ell_{\mathrm{tsp}}\cdot N_I^2 + (L-\ell_{\mathrm{tsp}})\cdot (\rho N_I)^2\bigr).
\]
The per-token decoding cost becomes \(O(H\gamma N d)\) rather than \(O(HNd)\).

The two hyper-parameters serve different budgets. The TSP rate \(\rho\in(0,1]\) sets the fraction of propagated tokens and therefore the prefill compute rate; smaller \(\rho\) increases prefill speedup but risks discarding critical tokens. The KV retention rate \(\gamma\in(0,1]\) sets the fraction of KV pairs stored per layer for decoding; smaller \(\gamma\) reduces memory bandwidth and generation cost. FastKV emphasizes that \(\rho\) and \(\gamma\) are independently tunable, unlike GemFilter and PyramidInfer where \(\gamma=\rho\) by design.

Empirically, on LLaMA-3.1-8B-Instruct with \(N_I=128\mathrm{K}\) tokens, FastKV reports prefill speedup up to \(1.82\times\) at \(\rho=20\%\) and \(\ell_{\mathrm{tsp}}=15\), decoding speedup up to \(2.87\times\) at \(\gamma=10\%\), and LongBench average accuracy within \(1\%\) of the full-context baseline, with one example reported as \(50.19 \rightarrow 49.07\) at \(\rho=20\%,\gamma=20\%\). Ablations show that accuracy plateaus around \(\rho\approx 20\%\), that large accuracy loss occurs if \(\ell_{\mathrm{tsp}}<12\), and that prefill savings vanish if \(\ell_{\mathrm{tsp}}>20\).

## 3. MambaMixer: selective token propagation through a data-dependent SSM

In MambaMixer, the component corresponding to TSP is the Selective Token Mixer, which operates on an input tensor \(x\in\mathbb R^{B\times L\times D}\) with expansion dimension \(E\) and uses data-dependent weights together with a selective SSM [2403.19888]. Its unidirectional layer-\(\ell\) update is:
\[
U=\mathrm{Linear}_1(x)\in\mathbb R^{B\times L\times E},
\]
\[
G=\sigma(\mathrm{Conv}_{1D}(U))\in\mathbb R^{B\times L\times E},
\]
\[
M=\mathrm{MLP}(x)=\mathrm{Linear}_2(\mathrm{GeLU}(\mathrm{Linear}_5(x)))\in\mathbb R^{B\times L\times E},
\]
\[
B_t=\mathrm{Linear}_3(x)\in\mathbb R^{B\times L\times E},
\quad
C_t=\mathrm{Linear}_4(x)\in\mathbb R^{B\times L\times E},
\]
\[
\Delta_t=\phi(\mathrm{Linear}_\Delta(x))\in\mathbb R^{B\times L\times E},
\]
\[
H=\mathrm{SSM}_{\bar A,B_t,C_t,\Delta_t}(G)\in\mathbb R^{B\times L\times E},
\]
\[
Y_{\mathrm{Token}}=H\odot M\in\mathbb R^{B\times L\times E}.
\]
Here the primary selection signal is the gate tensor
\[
G=\mathrm{sigmoid}(\mathrm{Conv}_{1D}(\mathrm{Linear}_1(x))),
\]
while \(B_t\), \(C_t\), and \(\Delta_t\) provide data-dependent selection of the continuous-time SSM. The stated interpretation is that tokens are first scored by depth-wise \(1\)D convolution plus sigmoid, then those scores are used to mask or compress the sequence before the selective SSM, and the MLP path adds non-recurrent mixing.

MambaMixer does not isolate token selection from the rest of the block. Instead, the Selective Token Mixer and the Selective Channel Mixer are coupled by weighted averaging across layers. If \(Y_{\mathrm{Token}}^{(\ell)}\) and \(Y_{\mathrm{Channel}}^{(\ell)}\) denote the outputs of the \(\ell\)-th token and channel mixers, then the next inputs are
\[
x_{\mathrm{Token}}^{(\ell+1)}
=
\sum_{i=0}^{\ell}\alpha_{\ell+1,i}Y_{\mathrm{Token}}^{(i)}
+
\sum_{i=0}^{\ell}\beta_{\ell+1,i}Y_{\mathrm{Channel}}^{(i)},
\]
\[
x_{\mathrm{Channel}}^{(\ell+1)}
=
\sum_{i=0}^{\ell+1}\theta_{\ell+1,i}Y_{\mathrm{Token}}^{(i)}
+
\sum_{i=0}^{\ell}\gamma_{\ell+1,i}Y_{\mathrm{Channel}}^{(i)},
\]
where \(\alpha,\beta,\theta,\gamma\) are scalar learnable weights and \(Y_{\mathrm{Token}}^{(0)}=Y_{\mathrm{Channel}}^{(0)}=x^{(0)}\). This gives later layers direct access to early features and to any earlier token- or channel-filtered representation.

The per-block complexity is reported as approximately \(O(BLE+EN)\) for the Token Mixer and \(O(BNE+EL)\) for the Channel Mixer, for total time \(O(E\cdot B\cdot (L+N))\). Working memory per block is approximately \(O(BLE + BNE + LN + \ell^2)\), with the text emphasizing linear dependence on \(L\) and \(N\). The implementation uses the GPU-friendly parallel associative scan for the recurrent S6 block, and memory is described as scaling roughly linearly in \(L\) and \(N\), without an \(L^2\) term as in self-attention.

As a proof of concept, MambaMixer instantiates Vision MambaMixer (ViM2) and Time Series MambaMixer (TSM2). On ImageNet-1K, ViM2-Tiny is reported at approximately \(20\)M parameters and \(4.3\)G FLOPs with \(82.7\%\) top-1 accuracy, ViM2-Small at approximately \(43\)M and \(8.1\)G with \(83.7\%\), and ViM2-Base at approximately \(75\)M and \(13.5\)G with \(83.9\%\). The comparison quoted in the summary includes ViM-Small at \(26\)M and approximately \(5\)G with \(80.5\%\), and swin-Tiny at \(29\)M and approximately \(4.1\)G with \(81.3\%\). For long-series forecasting, TSM2 is reported to outperform prior transformers and MLP mixers on \(8\) public benchmarks, often reducing MSE by \(5\)–\(15\%\) while using \(30\)–\(50\%\) fewer FLOPs.

## 4. TPC-ViT: pause, restart, and cumulative halting

TPC-ViT formulates TSP as an adaptive halting mechanism in which a token can be paused in one layer and restarted in a later layer, rather than being discarded permanently [2401.01470]. For token \(k\) at layer \(i\), two Bernoulli variables are introduced: \(X_i^k\in\{0,1\}\) for pause, with \(\mathrm{Prob}(X_i^k=1)=p_i^k\), and \(Y_i^k\in\{0,1\}\) for restart, with \(\mathrm{Prob}(Y_i^k=1)=r_i^k\). The break probability is
\[
b_i^k
=
\mathrm{Prob}(X_i^k=1,Y_i^k=0)
=
p_i^k(1-r_i^k).
\]

The probabilities \(p_i^k\) and \(r_i^k\) are derived from two scalar components of the token representation \(t_i^k\in\mathbb R^d\), with no new parameters except learned scale \(y\) and shift \(\beta\):
\[
p_i^k=\sigma(y\cdot t_i^{k,[1]}+\beta),
\qquad
r_i^k=\sigma(y\cdot t_i^{k,[2]}+\beta).
\]
Cumulative break determines halting. Once
\[
\sum_{j=1}^{i} b_j^k \ge 1-\epsilon,
\]
token \(k\) is removed at layer \(i\).

TPC-ViT adds two stabilizing components. First, a smoothing mechanism regularizes \(b_i^k\) toward the layerwise mean over the active token set \(K_i\). With
\[
\mu_i=\frac{1}{|K_i|}\sum_{\ell\in K_i} b_i^\ell,
\]
the smoothed break probability is
\[
\tilde b_i^k = \alpha b_i^k + (1-\alpha)\mu_i,
\qquad \alpha\in[0,1].
\]
An equivalent penalty is
\[
L_{\mathrm{smooth}}
=
\lambda_s\sum_{i,k\in K_i}(b_i^k-\mu_i)^2.
\]
Second, a model stabilizer injects local bias by sparse attention: for each query \(q_i\), only the \(k_s\) nearest keys by Euclidean distance are attended, with
\[
A_i=\mathrm{softmax}\Bigl([q_i\cdot k_j/\sqrt d]_{j\in S_i(q)}\Bigr),
\qquad
V_i=A_i\cdot [v_j]_{j\in S_i(q)}.
\]

Training combines classification loss, a ponder loss that encourages early stopping, a layer-level distribution loss that encourages a bell-shaped halting distribution, and the smoothing regularizer:
\[
L = L_{\mathrm{task}} + \lambda_p L_{\mathrm{ponder}} + \lambda_d L_{\mathrm{dist}} + \lambda_s L_{\mathrm{smooth}}.
\]
The simplified algorithm initializes the token mask, cumulative break, and remainder; applies StabilizerAttention at each layer; computes \(p_i^k\), \(r_i^k\), and \(b_i^k\); smooths them; updates cumulative break; and uses the resulting mask in the following self-attention.

On ImageNet-1K, the reported results include DeiT-S \(\rightarrow\) TPC-DeiT-S, with top-1 accuracy \(79.8\%\rightarrow 80.8\%\), GFLOPs \(4.6\rightarrow 2.8\), throughput at \(224^2\) \(26\mathrm{K}\,\mathrm{img/s}\rightarrow 65\mathrm{K}\,\mathrm{img/s}\), and average tokens per layer from \(197\) to approximately \(100\). For DeiT-T, accuracy is reported as \(72.2\rightarrow 73.0\) and GFLOPs as \(1.3\rightarrow 0.6\). For Swin-T, the summary reports \(81.2\rightarrow 81.2\) with GFLOPs \(4.5\rightarrow 3.6\). For LV-ViT-S, it reports \(83.2\rightarrow 83.2\) with GFLOPs \(6.6\rightarrow 4.3\). Ablations show that on DeiT-S, the stabilizer improves top-1 from \(79.7\%\) to \(80.1\%\), and the token distribution regularizer improves TPC-DeiT-S from \(80.1\%\) to \(80.8\%\). On DeiT-T, \(k_s=100\) yields the best reported top-1 accuracy of \(73.0\%\), compared with \(72.4\%\) at \(k_s=60\).

## 5. GTP-ViT: graph-based propagation after token selection

GTP-ViT instantiates token propagation as graph summarization: less significant tokens are not simply removed, but their features are propagated to spatially and semantically connected retained tokens [2311.03035]. After patch embedding, the input is \(X\in\mathbb R^{N\times C}\), where each token \(x_i\in\mathbb R^C\) becomes a node \(v_i\) in a weighted graph \(G=(V,E)\).

Two sparse graphs are built and then combined. The spatial graph uses adjacency \(A^{\mathrm{spatial}}\) with \(A^{\mathrm{spatial}}_{ij}=1\) when tokens \(i\) and \(j\) correspond to spatially adjacent patches and \(0\) otherwise. The semantic graph uses cosine similarity in the initial embedding \(X_0\):
\[
\mathrm{CosSim}(x_i,x_j)=\frac{x_i\cdot x_j}{\|x_i\|\|x_j\|}.
\]
If \(T_i\) is the \(M\)-th largest cosine similarity between \(x_i\) and all other tokens, then
\[
A^{\mathrm{semantic}}_{ij}
=
\begin{cases}
1 & \text{if } \mathrm{CosSim}(x_i,x_j)\ge T_i \text{ and } i\ne j,\\
0 & \text{otherwise.}
\end{cases}
\]
The mixed adjacency is
\[
A = A^{\mathrm{spatial}} \lor A^{\mathrm{semantic}},
\qquad
\hat A = D^{-1/2} A D^{-1/2},
\]
with \(D_{ii}=\sum_j A_{ij}\).

Token importance is computed from the multi-head attention map in a transformer block. With a permutation-invariant reducer \(\oplus\) across heads, regeneration difficulty is
\[
\gamma_i=\oplus_{\mathrm{heads}}(A_{ii}),
\]
broadcasting ability is
\[
\psi_i=\oplus_{\mathrm{heads}}\Bigl(\sum_{j\ne i}A_{ij}\Bigr),
\]
and the combined score is
\[
\mathrm{Score}_i=\gamma_i\times \psi_i.
\]
If \(P\) tokens are to be pruned from \(N\), GTP sorts tokens by \(\mathrm{Score}_i\), excludes the \([CLS]\) token, retains the top \(N-P\) tokens as \(X^k\in\mathbb R^{(N-P)\times C}\), and marks the bottom \(P\) as propagated tokens \(X^p\in\mathbb R^{P\times C}\).

Propagation is then a single sparse-matrix update. If \(\hat A^p\in\mathbb R^{(N-P)\times P}\) is the kept-by-pruned block of \(\hat A\), the summarized tokens are
\[
X^s = X^k + \alpha\cdot \hat A^p X^p,
\]
with \(\alpha>0\) as a small hyper-parameter. This avoids one-to-one token matching: each kept token can absorb information from all adjacent pruned tokens.

The complexity analysis contrasts GTP with vanilla MHSA and token merging. Vanilla MHSA costs \(O(N^2C + NC^2)\) per layer. After pruning to \(N-P\) tokens, self-attention costs
\[
O((N-P)^2C + (N-P)C^2),
\]
plus propagation cost
\[
O((N-P)\cdot P\cdot C).
\]
The summary states that token-selection cost \(O(HN)\) is negligible and that, with \(P\ll N\), GTP reduces quadratic attention cost by approximately \(2NP C\) per layer.

On ImageNet-1K, for DeiT-S with baseline top-1 \(79.8\%\), pruning \(P=8\) tokens per layer yields \(3.4\) GMACs, a \(26\%\) reduction from \(4.6\), throughput \(+25\%\), and top-1 \(79.5\%\), corresponding to \(-0.3\%\). For DeiT-B with baseline \(81.8\%\), the same \(P=8\) yields \(13.1\) GMACs, \(26\%\) reduction, throughput \(+28\%\), and top-1 \(81.5\%\), also \(-0.3\%\). The summary further states that at approximately \(2.6\) GMACs, GTP on DeiT-S achieves \(79.1\%\) without finetuning, outperforming ToMe at \(78.6\%\) and EViT at \(78.9\%\) at similar or higher throughput.

## 6. ToSA: selective attention with bypassed tokens and full-token reconstruction

ToSA implements token selectivity on pairs of consecutive ViT layers, where the first layer remains standard and the second is replaced by a ToSA transformer layer [2406.08816]. Between them, a token selector takes the unnormalized attention maps
\[
B_i^h = Q_i^h (K_i^h)^\top \in \mathbb R^{L\times L},
\]
predicts the next-layer attention maps \(\hat A_{i+1}^h\), and extracts token-importance scores from those predictions.

For each head \(h\),
\[
\hat A_{i+1}^h
=
\mathrm{LogSoftmax}\Bigl(
\mathrm{Conv}_2\bigl(\mathrm{ReLU}(\mathrm{Conv}_1(B_i^h))\bigr)
\Bigr)\in\mathbb R^{L\times L}.
\]
For token index \(j\), the importance score is the column sum
\[
s_j^h=\sum_{k=1}^L \hat A_{i+1}^h(k,j),
\qquad
s_j=\sum_{h=1}^H s_j^h,
\]
after which the top
\[
K=\lceil r\cdot L\rceil
\]
tokens are selected as attention tokens. The remaining tokens are skip tokens.

If \(X_{i+1}\in\mathbb R^{L\times D}\) is the output of the first layer, the selected and skipped subsets are
\[
X_{i+1}^a=X_{i+1}[\mathcal I_a,:]\in\mathbb R^{K\times D},
\qquad
X_{i+1}^s=X_{i+1}[\mathcal I_s,:]\in\mathbb R^{(L-K)\times D}.
\]
Only \(X_{i+1}^a\) participates in the second layer’s self-attention:
\[
Q',K',V' = W_QX_{i+1}^a,\;W_KX_{i+1}^a,\;W_VX_{i+1}^a,
\]
\[
A'=\mathrm{softmax}(Q'K'^\top/\sqrt D),
\qquad
X_{i+2}^a=A'V'.
\]
The skip tokens bypass the layer unchanged,
\[
X_{i+2}^s = X_{i+1}^s,
\]
and the full token set is reconstructed by concatenation:
\[
X_{i+2}
=
W_O\,\mathrm{Concat}\bigl(X_{i+2}^a,\;X_{i+2}^s\bigr)\in\mathbb R^{L\times D}.
\]
This preservation of one feature per original patch is the stated reason ToSA remains compatible with dense prediction tasks.

Training of the selector uses a frozen pretrained ViT to provide ground-truth attention maps \(A_{i+1}^h\), with selector loss
\[
\mathcal L_{i,i+1}
=
\sum_{h=1}^H
\mathrm{KL}\bigl(\hat A_{i+1}^h \,\|\, A_{i+1}^h\bigr).
\]
Only the small convolutional selector is updated at that stage; the full ToSA-modified network is then finetuned end-to-end.

The FLOPs reduction follows directly from replacing \(L\)-token attention by \(K=rL\)-token attention in each ToSA layer. A standard multi-head self-attention layer costs approximately \(2L^2D\), whereas a ToSA layer costs approximately \(2r^2L^2D\). If ToSA is applied to \(M\) out of \(N\) self-attention layers, the total self-attention cost becomes
\[
2L^2D\bigl[(N-M)+Mr^2\bigr],
\]
for a relative reduction
\[
\frac{M(1-r^2)}{N}.
\]
The summary gives an example for DeiT-Tiny with \(N=12\), \(M=5\), and \(r=0.8\), yielding a \(15\%\) self-attention FLOPs reduction.

The reported empirical configuration applies ToSA to DeiT-Tiny layers \(2,4,6,8,10\) with \(r=0.8\). On ImageNet-1K, the baseline DeiT-Tiny has top-1 \(72.2\%\) and \(1.3\) GFlops, whereas the ToSA version has top-1 \(73.9\%\) and \(1.0\) GFlops, corresponding to \(+1.7\) points and \(-25.8\%\) GFlops. On NYU Depth V2 with a NeWCRFs decoder, the baseline DeiT-Tiny encoder yields \(\mathrm{Abs\ Rel}=0.101\) and \(\delta<1.25=0.896\), while the ToSA encoder yields \(\mathrm{Abs\ Rel}=0.103\) and \(\delta<1.25=0.891\). The summary states that KITTI results are virtually identical.

## 7. Comparative interpretation and points of clarification

The literature shows that TSP is not synonymous with a single operation such as irreversible token pruning. FastKV performs one-time token selection after full-context early processing and continues later layers on a reduced hidden state, while separately compressing the KV cache [2502.01068]. TPC-ViT allows pausing and later restarting, making token exclusion explicitly reversible at the architectural level [2401.01470]. GTP-ViT propagates features from removed tokens to retained tokens rather than bypassing them unchanged [2311.03035]. ToSA bypasses skipped tokens and re-concatenates them, so token count in the representation remains constant even when only a subset participates in self-attention [2406.08816]. MambaMixer, by contrast, uses token selection as a gating operation inside a selective SSM rather than as an attention-pruning rule [2403.19888].

A second recurring distinction is between selection for computation and selection for memory. FastKV makes this explicit by decoupling the TSP rate \(\rho\) from the KV retention rate \(\gamma\), arguing that prefill compute reduction need not be tied to the decoding KV budget [2502.01068]. This suggests a broader interpretation of TSP as a resource-allocation mechanism that can target hidden-state propagation, attention participation, state-space scanning, or cache storage, depending on the model family.

A third point concerns when token saliency should be measured. FastKV chooses the earliest layer at which critical-token sets have stabilized, and reports substantial degradation when TSP is applied too early. TPC-ViT states that redundancy at one layer does not imply redundancy at later layers. ToSA predicts next-layer attention from current attention, and GTP-ViT computes importance within each block from attention-derived criteria. Taken together, these results suggest that token selectivity is fundamentally depth-aware, and that methods assuming a fixed early pruning boundary or permanent redundancy may be mismatched to the observed dynamics.

Finally, the surveyed works reflect two distinct preservation strategies once full participation is curtailed. One strategy preserves information by carrying fewer but more salient units forward, as in FastKV and MambaMixer. The other preserves information by redistributing or retaining the representation of omitted tokens, as in GTP-ViT and ToSA. TPC-ViT occupies an intermediate position, since paused tokens are removed from immediate computation yet remain eligible for later restart. This suggests that the phrase “Token-Selective Propagation” functions less as a single named primitive than as a family resemblance term for architectures that make token flow conditional, depth-dependent, and explicitly tied to efficiency.

Source: https://www.emergentmind.com/topics/token-selective-propagation-tsp