Self-adaptive Attention Allocation (SAA)

• Self-adaptive Attention Allocation (SAA) is a family of methods that modulate attention distribution in neural networks based on contextual relevance, structural regimes, and task states.
• It incorporates various mechanisms such as per-step temperature control, adaptive span gating, and sparse graph routing to optimize both efficiency and performance.
• SAA has broad applications across translation, long-context inference, vision-language models, recommendation systems, and robotics, addressing challenges like attention overload and representational collapse.

Self-adaptive Attention Allocation (SAA) denotes a family of mechanisms in which attention is not treated as a fixed dense operation with uniform behavior across timesteps, heads, layers, or inputs, but is modulated according to contextual relevance, structural regime, or task state. In the literature, this principle appears in several distinct forms: per-step temperature control in neural machine translation, learnable attention spans, learned sparse edge construction, head-specific sparsity budgets for long-context inference, adaptive normalization that suppresses irrelevant tokens, spatially masked cross-attention scaling in diffusion-based image generation, pre-LLM token-budget allocation in vision-LLMs, loss-level reweighting in recommendation, and learnable linear-attention replacements in robotics and sequence modeling (Lin et al., 2018, Sukhbaatar et al., 2019, Li et al., 2020, You et al., 10 Dec 2025, Fu et al., 1 Jan 2026, Zhou et al., 22 Jul 2025, Han et al., 16 Jan 2025, Liu et al., 2021, Leal et al., 2023). The unifying idea is that attention should decide not only where to look, but also how sharply, over what span, with what computational budget, and, in some settings, at what training weight.

1. Conceptual scope and historical development

An early explicit formulation appears in neural machine translation, where the central objection was that “the conventional attention mechanism treats the decoding at each time step equally with the same matrix,” even though function words and content words require different attention softness (Lin et al., 2018). That line of work framed SAA as dynamic concentration versus diversion of attention mass. Soon afterward, adaptive-span Transformers reinterpreted the same principle as learning how much historical context each head should consume, thereby allocating receptive field rather than only attention weights (Sukhbaatar et al., 2019). Sparse Adaptive Connection generalized the idea further by learning an input-dependent sparse attention graph under a fixed edge budget, making allocation a routing problem over edges rather than a redistribution within a dense matrix (Li et al., 2020).

Later work broadened the scope in two directions. One direction emphasized efficiency at long context: head-specific sparsity budgets, asymmetric key-query indexing, elastic sparsification, and adaptive node allocation all treat SAA as dynamic selection of the subset over which attention is computed (You et al., 10 Dec 2025, Mazaré et al., 12 Feb 2025, Fu et al., 1 Jan 2026, Kiruluta, 1 Jun 2025). The other direction emphasized modality-specific conflicts: in unified image generation, SAA became per–sub-instruction scaling of cross-attention under image-token interference; in visual-LLMs, it became adaptive splitting of a token budget between visual saliency and text-to-image similarity; in recommendation, it became context-conditioned reweighting of observed loss terms rather than query-key-value attention proper (Zhou et al., 22 Jul 2025, Han et al., 16 Jan 2025, Liu et al., 2021).

A later unifying account argues that two canonical Transformer pathologies—representational collapse and attention sink—share a common root in improper attention allocation. It distinguishes attention overload, where too many tokens receive comparable high weights, from attention underload, where no token is semantically relevant but normalization still forces nonzero mass, producing spurious focus such as sink behavior (Fu et al., 1 Jan 2026). This perspective suggests that SAA is not a single algorithmic family but a broader design principle for matching the structure of attention to the signal structure of the task.

Mechanism class	Representative paper	Allocation target
Temperature-scaled attention	(Lin et al., 2018)	Per decoding step
Learnable span gating	(Sukhbaatar et al., 2019)	Per head, optionally per token
Sparse graph construction	(Li et al., 2020)	Per edge under a budget
Context-adaptive sparse retention	(You et al., 10 Dec 2025)	Per head, per block, per input
Asymmetric partition routing	(Mazaré et al., 12 Feb 2025)	Per query, over key partitions
Elastic normalization	(Fu et al., 1 Jan 2026)	Per head and query
Spatial cross-attention scaling	(Zhou et al., 22 Jul 2025)	Per sub-instruction token group
Cross-modality token budgeting	(Han et al., 16 Jan 2025)	Per visual token under fixed budget
Loss-level weighting	(Liu et al., 2021)	Per observed user–item term

2. Mathematical forms of allocation

The most direct formulation of SAA modifies softmax concentration. In Self-Adaptive Control of Temperature (SACT), attention weights are computed as

$$\tilde{\alpha}_{t,i} = \frac{\exp\left(\tau_t^{-1} e_{t,i}\right)}{\sum_{j=1}^{n}\exp\left(\tau_t^{-1} e_{t,j}\right)},$$

with context vector

$$\tilde{c}_t = \sum_{i=1}^{n}\tilde{\alpha}_{t,i} h_i.$$

The temperature is learned at each decoding step from the previous context and current decoder output,

$$\beta_t = \tanh\left(W_c \tilde{c}_{t-1} + U_s s_t\right), \qquad \tau_t = \lambda^{\beta_t},$$

so that $\tau_t \in (1/\lambda,\lambda)$ (Lin et al., 2018). Small $\tau_t$ sharpens attention; large $\tau_t$ flattens it.

Adaptive Attention Span changes a different variable: not the entropy of the attention distribution, but the accessible history. It introduces a soft multiplicative mask

$$m_z(x)=\min\left[\max\left[\frac{1}{R}\left(R+z-x\right), 0\right], 1\right],$$

which gates attention as a function of distance $x=t-r$ and learned span parameter $z$. The masked weights become

$$a_{tr} = \frac{m_z(t-r)\exp\left(s_{tr}\right)}{\sum_{q=t-S}^{t-1}m_z(t-q)\exp\left(s_{tq}\right)},$$

and span length is regularized by

$$\tilde{c}_t = \sum_{i=1}^{n}\tilde{\alpha}_{t,i} h_i.$$0

This makes attention allocation a problem of differentiable receptive-field control (Sukhbaatar et al., 2019).

Sparse routing formulations replace dense matrices by learned or selected subgraphs. In Sparse Adaptive Connection, attention is restricted to neighbors in a learned edge set $$\tilde{c}_t = \sum_{i=1}^{n}\tilde{\alpha}_{t,i} h_i.$$1, so the update is

$$\tilde{c}_t = \sum_{i=1}^{n}\tilde{\alpha}_{t,i} h_i.$$2

with a hard budget of $$\tilde{c}_t = \sum_{i=1}^{n}\tilde{\alpha}_{t,i} h_i.$$3 directed edges per layer learned by a recurrent Edge Predictor and optimized via REINFORCE (Li et al., 2020). In TCA-Attention, the allocation target is a subset of global and local tokens:

$$\tilde{c}_t = \sum_{i=1}^{n}\tilde{\alpha}_{t,i} h_i.$$4

where the global subset is selected online by a redundancy metric and the local subset is the most recent $$\tilde{c}_t = \sum_{i=1}^{n}\tilde{\alpha}_{t,i} h_i.$$5 tokens (You et al., 10 Dec 2025). Saap applies a related idea at inference time, but with asymmetric partitions for keys and queries; masked attention is

$$\tilde{c}_t = \sum_{i=1}^{n}\tilde{\alpha}_{t,i} h_i.$$6

where $$\tilde{c}_t = \sum_{i=1}^{n}\tilde{\alpha}_{t,i} h_i.$$7 sets nonselected partitions to $$\tilde{c}_t = \sum_{i=1}^{n}\tilde{\alpha}_{t,i} h_i.$$8 (Mazaré et al., 12 Feb 2025).

A separate line changes normalization itself. Lazy Attention keeps a RoPE-based score function with learned distance-dependent head biases, then applies

$$\tilde{c}_t = \sum_{i=1}^{n}\tilde{\alpha}_{t,i} h_i.$$9

which can exactly zero out negligible weights and thereby address attention underload (Fu et al., 1 Jan 2026). In another replacement-style formulation, adaptive two-sided short-time Laplace transforms build a relevance matrix

$$\beta_t = \tanh\left(W_c \tilde{c}_{t-1} + U_s s_t\right), \qquad \tau_t = \lambda^{\beta_t},$$0

where each learned Laplace node $$\beta_t = \tanh\left(W_c \tilde{c}_{t-1} + U_s s_t\right), \qquad \tau_t = \lambda^{\beta_t},$$1 determines a decay rate, oscillatory frequency, and effective half-life

$$\beta_t = \tanh\left(W_c \tilde{c}_{t-1} + U_s s_t\right), \qquad \tau_t = \lambda^{\beta_t},$$2

turning SAA into adaptive allocation over interpretable temporal kernels rather than explicit attention heads (Kiruluta, 1 Jun 2025).

Finally, not all SAA mechanisms operate at the attention-matrix level. In SAM for recommendation, the allocation variable is a context-conditioned weight on each observed squared-error term:

$$\beta_t = \tanh\left(W_c \tilde{c}_{t-1} + U_s s_t\right), \qquad \tau_t = \lambda^{\beta_t},$$3

with $$\beta_t = \tanh\left(W_c \tilde{c}_{t-1} + U_s s_t\right), \qquad \tau_t = \lambda^{\beta_t},$$4 and the auxiliary objective

$$\beta_t = \tanh\left(W_c \tilde{c}_{t-1} + U_s s_t\right), \qquad \tau_t = \lambda^{\beta_t},$$5

Here SAA is realized as adaptive loss reweighting driven by item text rather than as token-token attention (Liu et al., 2021).

3. Sequence modeling and translation

The original neural machine translation formulation of SAA is explicitly linguistic. SACT computes a temperature $$\beta_t = \tanh\left(W_c \tilde{c}_{t-1} + U_s s_t\right), \qquad \tau_t = \lambda^{\beta_t},$$6 from the previous attention-derived context and the current decoder state, then uses that temperature to sharpen or soften attention according to the current decoding need (Lin et al., 2018). The reported qualitative behavior is highly structured: higher $$\beta_t = \tanh\left(W_c \tilde{c}_{t-1} + U_s s_t\right), \qquad \tau_t = \lambda^{\beta_t},$$7 appears for function and syntax-relevant words such as “to,” “from,” “they,” and punctuation, whereas lower $$\beta_t = \tanh\left(W_c \tilde{c}_{t-1} + U_s s_t\right), \qquad \tau_t = \lambda^{\beta_t},$$8 appears for content-bearing words and phrases with direct correspondences such as “pay attention,” “nuclear,” “paris,” “xinhua,” “wang,” and “french.” On Chinese–English translation, Seq2Seq + SACT improved average BLEU from 34.91 to 37.85, a gain of +2.94 BLEU over Seq2Seq; on English–Vietnamese, Seq2Seq + SACT improved BLEU from 26.93 to 29.12, a gain of +2.19 over Seq2Seq and +1.43 over NPMT (Lin et al., 2018). A fixed-temperature ablation on English–Vietnamese further reported that adaptive SACT reached 28.54 in the comparison figure, while all fixed $$\beta_t = \tanh\left(W_c \tilde{c}_{t-1} + U_s s_t\right), \qquad \tau_t = \lambda^{\beta_t},$$9 underperformed.

Adaptive Attention Span addresses a different bottleneck in sequence modeling: fixed context windows waste compute on heads that only need local information while starving heads that need long-range dependencies (Sukhbaatar et al., 2019). By learning per-head spans and regularizing them toward short values, it extends maximum context to 8k characters while keeping average span small. On text8, small Adaptive-Span with $\tau_t \in (1/\lambda,\lambda)$0 achieved test 1.11 bpc and large Adaptive-Span achieved 1.07 bpc; on enwik8, large Adaptive-Span achieved 0.98 bpc, surpassing Transformer-XL’s 0.99 test (Sukhbaatar et al., 2019). The learned span distribution is strongly stratified: with $\tau_t \in (1/\lambda,\lambda)$1, most heads in lower layers converge to the minimum span at $\tau_t \in (1/\lambda,\lambda)$2, while a small number of higher-layer heads grow to spans of several thousand. Average span remains 314 in small models and 245 in large models even when the limit is 8192.

Sparse Adaptive Connection generalizes the same principle from span selection to graph construction. Instead of truncating by distance, it learns a sparse attention graph with $\tau_t \in (1/\lambda,\lambda)$3 edges per layer through a recurrent policy and REINFORCE, so that attention complexity becomes $\tau_t \in (1/\lambda,\lambda)$4 rather than $\tau_t \in (1/\lambda,\lambda)$5 (Li et al., 2020). This turns previous fixed sparse patterns into constrained special cases. On WMT14 En–De, SAC Large with 18 blocks and dependency-based distance encodings reached test BLEU 29.5, surpassing Transformer big at 28.4; in character-level language modeling it achieved 1.00 on Enwiki8 and 1.06 on Text8 in the head-adaptive setting (Li et al., 2020). The important conceptual move is that SAA here is neither entropy control nor span control, but allocation of a finite edge budget to the most task-relevant connections.

4. Long-context language modeling and efficient inference

In long-context inference, SAA increasingly becomes a resource-allocation problem over memory bandwidth, KV cache growth, and approximate attention error. TCA-Attention is a training-free sparse attention mechanism that performs an offline calibration phase to determine head-specific sparsity budgets and an online token-selection phase driven by a redundancy metric (You et al., 10 Dec 2025). The method retains an adaptive global subset plus a fixed local window, and the retained attention mass threshold $\tau_t \in (1/\lambda,\lambda)$6 gives a bounded approximation guarantee,

$\tau_t \in (1/\lambda,\lambda)$7

with $\tau_t \in (1/\lambda,\lambda)$8 under calibration. Empirically, at 128K context length it achieved a 2.8$\tau_t \in (1/\lambda,\lambda)$9 prefilling speedup, a 2.1$\tau_t$0 decoding speedup, and 61% KV cache reduction while maintaining performance comparable to full attention across LongBench-E, RULER, short-context benchmarks, OlympiadBench, and MT-Bench-101 (You et al., 10 Dec 2025).

Saap also targets inference-time sparsification, but its diagnosis is different: standard symmetric partitioning fails because keys and queries follow different distributions and RoPE hinders bucket assignment (Mazaré et al., 12 Feb 2025). It therefore uses de-roped spherical k-means on keys and a separately trained small query classifier to route each query to a content-adaptive subset of key partitions. On a long-context Llama 3.1-8B model with sequences ranging from 100k to 500k tokens, Saap typically reduced by a factor of 20 the fraction of memory that needs to be looked up, translating to a time saving of 60% compared with FlashAttention-v2 (Mazaré et al., 12 Feb 2025). On Needle-in-a-Haystack, Saap reached 100% accuracy up to 128k with $\tau_t$1 and 100% at 500k with $\tau_t$2.

Lazy Attention shifts the discussion from efficiency alone to failure modes intrinsic to attention allocation. It argues that representational collapse and attention sink reflect attention overload and attention underload, respectively, and addresses them with two components: positional discrimination across heads and dimensions, and Elastic-Softmax, which applies a post-softmax shift-and-ReLU filter (Fu et al., 1 Jan 2026). The reported results show competitive performance on FineWeb-Edu pretraining across eight downstream tasks while reaching up to 59.58% attention sparsity and reducing sink ratio dramatically, with one reported best ablation pushing sink ratio down to approximately 0.18% (Fu et al., 1 Jan 2026). Unlike static sparse masks, this is a content-regime-sensitive redistribution of probability mass within otherwise flexible attention.

Adaptive two-sided Laplace transforms go further by replacing self-attention with learned short-time Laplace nodes (Kiruluta, 1 Jun 2025). Each node learns a decay rate $\tau_t$3, oscillatory frequency $\tau_t$4, and window bandwidth $\tau_t$5, and an adaptive node-allocation mechanism determines the effective number of active nodes $\tau_t$6. On WikiText-103, the adaptive model reported perplexity 23.8 with $\tau_t$7; on NarrativeQA it reported F1 40.5 while streaming 128k contexts; on long Project Gutenberg evaluation it improved from 31.5 for fixed $\tau_t$8 to 30.2 in the adaptive long setting (Kiruluta, 1 Jun 2025). This suggests that SAA can be interpreted not only as sparse masking or normalization control, but also as adaptive allocation over learned temporal kernels with explicit decay and frequency semantics.

5. Vision and multimodal instantiations

In unified image generation and editing, SAA appears as a response to cross-attention conflicts between text sub-instructions and input-image activations. Self-Adaptive Attention Scaling (SaaS) operates at inference time in OmniGen by extracting the cross-attention slice from noise latents to conditioning tokens, identifying per–sub-instruction spatial masks, and computing a dynamic scaling factor

$\tau_t$9

which is then applied to the corresponding sub-instruction tokens at the next denoising step and renormalized (Zhou et al., 22 Jul 2025). The method is restricted to early denoising steps and deeper layers, based on perturbation analysis showing that early steps are critical and deeper layers are vital. On multiple-sub-instruction editing, OmniGen improved from CLIP-T 0.276 and PickScore 0.244 to CLIP-T 0.315 and PickScore 0.513 with SaaS; in a user study, preference for SaaS reached 65.2% versus 21.0% for OmniGen on multiple sub-instructions (Zhou et al., 22 Jul 2025). The added latency was reported as 0.3 s, or 1.03%, with approximately 2 MB VRAM overhead.

AdaFV addresses a related allocation problem in vision-LLMs, but before the LLM rather than inside diffusion attention. Its self-adaptive cross-modality attention mixture allocates a fixed visual-token budget between [CLS]-based visual saliency and text-to-image cosine similarity computed in the pre-LLM space (Han et al., 16 Jan 2025). The adaptive split is obtained by maximizing a geometric mean over cumulative saliency and similarity scores under a budget constraint. On LLaVA-NEXT-7B, AdaFV reported 98.49% average retention at approximately 75% reduction, 96.00% at approximately 90%, and 94.35% at approximately 95%, outperforming FasterVLM, FastV, and SparseVLM at aggressive reduction rates (Han et al., 16 Jan 2025). The paper also reports that text-to-image similarity covers prompt-relevant regions with fewer reserved tokens than saliency alone, while TextVQA-like settings can favor saliency more strongly.

SAEViT provides a different vision interpretation. In that work, the acronym SAA refers to Sparsely Aggregated Attention, but the module is explicitly described as achieving self-adaptive attention allocation through blockwise average-pooling aggregation, reduced-resolution attention, and depth-wise transposed-convolution reconstruction (Zhang et al., 23 Aug 2025). At stage resolutions 56×56, 28×28, 14×14, and 7×7, the stride schedule $\tau_t$0 increases sparsification where redundancy is highest and removes it where detail is scarce. On ImageNet-1K, SAEViT-T achieved 76.3% Top-1 at 0.8 GFLOPs and SAEViT-XS achieved 79.6% at 1.3 GFLOPs; controlled efficiency measurements at 56×56, $\tau_t$1, and 8 heads reported SAA FLOPs 679.18, lower than SRA, shifted-window, cross-attention, and MSC (Zhang et al., 23 Aug 2025). Here SAA is realized through reversible spatial aggregation rather than through content-adaptive token ranking.

6. Recommendation and robotics

SAM extends the meaning of SAA beyond token-token attention altogether. In context-aware recommendation, the attention variable is a bias-corrected weight matrix learned from item text and applied to observed user–item losses (Liu et al., 2021). The module learns $\tau_t$2 from item context and encourages the reweighted observation mask $\tau_t$3 to approach the all-ones matrix in spectral norm, under the constraint $\tau_t$4. Integrated with MF, ConvMF, FTMF, and RCNNMF, SAM consistently improved RMSE on ML-100K, ML-1M, and Sogou News; the aggregate improvements reported by the authors were 0.33%, 0.59%, and 0.17%, and the gains were larger in sparser regimes (Liu et al., 2021). Under 20% training ratio on ML-100K, for example, RCNNMF improved from 1.163 to 1.021 with SAM.

SARA-RT brings SAA into robotics Transformers as a learnable linear-attention replacement. It defines feature maps

$\tau_t$5

and uses an up-training procedure to convert quadratic-attention robotic policies into linear-attention counterparts while preserving task behavior (Leal et al., 2023). The theoretical analysis connects exp-based random features to unbiased approximation of the softmax kernel for fixed-norm inputs and provides a theorem stating that there exist $\tau_t$6, $\tau_t$7, $\tau_t$8, and $\tau_t$9 such that the approximate attention matrix is uniformly close to the original one. Empirically, regular RT-2 had 53.2 ms forward-pass latency on TPU, while SARA reduced this to 45.7 ms, a 14% speedup; mean accuracy in the no-history, action-token setting was 65.8% for RT-2 and 65.1% for SARA-RT-2, while SARA-RT-2 with history and vector representation reached 76.4% mean accuracy (Leal et al., 2023). For Point Cloud Transformer policies, the reported AB-test average reward improved from 0.64 to 0.75, and SARA-PCT inference time remained approximately 100 ms regardless of point-cloud size (Leal et al., 2023).

These two cases are methodologically distant but conceptually aligned. In SAM, SAA means reallocating statistical weight across observations to reduce selection bias. In SARA-RT, SAA means learning low-dimensional feature maps that preserve selective attention behavior under linear-time computation. A plausible implication is that the core object of adaptation in SAA is not fixed: it may be probability mass, span, sparsity pattern, token budget, region mask, or training weight, provided that the allocation changes with contextual relevance.

7. Limitations, misconceptions, and open directions

A common misconception is that SAA names a single mechanism. The literature does not support that view. SAA can denote per-step softmax temperature, adaptive span length, sparse graph routing, head-specific token retention, post-softmax clipping, spatial cross-attention scaling, loss-level reweighting, or even a replacement of self-attention by adaptive Laplace analyzers (Lin et al., 2018, Sukhbaatar et al., 2019, Li et al., 2020, You et al., 10 Dec 2025, Fu et al., 1 Jan 2026, Zhou et al., 22 Jul 2025, Liu et al., 2021, Kiruluta, 1 Jun 2025). Another misconception is that SAA is necessarily efficiency-driven. Some works chiefly target quality or robustness, such as SACT’s differentiation between content and function words, Lazy Attention’s mitigation of overload and underload, and SaaS’s improvement of multi-part instruction following (Lin et al., 2018, Fu et al., 1 Jan 2026, Zhou et al., 22 Jul 2025).

The limitations are correspondingly heterogeneous. SACT constrains temperature to $$m_z(x)=\min\left[\max\left[\frac{1}{R}\left(R+z-x\right), 0\right], 1\right],$$0 and reports low sensitivity to $$m_z(x)=\min\left[\max\left[\frac{1}{R}\left(R+z-x\right), 0\right], 1\right],$$1, but does not provide explicit gradients or a closed-form stability analysis (Lin et al., 2018). TCA-Attention notes that tasks requiring truly global, dense dependencies may be sensitive to aggressive sparsification, and that last-query scoring can underweight earlier context relevant to future tokens (You et al., 10 Dec 2025). Saap is vulnerable to distribution shift, especially numerical or mathematical prompts, and its benefits diminish at short sequence lengths where routing overheads dominate (Mazaré et al., 12 Feb 2025). SaaS can still fail on highly conflicting or mutually exclusive sub-instructions, ambiguous text, or extreme edits requiring global restructuring (Zhou et al., 22 Jul 2025). SAEViT acknowledges that aggressive pooling with large stride can over-compress small objects or fine-grained details (Zhang et al., 23 Aug 2025). SAM states that item-context-driven weights approximate inverse propensities, but does not provide a formal proof of unbiasedness for the spectral-norm surrogate and may be insufficient when exposure depends strongly on user context or user–item interactions (Liu et al., 2021). Lazy Attention remains $$m_z(x)=\min\left[\max\left[\frac{1}{R}\left(R+z-x\right), 0\right], 1\right],$$2 in time even though it induces substantial sparsity (Fu et al., 1 Jan 2026).

The open directions in the cited work are unusually explicit. The NMT temperature line proposes application to the Transformer model and suggests token-level, head-level, or layer-level temperatures, as well as reinforcement learning or meta-learning for $$m_z(x)=\min\left[\max\left[\frac{1}{R}\left(R+z-x\right), 0\right], 1\right],$$3 or $$m_z(x)=\min\left[\max\left[\frac{1}{R}\left(R+z-x\right), 0\right], 1\right],$$4 (Lin et al., 2018). TCA-Attention suggests mixing policies across heads and layers and tuning $$m_z(x)=\min\left[\max\left[\frac{1}{R}\left(R+z-x\right), 0\right], 1\right],$$5, $$m_z(x)=\min\left[\max\left[\frac{1}{R}\left(R+z-x\right), 0\right], 1\right],$$6, and $$m_z(x)=\min\left[\max\left[\frac{1}{R}\left(R+z-x\right), 0\right], 1\right],$$7 according to task dependence on global versus local context (You et al., 10 Dec 2025). AdaFV points to uncertainty-aware selection and non-CLIP encoders (Han et al., 16 Jan 2025). Saap raises prefill acceleration, KV compression, and more robust out-of-distribution routing as next steps (Mazaré et al., 12 Feb 2025). STLT emphasizes the interpretability of learned half-lives and frequencies and the role of adaptive node allocation in scaling beyond 100k contexts (Kiruluta, 1 Jun 2025). Taken together, these directions suggest that future SAA research will likely continue moving away from the single dense softmax matrix as the unquestioned default and toward architectures in which allocation itself is an explicit, learned, and task-structured object.