Unnested Fixed Point in Elastic TTT

• Unnested Fixed Point (UFXP) is a design principle that enables conditional adaptation in test-time training by dynamically allocating attention and consolidation resources.
• It leverages mechanisms such as sparse routing for high-surprisal tokens, Fisher-weighted stabilization for parameter updates, and prompt-adaptive subspace selection.
• Empirical results indicate that UFXP methods improve exact recall and multi-chunk stability, mitigating issues like catastrophic forgetting and overfitting in streaming data.

Searching arXiv for papers on Elastic Test-Time Training and closely related TTT work. Elastic Test-Time Training denotes a family of test-time adaptation methods that seek to preserve the scalability advantages of Test-Time Training (TTT) while reducing instability, catastrophic forgetting, and exact-recall failures. In the recent literature, the term is used in at least three technically distinct but conceptually related senses: sparse exact-attention allocation for incompressible tokens in LLMs, Fisher-weighted stabilization of fast-weight updates in chunked spatial reconstruction, and decision-theoretic per-prompt adaptation of update magnitude and direction (P, 26 Feb 2026, Ma et al., 8 Apr 2026, Wakayama, 14 Jun 2026). Across these formulations, the unifying idea is dynamic allocation of adaptation capacity: exact attention is invoked only for routed anomalies, consolidation strength is modulated by estimated parameter importance, or update horizons and subspaces are selected according to prompt geometry and signal-to-noise structure. This suggests that “elasticity” in TTT is best understood not as a single algorithm, but as a design principle for controlling the stability–plasticity trade-off at inference time.

1. Conceptual scope and problem setting

Classical TTT replaces the standard exact-attention KV cache with hidden-state or parameter-based “fast weights” updated online during inference via a self-supervised objective, yielding an $O(1)$ memory footprint with theoretically unbounded context length because no per-token KV entries are retained (P, 26 Feb 2026). In this formulation, the model continuously compresses the incoming sequence into a fixed-size hidden state or adapter-like parameters and updates that state on the fly. The resulting architecture is attractive for long-context processing because memory does not scale with sequence length.

The central difficulty is that fully plastic inference-time updates are unstable over long streams. In language modeling, rare high-surprisal tokens can be overwritten by subsequent low-entropy tokens, leading to catastrophic failures on exact-recall tasks such as Needle-in-a-Haystack and passkey retrieval (P, 26 Feb 2026). In large-chunk spatial reconstruction, unconstrained fast-weight drift across chunks causes overfitting, catastrophic forgetting, temporal ghosting artifacts, and a tendency to rely on camera interpolation rather than true view-conditioned 4D reconstruction (Ma et al., 8 Apr 2026). In prompt-level TTT analyses, fixed update steps and fixed update subspaces are brittle under distribution shift, and infinite-step adaptation can overfit prompt residuals (Wakayama, 14 Jun 2026).

Elastic TTT addresses these failure modes by making adaptation conditional. In one line of work, capacity is elastic in memory: exact attention is reserved for tokens that cannot be safely compressed (P, 26 Feb 2026). In another, capacity is elastic in parameter drift: chunk-wise updates are softly pulled toward an evolving anchor with Fisher-weighted consolidation (Ma et al., 8 Apr 2026). In a third, capacity is elastic in optimization trajectory: the number of update steps and the update subspace are chosen per prompt according to evidence or query relevance (Wakayama, 14 Jun 2026).

2. Elasticity as selective memory allocation in sequence models

In "SR-TTT: Surprisal-Aware Residual Test-Time Training" (P, 26 Feb 2026), elastic test-time training is instantiated as a hybrid between pure fast-weight compression and sparse exact attention. The motivating claim is that pure TTT suffers an information bottleneck: the compressed fast-weight state must simultaneously represent all past tokens, so gradient updates driven by frequent background tokens overwrite the contributions of rare, unique, high-surprisal tokens such as names, IDs, or passkeys. Elasticity is introduced by routing only those tokens deemed “incompressible” to a sparse Residual Cache, while retaining the rest in the $W_{\text{fast}}$ path.

The routing signal is based on per-token reconstruction loss in the TTT inner loop,

$L_t = \|z_t - v_t\|^2,$

together with an adaptive threshold $\tau_{\text{EMA}}$, defined as the EMA-smoothed 95th percentile of recent losses (P, 26 Feb 2026). A token at position $t$ is routed only if both a token-level and a chunk-level anomaly criterion hold:

$$g_t = \mathbf{1}[L_t > \tau_{\text{EMA}}] \cdot \mathbf{1}[L_{\text{chunk}} > 0.8 \cdot \tau_{\text{EMA}}].$$

Routed tokens contribute post-RoPE key/value pairs to a fixed-capacity Residual Cache with priority-based eviction, whereas non-routed tokens contribute only through the fast weights.

Exact recall is then performed only over the routed set:

$$\mathrm{Attn}(Q_t, K_{\text{res}}, V_{\text{res}}) = \mathrm{softmax}(Q_t K_{\text{res}}^\top / \sqrt{d_k})V_{\text{res}},$$

where $K_{\text{res}}$ and $V_{\text{res}}$ are stacked only from tokens with $g_u=1$ for $W_{\text{fast}}$0 (P, 26 Feb 2026). The fast-weight state is updated in the usual TTT manner,

$W_{\text{fast}}$1

and the final output combines the compressive and exact-recall pathways:

$W_{\text{fast}}$2

with

$W_{\text{fast}}$3

This formulation yields an elastic memory profile. For low-entropy streams where losses rarely exceed $W_{\text{fast}}$4, $W_{\text{fast}}$5 for most tokens, so the model behaves like pure TTT with $W_{\text{fast}}$6 memory and throughput (P, 26 Feb 2026). Memory grows only with the number of routed anomalies; cache attention scales as $W_{\text{fast}}$7 in the number of routed tokens rather than $W_{\text{fast}}$8 in the full sequence length. A plausible implication is that this mechanism treats exact attention as an exception handler rather than the default memory substrate.

3. Elasticity as consolidation of fast-weight dynamics across chunks

In "Fast Spatial Memory with Elastic Test-Time Training" (Ma et al., 8 Apr 2026), Elastic Test-Time Training is formulated as a stabilization mechanism for Large Chunk Test-Time Training (LaCT). Here the core problem is not sparse exact recall but multi-chunk adaptation for 3D and 4D reconstruction. LaCT updates fast weights over large chunks to amortize throughput, but fully plastic chunk-wise dynamics accumulate instability over long sequences. The proposed Large-Chunk Elastic Test-Time Training (LaCET) augments the LaCT update with a Fisher-weighted elastic prior around an anchor state.

The LaCT chunk update aggregates per-token losses over a chunk of $W_{\text{fast}}$9 tokens:

$L_t = \|z_t - v_t\|^2,$0

and LaCET applies an elastic consolidation step afterward:

$L_t = \|z_t - v_t\|^2,$1

where $L_t = \|z_t - v_t\|^2,$2 is the intermediate post-LaCT fast weight, $L_t = \|z_t - v_t\|^2,$3 is the anchor, $L_t = \|z_t - v_t\|^2,$4 is a diagonal importance estimate, and $L_t = \|z_t - v_t\|^2,$5 controls elastic strength (Ma et al., 8 Apr 2026).

The importance matrix is maintained by an EMA over chunks,

$L_t = \|z_t - v_t\|^2,$6

with estimator variants derived from MAS-, EWC-, or SI-style update statistics (Ma et al., 8 Apr 2026). The anchor itself may be global, streaming, or streaming-EMA; the reported best practice is streaming-EMA at chunk boundaries,

$L_t = \|z_t - v_t\|^2,$7

which forms a low-pass constraint on the fast-weight trajectory.

The stated interpretation is explicitly EWC-inspired: chunks are treated as a stream of tasks at test time, and consolidation acts continuously rather than as an offline post hoc regularizer (Ma et al., 8 Apr 2026). Slow weights remain fixed at inference, while fast weights, anchor weights, and importance statistics are maintained per LaCET block. In this formulation, elasticity means that parameters with large estimated importance are resistant to drift, while parameters with small importance remain plastic.

This version of elasticity directly targets the stability–plasticity trade-off. Smaller chunks reduce activation-memory cost, but would ordinarily exacerbate forgetting across chunk boundaries. The elastic prior softens that trade-off by permitting multi-chunk adaptation while preserving useful information from earlier chunks, such as static background or longer-range spatiotemporal structure (Ma et al., 8 Apr 2026).

4. Elasticity as prompt-adaptive update magnitude and direction

In "A Decision-Theoretic View of Test-Time Training: When, How Far, and Which Directions to Adapt" (Wakayama, 14 Jun 2026), elasticity is not defined by a specific architecture but by a statistical interpretation of TTT in the local linearized kernel regime. TTT is treated as implicit Bayesian inference, where the adaptation horizon and update subspace define a prompt-induced prior. The central claim is that instability arises because fixed update steps and fixed subspaces cannot be optimal across shifts in prompt signal-to-noise ratio and query relevance.

Let updates lie in a $L_t = \|z_t - v_t\|^2,$8-dimensional subspace $L_t = \|z_t - v_t\|^2,$9, with linearized prompt features $\tau_{\text{EMA}}$0 and prompt-induced kernel $\tau_{\text{EMA}}$1. Gradient descent on a quadratic prompt loss yields the prompt correction

$\tau_{\text{EMA}}$2

and the query correction

$\tau_{\text{EMA}}$3

where

$\tau_{\text{EMA}}$4

for $\tau_{\text{EMA}}$5 (Wakayama, 14 Jun 2026). In the Gaussian benchmark, the Bayes-optimal correction is

$\tau_{\text{EMA}}$6

with $\tau_{\text{EMA}}$7. The paper shows that early-stopped gradient descent corresponds to a Gaussian posterior mean under an implicit prompt-dependent prior, so changing $\tau_{\text{EMA}}$8 or the update subspace changes the effective prior strength and directions (Wakayama, 14 Jun 2026).

The spectral statement is that large kernel eigenvalues are fit quickly and small ones more slowly, whereas Bayes-optimal shrinkage depends on prompt SNR. The conditional Bayes-gap identity weights errors in the update filter by terms

$\tau_{\text{EMA}}$9

showing that only eigen-directions aligned with the query matter for prediction (Wakayama, 14 Jun 2026). This yields two forms of elasticity: horizon selection, determining how far to adapt, and subspace selection, determining which directions to adapt.

A PAC-Bayes result is given for selecting the update horizon by prompt evidence, with normalized negative log-evidence

$t$0

where $t$1 (Wakayama, 14 Jun 2026). Separately, under a linear-Gaussian correction model, the Bayes-optimal rank-$t$2 update subspace is the top-$t$3 eigenspace of

$t$4

leading to a query-aware scoring rule for Transformer blocks and heads (Wakayama, 14 Jun 2026).

This perspective broadens the meaning of elastic TTT. Rather than allocating memory or consolidation strength alone, elasticity here refers to prompt-dependent control of adaptation geometry itself. A plausible implication is that architectural elasticity and optimization-theoretic elasticity can be viewed as complementary mechanisms operating at different levels.

5. Training procedures, practical mechanisms, and reported implementations

The language-model variant SR-TTT uses a per-token inference loop with loss computation, adaptive threshold update, chunk-level anomaly estimation, routing, fast-weight update, sparse cache attention, and gated fusion (P, 26 Feb 2026). Stability is attributed to routing only high-surprisal tokens to the external cache and to a Two-Stage Curriculum during training. The reported curriculum is: Stage 1 (1–7k) pure TTT with the cache disabled, and Stage 2 (7,001–10k) freezing the TTT backbone, enabling the cache, and training the fusion gates $t$5 (P, 26 Feb 2026). This is described as preventing “cold-start” collapse and helping the gate learn to pass residual attention.

LaCET follows a chunk-wise procedure: initialize slow weights, fast weights, anchor, and importance statistics; split the sequence into chunks; compute chunk losses; apply the LaCT surrogate update; update $t$6 by EMA; apply elastic consolidation; update the anchor by streaming-EMA; and then apply the updated fast weights to target queries in the chunk (Ma et al., 8 Apr 2026). Practical stabilizers include L2 weight normalization along the input dimension, optional Muon-style Newton–Schulz iteration, and QK-Norm in attention (Ma et al., 8 Apr 2026). The reported default settings in the appendix are $t$7, $t$8, and $t$9, with SI-style importance as default (Ma et al., 8 Apr 2026).

The decision-theoretic framework is more prescriptive than architectural. It recommends computing the prompt-space kernel $$g_t = \mathbf{1}[L_t > \tau_{\text{EMA}}] \cdot \mathbf{1}[L_{\text{chunk}} > 0.8 \cdot \tau_{\text{EMA}}].$$0, selecting $$g_t = \mathbf{1}[L_t > \tau_{\text{EMA}}] \cdot \mathbf{1}[L_{\text{chunk}} > 0.8 \cdot \tau_{\text{EMA}}].$$1 per prompt by minimizing $$g_t = \mathbf{1}[L_t > \tau_{\text{EMA}}] \cdot \mathbf{1}[L_{\text{chunk}} > 0.8 \cdot \tau_{\text{EMA}}].$$2 over a candidate grid, and choosing update directions using query-aware block or head scores such as

$$g_t = \mathbf{1}[L_t > \tau_{\text{EMA}}] \cdot \mathbf{1}[L_{\text{chunk}} > 0.8 \cdot \tau_{\text{EMA}}].$$3

when a query-aware design is feasible (Wakayama, 14 Jun 2026). The stability condition $$g_t = \mathbf{1}[L_t > \tau_{\text{EMA}}] \cdot \mathbf{1}[L_{\text{chunk}} > 0.8 \cdot \tau_{\text{EMA}}].$$4 is emphasized so that the filter remains monotone, and the continuous-time proxy $$g_t = \mathbf{1}[L_t > \tau_{\text{EMA}}] \cdot \mathbf{1}[L_{\text{chunk}} > 0.8 \cdot \tau_{\text{EMA}}].$$5 is used to interpret early stopping (Wakayama, 14 Jun 2026).

These procedures represent different operationalizations of elasticity. SR-TTT decides which tokens deserve exact attention. LaCET decides which parameters should be constrained toward an anchor. The decision-theoretic approach decides how long and in which subspace adaptation should occur. Despite their differences, all three introduce a control variable that suppresses indiscriminate plasticity.

6. Empirical evidence and observed failure modes

The empirical evidence for SR-TTT is centered on exact recall in a Needle-in-a-Haystack protocol using TinyStories with embedded 8-character alphanumeric needles. The reported model has 15.8M parameters, 4 layers, and $$g_t = \mathbf{1}[L_t > \tau_{\text{EMA}}] \cdot \mathbf{1}[L_{\text{chunk}} > 0.8 \cdot \tau_{\text{EMA}}].$$6, trained for 10,000 steps with maximum sequence length 2048 (P, 26 Feb 2026). At sequence length 2048, exact match improves relative to pure TTT by +23% at needle depth 0.50, from 10% to 33%, and by +20% at needle depth 0.75, from 17% to 37% (P, 26 Feb 2026). The qualitative description is that pure TTT forgets needles outside a recent sliding window, whereas SR-TTT’s Surprisal Filter routes the needle to cache and retrieves it later. The cache gates opened to approximately 10% at deeper semantic layers, and the cache exhibited saturation consistent with priority-based eviction (P, 26 Feb 2026). At sequence length 4096, however, both SR-TTT and pure TTT collapsed to 0% exact match because of known RoPE extrapolation limits rather than the routing mechanism itself (P, 26 Feb 2026).

The empirical evidence for LaCET is broader and concerns reconstruction quality, stability, and chunking behavior. On Stereo4D ablations with a 12-layer LVSM at $$g_t = \mathbf{1}[L_t > \tau_{\text{EMA}}] \cdot \mathbf{1}[L_{\text{chunk}} > 0.8 \cdot \tau_{\text{EMA}}].$$7, 32 input and 32 output views, and 32K training steps, LaCT without EWC yielded PSNR 26.908, LPIPS 0.0988, and SSIM 0.814, whereas LaCET with streaming-EMA anchors achieved PSNR 29.922, LPIPS 0.0519, and SSIM 0.899 for MAS, PSNR 29.989, LPIPS 0.0517, and SSIM 0.903 for SI, and PSNR 29.781, LPIPS 0.0537, and SSIM 0.897 for EWC (Ma et al., 8 Apr 2026). Streaming-only anchors performed worse than streaming-EMA, and global anchors were intermediate (Ma et al., 8 Apr 2026). On Stereo4D at $$g_t = \mathbf{1}[L_t > \tau_{\text{EMA}}] \cdot \mathbf{1}[L_{\text{chunk}} > 0.8 \cdot \tau_{\text{EMA}}].$$8, FSM-LVSM reported PSNR 32.16, LPIPS 0.043, and SSIM 0.931, while on NVIDIA it reported PSNR 23.90, LPIPS 0.105, and SSIM 0.747 (Ma et al., 8 Apr 2026). The qualitative findings are that LaCET suppresses temporal ghosting, stabilizes motion under sparse view-time inputs, and reduces reliance on the camera-interpolation shortcut (Ma et al., 8 Apr 2026).

The decision-theoretic paper provides evidence on synthetic and supplementary real-data settings. On a digit-shift regression task using distilgpt2 with 2000 tasks and $$g_t = \mathbf{1}[L_t > \tau_{\text{EMA}}] \cdot \mathbf{1}[L_{\text{chunk}} > 0.8 \cdot \tau_{\text{EMA}}].$$9 labeled prompt pairs, averaged MSE was 0.0747 ± 0.0031 for ICL ($$\mathrm{Attn}(Q_t, K_{\text{res}}, V_{\text{res}}) = \mathrm{softmax}(Q_t K_{\text{res}}^\top / \sqrt{d_k})V_{\text{res}},$$0), 0.0734 ± 0.0030 for fixed TTT ($$\mathrm{Attn}(Q_t, K_{\text{res}}, V_{\text{res}}) = \mathrm{softmax}(Q_t K_{\text{res}}^\top / \sqrt{d_k})V_{\text{res}},$$1), 0.0722 ± 0.0030 for evidence selection with fixed $$\mathrm{Attn}(Q_t, K_{\text{res}}, V_{\text{res}}) = \mathrm{softmax}(Q_t K_{\text{res}}^\top / \sqrt{d_k})V_{\text{res}},$$2, and 0.0719 ± 0.0029 for evidence selection with MLE-$$\mathrm{Attn}(Q_t, K_{\text{res}}, V_{\text{res}}) = \mathrm{softmax}(Q_t K_{\text{res}}^\top / \sqrt{d_k})V_{\text{res}},$$3 (Wakayama, 14 Jun 2026). The ECDF of $$\mathrm{Attn}(Q_t, K_{\text{res}}, V_{\text{res}}) = \mathrm{softmax}(Q_t K_{\text{res}}^\top / \sqrt{d_k})V_{\text{res}},$$4 varied across prompts and correlated with prompt noise, with smaller $$\mathrm{Attn}(Q_t, K_{\text{res}}, V_{\text{res}}) = \mathrm{softmax}(Q_t K_{\text{res}}^\top / \sqrt{d_k})V_{\text{res}},$$5 in the noisy regime (Wakayama, 14 Jun 2026). Under a budget of $$\mathrm{Attn}(Q_t, K_{\text{res}}, V_{\text{res}}) = \mathrm{softmax}(Q_t K_{\text{res}}^\top / \sqrt{d_k})V_{\text{res}},$$6 heads among 48 candidates, Query-Aware head selection outperformed Random on query MSE, while Trace-TopK improved prompt fit but not necessarily query prediction (Wakayama, 14 Jun 2026).

Across the three settings, the failure modes are consistent with the underlying definitions of elasticity. SR-TTT can experience worst-case memory growth if many tokens are surprising, and poorly calibrated $$\mathrm{Attn}(Q_t, K_{\text{res}}, V_{\text{res}}) = \mathrm{softmax}(Q_t K_{\text{res}}^\top / \sqrt{d_k})V_{\text{res}},$$7 can cause routing oscillations (P, 26 Feb 2026). LaCET remains sensitive to the quality of the importance estimate $$\mathrm{Attn}(Q_t, K_{\text{res}}, V_{\text{res}}) = \mathrm{softmax}(Q_t K_{\text{res}}^\top / \sqrt{d_k})V_{\text{res}},$$8 and to anchor and EMA hyperparameters; under large camera or view extrapolation, motion consistency may fail and stale gestures or ghosting may remain (Ma et al., 8 Apr 2026). The decision-theoretic framework formalizes failure under fixed $$\mathrm{Attn}(Q_t, K_{\text{res}}, V_{\text{res}}) = \mathrm{softmax}(Q_t K_{\text{res}}^\top / \sqrt{d_k})V_{\text{res}},$$9, overfitting as $K_{\text{res}}$0, and irreducible error due to subspace mismatch (Wakayama, 14 Jun 2026).

7. Relation to broader test-time adaptation and future directions

Elastic TTT occupies a middle ground between purely compressive recurrent-style memory and fully retained attention. SR-TTT explicitly differs from recurrent memory transformers and state-space models such as Mamba and RWKV by bypassing the recurrent bottleneck for anomalies through external storage rather than solely gating recurrent updates (P, 26 Feb 2026). It is also described as complementary to sliding windows, retrieval augmentation, KV-cache compression, and long-context acceleration methods such as dynamic sparse attention (P, 26 Feb 2026). This suggests an architectural interpretation of elasticity as a selective exception mechanism layered on top of efficient compression.

LaCET, by contrast, is framed through continual-learning analogies, specifically elastic weight consolidation, but adapted to a streaming inference setting where anchors evolve and importance is estimated online (Ma et al., 8 Apr 2026). From a Bayesian online learning perspective, the consolidation term approximates a Gaussian posterior centered at the anchor with diagonal precision $K_{\text{res}}$1, reinterpreting chunks as tasks at test time (Ma et al., 8 Apr 2026). This places elastic TTT in direct dialogue with continual-learning theory and with the problem of maintaining useful state under sustained adaptation.

The decision-theoretic framework generalizes elasticity still further by viewing TTT as implicit Bayesian correction to a base predictor under prompt-induced priors (Wakayama, 14 Jun 2026). In that account, “elastic” means prompt-adaptive shrinkage in both magnitude and direction, and the main lesson is that good prompt fit is insufficient unless the adapted directions are aligned with the query. This suggests that future elastic TTT systems may combine architectural routing, consolidation priors, and query-aware optimization schedules rather than relying on a single control mechanism.

Several extensions are explicitly proposed in the literature. SR-TTT identifies multimodal routing, continual learning with surprisal-gated episodic memory, adaptive caching and eviction, and positional robustness through YaRN or Dynamic NTK as future directions (P, 26 Feb 2026). LaCET points to structured Fisher or low-rank approximations, meta-learned anchors, adaptive elasticity, and added geometric supervision such as depth, correspondence, multi-view consistency, and optical flow (Ma et al., 8 Apr 2026). The decision-theoretic work highlights the need to move beyond the local linearized and squared-loss setting, especially for self-supervised TTT objectives such as entropy minimization, pseudo-labeling, and other nonlinear update rules (Wakayama, 14 Jun 2026).

Taken together, these works indicate that Elastic Test-Time Training is becoming a general framework for conditional adaptation at inference. The shared thesis is that test-time learning is effective only when plasticity is made selective: over tokens, over parameters, over chunks, or over update directions.