LaCT: Large Chunk Test-Time Training

• LaCT is a test-time training paradigm that uses extremely large token chunks for efficient fast-weight adaptation and long-context processing.
• It aggregates up to one million tokens per update, shifting the bottleneck from memory bandwidth to compute and maximizing accelerator utilization.
• LaCT supports multimodal applications in language, images, and video, yielding significant improvements in throughput and overall performance.

Large Chunk Test-Time Training (LaCT) is a paradigm in Test-Time Training (TTT) that utilizes extremely large, hardware-efficient chunk sizes for online fast-weight adaptation. Contrary to classical approaches where updates occur every few tokens (e.g., 16–64), LaCT leverages updates on orders of magnitude more tokens per chunk (2,000–1,000,000), improving computational throughput, scaling nonlinear fast-weight architectures, and enabling test-time adaptation across diverse data modalities including sequences, sets, images, and video. This paradigm shifts the core bottleneck from memory bandwidth to compute, maximizes accelerator utilization, and is implementable without custom kernels, fostering long-context modeling with high state capacity (Zhang et al., 29 May 2025, Li et al., 10 Nov 2025).

1. Motivation and Theoretical Foundations

Traditional TTT adapts a subset of model weights—“fast weights”—via online optimization on each incoming token, storing temporary contextual memory. However, when applied at small chunk sizes, this induces low GPU utilization (<5% TFLOPs on modern accelerators) and limits nonlinear fast-weight capacity. These deficiencies are exacerbated for high-dimensional data such as images and videos. By contrast, LaCT aggregates large numbers of tokens per update, driving hardware utilization (>70% TFLOPs), increasing state capacity (up to 40% of model parameters), and enabling practical scaling for multimodal, long-context workloads (Zhang et al., 29 May 2025).

2. Chunkwise Test-Time Memorization—Algorithmic Structure

Let $f_{W_f}: \mathbb R^d \to \mathbb R^d$ denote a fast-weight network parameterized by dynamic weights $W_f$ (adapting at test time) and static “slow” weights $\theta$. For a sequence of N tokens, LaCT divides inputs into C disjoint chunks of size M tokens. Within each chunk, the APPLY step computes outputs for all queries using the current $W_f$; the UPDATE step accumulates gradients over all key–value pairs and applies the fast-weight optimizer. The core update for chunk j is: $$L_{\text{chunk}}(W_f) = \sum_{i=1}^M \ell\left(f_{W_f}(k_i), v_i\right)$$

$$\nabla_{W_f}L_{\text{chunk}} = \sum_{i=1}^M \nabla_{W_f} \ell\left(f_{W_f}(k_i), v_i\right)$$

The fast-weight update typically uses a non-linear optimizer such as Muon: $$W_f^{(t+1)} = \mathrm{L2norm}\left(W_f^{(t)} - \mathrm{Muon}(g)\right)$$ where Muon applies five Newton–Schulz iterations to orthogonalize gradients (Zhang et al., 29 May 2025). This formulation supports both permutation invariance within chunks and fine-grained causal modeling via optional windowed attention layers.

3. TNT: Decoupled Two-Stage Training for Inference Performance

The TNT training paradigm (Li et al., 10 Nov 2025) resolves the large-chunk accuracy–efficiency trade-off by decoupling pre-training and inference chunk sizes. Stage 1 uses a hierarchical memory to maximize efficiency:

• Global memory $V$ processes large chunks ($C_G$), handling long-range context.
• Local memories $W^{(i)}$ operate on fine-grained chunks ($C_L$), supporting expressivity.
• Periodic resets ($S_L \gg C_L$) break sequential dependencies, unlocking parallel processing.

During Stage 2, a brief fine-tuning phase adapts only the local memory to a smaller chunk size ($C'_L$), restoring/optimizing accuracy: $$\tilde W_t = \begin{cases} W_{\text{init}} \quad & t \equiv 1 \pmod{S_L} \ \tilde W_{t-1} - \sum_{\tau = \xi(t, C'_L)}^t \eta_\tau \nabla_W \ell(f(\tilde W_{\xi(t, C'_L)}, k_\tau), v_\tau) & \text{otherwise} \end{cases}$$ This two-stage approach accelerates training by up to 17× while matching or exceeding fine-chunk accuracy in downstream tasks (Li et al., 10 Nov 2025).

Algorithmic Summary Table

Stage	Memory Module	Chunk Size
Stage 1	Global & Local	Large ($C_G, C_L$)
Stage 2	Local Only	Small ($C'_L$)

4. Hardware Utilization and Scaling Properties

LaCT achieves favorable compute-to-memory ratios by increasing chunk size $b$ per fast-weight update. For fast-weight multiplication (shape $h \times h$ × $b \times h$): $r = \frac{2 h^2 b}{2 h^2 + 4 h b}$ For large $b$ (2,048–1,000,000), the operation becomes compute-bound, driving hardware efficiency from <5% (small chunks) to >70% (large chunks) of peak FLOPs on H100 accelerators (Zhang et al., 29 May 2025). This efficiency enables the inclusion of highly expressive, nonlinear fast-weight architectures (e.g., SwiGLU MLPs), dramatically increasing adaptive state capacity without custom kernels.

5. Empirical Validation across Modalities

Experiments validate LaCT in image novel view synthesis, language modeling, and autoregressive video diffusion (Zhang et al., 29 May 2025):

• Image Sets: LaCT matches full attention in PSNR (37.9 dB @48 views), reduces prefill latency (16s → 1.4s), and outperforms 3D Gaussian Splatting with sparse views at 1M-token context.
• Language Modeling: Hybrid LaCT + sliding-window attn (M=2048, 4096) yields lower per-token loss and up to +20pp in retrieval accuracy vs. GLA/DeltaNet at long positions.
• Autoregressive Video Diffusion: On 14B Wan 2.1 backbone, LaCT matches full causal attention in denoising loss and outperforms Mamba2+SWA (Zhang et al., 29 May 2025).

Recent TNT benchmarks (Li et al., 10 Nov 2025) demonstrate:

• 17.4× training speedup for Titans at $C_L=64$ versus baseline at $C=8$.
• Perplexity improvement by 2 points and +1.9 pp accuracy on Commonsense QA after Stage 2 fine-tuning.

Representative Results Table (from (Li et al., 10 Nov 2025))

Model (Chunk)	Time (hr)	Speed-up ×	Perplexity	CSQA Acc.
Titans (8)	19.48	1	25.07	39.0%
Titans (64)	4.18	4.67	–	–
TNT {64}	1.12	17.37	–	–
TNT Stage 2 (8)	–	–	23.09	40.9%

6. Generalization and Applicability

A key insight is that LaCT, and by extension TNT, is agnostic to the type of deep memory module (e.g., delta rule, gated RNN) and modality, provided the architecture supports chunkwise parallel retrieval and domain-aligned query projection. By decoupling training and inference chunk sizes, and utilizing multi-resolution hierarchical memory plus resets, the paradigm is applicable in sequence tasks, N-D grids, and set-based problems. This approach eliminates the fundamental barrier in linearly-scaling RNN-style TTT models, paving the way for high-context, high-precision adaptation strategies (Li et al., 10 Nov 2025, Zhang et al., 29 May 2025).

7. Conclusion and Research Directions

Large Chunk Test-Time Training enables scalable, hardware-efficient adaptation for long-context workloads. Its core contributions are the use of extremely large chunk sizes for fast-weight updates, hierarchical memory with domain-aligned retrieval, and two-stage (pre-train/fine-tune) decoupling for accuracy and speed. LaCT matches or exceeds state-of-the-art in multiple domains, and its implementation simplicity (<100 lines of PyTorch) facilitates broad experimental access and future research in both architecture and optimizer design.

A plausible implication is that further exploration of chunkwise hybridization (e.g., integration with diffusion models, multi-directional scanning) may yield additional gains in vision and multimodal generative models. The paradigm is positioned to advance the efficiency frontier in test-time training and long-context neural architectures (Zhang et al., 29 May 2025, Li et al., 10 Nov 2025).