WAM-TTT: Steering World-Action Models by Watching Human Play at Test Time

Abstract: Steering robot foundation models (RFMs) toward new task variants or user-preferred behaviors remains challenging, often requiring additional robot demonstrations, task-specific fine-tuning, or long-context conditioning. We present WAM-TTT, a test-time training framework for steering world action models from raw human videos. Rather than treating human videos as trajectories to imitate, WAM-TTT absorbs them into a lightweight adaptive memory inside a frozen WAM through self-supervised video prediction. To make this memory useful for control, we introduce a meta-training stage that aligns human demonstrations with robot behaviors using paired human-robot data and a key--value memory reconstruction objective. At test time, only unlabeled human videos are required to adapt the memory, while the pretrained WAM remains frozen. This enables efficient and reusable steering without robot actions, human-side annotations, or task-specific fine-tuning, while preserving the generalization ability of the foundation model. Extensive experiments show that WAM-TTT consistently outperforms in-context human-video conditioning baselines across diverse manipulation tasks and generalization settings.

• The paper introduces WAM-TTT, which adapts a frozen world-action model using video-side fast-weight memory and test-time training on unlabeled human videos, achieving 46.2% average progress in unseen households versus 32.5% for the pretrained LDA baseline.
• The method meta-trains lightweight key–value memory updates with phase-aligned human–robot episodes, then adapts only fast weights at deployment while preserving the pretrained action stream and avoiding robot demonstrations, pose estimation, retargeting, or foundation-model fine-tuning.
• Ablations show that meta-training and key–value reconstruction are essential, while generic LoRA adaptation and pseudo-action supervision hurt performance; WAM-TTT also retains 76% of its in-cubicle performance under household shifts, though failures remain under severe geometric misalignment.

WAM-TTT addresses a specific deployment problem for robot foundation models (RFMs): steering a pretrained World Action Model (WAM) toward new task variants using only unlabeled human videos, without robot demonstrations, retargeting annotations, or fine-tuning of the foundation model. The approach is built on the observation that existing mechanisms for injecting human data—co-training, in-context conditioning, or supervision via hand pose and retargeted trajectories—either degrade the pretrained policy, collapse under distribution shift, or require costly human-side annotation. WAM-TTT instead treats human videos as test-time training (TTT) signal absorbed into lightweight fast-weight memory inside a frozen WAM (2607.06988).

Method

The framework is instantiated on LDA, a pretrained world-action model whose diffusion transformer blocks contain coupled video and action experts communicating through joint attention (Lyu et al., 12 Feb 2026). The architectural modification is minimal: each video expert receives a residual TTT branch producing $$\Delta \bm{z}_{\mathrm{TTT}} = \theta_O f_W(\theta_Q(\bm{z}))$$, where $f_W$ is a fast-weight MLP with slow projections $\theta_{K,V,Q,O}$ and a learnable initialization $W_{\mathrm{init}}$. The action stream is untouched by the residual, which places all test-time adaptation on the video side—the only modality that action-free human videos can supervise.

Training proceeds in two stages. In human-robot meta-training, paired human-robot episodes are phase-aligned offline (nearest normalized phase frame), and the fast weights are adapted by inner SGD on a combined human-side objective: the standard video-prediction loss plus a per-layer key–value memory reconstruction loss $\mathcal{L}_{\mathrm{KVM}}$ that measures how well $f_W$ maps human keys to human values. The outer loop then optimizes the WAM parameters, slow projections, and $W_{\mathrm{init}}$ through the standard robot multitask diffusion loss, backpropagating through the analytical inner update. At test time, only the fast weights update: given a small batch of unseen-task human videos, inner SGD minimizes the same combined objective, after which the adapted memory steers action generation during rollout while every other parameter remains frozen.

The appendix provides an instructive analysis of why this works: in the linear special case, minimizing $\mathcal{L}_{\mathrm{KVM}}$ has a closed-form least-squares solution that reduces, under isotropy of $\bm{K}_h^\top \bm{K}_h$, to a Hebbian outer-product memory whose readout is exactly a softmax-free linear-attention retrieval against the stored human keys and values. The nonlinear deployed MLP is presented as the smooth analog of this witness. This is stated as a sanity check rather than a strict modeling assumption, and the closed-form decomposition holds only in the linear case.

Experimental setup

Evaluation covers three embodiments (Unitree G1 humanoid, Galbot two-finger gripper, Galbot 22-DoF dexterous sharpa) across nine manipulation tasks, with 2,286 phase-paired human-robot episodes collected via teleoperation in a standardized cubicle and GoPro egocentric human recordings without any pose estimation. Each task is scored by partial-credit progress over 25 trials under two settings: Orig. (in-cubicle) and New (unseen households where lighting, table height, and objects shift jointly). Baselines include the raw LDA backbone, WAM-Cotrain (human data mixed into the multitask loss), WAM-ICL (human videos as in-context tokens), EgoScale, and $\pi_{0.5}$.

Main results

In the unseen-household setting, WAM-TTT averages 46.2% progress versus 32.5% for LDA (+13.7 pts), 25.3% for WAM-Cotrain, 15.0% for EgoScale, 14.8% for $f_W$0, and only 7.1% for WAM-ICL. It wins seven tasks outright, ties Flip Steak, and loses Stamp Paper (8.3 vs. LDA's 33.3), attributed to a geometrically tight stamp pose whose alignment the household perturbation breaks without visible correction from the human videos.

Method	Avg. progress (New, %)
WAM-TTT	46.2
LDA	32.5
WAM-Cotrain	25.3
EgoScale	15.0
π₀.₅	14.8
WAM-ICL	7.1

Two contrasts carry most of the evidential weight. First, the gap against WAM-ICL is the strongest support for the core hypothesis: feeding identical human videos as in-context tokens fails to transfer skill under scene shift, whereas absorbing them as fast-weight memory succeeds. Second, the full appendix table reveals a striking retention asymmetry: WAM-ICL collapses from 48.4% (Orig.) to 7.1% (New)—a 15% retention ratio and −41.3 points—while WAM-TTT retains 76% of its leading in-cubicle score of 61.1%. Notably, WAM-Cotrain falls below even the no-human baselines in-distribution, indicating that diluting robot supervision with unaligned human data actively damages the pretrained policy; WAM-TTT's residual design avoids this because the fast-weight memory does not perturb the policy stream on unrelated trajectories.

Ablations

Component ablations on Table Bussing and Swap Place isolate each design choice. Removing meta-training collapses performance entirely (9.0 and 0.0), removing the key–value reconstruction loss costs substantially (66.7 and 72.0 vs. 100.0 and 88.9), and replacing TTT with generic LoRA adaptation yields 30.0 and 0.0—evidence that the benefit comes specifically from the TTT memory structure rather than parameter-efficient tuning in general.

Three further ablations are informative:

• Data-ratio sweep: at fixed total budget of 200 episodes, (100,100) at 74.1 is statistically indistinguishable from (200,0) at 73.7, so paired human data substitutes roughly one-for-one for expensive robot teleoperation. However, the cheap-human extreme (10,190) drops to 51.4, confirming that some robot grounding is necessary—human data is a substitute, not a replacement, for action-conditioned signal.
• Architecture: freezing the pretrained VLM conditioning backend costs −46 points and underperforms even a DiT-only configuration, indicating joint VLM-DiT adaptation is required once human-robot alignment must modify conditioning features.
• Action pseudolabels: adding MANO-derived pseudo-actions and a forward-dynamics loss is uniformly harmful, dropping the four-task average from 72.3 to 28.9 (−43.4 pts). Even on the dexterous embodiment most geometrically compatible with MANO output, single-view retargeting noise corrupts the learned dynamics. This result directly supports keeping human videos action-free.

Generalization preservation tests address the concern that test-time adaptation might overfit the demonstrated trajectory: on Deliver Drink, WAM-TTT outperforms all baselines under both lighting (66.0) and spatial (56.0) perturbations, consistent with the claim that adapting only video-side fast weights preserves the frozen action prior. Additional stress tests show direct transfer to unseen lab scenes with no in-scene human videos at all, driven purely by the meta-trained slow projections.

Limitations

The paper concedes three caveats explicitly. Meta-training assumes phase-paired human episodes cover the same skill-phase distribution as robot episodes; misalignment degrades the inner adaptation signal silently, without any loss-based flag. Deployment-time adaptation is bounded by the expressiveness of the fast-weight network and the slow projections fixed at meta-training, and the boundary of degradation as deployment tasks drift from the meta-training pairing distribution has not been characterized empirically. Finally, the interface accepts only egocentric RGB frames and does not exploit hand-pose, contact, or 3D cues shown useful elsewhere. One further open question follows from the Stamp Paper failure: the mechanism offers no diagnostic for when human videos fail to correct a task-specific geometric misalignment.

Conclusion

WAM-TTT formulates human-video steering of world-action models as test-time training, coupling a plug-and-play fast-weight memory with a meta-trained key–value reconstruction objective that makes deployment-time video prediction useful for control. The empirical signature—highest in-distribution performance combined with the highest meaningful OOD retention among human-data injection methods—supports the central design claim that task-specific adaptation can be decoupled from the pretrained policy through residual, video-side-only fast-weight updates. The remaining questions are quantitative rather than conceptual: characterizing the drift boundary of the meta-trained interface, and detecting or correcting silent phase-misalignment failures.