CKT-WAM: Parameter-Efficient Context Knowledge Transfer Between World Action Models

Abstract: World action models (WAMs) provide a powerful generative framework for embodied control, yet transferring knowledge across heterogeneous WAMs remains challenging due to mismatched latent interfaces, high adaptation cost, and the rigidity of conventional distillation objectives. We propose \textbf{CKT-WAM}, a parameter-efficient \textbf{C}ontext \textbf{K}nowledge \textbf{T}ransfer framework that transfers teacher WAM's knowledge into a student WAM through a compact context in the text embedding space, rather than output imitation or dense hidden-state matching. Specifically, CKT-WAM extracts intermediate teacher hidden states, reduces the number of tokens via compressors' learnable-query cross attention (LQCA), and transforms them through an always-on generalized adapter, a lightweight router, and sparsely activated specialized adapters. The resulting context is then appended to the student's conditioning textual embeddings, thereby injecting the transferred knowledge into the student with minimal architectural modification. Experiments show that CKT-WAM consistently improves zero-shot generalization and achieves the best overall performance on LIBERO-Plus, reaching 86.1\% total success rate with only 1.17\% trainable parameters, while approaching full fine-tuning performance. Beyond simulation, CKT-WAM also demonstrates strong real-world long-horizon manipulation ability, achieving the best average success rate of 83.3\% across four multi-step and long-horizon tasks. Code is available at https://github.com/YuhuaJiang2002/CKT-WAM.

• The paper demonstrates that extracting and compressing teacher WAM intermediate representations enables efficient knowledge transfer with only 1.17% trainable parameters.
• It employs learnable-query cross-attention compressors and dynamic adapter-routing to inject optimized context into frozen student models.
• Empirical results show an 86.1% zero-shot success rate on LIBERO-Plus and strong real-world performance across diverse robotic manipulation tasks.

Parameter-Efficient Context Knowledge Transfer Between World Action Models: An Analysis of CKT-WAM

Introduction and Motivation

Embodied agents increasingly rely on world action models (WAMs) to decouple perception from action and to explicitly model how visual observations evolve under action. WAMs, unlike vision-language-action (VLA) models that prioritize input-to-action reactiveness, adopt generative modeling of physical dynamics for robust control. A significant bottleneck, however, remains: efficiently transferring knowledge between heterogeneous WAMs, particularly from computationally intensive teacher WAMs to lightweight student variants. This knowledge transfer is impeded by incompatibility at latent interfaces, adaptation costs, and the rigidity of canonical distillation objectives.

CKT-WAM introduces a novel approach for parameter-efficient context-based knowledge transfer between WAMs, bypassing conventional output imitation or dense hidden-state alignment. Instead, it extracts, compresses, and injects intermediate teacher representations into the student's native text embedding space through a specialized adapter-routing mechanism. This approach seeks to minimize architectural intervention and computational burden while maximizing generalization and robustness, with substantial implications for practical robotic control systems operating under severe resource constraints.

Figure 1: High-level schematic of CKT-WAM, showing extraction and compression of teacher intermediate hidden states, routing and adapter architecture, and context injection into the student WAM’s textual embedding sequence.

Methodology

CKT-WAM's methodology can be divided into two core phases: teacher-side context construction and student-side context injection. The process initiates by selecting semantically rich, action-aware intermediate hidden states from the teacher WAM (typically from an intermediate transformer layer, balancing informativeness and computational cost).

To transform these hidden states, the framework employs learnable-query cross-attention (LQCA) compressors, generating a fixed-size context by summarizing the most salient teacher representations. This is followed by:

• An always-on generalized adapter for extracting task-agnostic information
• A lightweight router that computes instance-dependent routing scores over a bank of specialized adapters
• Sparsely activated specialized adapters, each parameter-efficient, capturing distinct instance- or stage-specific knowledge

The final transferred context—obtained by concatenating generalized and specialized outputs—is injected into the student WAM by extending its conditioning textual embedding sequence. Both backbones remain frozen; only adapters and routing are optimized. This design ensures efficient one-pass context construction during inference and avoids any test-time need for explicit video synthesis or teacher-student full hidden state matching.

Figure 2: Quantitative trade-off between success rate and inference latency for CKT-WAM as a function of teacher layer depth selected for context extraction.

Empirical Performance

LIBERO-Plus Zero-Shot Generalization

CKT-WAM demonstrates strong results on the LIBERO-Plus benchmark, engineered for assessing robustness across severe out-of-distribution (OOD) axes (camera, robot, language, lighting, background, noise, layout). With only 1.17% of parameters trainable, CKT-WAM yields an aggregate zero-shot success rate of 86.1%, surpassing a wide array of VLA and WAM baselines, and exhibiting significant robustness under semantic and scene distribution shifts. It consistently attains or surpasses state-of-the-art performance in several OOD axes, matching or even closely approaching full fine-tuning baselines with a fraction of the parameter update cost.

Comparison Against Parameter-efficient Adaptation Baselines

Experiments contrast CKT-WAM with a range of PEFT methods, including LoRA, PiSSA, AdaMoLE, and other recent variants. CKT-WAM achieves the highest aggregate success rate, with only 1.17% of parameters updated—a substantially lower budget than LoRA and others. The results underline that CKT-WAM's performance stems not from sheer adaptation capacity, but from its architectural mechanism allowing for efficient and effective knowledge transfer into frozen WAMs.

Ablation Study

Component-wise ablations indicate that both the generalized and specialized adapter branches contribute materially to OOD robustness, with specialization being particularly important for handling diverse shifts. The auxiliary load-balancing loss further stabilizes routing and enhances adapter utilization, although its removal yields smaller degradation compared to loss of specialized branches.

Figure 3: Specialized adapter selection probabilities across four manipulation stages in cube catching, demonstrating functional specialization and dynamic routing.

Real-World Long-Horizon Manipulation

CKT-WAM is tested on an array of real-world, multi-stage manipulation tasks, including clothes folding, fruit sorting, cube catching, and unmanned retail (object picking in clutter). In a rigorous evaluation of 45 trials per task, CKT-WAM achieves an average success rate of 83.3%, outperforming leading baselines in all but the most challenging deformable object task (clothes folding), where it remains highly competitive.

Figure 4: Real-world evaluation suite—visualizations of four multi-stage manipulation tasks tested, highlighting temporal and compositional complexity.

Figure 5: Intermediate state illustrations for Clothes Folding and Fruit Sorting tasks, evidencing successful sequential subgoal completion.

Figure 6: Representative frames for Cube Catching and Unmanned Retail, showing both intermediate and final task states.

Optimization and Training Dynamics

CKT-WAM isolates training to the context generation and routing modules, preserving all teacher and student weights intact for stability and transferability. The loss comprises the student's original diffusion-style latent objective—with respect to both future video and action latents—augmented by a mixture-of-experts load-balancing regularizer that statistically enforces broader router utilization. Training curves confirm stability in both total and auxiliary loss metrics and show minimization of the balancing term, driving adapter utilization toward the uniform regime.

Figure 7: Training loss curves, highlighting rapid optimization and effective balancing of adapter utilization.

Analysis and Implications

CKT-WAM establishes that parameter-efficient, context-based transfer from a powerful generative WAM to a lighter counterpart suffices for robust generalization in both simulated and real-world control tasks. By introducing a structural separation of task-agnostic and instance-specific transfer (generalized vs. specialized adapters), and performing injection via the native conditioning route, CKT-WAM circumvents architectural incompatibilities and inefficiencies of deep distillation or output imitation protocols.

Notably, CKT-WAM nearly closes the gap to full fine-tuning while maintaining strict frozen-backbone constraints and requiring minimal compute at adaptation time. Its sparse route-first expert selection is functionally validated through trajectory-stage-specific adapter activations, evidencing meaningful decomposition of complex tasks.

Theoretical Implications

• The success of context-based transfer via intermediate teacher features supports the sufficiency of partial, carefully-compressed teacher knowledge as a source of generalization, without explicit policy or action output imitation.
• The modular, MoE-structured adaptation design—featuring route-first sparse execution—could generalize as an effective principle across heterogeneous model transfer scenarios beyond robotics.

Practical Implications

• Facilitates the deployment of advanced control abilities from large models onto hardware-constrained platforms, democratizing real-world robotics.
• Provides a template for high-efficiency, low-latency adaptation strategies crucial for edge robotics, continual learning, and multi-platform system integration.

Open Questions and Future Directions

While CKT-WAM focuses on WAM-to-WAM transfer, its architectural template suggests further exploration in broader multimodal settings (e.g., vision-language or pure LLMs), especially where alignment of latent representations is structurally challenging. The efficacy of such context-based transfer for domains such as VQA or language modeling, as well as under more extreme model heterogeneity and task diversity, remains an open research frontier.

Conclusion

CKT-WAM demonstrates that compact, adapter-mediated contextual transfer outperforms state-of-the-art PEFT baselines, efficiently bridges the performance gap to full fine-tuning, and delivers robust performance on both simulated and real-world long-horizon tasks. Its design offers a general, parameter-efficient framework for transferring embodied world knowledge under minimal architectural intervention, with theoretical and practical repercussions for model distillation, lifelong learning, and adaptable AI systems in dynamic environments.