UAM: A Dual-Stream Perspective on Forgetting in VLA Training

Abstract: Vision--language--action (VLA) models are typically built by fine-tuning a pretrained vision--LLM (VLM) on action data. However, we show that this standard recipe systematically erodes the VLM's multimodal competence, a side effect we call the embodiment tax. But do VLAs have to forget? Inspired by the two-stream organization of biological vision, we trace this degradation to a structural bottleneck: current VLAs ask a single encoder to support both language-grounded semantics and control-relevant visual features, whereas biological vision separates recognition and visuomotor control into distinct pathways. Building on this view, we propose the Unified Action Model (UAM), which adds a parallel Dorsal Expert, an analog of the brain's dorsal pathway. To make the Dorsal Expert an effective second pathway and reduce the control-learning burden on the VLM, we initialize it from a pretrained generative model and train it with a mid-level reasoning objective that predicts visual dynamics. This design allows us to train the whole VLA end-to-end on action data alone: with no parameter freezing, no gradient stopping, and no auxiliary VL co-training, UAM retains over $95\%$ of the underlying VLM's multimodal capability and at the same time achieves the highest average success rate among baselines on a variety of manipulation tasks that probe out-of-distribution generalization, including unseen objects, novel object--target compositions, and instruction variation. Together, these results suggest that semantic preservation in VLAs can emerge from architectural separation itself, rather than being enforced by frozen weights or auxiliary data replay, and that this preserved semantic capability can naturally transfer from VLMs to semantic generalization in actions.

• The paper introduces a dual-stream UAM architecture that decouples semantic understanding from visuomotor control to significantly reduce catastrophic forgetting.
• It demonstrates that a generative dorsal expert, pre-trained for visual dynamics, retains over 95% of the original multimodal benchmark scores during action fine-tuning.
• The approach enables an end-to-end trainable VLA model that maintains semantic grounding while excelling in robot policy performance and out-of-distribution generalization.

A Dual-Stream Architecture for Semantic Retention in Vision-Language-Action Fine-Tuning

Introduction

Vision-Language-Action (VLA) models, which extend pre-trained Vision-LLMs (VLMs) with action heads for language-conditioned robot policy learning, incur a systematic degradation in their inherited multimodal understanding during action fine-tuning—a phenomenon termed the embodiment tax. "UAM: A Dual-Stream Perspective on Forgetting in VLA Training" (2605.15735) presents a comprehensive analysis of this forgetting effect. The authors introduce the Unified Action Model (UAM), a dual-pathway VLA framework motivated by the two-stream architecture of biological vision, aiming to decouple semantic and control demands through a Dorsal Expert and to empirically verify that such representational separation enables robust multimodal retention without parameter freezing or auxiliary data.

Figure 1: UAM allows for end-to-end action learning while greatly mitigating the semantic-act gap and embodiment tax by introducing a visual-dynamics bridge via a Dorsal Expert.

The Embodiment Tax and Structural Bottlenecks

Standard VLA models fine-tuned exclusively on action data overwrite VLM semantic representations, eroding their open-vocabulary recognition, spatial reasoning, and instruction-following abilities—capabilities crucial for robust generalization, especially in out-of-distribution (OOD) scenarios. The paper's empirical analysis reveals that this loss is not unique to any specific VLM or downstream architecture; both Mixture-of-Transformers (MoT) and MLP-based action coupling suffer, with unfrozen backbones losing as much as 60% of their pre-trained multimodal benchmark scores, despite superior action accuracy compared to frozen counterparts.

The paper hypothesizes that the root of catastrophic forgetting lies in a representational bottleneck: conventional architectures conflate semantic understanding (ventral, “what”) and visuomotor control (dorsal, “how/where”) in a single encoder, contrary to biological vision which maintains a functional dissociation. This hypothesis is supported by controlled ablations and forms the basis for motivating an architectural, rather than procedural or data-centric, solution.

The Unified Action Model (UAM) Architecture

UAM introduces a Dorsal Expert to instantiate an explicit control pathway, operating in parallel with the semantic VLM expert. Both streams are coupled to a shared action expert via parallel MoT routing. The design space of the Dorsal Expert is systematically studied along two axes: prior (random, VLM-based, generative/world model-based) and input modality (raw image tokens or learnable queries). The generative expert, pre-trained with visual-dynamics objectives and world modeling signals, emerges as the only variant to maintain high action accuracy while virtually eliminating the embodiment tax.

Figure 2: The UAM macro-architecture with three experts: semantic ($E_{\text{sem}}$), dorsal ($E_{\text{dor}}$), and action; top: design space of the dorsal pathway.

The most effective configuration leverages a generative expert initialized from a foundation model trained for visual dynamics prediction, further supervised with a visual-dynamics loss during action fine-tuning. This dorsal pathway relieves the VLM of the burden of encoding spatial-temporal control signals, allowing it to maintain semantic grounding.

Figure 3: Design space sweep showing only UAM’s generative dynamics-expert suffices for robust OOD action and retention of semantics under the frozen-probe protocol.

Empirical Results and Analysis

Multimodal Competence under Action Fine-Tuning

Evaluations on several challenging multimodal understanding benchmarks (MMMU, MME, MMBench, etc.) show that UAM retains over 95% of the original backbone score without any co-training, parameter freezing, or gradient blocking, starkly contrasting with baselines where conventional fine-tuning or simple addition of a dorsal module leads to near-total collapse of VQA and reasoning performance.

Robot Manipulation and OOD Generalization

On the ALOHA bimanual system, UAM demonstrates strong zero-shot generalization to OOD manipulation and instruction variations, outperforming both no-dorsal and VLM-initialized dorsal ablations. Notably, the dorsal pathway initialized from a generative world model critically enables transfer of VLM semantic priors to the policy, while VLM-initialization or random prior are insufficient for robust transfer.

Figure 4: UAM achieves substantially higher success under OOD tasks requiring semantic generalization, including multilingual and code-mixed instructions.

Representational Decoupling and Visual Routing

Attention map analysis during action generation shows that the action head automatically routes object-related and linguistic entities to the semantic stream, while delegating end-effector, scene dynamics, and interaction cues to the dorsal stream. This emergent bifurcation is not hand-coded but an architectural consequence of the dual-pathway design.

Figure 5: Visualization of action attention; $E_{\text{sem}}$ focuses on semantic entities, $E_{\text{dor}}$ on robot arms and interaction zones.

Implications and Future Directions

UAM provides evidence that architectural induction of functional separation between semantics and control, rather than freezing or complex multi-tasking protocols, is a principal mechanism for mitigating catastrophic forgetting in embodied VLA policies. This result challenges current practices that rely on data or gradient gating for VLM retention and suggests that structural modularity is both necessary and sufficient for dual competence.

Practically, UAM enables fully end-to-end trainable VLA agents with strong OOD robustness, unlocking new application domains where semantic reasoning and action generalization are simultaneously required. Theoretically, it opens new directions in designing embodied agents with factorial representational spaces tuned for both understanding and control, with direct analogies to biological dual-stream processing.

Future developments may extend UAM to multi-granular reasoning tasks, ongoing interaction, or hierarchical planning and control. Open questions pertain to longitudinal stability, resource efficiency with increasing model scale, and extensions to interactive and self-improving embodied agents.

Conclusion

"UAM: A Dual-Stream Perspective on Forgetting in VLA Training" demonstrates that catastrophic forgetting in VLA fine-tuning is primarily a structural issue. Through the introduction of a generative, dynamics-aware dorsal pathway and systematic architectural analysis, the authors show that semantic retention and action performance in VLAs are both achievable without procedural handholds. This work substantiates the necessity of dual-stream architectures for robust embodied learning and informs the next generation of VLA design.