Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data

Abstract: Vision-LLMs (VLMs) are powerful general-purpose reasoners, yet converting them into robot control policies (VLAs) is surprisingly difficult. The root cause is a two-fold gap: VLMs are trained on internet-scale images with language-understanding objectives, while VLAs must perceive robot scenes and predict motor actions. Fine-tuning a VLM directly on robot action data forces the model to cross both gaps at once -- the learning curve is steep and the rich generalizations learned during pretraining tend to degrade rather than transfer. We argue that this gap can be bridged gradually with the right intermediate data. We introduce \emph{embodied trajectory-coupled (ETC) data} -- vision-language supervision derived from the same robot scenes and trajectories used for action learning. Because ETC data shares the visual context of robot operation while retaining familiar language-understanding objectives, it provides a natural stepping stone between VLM pretraining and VLA fine-tuning. Building on this, we design a three-stage training recipe. Distribution Bridging first adapts the VLM to embodied visual-language semantics. Objective Bridging then gradually shifts the model toward action prediction while preserving the acquired representations. Retentive Adaptation finally specializes the policy to the target deployment domain. We further show that mixing task-relevant out-of-distribution ETC data with a small amount of action data enables the model to generalize to novel visual-language conditions without requiring additional robot demonstrations. Simulation and real-robot experiments confirm that this gradual bridging strategy is the key to transferring VLM generalization into robust, deployable robot policies.

• The paper introduces an ETC-guided staged adaptation process that bridges the input distribution and supervision gaps between VLMs and VLAs.
• It employs automated ETC data—categorized into scene-grounding, task-oriented, and action-aligned planning—to align vision-language models with embodied actions.
• Extensive experiments show significant performance gains in both simulated and real-robot tasks, highlighting improvements in success rates and compositional generalization.

ETC-Guided Bridging for Robust VLM-to-VLA Adaptation

Introduction

Adapting large-scale Vision-LLMs (VLMs) into effective Vision-Language-Action (VLA) robot policies is fundamentally limited by the dual orthogonal gaps in input distribution and supervision objectives. VLMs are pretrained on internet-scale visual-language data with next-token prediction, whereas VLAs demand embodied scene understanding and direct action generation. Attempting to bridge both gaps simultaneously via direct fine-tuning leads to steep optimization landscapes and catastrophic degradation of pretrained generalization. The paper "Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data" (2606.08520) introduces a principled, data-driven adaptation strategy—centering on Embodied Trajectory-Coupled (ETC) supervision—to separate and gradually bridge these gaps.

Figure 1: Overview of the ETC-guided VLM-to-VLA paradigm with staged adaptation and the positioning of ETC data between general VLM and robot action data.

Embodied Trajectory-Coupled (ETC) Data

ETC data is formulated as multimodal VQA-style supervision, tightly coupled both to the embodied observations and the motion-generation goals of downstream robot tasks. Three hierarchically organized ETC categories are established:

• Scene-Grounding ETC: Probes spatial and semantic object relationships, such as cross-view correspondences and manipulation-oriented attributes.
• Task-Oriented ETC: Targets affordance reasoning and decomposing high-level instructions into sub-actions, planning stages, and validating grasp points.
• Action-Aligned Planning ETC: Directly forecasts low-level kinematics, e.g., key points along the future trajectory in the pixel/image plane.

Crucially, ETC construction is fully automated—using geometric computation from trajectories and LLM-based semantic labeling—requiring no hand annotation, which ensures scalability and alignment with action data. The ETC objective remains next-token prediction, matching VLM pretraining.

Three-Stage ETC-Based Adaptation Pathway

The adaptation strategy consists of three sequential stages, each targeting a specific axis of the VLM-to-VLA gap while retaining a unified modeling objective:

Distribution Bridging

This first stage fine-tunes the VLM backbone exclusively on ETC data (all categories), aligning both vision and language representations with the embodied scene distribution. The ViT is unfrozen, permitting vision-language grounding on manipulation-specific geometries.

Figure 3: Distribution Bridging retains VLM structure while embedding embodied task semantics; left: CKA similarity; right: t-SNE shows increased task separation post-bridging.

Objective Bridging

Subsequent training introduces action-token prediction on the already aligned embodied representation, with ETC co-training from the same scenes. This phase ensures the action objective extends capabilities without erasing the embodied alignment, mitigating representational drift.

Figure 5: Objective Bridging with ETC co-training preserves representational alignment as measured by CKA and attention maps, outperforming action-only supervision.

Retentive Adaptation

The final phase adapts the VLA policy to deployment-specific domains, maintaining ETC co-supervision on new objects and layouts. This ensures capability transfer to new settings without catastrophic forgetting or knowledge erosion.

Figure 6: Retentive Adaptation extends real-world task competence and yields robust attention on OOD targets, demonstrating strong transfer without knowledge collapse.

Compositional Induction via OOD ETC

ETC co-training additionally enables compositional generalization. By introducing OOD ETC corresponding to new visual-language compositions, the policy can bind unseen scene factors to previously learned action behaviors—even in the absence of supporting OOD action data. The generation of OOD ETC is shown feasible through LLM or diffusion-based visual synthesis, thus scaling generalization without new data collection.

Figure 8: OOD ETC induces representations supportive of compositional generalization—shown via planning probes, t-SNE, and attention visualizations shifting focus to unseen targets.

Experimental Analysis

Extensive evaluations on diverse simulation (VLABench, LIBERO, SimplerEnv) and real-robot platforms (WidowX, coffee-bag and flower tasks) validate the following:

• Distribution-Bridged VLM initialization consistently yields higher success rates and better embodied understanding, with, for example, +7.1 SR on VLABench T1 versus direct adaptation.
• ETC co-training in Objective Bridging raises execution metrics (SR, PS) by up to 20% over action-only baselines; each ETC subcategory confers specialized improvements (e.g., action-aligned ETC drives execution, task-oriented ETC improves intention).
• Retentive Adaptation is only effective when preceded by ETC conditioning; skipping prior ETC leads to performance deficits of up to 30% in real robot deployment.
• OOD ETC, when mixed at even low ratios (e.g., 6%), yields marked improvements in OOD task performance—e.g., coffee-bag placement OOD accuracy rises from 41.67 to 91.67 (when leveraging completely synthetic OOD ETC).

Theoretical and Practical Implications

The work provides strong empirical confirmation that input and objective alignment must be decoupled during VLM-to-VLA adaptation. ETC enables staged, gradient-preserving optimization that avoids representational collapse, a challenge unaddressed by conventional multimodal co-training. The modular ETC construction pipeline affords scalable, platform-agnostic deployment across new robotic embodiments and tasks.

From a theoretical stance, the ETC paradigm systems the critical role of distributionally matched, objective-aligned intermediate supervision in transferring foundation model generalization to embodied control. By isolating and analyzing each training stage, the authors establish a recipe for regularization and extension of pretraining knowledge that could generalize to other foundation model adaptation scenarios (e.g., language-to-language-action, multimodal to control).

Limitations and Future Directions

The ETC-based pathway increases training compute cost due to additional backbone updates per stage, and cross-embodiment generalization is not yet validated. The approach opens avenues for sample-efficient, compositionally generalizable robot skill acquisition—future work should explore ETC-guided adaptation to other sensorimotor domains, continual learning with life-long ETC construction, and invariant ETC pipelines for transfer among diverse robotic morphologies.

Conclusion

ETC-guided staged adaptation provides a robust and principled solution to the VLM-to-VLA transfer bottleneck, yielding policies that preserve and extend pretraining generalization, effectively bridge data and objective gaps, and enable compositionally efficient robot learning. The automatic construction of ETC supervision positions this approach as a scalable foundation for next-generation generalist VLA models (2606.08520).