FALCON: Actively Decoupled Visuomotor Policies for Loco-Manipulation with Foundation-Model-Based Coordination (2512.04381v1)

Abstract: We present FoundAtion-model-guided decoupled LoCO-maNipulation visuomotor policies (FALCON), a framework for loco-manipulation that combines modular diffusion policies with a vision-language foundation model as the coordinator. Our approach explicitly decouples locomotion and manipulation into two specialized visuomotor policies, allowing each subsystem to rely on its own observations. This mitigates the performance degradation that arise when a single policy is forced to fuse heterogeneous, potentially mismatched observations from locomotion and manipulation. Our key innovation lies in restoring coordination between these two independent policies through a vision-language foundation model, which encodes global observations and language instructions into a shared latent embedding conditioning both diffusion policies. On top of this backbone, we introduce a phase-progress head that uses textual descriptions of task stages to infer discrete phase and continuous progress estimates without manual phase labels. To further structure the latent space, we incorporate a coordination-aware contrastive loss that explicitly encodes cross-subsystem compatibility between arm and base actions. We evaluate FALCON on two challenging loco-manipulation tasks requiring navigation, precise end-effector placement, and tight base-arm coordination. Results show that it surpasses centralized and decentralized baselines while exhibiting improved robustness and generalization to out-of-distribution scenarios.

• The paper presents decoupled, diffusion-based policies for locomotion and manipulation that achieve 100% task success across varied conditions.
• It leverages a frozen CLIP model and a phase-progress head to semantically align subsystem actions without manual phase annotations.
• Experimental results demonstrate superior robustness and modularity, outperforming centralized baselines in both in-distribution and OOD scenarios.

FALCON: Decoupled Visuomotor Policies with Foundation Model Coordination for Loco-Manipulation

Introduction

FALCON introduces a novel framework for robot loco-manipulation that decouples locomotion and manipulation into dedicated diffusion policies, coordinated via a vision-language foundation model. This design directly addresses the instability and performance degradation encountered when monolithic policies are forced to fuse heterogeneous sensor observations and control signals on legged mobile manipulators. The guiding hypothesis is that explicit decoupling, with restored semantic coordination through foundation model embeddings, produces more robust, generalizable, and modular policies for challenging whole-body tasks.

System Architecture and Design

FALCON implements an actively decoupled control paradigm. Locomotion and arm manipulation are governed by independent diffusion-based visuomotor policies, acting on distinct observation and control spaces. The quad's locomotion policy outputs high-level velocity and pose commands, leveraging a reinforcement-learning-trained low-level controller for stable execution. The arm policy produces end-effector targets converted via IK to joint commands. Critical to the framework is the coordination layer: a frozen CLIP model (with adapters) encodes global RGB and proprioception, along with language instructions and phase prompts, into a shared latent $z_t$. This latent conditions both subsystems for semantically aligned behavior.

Figure 1: FALCON architecture showing the decoupled manipulator/quadruped diffusion policies acting on their respective observations, coordinated via a shared VLM-derived latent.

The architecture further incorporates a phase-progress head that infers discrete task phase and continuous progress from language prompts, operating zero-shot with no manual phase annotations. For explicit cross-modality consistency, a coordination-aware contrastive loss is applied, aligning the CLIP embedding with paired arm-base actions while repelling mismatched pairings. This mechanism leverages negative samples generated by shuffling subsystem action pairs across the batch, enforcing semantic and behavioral compatibility in the shared latent.

Experimental Tasks and Setup

Experiments focus on two demanding real-robot loco-manipulation settings with an arm-mounted quadruped (Unitree Go2 base, Airbot arm, RGB-only tri-camera configuration):

• Task 1 (Precise Manipulation): Navigate to a cabinet, open a closed drawer, pick up a toy, and place it into the drawer.
• Task 2 (Mobile Manipulation): Navigate with a held toy to an open drawer, place the toy, and close the drawer.

Data collection emphasizes diversity and OOD robustness by randomizing initial positions (with most demonstrations from the center region) and manipulating drawer states and toy locations.

Figure 2: Task stages visualized—left, the system navigates, opens a drawer, and places a toy; right, mobile navigation and placing.

Figure 3: Evaluation regions for Task 2, highlighting OOD starting positions—center-trained, left/right as OOD test.

Baselines include centralized diffusion policies (CDP), Action Chunking Transformers (ACT), and the decentralized LatentToM diffusion architecture.

Results and Analysis

Numerical Results

FALCON achieves 100% task-level success rates across all initializations in both tasks, surpassing all baselines. Notably, centralized baselines (CDP, ACT) exhibit severe failure modes, especially in OOD settings and tasks requiring precise base-arm coordination. ACT fails to achieve any complete task success in Task 1, while FALCON and LatentToM (decentralized) show markedly higher robustness.

Figure 4: Success rates for both tasks across regions and methods—FALCON consistently outperforms all baselines, especially on OOD.

Key ablations demonstrate that both the language-inferred phase/progress head and the contrastive coordination loss contribute substantially to final success rates and robustness, especially in in-distribution trials. The absence of contrastive loss allows greater navigation adaptability in some OOD conditions but marginally reduces overall robustness, highlighting a trade-off between coupling and subsystem autonomy.

Teleoperation and Subsystem Robustness

Human-in-the-loop experiments further stress the modularity of the architecture. Swapping in teleoperated base or arm control demonstrates that FALCON's policies maintain 100% subsystem success rate when paired with human-operated counterparts, confirming robust inter-policy coordination independent of control regime.

Figure 5: Teleoperation success rates—FALCON versus LatentToM for subsystem-level control, showing superior robustness.

Subsystem analysis indicates that FALCON's manipulation policy is robust even under noisy, off-distribution navigation, with phase/progress-informed latent context enabling stable manipulation. Conversely, when teleoperating the arm, the leg policy autonomously achieves base configurations that facilitate human manipulation.

Coordination Mechanisms

The contrastive loss significantly improves coherence between decoupled policies, aligning the CLIP-extracted context with joint arm-base actions. The phase/progress head, grounded only in task- and phase-specific language prompts, introduces a weak supervision scaffold stabilizing training and decomposing highly multimodal action distributions into tractable components.

Figure 6: Task 1 language instruction and phase prompt design.

Figure 7: Task 2 language instruction and phase prompt design.

Implications and Future Directions

FALCON's results establish that decoupling coupled with foundation-model-guided latent alignment can outperform monolithic architectures, particularly as task complexity, embodiment asymmetry, and distribution shift increase. Practically, this yields modular, interoperable subsystems, facilitating human-collaborative or plug-and-play approaches and promising a path toward generalizable robot learning architectures. Theoretically, foundation models as coordination relays (rather than as action decoders) appear more data- and compute-efficient, enabling specialization and robustness while maintaining semantic coherence.

Potential future advances include:

• Automatic phase discovery and richer phase prompt generation via LLMs, reducing reliance on handcrafted language prompts for multi-phase and long-horizon tasks.
• Expanding to multi-modal sensor modalities and broader manipulation taxonomies, assessing compositional generalization capabilities.
• Transfer of trained subsystems across diverse robot morphologies and tasks, exploiting the decoupling for rapid domain adaptation.

Conclusion

FALCON demonstrates that deliberate subsystem decoupling paired with foundation-model-informed coordination yields robust, sample-efficient, and highly modular visuomotor policies for loco-manipulation on complex mobile manipulation platforms. The architecture outperforms centralized and prior decentralized baselines, exhibiting strong generalization in out-of-distribution scenarios and impressive human-collaborative flexibility. This work supports a shift toward modular, foundation model-coordinated architectures as critical for scaling up real-world robotic autonomy.