Force-Sensitive Manipulation Tasks (ForceVLA)

Last updated: June 9, 2025

ForceVLA introduces a robust, generalizable framework for executing force-sensitive, contact-rich manipulation tasks ° by tightly integrating force sensing ° into the heart of modern vision-language-action ° (VLA °) robotic models. Here’s a structured, implementation-focused synthesis of its approach and contributions across all requested dimensions:

1. ForceVLA Framework: Unified Architecture with Force Sensing

ForceVLA extends the classical VLA paradigm (vision-language-action) by treating 6D external force/torque as a primary, synchronous input modality alongside vision, language, and proprioception °. The system is centered on the following key architectural innovations:

• Multi-modal Input: Observations per timestep are concatenated:
  • Vision: Synchronized wrist-mounted and third-person RGB-D images °.
  • Language: Task instructions or queries.
  • Proprioception: End-effector ° pose, joint states, gripper aperture.
  • Force/Torque: Real-time, 6D F/T vector at the robot end-effector.
• Pretrained Vision-Language Backbone: A large SigLIP-based VLM ° (e.g., PaliGemma) encodes the vision-language input into task-conditioned embeddings.
• Force Token Integration: The 6-axis force input is linearly projected into a fixed-size embedding (“force token”), appended to the sequence output of the VLM.
• Fusion Module ° (FVLMoE): The appended sequence, now spanning visual, language, and force tokens, is sent to the Force-aware Vision-Language Mixture-of-Experts ° (FVLMoE) for dynamic multimodal fusion ° (see section 2).
• Action Decoder: Conditioned on the fused multimodal context, a flow-based decoder (flow-matching or diffusion) generates full action chunks (sequences of end-effector position/gripper commands).
• Closed-Loop Execution: Generated actions are enacted with direct feedback from force signals, providing reactive adaptation to subtle contact dynamics °.

2. FVLMoE Module: Deep, Phase-Aware Force-Visual-Language Fusion

The core innovation enabling effective exploitation of force sensors in the high-dimensional policy space ° is the Force-aware Mixture-of-Experts:

• MoE Structure: FVLMoE contains multiple expert MLPs ° (E=4 found optimal).
• Expert Routing: Instead of statically mixing sensor modalities, a learned “router °” determines, for every token (vision, language, force), which expert(s) to activate, based on joint context (task phase, detected contact, etc.).

# Pseudocode illustrating dynamic expert routing for a token x
dispatch_weights = router(x)  # Token/context dependent
routed_output = sum(dispatch_weights[i] * expert_MLP[i](x) for i in range(E))

• Late Fusion ° Design: Force signals are fused downstream of the VLM after vision-language context ° is resolved, preserving pretrained feature distributions ° and maximizing force-action context synergy. Early fusion ° (injecting force into the VLM inputs) was empirically shown to degrade performance.
• Phase-Specific Specialization: Analysis of router statistics revealed that certain experts specialize in approach, contact, and insertion/force-intensive task phases. This dynamic specialization handles the challenge of contact timing and fine-grained force adaptation.
• Guidance to Decoder: The output of FVLMoE is used as direct guidance to the flow-based action decoder, enabling tokens from vision, language, and force to jointly determine the full action sequence.

3. ForceVLA-Data: Synchronized Multimodal Dataset

Force-sensitive manipulation policy learning ° and evaluation are powered by a new dataset:

• Sensor Streams:
  • Vision: Both egocentric (wrist) and allocentric (third-person) RGB-D °.
  • Force/Torque: 6D readings at robot end-effector in world frame.
  • Proprioception: TCP ° pose, gripper width, joint states.
  • Annotated Actions: Stepwise end-effector/gripper targets.
• Tasks: Five diverse, contact-rich scenarios:
  1. Bottle pumping (vertical force control)
  2. Plug insertion (fine alignment, then pushing)
  3. USB ° insertion
  4. Whiteboard wiping (continuous force regulation over trajectory)
  5. Cucumber peeling (strong sustained contact and adaptation)

• Scale: 244 human expert teleoperated trials, yielding 140,000 aligned multimodal timesteps.

• Collection Method: VR teleoperation ° of a Flexiv 7-DOF arm, ensuring natural, context-rich demonstrations.

Code, data, and preprocessing pipelines are announced for open release, enabling benchmarking and rapid experimentation.

4. Performance: ForceVLA vs. Baselines

Performance metrics and results highlight the impact of robust force sensing and fusion:

• Average Success Rate Across Tasks (in real-world hardware eval):
  • ForceVLA: 60.5%
  • Best pi₀-base w/ F baseline: 40.2%
  • pi₀-base w/o F: 37.3%
  • Naive or early fusion MoE: ≤55%
  • ForceVLA (FVLMoE) best-case: 80% (plug insertion)
• Task-Level Highlights:
  • Plug insertion: 80% (ForceVLA), notably robust to misalignment and visual occlusion.
  • Cucumber peeling: Longest average peel per stroke (14.12 cm), fewest strokes (7 vs. 14 for baseline), most reliable completion.
• Generalization:
  • In occlusion and object-variation experiments, ForceVLA achieved 80–90% success (vs ≤60% for baselines).
• Ablation studies: Confirmed drastic performance drops with early fusion or naive force mixing; only late, expert-based force fusion (FVLMoE) provides consistent, phase-aware improvements.

5. Practical Applications and Future Directions

Practical Application Scenarios:

• Industrial assembly: Plug/connector/USB insertion where visual information is missing or ambiguous, necessitating fine force feedback °.
• Assistive & household robotics: Cleaning, food prep, or delicate grasping where tactile cues drive safe, successful manipulation.
• Tool use: Polishing, peeling, surface finishing—tasks where continuous force regulation is essential and vision alone is unreliable or occluded.

Research Implications & Future Work:

• Multimodal Policy Design: The success of phase-aware, expert-based fusion sets a new design paradigm for integrating disparate sensing modalities ° in robotics.
• Reconfigurable Sensors: Adapting the FVLMoE to fuse additional touch or slip sensors could further enhance manipulation robustness for generalized hands.
• Temporal Expert Routing: Extending dynamic expert specialization ° to handle longer, multi-phase tasks or to learn “skill modules” for subtasks.
• Extension to Lower-cost Sensing: The force-first-class paradigm could be ported to lower-cost or wearable sensors, enabling broader adoption.
• Community Benchmarking: The dataset and code will serve as a standard for evaluating future multimodal manipulation policies, accelerating development.

6. Code, Data, and Reproducibility

• Code & Models: Training scripts, model weights, FVLMoE implementation, and data preprocessing will be fully open sourced.
• ForceVLA-Data: Synchronized, multi-modal sequences will enable replicable development and fair comparison.
• Usage Tools: VR teleoperation and demonstration capture tools are being provided.

Project and resource link: https://sites.google.com/view/forcevla2025/

Summary Table

In summary:

ForceVLA establishes a practical, scalable, and robust paradigm for force-sensitive manipulation by elevating force as a fundamental policy input, using late expert-based fusion to adaptively and intelligently handle the challenges of physical contact. The methodology, code, and public benchmarks ° create a solid foundation for the next generation of physically intelligent robotic control °.