BORA: Bridging Offline Reinforcement Learning and Online Residual Adaptation for Real-World Dexterous VLA Models

Abstract: Vision-Language-Action (VLA) models have emerged as a promising paradigm for grounding visual-language understanding into real-world robotic manipulation. However, dexterous manipulation remains challenging for VLA policies due to high-dimensional hand control and compounding execution errors, which makes real-world RL post-training essential for bridging the gap between visually grounded action generation and physically reliable dexterous execution. However, high-dimensional dexterous exploration often triggers temporal inconsistency, sample inefficiency and hardware risks in the real world. To address these challenges, we propose BORA, an offline-to-online RL post-training framework designed for real-world dexterous VLA models. In the offline phase, BORA constructs a critic that takes both the VLM's cognition tokens and action chunks as inputs. This design enables action-conditioned value guidance, allowing the critic to evaluate dexterous hand motions beyond visual context alone. During the subsequent online phase, BORA freezes the VLA base and introduces a lightweight, Human-in-the-Loop (HiL) chunk-wise residual adaptation mechanism to mitigate real-world execution errors and further correct the offline-learned intents within the actual physical environment. By inheriting the offline critic and employing intervention-driven rewards, BORA effectively corrects execution discrepancies and adapts to real-world physical variances while preserving the pretrained policy as a stable prior. Extensive evaluations across five complex real-world dexterous tasks demonstrate that BORA significantly outperforms pure imitation learning and traditional decoupled RL baselines, achieving a 33% absolute increase in average success rate under standard settings and up to a 43% improvement in unseen object generalization.

• The paper introduces BORA, a framework that fuses offline RL with online HiL residual adaptation for precise dexterous VLA manipulation.
• It leverages a consistency policy actor and action-conditioned critic to overcome challenges in high-dimensional actions and visual occlusion.
• Empirical results show significant improvements with success rates up to 86% in standard tasks and robust generalization in unseen scenarios.

Bridging Offline RL and Online Residual Adaptation for Real-World Dexterous VLA Models

Overview and Motivation

The paper "BORA: Bridging Offline Reinforcement Learning and Online Residual Adaptation for Real-World Dexterous VLA Models" (2605.30226) introduces BORA, a unified offline-to-online reinforcement learning (RL) framework tailored for Vision-Language-Action (VLA) models performing dexterous manipulation. VLA models, leveraging vision and language grounding for robotic control, face severe challenges in dexterous manipulation due to high-dimensional action spaces, action diversity, visual occlusion, and compounding execution errors. Existing approaches—imitation learning and decoupled post-training RL—struggle with temporal consistency, inefficient exploration, critic overfitting, and catastrophic representation drift during real-world deployment.

BORA addresses these limitations by fusing action-conditioned value guidance during offline RL with a lightweight human-in-the-loop (HiL) chunk-wise residual adaptation in the online phase. This composition enables precise physical intent extraction, conservative online adaptation, and robust deployment performance without sacrificing generalization or stability.

Methodology

Offline RL with Action-Conditioned Critic

BORA's offline stage centers on a consistency policy actor and a joint action-conditioned critic:

• Consistency Policy Actor: Unlike diffusion-based action heads requiring hundreds of denoising iterations (leading to gradient noise and credit assignment failures), the actor leverages a consistency policy to generate continuous action chunks in 1-3 steps. This truncation efficiently routes RL gradients back into the vision-language backbone, supporting robust intent extraction amid high-dimensional, multimodal action spaces.
• Action-Conditioned Critic: The critic is explicitly conditioned on both VLM cognition tokens and action chunks. This fusion avoids overfitting to visual artifacts and ensures value estimation hinges on physical contact dynamics. The critic outputs step-wise value vectors, propagates credit within each action chunk via shifted value bootstrapping, and utilizes a masked Bellman residual alongside an IQL-style expectile value objective, facilitating conservative policy improvement while regularizing against suboptimal demonstrations.

Online Residual Adaptation

Upon real-world deployment, BORA initiates an online adaptation phase:

• Frozen VLA Base: The offline-trained VLA model is frozen, preventing catastrophic feature drift typical in end-to-end online RL fine-tuning.
• Residual Chunk Actor: A lightweight residual actor, parameterized using an MLP, generates corrective action chunks based on proprioceptive state, base action, and VLM tokens. Final actions are composed via $A_{final} = A_{base} + \alpha A_{res}$, where $\alpha$ regulates intervention magnitude to maintain smooth action manifolds.
• Critic Inheritance and HiL Intervention: The inherited critic landscape from offline RL provides conservative value guidance during online residual policy optimization. Human intervention trajectories are integrated through a reward-driven RLPD protocol at a 1:1 ratio with offline data, enforcing sample efficiency and robust adaptation under covariate shifts. An asymmetric reward structure penalizes OOD drift, incentivizes recovery, and aggressively regularizes against high-risk states.

Empirical Evaluation

BORA was evaluated on five dexterous manipulation tasks using a Franka arm with a 12-DoF hand, comparing against baselines including pure imitation learning, diffusion-based VLA models, consistency policy without RL, and decoupled offline RL critics.

Key results:

• Offline RL Improvements: BORA-Offline delivers strong intent alignment and improved generalization, raising average success rates from 53.0% (IL baseline) to 67.0% (standard) and from 27.0% to 52.0% (object-unseen).
• Online Adaptation Gains: BORA-Full, combining offline RL and HiL residual adaptation, elevates average success rates to 86.0% (standard) and 70.0% (object-unseen), representing a 33% absolute increase and up to 43% improvement in unseen object generalization relative to baselines.
• Sample-Efficiency: Adaptation converges rapidly with minimal human intervention (1-2 per task, 20% trajectory time).
• Critic Inheritance Robustness: The value function inherited from offline RL demonstrates stable, discriminative estimation, supporting safe residual policy improvement even under severe visual occlusion and distributional shifts.

These outcomes directly affirm the efficacy of BORA's architectural choices: truncating denoising for viable gradient flows, semantic token-action conditioning for critic robustness, and conservative online adaptation.

Analysis of Failure Modes and Limitations

Failure analysis reveals that:

• Pure IL and decoupled RL baselines frequently suffer from unstable grasps, off-center contacts, and susceptibility to occlusion-driven misguidance.
• BORA's chunk-level adaptation and inherited critic significantly mitigate such errors, but residual limitations remain—primarily reliance on visuo-proprioceptive inputs (absence of dense tactile sensing) and evaluation constrained to a single hand-arm topology.

These limitations suggest paths for future research, including integration of tactile arrays for enhanced perception-action coupling and broader embodiment generalization.

Implications and Future Directions

BORA's demonstrated improvements in dexterous VLA policy reliability and generalization have several implications:

• Practical: The framework is highly relevant for real-world robotic manipulation tasks requiring fine-grained, contact-rich execution, minimal retraining, and safe adaptation to new object instances without full model retraining or extensive exploration.
• Theoretical: BORA advances RL post-training methodology by fusing offline semantic intent extraction and online conservative residual adaptation. The action-conditioned critic architecture and residual actor design inform future research into value-guided RL in high-dimensional, multimodal action spaces, especially in the presence of occlusion and noisy demonstrations.

Moving forward, extending BORA to multi-modal sensory integration (e.g., tactile feedback), validating cross-embodiment generalization, and scaling to long-horizon tasks will further push the boundaries of dexterous manipulation with vision-language-action models.

Conclusion

BORA presents a principled offline-to-online RL post-training framework for dexterous VLA models, resolving key challenges in physical intent extraction, critic regularization, and real-world adaptation. Through action-conditioned critics, consistency-based chunk generation, and lightweight HiL residual correction, BORA achieves robust performance and strong generalization with minimal human intervention, outperforming standard IL and decoupled RL baselines. The approach constitutes a compelling solution for reliable, adaptive dexterous manipulation in robotics, with clear theoretical underpinnings and practical efficacy (2605.30226).