Papers
Topics
Authors
Recent
Search
2000 character limit reached

Learning Hybrid-Control Policies for High-Precision In-Contact Manipulation Under Uncertainty

Published 21 Apr 2026 in cs.RO, cs.AI, and cs.LG | (2604.19677v1)

Abstract: Reinforcement learning-based control policies have been frequently demonstrated to be more effective than analytical techniques for many manipulation tasks. Commonly, these methods learn neural control policies that predict end-effector pose changes directly from observed state information. For tasks like inserting delicate connectors which induce force constraints, pose-based policies have limited explicit control over force and rely on carefully tuned low-level controllers to avoid executing damaging actions. In this work, we present hybrid position-force control policies that learn to dynamically select when to use force or position control in each control dimension. To improve learning efficiency of these policies, we introduce Mode-Aware Training for Contact Handling (MATCH) which adjusts policy action probabilities to explicitly mirror the mode selection behavior in hybrid control. We validate MATCH's learned policy effectiveness using fragile peg-in-hole tasks under extreme localization uncertainty. We find MATCH substantially outperforms pose-control policies -- solving these tasks with up to 10% higher success rates and 5x fewer peg breaks than pose-only policies under common types of state estimation error. MATCH also demonstrates data efficiency equal to pose-control policies, despite learning in a larger and more complex action space. In over 1600 sim-to-real experiments, we find MATCH succeeds twice as often as pose policies in high noise settings (33% vs.~68%) and applies ~30% less force on average compared to variable impedance policies on a Franka FR3 in laboratory conditions.

Summary

  • The paper introduces MATCH, an action-distribution design that conditions policy learning on the selected position or force mode, eliminating misleading gradients from unused action outputs.
  • The paper shows that supervised mode selection improves exploration and performance, with MATCH achieving 97.0% simulated success, zero breaks in 500 trials, and strong robustness to ±7.5 mm localization noise.
  • The paper demonstrates zero-shot transfer across 1,600 Franka FR3 insertions, where MATCH reached 68% success versus 33% for pose control under high noise while applying about 30% less force than VICES.

Overview

This paper addresses a long-standing gap between classical hybrid position-force control and modern model-free reinforcement learning (RL). The authors, Brown, Hollinger, and Lee of Oregon State University, present what they describe as the first framework to fully integrate hybrid position-force control — with discrete per-axis mode selection — into on-policy, model-free RL. Their central contribution is Mode-Aware Training for Contact Handling (MATCH), an action-distribution formulation that aligns the policy's probability structure with how the underlying hybrid controller actually consumes its outputs. They validate the approach on fragile peg-in-hole (fPiH) tasks under substantial state-estimation uncertainty, in simulation and across 1600 sim-to-real insertions on a Franka FR3.

Motivation and problem setting

Pose-based RL policies output end-effector pose targets executed by a stiff PD controller. Such policies have no explicit force authority and rely on carefully tuned low-level gains to avoid damaging contact forces. Variable impedance control (VICES) offers indirect force regulation by letting the policy modulate controller gains, but direct force control — argued as essential for contact-rich tasks since Raibert and Craig's hybrid formulation — has not been fully exploited in model-free RL due to three obstacles: training in a mixed discrete-continuous action space, controller stability, and sim-to-real transfer. Prior work sidestepped these difficulties either by fixing the selection matrix Λ\Lambda a priori (Anand et al.) or by blending position and force commands continuously via parallel controllers or admittance schemes, both of which limit expressiveness or permit contradictory objectives on the same axis.

The task is fPiH: insert a peg into a hole with ~0.5 mm clearance without exceeding a break-force threshold FthF_{th} (10 N in simulation, 20 N on hardware), cast as a POMDP trained with PPO using asymmetric actor-critic in the Factory/IsaacLab simulator. Observations are noisy estimates of relative end-effector/hole pose, velocity, and forces; the critic receives privileged ground-truth state.

Hybrid action space and the MATCH formulation

The policy outputs, per control dimension ii: a Bernoulli selection probability ϕi(ot)\phi_i(o_t) over Λii\Lambda_{ii}, a pose-target distribution, and a force-target distribution, each scaled tanh-transformed Gaussians. The low-level controller executes pose control where Λii=1\Lambda_{ii}=1 and proportional force control (kf(fd−fee)k_f(f_d - f_{ee})) elsewhere; orientation remains pose-controlled throughout due to torque-sensing unreliability.

Under a standard mixed discrete-continuous formulation ("Hybrid-Basic"), the joint action probability factors into independent products of all outputs regardless of which mode was selected. The authors identify this as a credit-assignment defect: gradients propagate through pose or force outputs that the controller never used, yielding uninformative updates. MATCH instead defines the per-dimension action distribution piecewise — conditioning on the sampled mode so that only the selected component's density contributes:

c(ai)={p(Λii∣ϕi) p(xdi)Λii=1 (1−p(Λii∣ϕi)) p(fdi)Λii=0c(a_i) = \begin{cases} p(\Lambda_{ii}\mid\phi_i)\, p(x_{d_i}) & \Lambda_{ii}=1 \ (1-p(\Lambda_{ii}\mid\phi_i))\, p(f_{d_i}) & \Lambda_{ii}=0 \end{cases}

This is the paper's core algorithmic insight, and it is supported empirically: Hybrid-Basic converges markedly slower than Pose, while MATCH trains at rates comparable to Pose despite its larger action space.

A second contribution is the supervised selection loss (SSL): binary cross-entropy between ϕi\phi_i and the true directional contact indicator ψi\psi_i, added to the PPO objective. The motivation is an exploration pathology — only 20–40% of training steps involve contact, and free-space force control risks unbounded acceleration, so early failures suppress exploration of force modes entirely. Ablations quantify this sharply: removing SSL costs Hybrid-Basic 11.2% and MATCH 15% in final success rate, and no-SSL variants select force control less than 0.1% of the time while making 3x less contact. Notably, SSL provides safety-relevant supervision without any task-specific penalty reward, avoiding the overly conservative local optima reported when force penalties appear in the reward (Huang et al.).

Simulation results

In nominal conditions, hybrid methods dominate: Hybrid-Basic reaches 98.2% success with 0.4% breaks, and MATCH reaches 97.0% success with zero peg breaks across all 500 evaluation trials. An interesting inversion appears here: hybrid policies apply higher average force than Pose (~3 N vs lower values), which the authors attribute to deliberate sustained contact against the hole rim during alignment — higher but still-safe forces that improve search stability.

Robustness experiments add uniform hole-localization noise up to ±7.5 mm on an 8 mm hole — error comparable to the hole diameter. At the extreme, hybrid policies maintain roughly 40% success versus ~30% for Pose (up to +10%), and hybrid break rates remain below half those of Pose across all noise levels. VICES performs statistically indistinguishably from the hybrid methods in simulation. A Pose variant trained with wider noise (σ=2.5 mm) proved difficult to train reliably, with some seeds collapsing into overly conservative behavior — a concrete instance of the fragility of reward-based approaches to uncertainty.

Two further analyses strengthen the claims. First, the learned mode-selection profile is semantically meaningful and mirrors hand-designed strategies such as spiral search: near-zero force probability in free space, a z-dominant force-control spike upon contact that persists through search, then a decline as position control takes over descent during insertion. Quantitatively, MATCH selects force control in 84±10.8% of in-contact steps and only 2.9±2.1% of free-space steps, indicating effective implicit contact detection. Second, an ablation with unbreakable pegs shows MATCH and VICES still outperform Pose by 17.1% and 21.1% respectively under noise, demonstrating that force regulation yields generally useful behaviors beyond satisfying explicit force constraints.

Sim-to-real transfer

Policies trained with dynamics randomization were deployed zero-shot on a Franka FR3 for 1600 total insertions. Two practical caveats deserve note: for Pose and Hybrid-Basic, predicted standard deviations had to be rescaled to 5% of their values to achieve high performance, and real force estimates derive from filtered joint-torque readings rather than a wrist sensor. Under increasing localization noise, MATCH outperforms Pose by up to +35 percentage points in the 7.5 mm condition, and in high-noise settings succeeds twice as often as Pose (68% vs 33%). Against VICES, MATCH achieves comparable success while applying ~30% less average force; VICES additionally exhibits substantially higher peak forces. Only one break occurred across all real trials (a Pose run).

Limitations and open questions

Several limitations are acknowledged or evident. Orientation control remains purely pose-based because wrist torque sensing proved unreliable, so the method does not yet exercise full six-dimensional hybrid selection. Real-world force control depends on filtered joint-torque estimates assumed to act at the end-effector frame, an approximation whose fidelity is not independently validated. The necessity of scaling down predicted action standard deviations for two of four methods on hardware suggests residual sim-to-real gaps in stochasticity calibration that the paper does not resolve. Evaluation is confined to circular pegs at a single clearance (~0.5 mm); generalization across geometries, clearances, and other in-contact tasks (welding, cable insertion) remains untested. Finally, whether the SSL's reliance on ground-truth contact indicators transfers to settings without reliable contact labels — e.g., from learned contact estimators — is left open, as is the question of whether discrete mode selection retains its advantage over continuous blending on tasks requiring simultaneous compliant tracking in multiple axes.

Conclusion

This paper demonstrates that hybrid position-force control with discrete, learned per-axis mode selection is trainable with model-free RL, provided the action distribution is structured to match controller consumption (MATCH) and mode exploration is scaffolded by contact-supervised selection losses. The empirical case is strong: parity with Pose in sample efficiency, zero simulated breaks in nominal conditions, halved break rates under extreme localization noise, and zero-shot hardware transfer where MATCH doubles Pose's success rate in high-noise regimes while matching VICES performance at ~30% lower force. The result positions direct force control as a practical alternative to variable impedance action spaces for fragile, uncertainty-laden manipulation.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.