Tune to Learn: How Controller Gains Shape Robot Policy Learning

Abstract: Position controllers have become the dominant interface for executing learned manipulation policies. Yet a critical design decision remains understudied: how should we choose controller gains for policy learning? The conventional wisdom is to select gains based on desired task compliance or stiffness. However, this logic breaks down when controllers are paired with state-conditioned policies: effective stiffness emerges from the interplay between learned reactions and control dynamics, not from gains alone. We argue that gain selection should instead be guided by learnability: how amenable different gain settings are to the learning algorithm in use. In this work, we systematically investigate how position controller gains affect three core components of modern robot learning pipelines: behavior cloning, reinforcement learning from scratch, and sim-to-real transfer. Through extensive experiments across multiple tasks and robot embodiments, we find that: (1) behavior cloning benefits from compliant and overdamped gain regimes, (2) reinforcement learning can succeed across all gain regimes given compatible hyperparameter tuning, and (3) sim-to-real transfer is harmed by stiff and overdamped gain regimes. These findings reveal that optimal gain selection depends not on the desired task behavior, but on the learning paradigm employed. Project website: https://younghyopark.me/tune-to-learn

• The paper shows that compliant, overdamped gains improve behavior cloning, achieving 85.1% versus 39.0% success on Block Stacking and 70.1% versus 45.6% on Dishwasher Opening, while also reducing sensitivity to action noise.
• The paper finds that reinforcement learning can succeed across widely varying gain regimes, including 99%+ success, when action-space and environment hyperparameters are tuned for each controller setting.
• The paper demonstrates that stiff, overdamped gains can harm zero-shot sim-to-real transfer despite producing better system-identification error, with trajectory error reaching 0.043 versus 0.010 in a representative reaching task.

Overview

This paper investigates a design decision that is pervasive in modern robot learning yet rarely examined: the choice of proportional–derivative (PD) gains in position controllers used to execute learned manipulation policies (2604.02523). The authors' central argument is that conventional gain-selection logic—choosing gains to achieve desired task compliance or stiffness—breaks down when controllers are paired with state-conditioned policies, because task-level impedance emerges from the interplay between learned reactions and control dynamics rather than from gains alone. They reframe gains as an inductive bias over closed-loop behaviors that shapes learnability, and systematically study how gain settings affect behavior cloning (BC), reinforcement learning (RL) from scratch, and zero-shot sim-to-real transfer across multiple tasks and embodiments.

The paper's three headline findings are: BC performs best under compliant, overdamped gains; RL can succeed across all gain regimes given compatible hyperparameter tuning; and sim-to-real transfer is degraded by stiff, overdamped gains. Together these imply that optimal gain selection depends on the learning paradigm employed, not on the desired task behavior—a conclusion that directly contradicts prevailing practice, since tracking analysis of DROID and Open X-Embodiment datasets shows stiff controllers are an implicit default in existing data collection pipelines.

Decoupling gains from task compliance

Before studying learning effects, the paper establishes empirically that task-level compliance is predominantly determined by learned reactions rather than underlying gains. Two counterintuitive RL experiments demonstrate this: a policy trained with a sharp distance-based reward achieves stiff task-level behavior despite $32\times$ lower actuator stiffness than a second policy trained with softened objectives and action-smoothness penalties, which yields compliant behavior despite stiff low-level gains. Force-displacement system identification confirms the stiff-behavior policy attains roughly $5\times$ higher effective Cartesian stiffness than the compliant-behavior policy. This result substantiates the paper's framing of gains as interface parameters rather than behavioral parameters, and motivates treating gain selection as a question about learnability.

Behavior cloning favors compliant, overdamped gains

A methodological contribution underpins the BC study: Torque-to-Position Retargeting (TPR), a two-stage procedure that generates demonstrations at 500 Hz using torque commands as a gain-agnostic action representation, then converts them into position targets for arbitrary $(\mathbf{K}_p, \mathbf{K}_d)$ settings via $$\mathbf{q}_{\text{des}}(t) = \mathbf{q}(t) + \mathbf{K}_p^{-1}(\boldsymbol{\tau}(t) + \mathbf{K}_d\dot{\mathbf{q}}(t))$$. TPR holds the state distribution nearly fixed while varying only the induced action distribution, isolating the causal effect of gains on learning. Validation confirms fidelity: retargeted trajectories maintain ${\geq}90\%$ success rate and joint-position MSE below $10^{-3}$ up to $25\times$ decimation, with degradation only for contact-rich phases at higher decimation—an acknowledged boundary of the method's trajectory-matching assumption.

Across six manipulation tasks spanning two robot embodiments, closed-loop success rates consistently concentrate in the compliant, overdamped regime. The effect is statistically robust: Bonferroni-corrected one-sided Barnard's exact tests reject the null hypothesis in all six tasks ($p \ll 0.0083$), with logistic regression confirming lower $\mathbf{K}_p$ and higher $\mathbf{K}_d$ as significant success predictors. Representative contrasts are stark—for Block Stacking, the compliant-overdamped region achieves 85.1% success versus 39.0% elsewhere; for Dishwasher Opening, 70.1% versus 45.6%. The preference persists across ablations of architecture (MLP vs. Transformer), policy class (regression, VAE, diffusion), chunking, action representation, control frequency, and dataset size.

Two observations sharpen the mechanism. First, policies trained under compliant gains exhibit higher validation MSE than those trained under stiff gains, yet achieve better closed-loop performance—lower imitation loss does not translate to better policy performance. Second, under identical injected action noise in open-loop execution, compliant gains maintain high success while stiff gains fail entirely. The appendix formalizes this with a theorem showing steady-state position error variance under stochastic action noise scales as $5\times$0, independent of mass: low stiffness reduces the force produced by a given action error, while high damping dissipates perturbations faster. The authors note this analysis assumes fixed noise variance, whereas prediction difficulty itself varies with gains; the attenuation effect dominating prediction difficulty is established empirically rather than analytically. Notably, compliant-overdamped gains also yield steeper data-scaling curves, so demonstration budgets return more per trajectory in this regime.

A user study (12 participants, 1,297 trials) addresses the practical concern that sluggish compliant robots might hinder teleoperation. With each gain configuration evaluated at its own optimized input mapping, compliant-overdamped settings achieve comparable or better success rates, completion times, and subjective ratings than stiffer settings. An end-to-end experiment on a real Franka Research 3—collecting 100 teleoperated demonstrations per corner gain configuration and training per-gain policies—confirms the composed effect: the learning advantage is not offset by distributional differences in teleoperated data. The implication is direct: adopting compliant, overdamped gains for imitation learning imposes no penalty on data collection, contradicting the stiff defaults found in DROID and RT-X datasets.

Reinforcement learning is gain-agnostic given environment shaping

For online RL, the paper controls for what it calls environment shaping—the strong dependence of RL outcomes on hyperparameters and environment design—by tuning action-space parameters per gain setting via Optuna, evaluating each configuration at its best achievable performance. Three results emerge:

• Solution existence: successful policies (99%+ success) were discovered across all four corner gain regimes for seven tasks spanning FR3 manipulation, Unitree G1 locomotion, and Allegro in-hand manipulation—gains spanning over two orders of magnitude in both $5\times$1 and $5\times$2. Unlike BC, on-policy RL trains on its own exploration data, apparently enabling compensatory behaviors without relying on controller error attenuation.
• Hyperparameter sensitivity: the gain setting modulates the optimization landscape, but no regime is consistently easier to optimize. The authors concede this could reflect genuine task-dependence or an artifact of optimizing only a low-dimensional slice of the full hyperparameter space.
• Sample efficiency: training reward curves and seed variance are comparable across regimes once a working configuration is identified, with one exception—the compliant, underdamped regime on G1 Track-Velocity yielded only one marginally successful configuration.

The practical implication is that gain choice need not constrain RL pipelines, provided hyperparameters are co-tuned; conversely, fixed hyperparameters inherited from one gain regime may silently fail under another.

Sim-to-real transfer degrades under stiff, overdamped gains

The sim-to-real study uses reaching tasks on a Franka Research 3 with gain-specific system identification (CMA-ES over per-actuator parameters, spectral MSE objective). A notable inversion appears here: system identification achieves its lowest modeling error under stiff, overdamped gains—over an order of magnitude lower than other regimes—yet these same gains produce the largest zero-shot sim-to-real trajectory error. Bonferroni-corrected Mann-Whitney U tests reject the null hypothesis in all three conditions ($5\times$3); e.g., Joint-Reach without domain randomization shows mean trajectory error of 0.043 in the stiff-overdamped region versus 0.010 elsewhere. The dominant failure mode is high-frequency oscillation during closed-loop policy execution, which persists despite domain randomization, even though the controller itself is stable under smooth commands. Policy-output divergence between matched sim and real states correlates strongly with trajectory error, indicating the gap arises from out-of-distribution observations rather than low-level instability.

The proposed explanation is that stiff, overdamped controllers amplify small modeling errors by responding to deviations with high torques; when the policy reacts to unmodeled dynamics, aggressive tracking pushes the system further from simulated state distributions. Consequently, choosing gains that minimize open-loop modeling error can paradoxically increase the closed-loop sim-to-real gap—a caution against SysID-driven gain selection. A frequency ablation offers a mitigation: jitter failures drop from 21.8% at 100 Hz policy frequency to 5.0% at 10 Hz, consistent with prior findings that longer zero-order-hold durations let joints settle before out-of-distribution transients are amplified, at the cost of temporal resolution. One acknowledged limitation is the fidelity gap between the real robot's 1 kHz torque loop and the simulator's 100 Hz physics rate, which contributes to the very discrepancy being studied.

Limitations and open questions

Several caveats bound the conclusions. The BC findings rest on TPR's trajectory-matching assumption, which weakens for contact-rich tasks at high decimation. The analytical error-variance result assumes i.i.d. Gaussian-like action errors and fixed noise variance, whereas actual prediction difficulty varies with gains. The RL hyperparameter-landscape analysis covers only a subset of tunable parameters, so claims of paradigm-agnosticism are conditional on the chosen shaping procedure. Sim-to-real experiments are limited to reaching tasks on a single embodiment, leaving contact-rich transfer untested. The paper also leaves open whether its trends extend to RL-trained whole-body tracking controllers used as low-level interfaces on humanoids, and to cross-embodiment imitation paradigms that treat observed next-state as the action label—an assumption of perfect tracking that the BC results suggest may be suboptimal.

Conclusion

This paper reframes position-controller gains as learning-interface parameters whose optimal values depend on the training paradigm: compliant, overdamped gains for behavior cloning; any regime for RL with compatible hyperparameters; and avoidance of stiff, overdamped configurations for sim-to-real transfer. By combining controlled dataset generation (TPR), statistical hypothesis testing, user studies, and hardware deployment, it provides actionable guidance against the implicit stiff-gain default observed in large-scale datasets, and identifies the mismatch between open-loop system identification quality and closed-loop transferability as a concrete pitfall in current practice.