Residual Copilot Policy

• Residual copilot policy is a framework that combines a fixed base controller with a learned residual component to adaptively correct actions and maintain safety.
• It employs diverse formulations such as additive corrections, weighted blending, and probabilistic mixtures across applications like shared autonomy and autonomous parking.
• By retaining domain-specific priors and optimizing through methods like PPO, TD3, and SAC, it enhances performance metrics and reduces exploration complexity.

Residual copilot policy denotes a class of control and decision architectures in which a primary source of action—such as a human operator, a conventional controller, a motion prior, a planner, a behavior policy, or a next-action model—is not replaced outright, but is instead modified, blended, or conditionally overridden by a learned residual component. In the cited literature, this pattern appears in shared autonomy, humanoid locomotion, autonomous parking, racing, industrial process control, teleoperation, congestion-mitigating driver advisory, enterprise workflow automation, offline reinforcement learning, antenna control, and online policy customization (Schaff et al., 2020, Li et al., 25 Sep 2025, Sohn et al., 2020). The central design choice is a decomposition of competence: the base component supplies structure, priors, or safety-relevant regularity, while the residual component supplies adaptation, correction, reactivity, or calibrated autonomy.

1. Canonical formulations

The most direct formulation is additive. In shared autonomy, the executed action is the sum of the human command and the residual correction,

$$a = a_h + a_r, \qquad a_h\sim \pi_h(\cdot\mid s),\quad a_r\sim \pi_r(\cdot\mid s,a_h),$$

with matching continuous action spaces $\mathcal A_h=\mathcal A_r=\mathbb R^d$ (Schaff et al., 2020). RuN uses the same structural idea in joint space: a pre-trained Conditional Motion Generator produces $q_{\mathrm{ref},t}$, and the residual policy outputs $\Delta a_t$, yielding

$$q_{\mathrm{target},t}=q_{\mathrm{ref},t}+\Delta a_t$$

for the low-level PD controller (Li et al., 25 Sep 2025).

Other works generalize residualization beyond plain vector addition. In teleoperation, the final slave-side command is a weighted blend

$X_{sc}=W_m X_{cpf} + (I-W_m)X_{mf},$

where the learned agent outputs the diagonal arbitration weights $W_m$ online (Wang et al., 2021). In BRPO, the learned policy is a state-action-dependent mixture of the behavior policy $\mu$ and a candidate policy $\rho$,

$$\pi(a\mid s)=(1-\lambda(s,a))\mu(a\mid s)+\lambda(s,a)\rho(a\mid s),$$

so the residual degree itself is learned as $\mathcal A_h=\mathcal A_r=\mathbb R^d$0 (Sohn et al., 2020). In $\mathcal A_h=\mathcal A_r=\mathbb R^d$1-RPO, the mean of the fused policy is

$\mathcal A_h=\mathcal A_r=\mathbb R^d$2

with $\mathcal A_h=\mathcal A_r=\mathbb R^d$3 ramped from $\mathcal A_h=\mathcal A_r=\mathbb R^d$4 to $\mathcal A_h=\mathcal A_r=\mathbb R^d$5 over the first $\mathcal A_h=\mathcal A_r=\mathbb R^d$6 PPO iterations (Trumpp et al., 13 Mar 2026). CoPark combines prior and residual in log-probability space, with separate reliance coefficients for acceleration and steering channels (Wei et al., 2 Jun 2026).

These formulations show that residual copilot policy is not confined to a single algebraic pattern. The recurring invariant is architectural: a privileged or pre-existing action source remains explicit, and the learned component acts through correction, arbitration, attenuation, or release rather than de novo control generation.

Setting	Base term	Residual combination
Shared autonomy	Human action $\mathcal A_h=\mathcal A_r=\mathbb R^d$7	$\mathcal A_h=\mathcal A_r=\mathbb R^d$8
Humanoid locomotion	CMG reference $\mathcal A_h=\mathcal A_r=\mathbb R^d$9	$q_{\mathrm{ref},t}$0
Teleoperation	$q_{\mathrm{ref},t}$1	$q_{\mathrm{ref},t}$2
Offline RL	Behavior policy $q_{\mathrm{ref},t}$3	$q_{\mathrm{ref},t}$4
Autonomous racing	Base controller $q_{\mathrm{ref},t}$5	$q_{\mathrm{ref},t}$6
Reactive parking	Offline plan prior	Prior–residual fusion in log-probability space

2. Sources of priors and task decomposition

The base component in a residual copilot system is typically chosen to encode structure that is difficult or undesirable to relearn online. In RuN, the frozen prior is a Conditional Motion Generator trained on a curated, retargeted human-motion dataset from AMASS, with approximately 400 sequences spanning $q_{\mathrm{ref},t}$7–$q_{\mathrm{ref},t}$8; it supplies a kinematically natural motion prior, while the residual policy only learns dynamic stabilization and task tracking (Li et al., 25 Sep 2025). The paper reports that this decoupling dramatically reduces the residual policy’s exploration dimension and yields approximately $q_{\mathrm{ref},t}$9–$\Delta a_t$0 faster convergence with superior final reward.

In CoPark, the prior is an offline geometric plan produced by Hybrid A* + Reeds–Shepp and tracked by a Stanley controller. The learned residual head supplies reactive corrections under self-play, while a partner-threat-modulated, channel-asymmetric release mechanism relaxes the longitudinal prior more strongly than the lateral prior, so yielding can occur without sacrificing sub-meter slot alignment (Wei et al., 2 Jun 2026). This suggests that residualization can be structured at the channel level rather than only at the whole-action level.

In process control, CoL-SDRPRL uses a conventional controller $\Delta a_t$1 as the expert baseline and activates the residual only inside a region of interest identified through an Input–Output HMM and a pretrained value function (Abbas et al., 2023). Outside that region, the expert controller remains in full control. In congestion mitigation, CARP uses a piecewise-constant nominal advisory policy that proposes a constant speed, and the residual policy adds a small offset to the nominal advice; the personalized variant conditions the residual on a latent driver-trait vector inferred by a VAE (Hasan et al., 2024). In enterprise support workflows, the next-action policy proposes actions over a schema-driven BPM interface, and the critic decides whether critical actions are executed automatically or deferred to the operator (Borovkov et al., 26 Apr 2026).

Across these systems, the prior is not merely an initialization. It carries domain-specific invariants: human-like kinematics in locomotion, slot-frame geometry in parking, plant-safe behavior in industrial control, advisory continuity in traffic, or workflow legality in BPM execution. The residual then specializes to whatever the prior omits: dynamics, threat response, abnormality handling, personalization, or calibrated abstention.

3. Optimization regimes

Residual copilot policies are trained with heterogeneous optimization strategies, and the choice of optimizer follows the structure of the base component and the residual’s operational role.

RuN trains the residual policy with on-policy PPO under

$\Delta a_t$2

with reward decomposition

$\Delta a_t$3

The imitation terms keep the realized motion close to the CMG reference, the task terms track commanded linear and angular velocity, and regularization includes alive bonus, joint-limit violations, torque effort, action smoothness, and foot-slip penalties (Li et al., 25 Sep 2025). The actor and critic are asymmetric two-layer LSTMs with 256 units each, and training converges in approximately $\Delta a_t$4 hours on a single RTX 4090.

Shared autonomy frames the residual as a constrained optimization problem: minimize $\Delta a_t$5 subject to achieving at least a threshold return under a goal-agnostic general reward, then solve the dualized problem with a constrained PPO variant (Schaff et al., 2020). CoPark also uses PPO, but with V-trace, parameter sharing, a scene-density curriculum, and $\Delta a_t$6 environment steps in multi-agent self-play (Wei et al., 2 Jun 2026). In $\Delta a_t$7-RPO, attenuation is integrated into PPO by conditioning the policy on $\Delta a_t$8 and using importance sampling to compensate the rollout/optimization mismatch; the paper identifies the “synchronization trick” as critical, and reports training collapse when it is disabled (Trumpp et al., 13 Mar 2026).

Other works use off-policy actor-critic. CoL-SDRPRL uses TD3, with behavioral cloning pretraining of the residual, critic warmup, online replay, and a composite Cycle-of-Learning loss

$\Delta a_t$9

(Abbas et al., 2023). MBRPL trains a stochastic residual policy with SAC while fitting a Gaussian dynamics model

$$q_{\mathrm{target},t}=q_{\mathrm{ref},t}+\Delta a_t$$0

to support imagined rollouts, thereby interleaving real interaction, model learning, and model-based actor-critic updates (Möllerstedt et al., 2022). Teleoperation uses DDPG both for delay compensation and for arbitration-weight learning (Wang et al., 2021). CARP uses TRPO for the residual advisory policy and a separate LSTM-based VAE for unsupervised trait inference (Hasan et al., 2024).

A distinct line appears in offline and online planning settings. BRPO derives a conservative lower bound on policy improvement and jointly optimizes the candidate policy $$q_{\mathrm{target},t}=q_{\mathrm{ref},t}+\Delta a_t$$1 and state-action confidence $$q_{\mathrm{target},t}=q_{\mathrm{ref},t}+\Delta a_t$$2 by coordinate ascent, using a relative-softmax update for $$q_{\mathrm{target},t}=q_{\mathrm{ref},t}+\Delta a_t$$3 and a concave quadratic program for $$q_{\mathrm{target},t}=q_{\mathrm{ref},t}+\Delta a_t$$4 (Sohn et al., 2020). Residual-MPPI performs no policy-network fine-tuning at all; it treats the prior policy as a black box, adds a residual control sequence through MPPI, and scores rollouts with a term that includes both the new reward and $$q_{\mathrm{target},t}=q_{\mathrm{ref},t}+\Delta a_t$$5 (Wang et al., 2024). A plausible implication is that “residual copilot policy” names an architectural decomposition more than a single learning algorithm.

4. Authority management, safety, and calibrated intervention

A defining issue in residual copilot systems is not only how residuals are learned, but when and how much authority they are granted. The literature presents several distinct mechanisms.

In shared autonomy, minimal intervention is part of the objective itself: the residual is trained to minimize correction magnitude while satisfying a return threshold under a goal-agnostic reward (Schaff et al., 2020). In CoL-SDRPRL, authority is region-gated: when the inferred IO-HMM hidden state belongs to the abnormal region $$q_{\mathrm{target},t}=q_{\mathrm{ref},t}+\Delta a_t$$6, the residual activates; otherwise the action is exactly $$q_{\mathrm{target},t}=q_{\mathrm{ref},t}+\Delta a_t$$7 (Abbas et al., 2023). In CoPark, authority is threat-modulated continuously. The threat signal

$$q_{\mathrm{target},t}=q_{\mathrm{ref},t}+\Delta a_t$$8

is converted into a saturated interaction intensity $$q_{\mathrm{target},t}=q_{\mathrm{ref},t}+\Delta a_t$$9, and the reliance scalars become

$X_{sc}=W_m X_{cpf} + (I-W_m)X_{mf},$0

with $X_{sc}=W_m X_{cpf} + (I-W_m)X_{mf},$1 so that longitudinal authority is released much more strongly than lateral authority (Wei et al., 2 Jun 2026).

In enterprise support automation, critical actions are governed by explicit abstention. The critic $X_{sc}=W_m X_{cpf} + (I-W_m)X_{mf},$2 is trained from operator accept/reject feedback, and the automation threshold $X_{sc}=W_m X_{cpf} + (I-W_m)X_{mf},$3 is calibrated offline to satisfy a target precision on critic-approved critical actions, then monitored online through rolling precision and session-level safety signals (Borovkov et al., 26 Apr 2026). If the critic score falls below threshold, control returns to the operator; after any human correction, execution resumes from the new UI state.

$X_{sc}=W_m X_{cpf} + (I-W_m)X_{mf},$4-RPO uses yet another authority schedule: early in training, $X_{sc}=W_m X_{cpf} + (I-W_m)X_{mf},$5, so the base controller dominates; as $X_{sc}=W_m X_{cpf} + (I-W_m)X_{mf},$6, the base controller is phased out entirely, producing a standalone residual network at deployment (Trumpp et al., 13 Mar 2026). This is an important distinction. Residual copilot architectures do not necessarily preserve the base component permanently; some use it as a transient scaffold, others as a persistent copilot, and others as a fallback only under uncertainty.

Safety claims are likewise heterogeneous. BRPO provides a lower-bound-based safe-improvement argument relative to the behavior policy under its surrogate objective (Sohn et al., 2020). MBRPL gives a performance bound in terms of KL divergence between the true and baseline dynamics and between the learned policy and the baseline policy (Möllerstedt et al., 2022). Other systems rely on domain randomization, gating, threshold calibration, deterministic fallbacks, or explicit rollback procedures rather than a single universal guarantee (Li et al., 25 Sep 2025, Borovkov et al., 26 Apr 2026).

5. Representative applications and reported performance

The empirical record is domain-specific, but several systems report substantial gains while keeping the base prior explicit.

Domain	Representative result	Reported value
Humanoid locomotion	Stable locomotion range	$X_{sc}=W_m X_{cpf} + (I-W_m)X_{mf},$7–$X_{sc}=W_m X_{cpf} + (I-W_m)X_{mf},$8
Humanoid locomotion	Full RuN FID	$X_{sc}=W_m X_{cpf} + (I-W_m)X_{mf},$9
Shared autonomy	Lunar Lander laggy pilot success	$W_m$0
Reactive parking	Success rate with reactive partners	$W_m$1 (DLP), $W_m$2 (DSC3D)
Autonomous racing	Zero-shot real-world single-lap average	$W_m$3
Enterprise support	Automated session coverage	$W_m$4
Enterprise support	Average handling time reduction	$W_m$5

RuN reports that the CMG prior improves over a GMP prior with FID $W_m$6 versus $W_m$7 and reconstruction loss $W_m$8 versus $W_m$9, and that the full system outperforms Humanoid-Gym, AMP, and GMP with FID $\mu$0 versus $\mu$1, joint-position error $\mu$2 versus $\mu$3, and joint-velocity error $\mu$4 versus $\mu$5; velocity-tracking error remains approximately $\mu$6 and the policy transfers zero-shot to the Unitree G1 with stable walk-run transitions and no policy resets (Li et al., 25 Sep 2025).

Shared-autonomy experiments report that, for a laggy pilot on Lunar Lander, success rises from approximately $\mu$7 to approximately $\mu$8 and crash rate drops from approximately $\mu$9 to $\rho$0. In human-subject studies with $\rho$1 users per domain, success on Lunar Lander and Lunar Reacher rises from approximately $\rho$2 to approximately $\rho$3, crash rate falls from approximately $\rho$4 to approximately $\rho$5, and participants rate the learned copilot as more helpful, trustworthy, and collaborative with $\rho$6 (Schaff et al., 2020).

CoPark reports zero-shot evaluation on Dragon Lake Parking and DeepScenario Open 3D with more than $\rho$7 episodes per dataset. With reactive partners, success rate is approximately $\rho$8 on DLP and approximately $\rho$9 on DSC3D, collision rate is approximately $$\pi(a\mid s)=(1-\lambda(s,a))\mu(a\mid s)+\lambda(s,a)\rho(a\mid s),$$0, off-road rate is $$\pi(a\mid s)=(1-\lambda(s,a))\mu(a\mid s)+\lambda(s,a)\rho(a\mid s),$$1 in all settings, and terminal precision is approximately $$\pi(a\mid s)=(1-\lambda(s,a))\mu(a\mid s)+\lambda(s,a)\rho(a\mid s),$$2 and $$\pi(a\mid s)=(1-\lambda(s,a))\mu(a\mid s)+\lambda(s,a)\rho(a\mid s),$$3 (Wei et al., 2 Jun 2026). The system also exhibits reverse-yielding, mid-maneuver yielding, tight-corridor passing, and queuing without behavior-specific reward.

In autonomous racing, $$\pi(a\mid s)=(1-\lambda(s,a))\mu(a\mid s)+\lambda(s,a)\rho(a\mid s),$$4-RPO reaches average race time $$\pi(a\mid s)=(1-\lambda(s,a))\mu(a\mid s)+\lambda(s,a)\rho(a\mid s),$$5 on train maps versus $$\pi(a\mid s)=(1-\lambda(s,a))\mu(a\mid s)+\lambda(s,a)\rho(a\mid s),$$6 for RPL and $$\pi(a\mid s)=(1-\lambda(s,a))\mu(a\mid s)+\lambda(s,a)\rho(a\mid s),$$7 for DRL, and $$\pi(a\mid s)=(1-\lambda(s,a))\mu(a\mid s)+\lambda(s,a)\rho(a\mid s),$$8 on test maps versus $$\pi(a\mid s)=(1-\lambda(s,a))\mu(a\mid s)+\lambda(s,a)\rho(a\mid s),$$9 for RPL, with zero collisions. On the Munich real-world track, its single-lap average is $\mathcal A_h=\mathcal A_r=\mathbb R^d$00 versus $\mathcal A_h=\mathcal A_r=\mathbb R^d$01 for RPL and $\mathcal A_h=\mathcal A_r=\mathbb R^d$02 for FTG, with $\mathcal A_h=\mathcal A_r=\mathbb R^d$03 inference latency on a Jetson Orin Nano Super (Trumpp et al., 13 Mar 2026).

CARP reports combined-factor improvements of $\mathcal A_h=\mathcal A_r=\mathbb R^d$04–$\mathcal A_h=\mathcal A_r=\mathbb R^d$05 over piecewise-constant-policy baselines across simulation and user study, depending on hold length and whether the residual is personalized (Hasan et al., 2024). In enterprise support, selective automation automated $\mathcal A_h=\mathcal A_r=\mathbb R^d$06 of sessions and reduced average handling time from $\mathcal A_h=\mathcal A_r=\mathbb R^d$07 to $\mathcal A_h=\mathcal A_r=\mathbb R^d$08, a $\mathcal A_h=\mathcal A_r=\mathbb R^d$09 reduction, without a statistically significant drop in blind quality scores (Borovkov et al., 26 Apr 2026).

These results do not establish a single performance law for residual copilot policy. They do, however, indicate that the pattern can preserve strong inductive bias from an existing controller while improving reactivity, sample efficiency, or operational throughput in markedly different environments.

6. Conceptual distinctions, limitations, and recurring failure modes

Residual copilot policy is often conflated with simple action correction, but the literature shows several distinctions. First, residualization is not always additive in Euclidean action space. It can be a probabilistic mixture, a logit-space fusion, a learned arbitration weight, or an abstention mechanism that hands control back to a human (Sohn et al., 2020, Wei et al., 2 Jun 2026, Borovkov et al., 26 Apr 2026). Second, the residual is not always permanent. In $\mathcal A_h=\mathcal A_r=\mathbb R^d$10-RPO the base controller is intentionally eliminated at deployment, whereas in shared autonomy, teleoperation, and enterprise workflows the base authority remains structurally present (Trumpp et al., 13 Mar 2026, Schaff et al., 2020).

Limitations recur across domains. Shared autonomy relies on surrogate human models during training, and performance may degrade if real pilots differ substantially; it also inherits the high sample complexity of model-free on-policy learning, and the additive form $\mathcal A_h=\mathcal A_r=\mathbb R^d$11 may be too restrictive for tasks that require more sophisticated arbitration (Schaff et al., 2020). CoL-SDRPRL depends on the correctness of IO-HMM specialization and on the baseline controller to encode plant limits outside the region of interest (Abbas et al., 2023). $\mathcal A_h=\mathcal A_r=\mathbb R^d$12-RPO reports that longer attenuation schedules of at least $\mathcal A_h=\mathcal A_r=\mathbb R^d$13 of total steps regress toward static RPL, and disabling the synchronization trick leads to training collapse (Trumpp et al., 13 Mar 2026).

Model-based variants introduce their own dependencies. MBRPL improves sample efficiency by imagined rollouts, but its theoretical guarantee is explicitly conditioned on bounded KL divergence in both dynamics and policy space (Möllerstedt et al., 2022). Residual-MPPI avoids retraining the policy network and can operate in zero-shot or few-shot settings, but it requires access to the prior action distribution and to a possibly learned dynamics model $\mathcal A_h=\mathcal A_r=\mathbb R^d$14 (Wang et al., 2024). Enterprise selective automation obtains calibrated autonomy only with monitoring, slice-aware guardrails, deterministic fallbacks, and automatic rollback when precision or business metrics drift (Borovkov et al., 26 Apr 2026).

A recurring misconception is that residual formulations are intrinsically conservative. The evidence is mixed. Some formulations explicitly enforce small corrections or human authority, as in shared autonomy and CARP (Schaff et al., 2020, Hasan et al., 2024). Others use the base policy only as a bootstrap and ultimately replace it, as in $\mathcal A_h=\mathcal A_r=\mathbb R^d$15-RPO (Trumpp et al., 13 Mar 2026). BRPO shows that conservatism itself can be made state-action dependent through the learned confidence $\mathcal A_h=\mathcal A_r=\mathbb R^d$16 (Sohn et al., 2020). This suggests that “residual” should be understood structurally rather than semantically: it specifies how policy components are composed, not a fixed magnitude of intervention.

In aggregate, the literature treats residual copilot policy as a mechanism for factoring control into prior-guided competence and learned adaptation. What varies is the source of the prior, the algebra of composition, the optimizer, and the authority schedule. That variability is not incidental; it is the main reason the same architectural idea transfers from humanoid locomotion and autonomous parking to process control, teleoperation, advisory driving, offline RL, and enterprise workflow automation.