Gated Egocentric Residual Policy

• The paper introduces GERP as a two-expert policy architecture that preserves wrist-view stability while employing an egocentric branch to correct local ambiguities.
• GERP uses a learned gating mechanism to blend wrist-based and head-view actions, ensuring robust control under occlusion and partial visibility.
• The staged training and architectural decomposition improve data efficiency, outperforming simple concatenation of multimodal inputs in challenging robotic tasks.

The Gated Egocentric Residual Policy (GERP) is a policy architecture introduced in EgoGuide: Egocentric Guidance for Efficient Robot-Free Demonstration Collection and Learning for imitation learning from synchronized wrist and head/egocentric observations in real-world robot-free demonstration pipelines (Xu et al., 12 Jun 2026). It is designed to address a specific failure mode of wrist-only Universal Manipulation Interface (UMI)-style learning: the wrist camera is stable and well aligned with the manipulation interface, but it is often too local to disambiguate the task when the object is occluded, partially out of view, or when the local appearance is ambiguous. GERP therefore retains a wrist-view policy as a strong base controller and adds an egocentric branch that can “nudge” the action when broader head-view context is useful. Its central design principle is to preserve the stability of wrist-view control while allowing global scene context to correct local ambiguity only when needed.

1. Definition and problem setting

GERP is situated within EgoGuide, a data-collection and learning framework that records synchronized wrist and head/egocentric observations and couples them with online visual-geometric data quality guidance (Xu et al., 12 Jun 2026). Within that setting, the policy problem is not simply multimodal fusion in the generic sense. The paper identifies an asymmetry between the two views: the wrist view is stable and tightly coupled to manipulation, whereas the head/egocentric view provides broader scene context but is viewpoint-varying and not always informative.

This asymmetry determines the architectural goal. EgoGuide does not replace the wrist-view policy with an egocentric policy. Instead, it keeps the wrist-view policy as a strong base controller and introduces a separate egocentric residual expert that can modify behavior selectively. The intended use case is precisely the regime in which local wrist perception becomes ambiguous: object occlusion, partial visibility, limited field of view, or visually confusing local appearances. In such states, the egocentric observation can supply information about object arrangement, workspace layout, and global task state that is unavailable to the wrist camera.

The policy is therefore best understood in the paper’s own terms as a two-expert, gated residual imitation policy. The wrist branch is the stable expert, the egocentric branch is the context-sensitive corrective expert, and a learned gate determines how much correction to apply. This framing is significant because it rejects a monolithic policy that concatenates all observations indiscriminately; the decomposition itself is the mechanism used to preserve wrist-view stability.

2. Architectural decomposition and input structure

GERP is trained in two stages, beginning with a wrist-only base policy (Xu et al., 12 Jun 2026). The first-stage policy is

$\mathbf{A}^b = \pi_b(I^W, T^W, \ell),$

where $I^W$ is the wrist image, $T^W$ is the wrist pose in the collection/world frame, and $\ell$ is the task instruction. The output $\mathbf{A}^b$ is an action chunk in the wrist-relative action space used by UMI.

The second stage freezes this base policy and adds an egocentric residual branch. That branch takes the head/egocentric image $I^H$, the wrist pose expressed in the head frame, and the task instruction:

$T^{H\leftarrow W} = (T^H)^{-1}T^W,$

$$(\mathbf{A}^r,\alpha) = \pi_r(I^H, T^{H\leftarrow W}, \ell).$$

The conditioning structure is deliberate. The residual branch receives the egocentric/head-view image $I^H$, the wrist pose relative to the head frame $T^{H\leftarrow W}$, and the task instruction $I^W$0. The pose transform is important because it expresses the wrist pose in the coordinate system of the head camera, making the pose input geometrically consistent with the egocentric image.

A key architectural fact is that there is no direct concatenation of wrist and head features into a single monolithic policy in the final design. The final system is explicitly a two-branch decomposition: a stable wrist policy plus a separate egocentric residual expert. This separation is the architectural choice that preserves wrist-view stability. A common misconception is that GERP is simply a multimodal policy with an additional camera input; the paper’s formulation instead treats egocentric context as a selective corrective signal layered on top of an already competent wrist-centric controller.

3. Gating mechanism and residual action composition

The defining mechanism in GERP is the scalar gate $I^W$1, which controls how much the final action deviates from the wrist base policy toward the egocentric candidate (Xu et al., 12 Jun 2026). The fused action is

$I^W$2

The interpretation given in the paper is explicit. When $I^W$3, the system trusts the wrist-only base policy almost entirely. When $I^W$4, it trusts the egocentric candidate more strongly. Intermediate values interpolate between the two. The gate is therefore a selective correction mechanism rather than merely an abstract confidence score: it is a learned mixing coefficient that decides when egocentric context is actually useful.

An important technical clarification concerns the meaning of “residual.” The residual branch does not predict an additive delta $I^W$5 explicitly. Instead, it predicts a complete action candidate $I^W$6 in the same action space as the base policy, and the final output is formed by interpolation. The paper highlights two consequences. First, $I^W$7, $I^W$8, and $I^W$9 all live in the same wrist-relative action space, so they can be mixed directly. Second, the egocentric branch does not overwrite the wrist policy; it only shifts the action toward a different candidate when the gate permits it.

The coordinate transform $T^W$0 is also used in a constrained way. The paper explicitly states that it is used only as residual input, not to change the action target. During training, the residual branch is supervised with the same ground-truth action chunk $T^W$1 as the base policy. This makes the residual branch a full action predictor in the same convention as the wrist policy, which in turn makes the gating-based interpolation straightforward.

4. Training objective, staged optimization, and implementation details

GERP uses a staged optimization procedure in which the wrist-view base policy is trained first, frozen, and then augmented with the egocentric residual branch (Xu et al., 12 Jun 2026). The objective for the second stage is

$T^W$2

Here, $T^W$3 is the same flow-matching objective used for the base diffusion policy, applied to the egocentric residual branch. The $T^W$4 term supervises the composed final action to match the demonstration action $T^W$5. The coefficients $T^W$6 and $T^W$7 balance the two terms.

The training schedule is likewise staged. The procedure is: train the wrist-view base policy first, freeze the base policy, train the egocentric residual branch, and use a curriculum on the composed-action loss. The curriculum starts with $T^W$8, linearly ramps it to 1, and then continues with both terms balanced. The paper states that this warm-up is important because it allows the residual branch to first learn to produce reasonable egocentric actions before being forced to coordinate with the gate and the frozen base policy.

The implementation details given for EgoGuide specify an action horizon $T^W$9, images resized to $\ell$0, training for 30K steps, batch size 128, and a cosine learning rate from $\ell$1 to $\ell$2. GERP residual training is also run for 30K steps, with the composed loss introduced after a 15K-step warm-up and ramped over 10K steps. These details situate GERP as a practically specified learning procedure rather than a purely conceptual fusion layer.

5. Inference behavior and robustness to viewpoint variation

At inference time, GERP computes the wrist-only base action $\ell$3, the egocentric candidate $\ell$4, the gate $\ell$5, and the final fused action $\ell$6 (Xu et al., 12 Jun 2026). The system therefore executes a gated correction, not a hard switch. The base policy remains the default behavior, and the egocentric branch only modulates it.

The robustness claim in the paper derives from the complementarity of the two views. The wrist view is stable and tightly coupled to the manipulator, so it is reliable for local control. The head/egocentric view provides broader scene context, including object arrangement, workspace layout, and global task state, which the wrist camera may miss under occlusion or limited field of view. Robustness arises in two ways: when the wrist image is ambiguous or occluded, the egocentric branch can supply a better action candidate; when the wrist image is already sufficient, the gate suppresses unnecessary corrections and keeps behavior stable.

The paper also argues that human head motion is often not a controllable action signal. For that reason, GERP is trained to use a fixed egocentric camera as auxiliary context rather than learning an active head policy. This choice is presented as more realistic for deployment and as a way to avoid dependence on the variability of active head motion.

A central empirical justification for the gating mechanism is that feeding the head image directly into the policy can hurt performance because the moving egocentric viewpoint may be mismatched with pretraining, noisy, or distracting. The ablation labeled Wrist+Ego Direct is reported to show that simply concatenating wrist and egocentric views can degrade performance, whereas GERP improves it. The paper further notes that the gate is especially high when the target object is occluded or missing from the wrist view, which matches the intended behavior.

6. Empirical interpretation, limitations of direct fusion, and broader significance

The paper reports that real-world experiments show that EgoGuide reduces the required number of data episodes and improves data efficiency, and that the residual policy further improves robustness under visual occlusion (Xu et al., 12 Jun 2026). Within the ablations specifically associated with GERP, Wrist+Ego Direct sometimes performs worse than Wrist Only, while GERP consistently outperforms both on tasks such as Pepper Sorting and Rubik’s Cube. The appendix additionally notes that the gate increases when the target object is occluded or missing from the wrist view, and that GERP is more stable than the direct baseline under different fixed egocentric camera placements.

These results support the paper’s justification for gating. The egocentric stream is useful but unreliable as a universal control input: the head camera viewpoint can vary, the egocentric stream may be redundant in many states, direct conditioning on it can distract training, and it can mismatch the wrist-centric policy prior. The gate solves two problems simultaneously: selectivity, by using egocentric context only when it helps, and stability, by preserving the wrist policy’s strong local control as the default.

Several misconceptions are explicitly addressed by the design. GERP is not a replacement of wrist-centric control with egocentric control. It is not a hard-switch controller. It is not a literal additive-residual method in which the second branch predicts only $\ell$7. Instead, it is a selective interpolation between two action candidates in the same wrist-relative action space. This distinction matters because it explains why the method can recover from ambiguity and occlusion without sacrificing the stability associated with wrist-view manipulation.

In the broader context of EgoGuide, GERP is tightly coupled to the data-collection interface itself. EgoGuide collects synchronized wrist and head/egocentric data, and GERP exploits those complementary views without abandoning the reliability of wrist-based manipulation control. A plausible implication is that the architecture is tailored to settings in which one modality is locally precise but globally incomplete, while another modality is globally informative but viewpoint-varying. In the formulation given by the paper, the result is a policy that remains anchored to stable wrist-view behavior while using egocentric context as a conservative, learned corrective signal.