Gated Egocentric Residual Policy (GERP)

• GERP is a modular policy architecture for robotic imitation that fuses dual-view inputs with a learned gating mechanism for context-aware control.
• It integrates a stable wrist-view policy with an egocentric residual branch to correct actions during occlusion and ambiguous scenes.
• Empirical results indicate that GERP improves success rates and task phase performance compared to wrist-only and direct fusion methods.

The Gated Egocentric Residual Policy (GERP) is a modular policy architecture introduced in the context of robot-free demonstration learning to robustly fuse complementary wrist-mounted and egocentric (head-mounted) sensory observations. By employing a residual correction mechanism gated by a learned scalar, GERP maintains the stability of a well-established wrist-view policy while leveraging global context from an egocentric perspective to improve behavioral robustness—particularly under challenging visual conditions such as local occlusion or ambiguous local geometry. GERP was presented as a component of EgoGuide, a dual-view demonstration collection and learning system targeting data-efficient robotic imitation from heterogeneous visual inputs (Xu et al., 12 Jun 2026).

1. Motivation and Problem Setting

Traditional Universal Manipulation Interface (UMI) pipelines are limited to recording from a wrist-mounted camera, resulting in inefficiencies for long-horizon or occlusion-prone tasks due to a lack of global scene awareness. EgoGuide addresses this by recording time-synchronized observations from both a wrist-mounted and a head-mounted (egocentric) camera, paired with precise pose information for each. The Gated Egocentric Residual Policy is the policy learning architecture designed for this dual-observation regime. It satisfies three principal constraints:

• Preserves reliability of the base wrist-view policy under ordinary conditions.
• Allows corrective actions based on egocentric/global context, especially during occlusions or ambiguous wrist perspectives.
• Prevents catastrophic interference or distraction from irrelevant egocentric observations via a learned gating function.

The architecture is explicitly designed for robust policy performance in demonstration-driven robotic manipulation where local visual cues may be insufficient or momentarily unreliable (Xu et al., 12 Jun 2026).

2. Mathematical Formulation

GERP operates on temporally aligned dual-view inputs consisting of wrist and head images and 6-DoF poses at each timestep, along with a task instruction embedding. The following summarizes the inputs, policy outputs, and action fusion mechanism:

Inputs at timestep $t$

• Wrist image: $I^W_t \in \mathbb{R}^{3 \times 224 \times 224}$
• Wrist pose: $T^W_t \in SE(3)$
• Head image: $I^H_t \in \mathbb{R}^{3 \times 224 \times 224}$
• Head pose: $T^H_t \in SE(3)$
• Task instruction: $\ell$ (either one-hot or language embedding)

Coordinate Transformation

• Wrist pose in head-frame: $T^{H\leftarrow W}_t = (T^H_t)^{-1} T^W_t \in SE(3)$

Action Representation

• Action chunk: $$\mathbf{A} \in \mathbb{R}^{K \times (d_{\Delta p} + d_{\Delta q} + d_g)}$$, where each step contains:
  • Relative translation $\Delta p \in \mathbb{R}^3$
  • Relative rotation $\Delta q \in \mathbb{R}^4$ (quaternion)
  • Gripper width $I^W_t \in \mathbb{R}^{3 \times 224 \times 224}$0

Policy Structure

1. Base policy (wrist-only):

$I^W_t \in \mathbb{R}^{3 \times 224 \times 224}$1

Trained via diffusion-based flow-matching objective to match demonstration actions.

2. Residual branch with gate:

$I^W_t \in \mathbb{R}^{3 \times 224 \times 224}$2

$I^W_t \in \mathbb{R}^{3 \times 224 \times 224}$3: residual action proposal; $I^W_t \in \mathbb{R}^{3 \times 224 \times 224}$4: scalar gate via sigmoid.

3. Action fusion:

$I^W_t \in \mathbb{R}^{3 \times 224 \times 224}$5

Training Protocol

• Stage 1: Train $I^W_t \in \mathbb{R}^{3 \times 224 \times 224}$6, freeze $I^W_t \in \mathbb{R}^{3 \times 224 \times 224}$7.
• Stage 2: Freeze $I^W_t \in \mathbb{R}^{3 \times 224 \times 224}$8, train $I^W_t \in \mathbb{R}^{3 \times 224 \times 224}$9 (including $T^W_t \in SE(3)$0).
• Loss:

$T^W_t \in SE(3)$1

• Curriculum on $T^W_t \in SE(3)$2: increases linearly during residual/gate training.

3. Network Architecture

GERP is implemented as two distinct policy networks with respective visual and pose encoders. The table below summarizes the architectural components.

Branch	Encoder	Pose Conditioning	Output Head(s)
Wrist-only base ($T^W_t \in SE(3)$3)	ResNet50-style CNN	$T^W_t \in SE(3)$4 through MLP, injected into UNet	Action chunk
Egocentric residual ($T^W_t \in SE(3)$5)	Separate, untied CNN	$T^W_t \in SE(3)$6 (translation + quaternion through 2-layer MLP)	Action chunk and scalar gate ($T^W_t \in SE(3)$7)

The residual branch fuses egocentric features, transformed pose embedding, and instruction embedding into a shallow UNet or MLP. The two parallel output heads respectively produce the residual action chunk proposal and a scalar logit for gating, which becomes $T^W_t \in SE(3)$8 after sigmoid activation.

Inference procedure:

$I^H_t \in \mathbb{R}^{3 \times 224 \times 224}$9

Optimization:

• Optimizer: AdamW, weight decay $T^W_t \in SE(3)$9
• Learning rate: cosine decay $I^H_t \in \mathbb{R}^{3 \times 224 \times 224}$0
• Batch size: 128 action chunks
• Stage lengths: 30k steps each for $I^H_t \in \mathbb{R}^{3 \times 224 \times 224}$1, then $I^H_t \in \mathbb{R}^{3 \times 224 \times 224}$2.

Preprocessing:

• Synchronize frames within 20 ms, resize to $I^H_t \in \mathbb{R}^{3 \times 224 \times 224}$3, ImageNet normalization, pose normalization (translation/rotation z-score, unit quaternion).

4. Empirical Evaluation

GERP was evaluated on real-world manipulation (e.g., pepper sorting) using 400 human demonstrations captured via EgoGuide. Metrics:

• Success Rate (SR)
• Task Phase Success (TPS)

Results:

Method	SR (%)	TPS (%)
Wrist Only	75	77.5
Wrist+Ego Direct	65	72.5
GERP	80	87.5

Notable findings:

• Under forced occlusion of the wrist camera, GERP’s gate $I^H_t \in \mathbb{R}^{3 \times 224 \times 224}$4 increases (approximately 0.8), selectively activating the egocentric branch, leading to a ~10% higher success rate versus wrist-only.
• Gating behavior: $I^H_t \in \mathbb{R}^{3 \times 224 \times 224}$5 remains low when the target is visible in the wrist camera; rises when occluded (see App. Fig. A.12 in (Xu et al., 12 Jun 2026)).
• Under test-time egocentric camera viewpoint shifts, GERP is less sensitive (<5% drop) versus direct input concatenation (up to 15% drop).

5. Ablations and Analysis

Ablation studies in (Xu et al., 12 Jun 2026) confirm the necessity of both the residual and gating components:

• No gate ($I^H_t \in \mathbb{R}^{3 \times 224 \times 224}$6): Performance degrades to wrist+ego direct, occasionally below wrist-only due to egocentric distractions.
• Gate only (no residual loss, $I^H_t \in \mathbb{R}^{3 \times 224 \times 224}$7): The gate shuts off ($I^H_t \in \mathbb{R}^{3 \times 224 \times 224}$8), reducing to wrist-only performance.
• Full GERP (with curriculum, both losses): Most robust across tasks and perturbations.

A plausible implication is that proper curriculum and dual loss functions are required for the residual policy to effectively complement, rather than disrupt, the base policy.

6. Extensions and Considerations

GERP's architecture decouples a locally stable controller from a context-driven corrector, moderated by a learned, state-dependent scalar gate. The gating mechanism is critical to avoid performance collapse when the egocentric view is either uninformative or noisy. The approach is flexible: additional sensing modalities (e.g., force-torque) could be accommodated by separate residual branches, each with an individualized gating function. Future research may consider replacing the scalar policy-level gate with a learned attention mask over the action dimensions, potentially enabling more selective cross-modal fusion (Xu et al., 12 Jun 2026).

GERP serves as a reference implementation for robust multi-view policy fusion in demonstration-driven robotic imitation, particularly under uncertain or partial local sensory conditions.