---
title: Gated Egocentric Residual Policy (GERP)
url: https://www.emergentmind.com/topics/gated-egocentric-residual-policy-gerp
type: topic
---

# Gated Egocentric Residual Policy (GERP)

The Gated Egocentric Residual Policy (GERP) is a modular policy architecture introduced in the context of robot-free demonstration learning to robustly fuse complementary wrist-mounted and egocentric (head-mounted) sensory observations. By employing a residual correction mechanism gated by a learned scalar, GERP maintains the stability of a well-established wrist-view policy while leveraging global context from an egocentric perspective to improve behavioral robustness—particularly under challenging visual conditions such as local occlusion or ambiguous local geometry. GERP was presented as a component of EgoGuide, a dual-view demonstration collection and learning system targeting data-efficient robotic imitation from heterogeneous visual inputs [2606.14665].

## 1. Motivation and Problem Setting

Traditional Universal Manipulation Interface (UMI) pipelines are limited to recording from a wrist-mounted camera, resulting in inefficiencies for long-horizon or occlusion-prone tasks due to a lack of global scene awareness. EgoGuide addresses this by recording time-synchronized observations from both a wrist-mounted and a head-mounted (egocentric) camera, paired with precise pose information for each. The Gated Egocentric Residual Policy is the policy learning architecture designed for this dual-observation regime. It satisfies three principal constraints:

- Preserves reliability of the base wrist-view policy under ordinary conditions.
- Allows corrective actions based on egocentric/global context, especially during occlusions or ambiguous wrist perspectives.
- Prevents catastrophic interference or distraction from irrelevant egocentric observations via a learned gating function.

The architecture is explicitly designed for robust policy performance in demonstration-driven robotic manipulation where local visual cues may be insufficient or momentarily unreliable [2606.14665].

## 2. Mathematical Formulation

GERP operates on temporally aligned dual-view inputs consisting of wrist and head images and 6-DoF poses at each timestep, along with a task instruction embedding. The following summarizes the inputs, policy outputs, and action fusion mechanism:

**Inputs at timestep $t$**

- Wrist image: $I^W_t \in \mathbb{R}^{3 \times 224 \times 224}$
- Wrist pose: $T^W_t \in SE(3)$
- Head image: $I^H_t \in \mathbb{R}^{3 \times 224 \times 224}$
- Head pose: $T^H_t \in SE(3)$
- Task instruction: $\ell$ (either one-hot or language embedding)

**Coordinate Transformation**

- Wrist pose in head-frame: $T^{H\leftarrow W}_t = (T^H_t)^{-1} T^W_t \in SE(3)$

**Action Representation**

- Action chunk: $\mathbf{A} \in \mathbb{R}^{K \times (d_{\Delta p} + d_{\Delta q} + d_g)}$, where each step contains:
  - Relative translation $\Delta p \in \mathbb{R}^3$
  - Relative rotation $\Delta q \in \mathbb{R}^4$ (quaternion)
  - Gripper width $g \in \mathbb{R}$

**Policy Structure**

1. **Base policy (wrist-only):**
   \[
   \mathbf{A}^b_t = \pi_b (I^W_t, T^W_t, \ell)
   \]
   Trained via diffusion-based flow-matching objective to match demonstration actions.

2. **Residual branch with gate:**
   \[
   (\mathbf{A}^r_t, \alpha_t) = \pi_r (I^H_t, T^{H\leftarrow W}_t, \ell)
   \]
   $\mathbf{A}^r_t$: residual action proposal; $\alpha_t \in [0, 1]$: scalar gate via sigmoid.

3. **Action fusion:**
   \[
   \hat{\mathbf{A}}_t = (1 - \alpha_t) \mathbf{A}^b_t + \alpha_t \mathbf{A}^r_t
   \]

**Training Protocol**

- Stage 1: Train $\pi_b$, freeze $\pi_r$.
- Stage 2: Freeze $\pi_b$, train $\pi_r$ (including $\alpha_t$).
- Loss:
  \[
  \mathcal{L}_{\text{GERP}} = \lambda_{\mathrm{res}} L_{\mathrm{FM}}(\pi_r; I^H, T^{H\leftarrow W}, \ell; \mathbf{A}^\star) + \lambda_{\mathrm{act}} \|\hat{\mathbf{A}} - \mathbf{A}^\star\|_2^2
  \]
- Curriculum on $\lambda_{\mathrm{act}}$: increases linearly during residual/gate training.

## 3. Network Architecture

GERP is implemented as two distinct policy networks with respective visual and pose encoders. The table below summarizes the architectural components.

| Branch                  | Encoder           | Pose Conditioning         | Output Head(s)         |
|-------------------------|-------------------|--------------------------|------------------------|
| Wrist-only base ($\pi_b$)    | ResNet50-style CNN | $T^W$ through MLP, injected into UNet | Action chunk           |
| Egocentric residual ($\pi_r$)| Separate, untied CNN | $T^{H\leftarrow W}$ (translation + quaternion through 2-layer MLP) | Action chunk and scalar gate ($\alpha$) |

The residual branch fuses egocentric features, transformed pose embedding, and instruction embedding into a shallow UNet or MLP. The two parallel output heads respectively produce the residual action chunk proposal and a scalar logit for gating, which becomes $\alpha$ after sigmoid activation.

**Inference procedure:**

```python
# Inputs: (I^W, T^W), (I^H, T^H), instruction ℓ

A_b = π_b(I^W, T^W, ℓ)
T_hw = inverse(T^H) @ T^W
A_r, α = π_r(I^H, T_hw, ℓ)
A_hat = (1 - α) * A_b + α * A_r
# Deploy A_hat on robot
```

**Optimization:**
- Optimizer: AdamW, weight decay $10^{-2}$
- Learning rate: cosine decay $2.5 \times 10^{-5} \rightarrow 2.5 \times 10^{-6}$
- Batch size: 128 action chunks
- Stage lengths: 30k steps each for $\pi_b$, then $\pi_r$.

**Preprocessing:**
- Synchronize frames within 20 ms, resize to $224 \times 224$, ImageNet normalization, pose normalization (translation/rotation z-score, unit quaternion).

## 4. Empirical Evaluation

GERP was evaluated on real-world manipulation (e.g., pepper sorting) using 400 human demonstrations captured via EgoGuide. Metrics:

- **Success Rate (SR)**
- **Task Phase Success (TPS)**

Results:

| Method                   | SR (%) | TPS (%) |
|--------------------------|--------|---------|
| Wrist Only               | 75     | 77.5    |
| Wrist+Ego Direct         | 65     | 72.5    |
| GERP                     | 80     | 87.5    |

Notable findings:

- Under forced occlusion of the wrist camera, GERP’s gate $\alpha$ increases (approximately 0.8), selectively activating the egocentric branch, leading to a ~10% higher success rate versus wrist-only.
- Gating behavior: $\alpha$ remains low when the target is visible in the wrist camera; rises when occluded (see App. Fig. A.12 in [2606.14665]).
- Under test-time egocentric camera viewpoint shifts, GERP is less sensitive (<5% drop) versus direct input concatenation (up to 15% drop).

## 5. Ablations and Analysis

Ablation studies in [2606.14665] confirm the necessity of both the residual and gating components:

- No gate ($\alpha \equiv 1$): Performance degrades to wrist+ego direct, occasionally below wrist-only due to egocentric distractions.
- Gate only (no residual loss, $\lambda_{\mathrm{res}} = 0$): The gate shuts off ($\alpha \rightarrow 0$), reducing to wrist-only performance.
- Full GERP (with curriculum, both losses): Most robust across tasks and perturbations.

A plausible implication is that proper curriculum and dual loss functions are required for the residual policy to effectively complement, rather than disrupt, the base policy.

## 6. Extensions and Considerations

GERP's architecture decouples a locally stable controller from a context-driven corrector, moderated by a learned, state-dependent scalar gate. The gating mechanism is critical to avoid performance collapse when the egocentric view is either uninformative or noisy. The approach is flexible: additional sensing modalities (e.g., force-torque) could be accommodated by separate residual branches, each with an individualized gating function. Future research may consider replacing the scalar policy-level gate with a learned attention mask over the action dimensions, potentially enabling more selective cross-modal fusion [2606.14665].

GERP serves as a reference implementation for robust multi-view policy fusion in demonstration-driven robotic imitation, particularly under uncertain or partial local sensory conditions.

Source: https://www.emergentmind.com/topics/gated-egocentric-residual-policy-gerp