Asymmetric Advantage Weighted Regression (AAWR)

• AAWR is a reinforcement learning method that extends advantage-weighted regression to address partial observability in robotic active perception using privileged training data.
• It leverages asymmetric advantage estimation by conditioning on both privileged and observed states, ensuring robust policy improvement in POMDP settings.
• AAWR demonstrates empirical improvements in simulation and real-world tasks by enhancing grasp rates and active perception behaviors compared to traditional methods.

Asymmetric Advantage Weighted Regression (AAWR) is a reinforcement learning algorithm tailored for active perception under partial observability. It systematically leverages privileged state information (unavailable at deployment) during offline and online training to enable efficient policy improvement for robotic tasks where critical environment information is partially observed or inferred. AAWR extends classical advantage-weighted regression paradigms by introducing an explicit asymmetry in how advantage estimation and policy learning are treated with respect to available information during training versus deployment. This approach yields robust information-seeking behaviors and high task performance in both simulated and real-world robotic manipulation settings (Hu et al., 1 Dec 2025).

1. Problem Setting: POMDPs, Privileged Information, and Policy Structure

Active perception tasks are formalized as partially observable Markov decision processes (POMDPs) specified by the tuple $$(\mathcal S, \mathcal A, \mathcal O, T, R, E, P, \gamma)$$. Here, $\mathcal S$ denotes the true, privileged state (e.g., robot and object poses), $\mathcal A$ the action space, and $\mathcal O$ the partial observations available at test time (camera images, proprioception). The dynamics $T(s_{t+1}|s_t, a_t)$, reward $R(r_t|s_t, a_t)$, and observation model $E(o_t|s_t)$ comprise task-specific models. The optimal policy in a POMDP depends on the entire observable-agent history $h_t = (o_0, a_0, \dots, o_t)$. This history is summarized as an agent state $z_t = f(h_t)$, typically via recurrent modules (e.g., LSTMs) or fixed-length sliding windows. The resultant deployable policy $\pi: \mathcal Z \rightarrow \Delta(\mathcal A)$ executes $a_t \sim \pi(\cdot | z_t)$.

AAWR specifically utilizes access to privileged state $s_t$ or privileged observations $o^p_t$ during training. Critic and value networks are conditioned on $(s_t, z_t)$, while the deployed policy operates exclusively on $z_t$ at test time. This separation is fundamental: privileged information is available only at training.

2. AAWR Objective: Advantage-Weighted Policy Regression with Privileged Weights

AAWR extends Advantage-Weighted Regression (AWR) to the POMDP regime by working with an equivalent MDP whose state is the joint $(s,z)$. The privileged Q-function and V-function for behavior policy $\mu$ are defined as: $$Q^\mu(s, z, a) = \mathbb E^\mu \Bigl[\sum_{k=0}^{\infty} \gamma^k r_{t+k} \,\Big|\, s_t = s, z_t = z, a_t = a \Bigr],$$

$$V^\mu(s, z) = \mathbb E^\mu \Bigl[\sum_{k=0}^{\infty} \gamma^k r_{t+k} \,\Big|\, s_t = s, z_t = z \Bigr].$$

The privileged advantage is $A^\mu(s, z, a) = Q^\mu(s, z, a) - V^\mu(s, z)$. The AAWR importance weights are then

$$w(s, z, a) = \exp \Bigl( A^\mu(s, z, a) / \beta \Bigr),$$

where $\beta > 0$ is an inverse temperature hyperparameter.

Policy learning proceeds by maximizing: $$\mathcal L_{\text{AAWR}}(\theta) = \mathbb E_{(s, z) \sim d_\mu, \, a \sim \mu(\cdot|z)} \bigl[ w(s, z, a) \, \log \pi_\theta(a | z) \bigr],$$ with $d_\mu(s, z)$ the discounted state visitation distribution.

Critic and value functions are learned via Implicit Q-Learning (IQL), employing a TD update and expectile regression:

• Q-loss (1-step TD): $$\mathcal L_Q(\phi) = \mathbb E_{(s,z,a,r,s',z') \sim \mathcal D} \bigl[ r + \gamma V_\theta(s',z') - Q_\phi(s,z,a) \bigr]^2$$
• V-loss (expectile regression):

$$\mathcal L_V(\theta) = \mathbb E_{(s,z,a) \sim \mathcal D} \Bigl| \tau - \mathbf{1}_{\{Q_\phi(s,z,a) - V_\theta(s,z) < 0\}} \Bigr| \cdot \bigl[ Q_\phi(s,z,a) - V_\theta(s,z) \bigr]^2$$

with $\tau = 0.7$.

3. Asymmetry in Advantage Weighting: Necessity and Theoretical Basis

Asymmetry arises by distinguishing the role of privileged (training-only) and unprivileged (deployment) information. In the $(s, z)$-MDP, policy improvement must depend on the precise advantage $A^\mu(s, z, a)$. A symmetric variant, SAWR, that instead uses $A^\mu(z, a) = \mathbb E_{s|z}[A^\mu(s, z, a)]$ (collapsing $s$) is theoretically unsound in POMDPs. By Jensen’s inequality, $$\exp(\mathbb E_{s|z}[A(s, z, a)/\beta]) \neq \mathbb E_{s|z}[\exp(A(s,z,a)/\beta)]$$; thus, SAWR does not effect the same constrained policy improvement as AAWR.

Furthermore, TD learning of unprivileged $Q(z, a)$ in mixture distributions is inconsistent—the correct advantage structure relates to $(s, z, a)$, not marginal $z, a$. Privileged TD on $(s, z)$ exhibits a unique fixed point, while unprivileged TD may not converge, especially under partial observability. This formal justification underscores why AAWR’s use of privileged advantage is necessary for effective policy improvement in POMDPs.

4. Algorithmic Workflow and Implementation

AAWR follows an offline-to-online training architecture:

1. Offline Phase:
   • Start with an offline buffer $\mathcal D_\text{off}$ (teleoperated/scripted demonstrations or prior policy rollouts).
   • For $N_\text{off}$ iterates: sample a batch from $\mathcal D_\text{off}$, update critic and value networks by minimizing $\mathcal L_Q + \mathcal L_V$, ascend $\mathcal L_\text{AAWR}$ to update the policy.
2. Online Phase:
   • For $N_\text{on}$ iterates: collect a trajectory using current $\pi_\theta$, store in $\mathcal D_\text{on}$.
   • Sample batches (50/50) from $\mathcal D_\text{off} \cup \mathcal D_\text{on}$, update critics/values and policy as in the offline phase.

At deployment, only $\pi_\theta(z)$ is used; privileged inputs $(s, o^p)$ are not required.

Key implementation details:

• Critic/value networks: three-layer MLP after state encoder.
• Policy: three-layer MLP after partial observation encoder (e.g., CNN, pretrained DINO-V2 + PCA).
• IQL expectile: $\tau = 0.7$.
• AWR temperature: $\beta \in \{1, 10\}$ (10 is default for most tasks).
• Learning rates: $3 \times 10^{-4}$, Adam optimizer.
• Batch size: 256.
• Demonstration count per task: 30–250 (task dependent).
• Training budgets: e.g., simulated (20K/80K or 100K/900K), real Koch (20K/1.2K), real Franka (100K offline only).

5. Theoretical Guarantees and Policy Improvement Properties

Theorem 3.1 demonstrates that constrained policy improvement in the $(s, z)$-MDP under a Kullback–Leibler (KL) divergence budget reduces to maximizing $\mathcal L_{\rm AAWR}(\pi)$. The symmetric version (SAWR) is generally an invalid surrogate for policy improvement in POMDPs due to aliasing and the failure of Jensen’s equality in this context. Privileged TD-learning on $(s, z)$ admits a unique fixed point, while TD-learning of unprivileged $Q(z, a)$ can fail to converge to meaningful values.

AAWR inherits sample efficiency and stability from AWR/AWAC but achieves correct advantage estimation under partial observability through its asymmetry. This is crucial for POMDPs, where standard methods often struggle.

6. Empirical Performance and Active Perception Behaviors

AAWR demonstrates significant performance gains over baseline algorithms across both simulated and real-world manipulation tasks involving active perception:

• Simulated Camouflage Pick (hidden marble): AAWR achieves approximately 2$\times$ the performance of AWR or behavioral cloning.
• Fully Observed Pick (block grasp): AAWR attains near-perfect performance; AWR/BC fail or produce mis-grasps.
• Active-Perception Koch task (narrow FOV): AAWR learns sequential scanning, grasping, and lifting, outperforming privileged-policy distillation (which plateaus at 80%, lacking scanning) and VIB (which collapses at test time).

Real World:

• Blind Pick on Koch: AAWR increases grasp rate from 88% to 94%; pick rate from 71% to 89% versus AWR’s 55%.
• Interactive Search+Handoff (Franka): Across Bookshelf-P/D, Shelf-Cabinet, Complex, AAWR improves search scores by 20–60 percentage points over AWR, doubles completion rates versus exhaustive search, AWR, BC, and VLM+$\pi_0$. Search efficiency improves by factors of 2–8 over exhaustive.

Learned active perception behaviors include: "zoom out" actions, vertical and lateral scans, and fixations on target regions; in handoff tasks, detection of object slip, re-scanning, and re-grasping.

7. Limitations and Open Questions

Empirical success notwithstanding, AAWR exhibits several limitations:

• Reliance on small demonstration sets; demo quality and diversity critically impact performance.
• Necessity of privileged sensors at training (must be labeled or estimated, e.g., via object detectors or masks).
• Scalability to tasks with very long temporal horizons and compounded partial observability remains an open challenge.
• Promising future research directions include integrating AAWR fine-tuning into foundation visuolinguistic action (VLA) policies, automatic representation learning for privileged features, and incorporation of alternative privileged signals (language, audio, haptics) (Hu et al., 1 Dec 2025).