---
title: Informed Asymmetric Actor-Critic
url: https://www.emergentmind.com/topics/informed-asymmetric-actor-critic
type: topic
---

# Informed Asymmetric Actor-Critic

Searching arXiv for the core papers and related formulations to ground the article.
Informed Asymmetric Actor-Critic is a family of actor-critic methods in which the actor is restricted to deployment-feasible observations or histories, while the critic is trained with privileged information available only during training. In the original image-based robot learning formulation, the actor consumes RGB-D images and the critic consumes the full simulator state; later work generalized the privileged input from full states to arbitrary privileged signals and contextual environmental factors. Across these formulations, the method is designed to exploit training-time information to improve value estimation, reduce variance in actor updates, and preserve a policy that remains executable without privileged inputs [1710.06542] [2509.26000].

## 1. Core formulation

The canonical setting is a POMDP or a partially observed MDP in which latent states $s \in S$ generate observations $o \in O$, actions are $a \in A$, and the deployed policy cannot access $s$. In the robotics formulation, observations are rendered from state by $o = f_{\mathrm{render}}(s)$, while the simulator exposes both $s$ and $o$ during training. The actor is therefore image-based, $\pi_\theta(a \mid o)$ or $\pi_\theta(o,g^o)$ in the multi-goal setting, whereas the critic is privileged, taking $V_\phi(s)$ or $Q_\phi(s,a)$, and in the multi-goal case $Q_\phi(s,a,g)$. The method is “informed” because the critic learns from simulator-only information unavailable at test time, and “asymmetric” because actor and critic consume different inputs [1710.06542].

A later generalization replaces full-state privilege with arbitrary privileged signals. In that formulation, the actor remains history-dependent, $\pi_\theta(a_t \mid h_t)$, but the critic conditions on $(h_t,i_t)$, where $i_t$ is sampled from an information function $\mathcal{I}(i_t \mid s_t)$ and need not equal the full state. This informed POMDP view makes the privileged channel strictly more general than simulator state access: it may encode noisy or partial internal variables, environmental annotations, or other state-dependent signals [2509.26000].

A related contextual formulation casts the environment as a CMDP with context $c$ fixed within an episode. There, the actor uses only the observation $o_t$, but the critic receives $(c,o_t)$, with $c=\mu_\phi(e)$ obtained from simulator-accessible environmental factors such as wind, frictions, inertias, or payload. The context encoder is trained only through the critic’s value loss; the actor remains blind to context during both training and deployment [2208.02376].

| Formulation | Actor input | Critic input |
|---|---|---|
| Image-based robot learning | $o$ or $(o,g^o)$ | $s$ or $(s,a,g)$ |
| Unbiased asymmetric POMDP learning | $h$ | $(h,s)$ or $(h,s,a)$ |
| IAAC with arbitrary privilege | $h$ | $(h,i)$ or $(h,i,a)$ |
| AACC in contextual RL | $o$ | $(c,o)$ |

## 2. Policy-gradient validity and the problem of bias

The central theoretical question is whether a privileged critic preserves the correct policy gradient under partial observability. For a history-based policy, the standard policy gradient is
$$
\nabla_\theta J(\theta)=\mathbb{E}\!\left[\sum_{t=0}^{\infty}\gamma^t\,Q^\pi(h_t,a_t)\,\nabla_\theta \log \pi_\theta(a_t\mid h_t)\right].
$$
In POMDPs, a common but generally invalid substitution is to replace $Q^\pi(h_t,a_t)$ by a state-only critic such as $Q^\pi(s_t,a_t)$ or $V^\pi(s_t)$. The 2021 analysis shows that these “timeless” state values are generally ill-defined or biased for history-based control, and that the naive equalities
$$
Q^\pi(h,a)\stackrel{?}{=}\mathbb{E}_{s\sim p(s\mid h)}[Q^\pi(s,a)],\qquad
V^\pi(h)\stackrel{?}{=}\mathbb{E}_{s\sim p(s\mid h)}[V^\pi(s)]
$$
are false in general POMDPs. The unbiased construction instead uses history-state values,
$$
Q^\pi(h,a)=\mathbb{E}_{s\sim p(s\mid h)}[Q^\pi(h,s,a)],\qquad
V^\pi(h)=\mathbb{E}_{s\sim p(s\mid h)}[V^\pi(h,s)],
$$
which leads to the asymmetric policy gradient
$$
\nabla_\theta J(\theta)=\mathbb{E}\!\left[\sum_{t=0}^{\infty}\nabla_\theta \log \pi_\theta(A_t\mid H_t)\,Q^\pi(H_t,S_t,A_t)\right].
$$
This preserves unbiasedness while exploiting privileged latent state during training [2105.11674].

The arbitrary-signal formulation extends the same idea beyond full-state access. Its key identities are
$$
\mathbb{E}_{i\mid h}[R(h,i,a)] = R(h,a),\qquad
\mathbb{E}_{i\mid h}[Q^\pi(h,i,a)] = Q^\pi(h,a),\qquad
\mathbb{E}_{i\mid h}[V^\pi(h,i)] = V^\pi(h).
$$
Hence the informed asymmetric policy gradient,
$$
\nabla_\theta^{\mathrm{IAAC}} J(\pi_\theta)=
\mathbb{E}\!\left[\sum_{t=0}^{\infty}\gamma^t\,Q^\pi(h_t,i_t,a_t)\,\nabla_\theta \log \pi_\theta(a_t\mid h_t)\right],
$$
equals the standard gradient. The same argument applies to advantages because $A^\pi(h,i,a)=Q^\pi(h,i,a)-V^\pi(h,i)$ satisfies the zero-mean baseline property [2509.26000].

A complementary finite-time justification with linear function approximation shows why the asymmetric critic can improve learning even when the actor still acts from a finite internal state $Z_t$. In that analysis, the symmetric critic incurs an additional aliasing term caused by the mismatch between the true belief $b_{k,m}(s\mid h_{k,m})$ and the approximate belief $\hat b_{k,m}(s\mid z_{k,m})$, whereas the asymmetric critic over $(S,Z)$ eliminates this term because $(S,Z)$ is Markov. The finite-time bound therefore sharpens the critic error term in natural actor-critic by removing the aliasing-induced contribution [2501.19116].

## 3. Algorithmic patterns and implementation

The original image-based robotic instantiation uses an off-policy DDPG base with HER for sparse rewards and UVFA-style goal conditioning. Replay stores aligned privileged and partial-observation tuples, $(s,o,a,r,s',o')$ and, in the multi-goal case, $(g,g^o)$. The critic is trained on $(s,a,r,s',g)$, while the actor is trained on $(o,g^o)$ and updated through a deterministic policy gradient that backpropagates through the informed critic. This alignment is explicit: at each simulator step the method renders $o_t=f_{\mathrm{render}}(s_t)$, stores both representations in replay, optionally relabels achieved goals by HER, and applies target networks, exploration noise, normalization, and replay [1710.06542].

The actor network in that system takes RGB or RGB-D images $o_t$ and goal images $g^o$, each of size $100\times 100\times C$ with $C=3$ or $4$. It uses two shared-architecture CNN towers with 4 convolutional layers, 64 filters, $2\times 2$ kernels, and ReLU activations, followed by concatenation and 3 fully connected layers with 512 units each. The output is tanh normalized and rescaled to action limits, and a preactivation penalty is added to avoid tanh saturation. An optional bottleneck constrains an intermediate layer to predict the full state $s_t$ through an auxiliary $L_2$ loss. The critic is a state-action MLP with 3 fully connected layers of 512 units each, taking concatenated $(s_t,g,a_t)$ and outputting scalar $Q(s,a,g)$ [1710.06542].

The reported hyperparameters in the same implementation are unusually specific and have been widely cited in later discussions of asymmetric design: replay buffer size $10^5$ transitions; minibatch size $128$; 16 parallel rollouts per iteration; 40 optimization steps per iteration; Adam with learning rate $0.001$; discount $\gamma=0.98$; horizon $T=50$; target-network Polyak averaging $\tau \approx 0.98$; and exploration with 20% probability of a uniform random action, otherwise i.i.d. Normal noise with standard deviation 5% of the action range. States are normalized by running means and standard deviations. Domain randomization is applied at rendering time over textures, lighting, camera pose and focal length, and depth noise, with viewpoint randomization identified as especially important for transfer [1710.06542].

In the contextual PPO variant, the algorithmic asymmetry is realized differently. The actor objective is the PPO clipped surrogate
$$
J_\pi(\theta)=\hat{\mathbb{E}}_t\!\left[\min\big(r_t(\theta)\hat A_t,\mathrm{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat A_t\big)\right],
$$
while the critic optimizes
$$
L_V(w)=\hat{\mathbb{E}}_t\!\left[\big(V_w(c,o_t)-\hat V_t\big)^2\right],
$$
with $c=\mu_\phi(e)$ and Monte Carlo targets $\hat V_t=\sum_{k=t}^{T-1}\gamma^{k-t}r_k$. Actor updates use only observations, whereas critic and context encoder updates use the privileged environmental factors through $c$ [2208.02376].

## 4. Empirical performance

In simulated robotics benchmarks, asymmetric critics improve both sample efficiency and final performance relative to symmetric baselines in which both actor and critic operate from images. The effect is reported on Particle, Reacher, and Fetch Pick. On Particle and Reacher, asymmetric DDPG/HER outperforms symmetric counterparts by large margins. Fetch Pick is harder, HER is essential, and asymmetric HER beats symmetric HER notably. An imitation-learning baseline using behavior cloning with DAgger learns faster initially but saturates at suboptimal performance; asymmetric HER eventually surpasses it on all tasks. Adding a full-state prediction bottleneck accelerates learning and stabilizes training, with substantial gains on Particle and Reacher and reduced variance on Fetch Pick [1710.06542].

The same study reports zero-shot sim-to-real transfer without training on any real-world data. With domain randomization and asymmetric HER, real robot experiments on Pick, Forward Push, and Block Move achieved 5/5 successes on each task across different block initializations and goals. The ablations are sharp: asymmetric DDPG without HER and symmetric HER each achieved 0/5; vanilla behavior cloning achieved 0/5 except Forward Push 1/5; DAgger behavior cloning achieved Pick 3/5 and 0/5 on the others; removing randomization yielded 0/5; and removing viewpoint randomization yielded Block Move 4/5 and 0/5 on the other tasks. The paper also reports robustness to distractors and appearance changes, and notes emergent push-grasping and re-grasping behaviors [1710.06542].

Under stronger partial observability, the unbiased history-state formulation also outperforms symmetric and biased asymmetric baselines. The evaluation includes Heaven-Hell-3/4, Shopping-5/6, Car-Flag, Cleaner, and Memory-Four-Rooms-7x7/9x9. Quasi-reactive baselines with short history windows often fail or converge slowly; biased asymmetric A2C with a state critic frequently underperforms or becomes unstable; and unbiased asymmetric A2C with a history-state critic consistently matches or exceeds the best baselines. In Shopping-5/6 it converges faster and to higher return; in Heaven-Hell-3/4 it is the only method to solve the tasks reliably; and in Memory-Four-Rooms and Cleaner it matches the best symmetric A2C [2105.11674].

The broader IAAC formulation was evaluated on six navigation tasks and synthetic informed POMDPs. On learning curves averaged over the last 100 episodes, IAAC was strong versus A2C and asym-A2C-s in Heaven-Hell-3, faster than asym-A2C-hs in Shopping-5 with comparable final returns, best in Car-Flag for both sample efficiency and final performance, stable in Cleaner, and substantially stronger than both asymmetric baselines in Memory-Four-Rooms. Its informativeness diagnostics are environment-dependent: in synthetic tests, the no-privilege condition $i_t=\varnothing$ gave $\bar I = 66.29 \pm 29.84$ and $p = 1.00 \pm 0.00$, while increasing noise generally increased $p$-values; for the $(\epsilon,\delta)$ criterion, median $\epsilon$ exceeded zero only at $\delta=0.1$ [2509.26000].

## 5. Generalizations and adjacent variants

AACC extends the asymmetric principle from observation-state asymmetry to dynamics-context asymmetry. It formalizes changing dynamics as a CMDP, samples training contexts from $p_{\mathrm{train}}(c)$ and deployment contexts from $p_{\mathrm{test}}(c)$, and conditions only the critic on a learned context vector. The empirical finding emphasized in that work is that feeding context to the actor is counterproductive: AACC-actor and AACC-hybrid underperform AACC, and larger actor-context dimension correlates with worse performance in Laikago. On hard tasks such as Gym-JSBSim, Hopper, and Laikago, AACC significantly outperforms baselines in final return and sample efficiency; under distribution shift it remains best, with examples including Gym-JSBSim (new distribution) at $8485 \pm 404$ versus Robust at $7688 \pm 713$, and Laikago (new distribution) at $3271 \pm 715$ versus Robust at $2380 \pm 912$ [2208.02376].

Another adjacent line of work studies asymmetry at the representation level rather than in privileged inputs. In on-policy PPO, PPG, and DCPG from pixels, fully separating actor and critic encoders induces specialization: the actor representation focuses on action-relevant, level-invariant cues, while the critic representation specializes in value and forward dynamics information. Relative to shared representations in PPO on Procgen, decoupling yields actor-side $I(Z;L)$ at $-20\%$, $I((Z,Z');A)$ at $+23\%$, $I(Z;Z')$ at $-96\%$, and $I(Z;V)$ at $+37\%$; critic-side $I(Z;L)$ at $+35\%$, $I((Z,Z');A)$ at $-48\%$, $I(Z;Z')$ at $+324\%$, and $I(Z;V)$ at $+41\%$. That work also argues that the critic shapes exploration and data collection through the advantages used in the actor objective [2503.06343].

Capacity asymmetry produces a different set of effects. In SAC and DrQ, shrinking the actor while keeping standard critics can degrade returns, increase critic overfitting $o_\phi$, reduce policy entropy, and induce value underestimation. The reported mitigation is to replace the conservative double-Q min aggregator by mean or max in both the critic target and actor objective. In the evaluated settings, mean aggregation recovers performance for small actors, increases validation Q-values, decreases critic overfitting, and increases effective rank, whereas critic regularization methods such as layer norm, spectral norm, weight decay, $L_2$-to-init, or output-layer resets help only modestly by comparison [2506.01016].

These variants suggest that “informed asymmetry” is not a single architecture but a design pattern. The privileged channel may be full simulator state, arbitrary partial signals, or contextual environmental factors; the asymmetry may appear in inputs, representations, or capacity; and the empirical benefit depends on whether the critic’s additional information improves actor learning without inducing harmful bias or overfitting.

## 6. Misconceptions, limitations, and open questions

A persistent misconception is that any state-conditioned critic is theoretically sound in a partially observable problem. The POMDP analyses reject that view. A state-only critic $Q^\pi(s,a)$ or $V^\pi(s)$ is generally not the correct object for a history-based actor, and naive asymmetric gradients are biased except under strong conditions such as full observability through a bijection between observations and states. The sound constructions are the history-state critic $Q^\pi(h,s,a)$ and its generalization $Q^\pi(h,i,a)$, or, in contextual RL, a critic over $(c,o)$ with an actor that remains independent of $c$ [2105.11674] [2509.26000].

Another misconception is that privileged information must equal the full state. The arbitrary-signal IAAC formulation explicitly challenges that assumption and proves unbiasedness for privileged partial information by the law of total expectation. This suggests that useful privileged signals may include noisy or partial internal variables rather than complete simulator state. A plausible implication is that asymmetric design is practical even when full-state access is unavailable, provided the privileged signal is informative for return prediction [2509.26000].

The limitations are equally explicit in the literature. Robotics formulations require accurate simulator full states; if $s$ is biased or incomplete, critic training may mislead the actor. Reality gaps in visuals and dynamics still cause failures, making domain randomization critical. Scaling to very complex tasks with long horizons or severe occlusions may require memory or richer observation models, and sparse rewards remain difficult even with HER. In contextual RL, the context is fixed per episode and privileged environmental factors must be available during training. In the finite-time theory, the current guarantees rely on linear critics, log-linear actors, bounded features, and natural actor-critic updates; extending the same results to nonlinear recurrent systems is nontrivial [1710.06542] [2208.02376] [2501.19116].

The current research directions are correspondingly diverse. Reported extensions include asymmetric SAC or TD3, learned context estimators from trajectory history, representation-learning objectives for the actor encoder, mixed-modality actor inputs, curriculum learning, dynamics randomization, dynamic selection of privileged signals via online HSCIC or bandit-style selection, off-policy IAAC with importance sampling under partial observability, and privilege encoders or gating mechanisms to filter noise. Across these proposals, the durable principle is narrow but robust: train an informed critic on information unavailable to the deployed policy, but do so in a way that preserves the actor’s test-time interface and the correctness of the policy update [2509.26000] [2208.02376].

Source: https://www.emergentmind.com/topics/informed-asymmetric-actor-critic