Privileged-State Critic in Reinforcement Learning

• Privileged-state critic is a training-time value estimator in partially observable reinforcement learning that leverages extra, simulator-side state information absent at deployment.
• Architectural variants include critics that use full state inputs, arbitrary privileged signals, or privileged latent representations to address value estimation challenges.
• Empirical results show that employing privileged-state critics improves training efficiency and stability by mitigating ambiguity from noisy and partial observations.

Searching arXiv for the cited papers and related "privileged-state critic" terminology. A privileged-state critic is a training-time value estimator for partially observable reinforcement learning that is given richer information than the deployable policy. In the canonical asymmetric actor-critic formulation, the actor is trained and executed from partial observations or observation histories, while the critic is trained with simulator-side state, history-state pairs, or other privileged signals unavailable at inference. Across the literature, this pattern appears in direct form as $Q(s,a)$ or $V(h,s)$, in generalized form as critics conditioned on arbitrary privileged signals $i_t$, and in hybrid form as privileged-latent critics or critics whose supervision targets are constructed from privileged geometry rather than privileged inputs (Cai et al., 2024, Ebi et al., 30 Sep 2025, Wang et al., 2024).

1. Formal setting and defining asymmetry

The basic setting is a POMDP. In the UAV navigation formulation, the environment is modeled as $\langle s,o,a,r,p,\gamma\rangle$, where the true state $s$ contains the full environment and UAV situation, but the policy only observes imperfect onboard sensing $o$ because depth and self-state estimates are noisy and incomplete (Wang et al., 2024). The theoretical analysis of partially observable RL with privileged information likewise distinguishes the execution-time history $\tau_h=(o_{1:h},a_{1:h-1})$ from the training-time augmented trajectory $\overline{\tau}_h=(s_{1:h},o_{1:h},a_{1:h-1})$ (Cai et al., 2024).

Within this setting, the defining asymmetry is not that the actor sees more information during training than at test time. It is that the critic sees more information than the actor during training. In DPRL, the actor action is $a_{i,t}=\pi_\theta(o_{i,t})+\epsilon$, while the critic evaluates $Q_{\omega_j}(s,a)$ on privileged state-action pairs (Wang et al., 2024). In GSAC and BA-GSAC, the deployable control actor receives only an observation history $V(h,s)$0, whereas the twin $V(h,s)$1-network is trained on full state $V(h,s)$2 under CTDE, alongside a full-state guiding actor $V(h,s)$3 (Haklidir, 24 May 2026).

Several papers refine this definition. The provable asymmetric actor-critic analysis argues that, under partial observability, a critic that depends only on latent state is biased for policy gradient; the corrected object is a history-and-state critic $V(h,s)$4, combined with a belief-weighted actor update (Cai et al., 2024). Scaffolder makes the same point in model-based form: its privileged critic is $V(h,s)$5, not merely $V(h,s)$6, because the target latent $V(h,s)$7 carries the agent’s historical belief and “is still required because $V(h,s)$8 contains info about the historical beliefs of the agent, which is not predictable from only the Markovian learned world model state $V(h,s)$9” (Hu et al., 2024). This suggests that “privileged-state critic” is most precise when the critic’s extra input supplements, rather than replaces, the actor’s information state.

2. Main architectural variants

The literature summarized here contains several distinct variants of privileged critics.

Variant	Critic-side privilege	Representative papers
Privileged-state-input critic	Critic receives full state $i_t$0 or history-state pairs $i_t$1 while actor uses $i_t$2 or $i_t$3	(Wang et al., 2024, Cai et al., 2024, Haklidir, 24 May 2026)
Privileged-signal critic	Critic receives arbitrary state-dependent signal $i_t$4, not necessarily full state	(Ebi et al., 30 Sep 2025)
Privileged-latent critic	Critic receives deployable latent plus privileged latent, e.g. $i_t$5	(Hu et al., 2024, Huang et al., 4 Aug 2025)
Privileged-target critic	Critic inputs remain deployable, but targets are built from privileged geometry or maps	(Cai et al., 13 May 2025)

The first row corresponds to the narrowest and most textbook meaning. DPRL is explicit: actor input is noisy depth and noisy self-state, critic input is accurate simulator-side state and action, and the actor never receives the privileged information at deployment (Wang et al., 2024). The provable POMDP analysis treats privileged state access during training as the central critic enhancement, but insists that the unbiased critic should depend on both partial-information summary and latent state (Cai et al., 2024).

The second row generalizes the paradigm. IAAC introduces an informed POMDP with privileged signal $i_t$6 and critic $i_t$7. Full-state critics appear as the special case $i_t$8, but the theory permits velocity, target distance, expanded local maps, masked state subsets, simulator internals, or other state-dependent side channels (Ebi et al., 30 Sep 2025).

The third and fourth rows broaden the design space without preserving the classical input asymmetry. Scaffolder and PIGDreamer move the critic into model-state space, where training-time privileged sensing shapes a second latent trajectory representation used only by the critic and auxiliary training components (Hu et al., 2024, Huang et al., 4 Aug 2025). NavDP goes further from the classical template: its critic is observation-conditioned at runtime, but its demonstration generation and critic regression targets are constructed from privileged global ESDF geometry in simulation (Cai et al., 13 May 2025).

3. Learning mechanisms and concrete instantiations

In direct asymmetric actor-critic form, DPRL adapts TD3 to privileged-state training for high-speed UAV navigation under partial observability. The actor consumes a resized depth image and an 8-D self-state vector, outputs $i_t$9, and is trained and executed only from noisy onboard-like observations. The critic uses privileged state $\langle s,o,a,r,p,\gamma\rangle$0 in the TD3 targets and losses, while the target action is still produced from the next observation $\langle s,o,a,r,p,\gamma\rangle$1. Privileged information also enters reward shaping through a global obstacle map, which is used to compute the closest obstacle distance $\langle s,o,a,r,p,\gamma\rangle$2 more accurately than depth-only estimation (Wang et al., 2024).

A graph-based instantiation appears in distributed multi-robot exploration. There, each actor reasons from its own partial informative graph $\langle s,o,a,r,p,\gamma\rangle$3 and compact learned messages $\langle s,o,a,r,p,\gamma\rangle$4, whereas the critic receives a ground-truth graph $\langle s,o,a,r,p,\gamma\rangle$5 derived from the true state $\langle s,o,a,r,p,\gamma\rangle$6. The critic explicitly removes the communication layer because “as the model has access to the global ground truth already, messages shared by other agents are not helpful for return estimation.” Training uses discrete-action SAC with double critics, and the privileged map is used to improve long-term return estimation rather than as a separate imitation target (Ma et al., 2024).

Model-based privileged critics integrate the asymmetry into latent imagination. Scaffolder trains a target world model on deployable observations $\langle s,o,a,r,p,\gamma\rangle$7, a scaffolded world model on $\langle s,o,a,r,p,\gamma\rangle$8, and a privileged critic $\langle s,o,a,r,p,\gamma\rangle$9 over both target and scaffolded latents. Policy improvement then uses TD($s$0) returns computed under the scaffolded world model while the deployed actor remains $s$1 (Hu et al., 2024). PIGDreamer instantiates a safe-RL analogue: the actor uses only the naive latent state $s$2, while both a reward critic $s$3 and a cost critic $s$4 use joint naive and privileged latents, with privileged representation alignment supplying an additional training-time channel from privileged information into the observation-derived latent state (Huang et al., 4 Aug 2025).

These examples show that privileged-state critics are often embedded in broader privileged-training systems. In DPRL, privileged information affects both critic inputs and reward computation (Wang et al., 2024). In Scaffolder and PIGDreamer, it also shapes world models, reward or cost predictors, and auxiliary representation losses (Hu et al., 2024, Huang et al., 4 Aug 2025). A plausible implication is that many empirical gains attributed to “the privileged critic” are actually produced by a compound privileged-training stack.

4. Theory, unbiasedness, and critic-side information design

Theoretical work on privileged-state critics has focused on two questions: when the asymmetry is unbiased, and what kind of privileged information is actually useful. The finite-horizon POMDP analysis in “Provable Partially Observable Reinforcement Learning with Privileged Information” proves that vanilla asymmetric actor-critic is still sample-inefficient in $s$5-observable POMDPs, and that the original empirical form where value depends only on $s$6 is biased for policy gradient under partial observability. Its correction is a belief-weighted asymmetric actor-critic in which the actor update uses $s$7, together with an approximate belief oracle that preserves filter stability under misspecified models. Under $s$8-observability, the paper gives polynomial sample complexity and quasi-polynomial computational complexity for learning an $s$9-optimal policy (Cai et al., 2024).

IAAC generalizes the same unbiasedness logic from full state to arbitrary privileged signals. Its key result is that the informed asymmetric policy gradient using $o$0 remains equal to the standard history-based policy gradient, because $o$1 (Ebi et al., 30 Sep 2025). This converts “privileged-state critic” into a broader “privileged-signal critic” framework and makes full-state critics a special case rather than a requirement.

The same paper also treats privileged-signal selection as an empirical design problem. It proposes a pre-training HSCIC-based conditional dependence measure asking whether the privileged signal explains return beyond $o$2, and a post-training return-prediction-error criterion comparing symmetric and asymmetric critics (Ebi et al., 30 Sep 2025). In the reported benchmarks, partial privileged signals often suffice to match or beat full-state critics, and full-state access is on average sometimes less informative than a noiseless partial signal. This reframes a common assumption: more privileged information is not automatically better if it is poorly aligned with the return-prediction problem.

A second theoretical lesson comes from safe RL. PIGDreamer formalizes ACPOMDPs as a relaxation of CPOMDPs in which the critic is allowed to access underlying state while the policy remains partially observable, and states that $o$3. The paper also argues that CPOMDP critics tend to underestimate future safety risks, whereas privileged critics provide more accurate estimations for reward and cost (Huang et al., 4 Aug 2025). This suggests that privileged-state critics are especially consequential when hidden state affects constraint satisfaction rather than only nominal performance.

5. Empirical roles across application domains

The most direct empirical evidence comes from UAV navigation. DPRL compares privileged learning plus distributed exploration against both ablations and a non-privileged TD3 baseline. In the training curves, DPRL stabilizes in success rate around 220k steps, while the ablated versions converge only after roughly 300k steps; DPRL reaches average success rate above 85% after 220,000 steps, whereas TD3 does not converge by the end of training. Over 30 final-evaluation episodes, DPRL reports $o$4 in the training environment and $o$5 in the random environment, versus $o$6 and $o$7 for TD3. The paper is explicit, however, that these gains are not solely attributable to the privileged critic, because DPRL also includes distributed exploration and privileged reward computation, although the ablation indicates that the privileged-learning component is the larger contributor (Wang et al., 2024).

In distributed multi-robot exploration, privileged map information is used to train a critic that sees a ground-truth graph while the deployed policies communicate only fixed-size learned messages. The paper reports that “the introduction of privileged map information significantly enhanced training performance by 23.8%,” with improved training stability, and that the full framework reduces communication by up to two orders of magnitude while sacrificing only a marginal 2.4% in total travel distance (Ma et al., 2024). Here the privileged critic primarily addresses long-horizon coordination under severe information bottlenecks.

Vision-centric variants show that privileged critics can also regularize perception. APRiL trains an image-based asymmetric actor-critic branch whose critic receives state, together with a separate privileged state-based actor-critic and an attention-alignment module. In extrapolated domains with additional unseen distractors, the reported JacoReach performance drop is 11% for APRiL, versus 42% for asym-DDPG and 48% for the state-map asym-DDPG baseline (Salter et al., 2019). Scaffolder extends this logic into model-based RL and reports that, across ten tasks, it bridges 79% of the gap between DreamerV3 trained only on $o$8 and privileged DreamerV3 trained and evaluated on $o$9, while outperforming the AAC baseline in 9 out of 10 tasks (Hu et al., 2024).

These application results indicate that privileged critics are not tied to a single algorithmic substrate. They appear in TD3-style continuous control, SAC-based graph policies, DDPG-style vision policies, and Dreamer-style latent imagination (Wang et al., 2024, Ma et al., 2024, Salter et al., 2019, Hu et al., 2024). The recurring empirical role is not merely faster learning, but reduction of value-estimation ambiguity induced by partial observability, noisy sensing, bandwidth limits, or distractor-ridden visual input.

6. Boundaries, failure modes, and adjacent paradigms

Not every method using privileged information during training is a privileged-state critic in the strict sense. NavDP is the clearest boundary case. Its deployed diffusion policy and critic are both conditioned only on local observation tokens and candidate trajectories at inference, while privileged simulation geometry is used offline to generate high-quality demonstrations and to compute ESDF-based critic targets with contrastive negative samples. The paper explicitly states that the critic is not a privileged-state critic in the classic sense of taking full simulator state as an input; the asymmetry lies in target generation rather than critic input (Cai et al., 13 May 2025).

PriPG-RL is another adjacent but distinct paradigm. It does not train a critic on privileged state while the actor uses partial observations. Instead, it trains a standard observation-based SAC actor-critic and transfers training-only privileged information through an anytime-feasible MPC teacher, dual replay buffers, behavior substitution, a logit-space imitation anchor, and an advantage-based gate. The privileged module is a planner-teacher, not a privileged critic (Amiri et al., 9 Apr 2026).

A different boundary appears when privilege is injected into the policy input rather than the critic. “Privileged to Predicted” studies an urban-driving RL agent trained directly on a privileged 15-channel BEV state, then progressively replaces parts of that state with learned predictors from sensors. The paper does not describe an asymmetric actor-critic with privileged critic inputs; privilege resides in the policy representation itself (Özsüer et al., 2023). PrivilegedDreamer also falls outside the strict critic-only definition, because both actor and critic are conditioned on an inferred hidden-parameter estimate $\tau_h=(o_{1:h},a_{1:h-1})$0, while true hidden parameters are used only as training targets (Byrd et al., 17 Feb 2025).

The literature also records failure modes for adaptive privileged guidance. BA-GSAC adds an uncertainty-aware schedule for teacher-student distillation on top of CTDE, but under severe occlusion the adaptive coefficient collapses to $\tau_h=(o_{1:h},a_{1:h-1})$1 within about 3K steps because the ensemble predicts partial observations rather than hidden state. The paper calls this observability blindness and reports that, in the severe POMDP, a simple deterministic linear decay schedule achieves the best performance across all metrics, with mean 116.5 and CV $\tau_h=(o_{1:h},a_{1:h-1})$2, compared with BA-GSAC mean 89.2 and CV $\tau_h=(o_{1:h},a_{1:h-1})$3 (Haklidir, 24 May 2026). This is a useful caution for privileged-state critics more generally: if privileged influence is modulated by an auxiliary uncertainty signal, the uncertainty model must track hidden-state ambiguity rather than only observable-signal predictability.

Taken together, these distinctions imply a narrow and a broad usage. In the narrow usage, a privileged-state critic is an asymmetric actor-critic where the critic consumes training-only privileged state unavailable to the actor. In the broader usage emerging from recent work, it is part of a family of privileged-information training schemes that may place privilege in critic inputs, critic targets, reward computation, world models, teacher actors, or planners. The narrow definition is still analytically useful, but the empirical systems increasingly combine several of these channels at once (Cai et al., 13 May 2025, Amiri et al., 9 Apr 2026, Haklidir, 24 May 2026).