---
title: Reinforcement Learning from Hierarchical Critics
url: https://www.emergentmind.com/topics/reinforcement-learning-from-hierarchical-critics-rlhc
type: topic
---

# Reinforcement Learning from Hierarchical Critics

Reinforcement Learning from Hierarchical Critics (RLHC) denotes a family of reinforcement learning (RL) architectures in which agents receive value signals from multiple cooperative critics organized in a hierarchical fashion. Hierarchical critics operate at distinct levels, typically distinguishing between local (fine-grained, agent-centric) and global (coarse, system-level) perspectives. By fusing evaluations from these levels, RLHC enhances credit assignment, accelerates learning, and achieves higher cumulative rewards, particularly in non-stationary, competitive, or multi-agent environments [1902.03079].

## 1. Motivation and Theoretical Foundations

Traditional actor–critic methods deploy a single critic per agent, yielding value functions $V_w(s)$ or action-value functions $Q_w(s,a)$ based exclusively on the agent's limited local observations and immediate rewards. In dynamic, multi-agent scenarios—such as competitive games—policies of opponents evolve during training, rendering the environment non-stationary from any one agent's viewpoint and impairing convergence. Decentralized critics, being myopic, often miss critical coordination signals necessary for robust collective performance.

By introducing hierarchical critics, RLHC structures injects additional value functions at higher levels, allowing agents to access not only granular feedback from their own observations but also broader coordination cues derived from the entire environment or agent team. The local critic $V^{\text{local}}$ provides feedback optimized for fine-tuned actions, while the global critic $V^{\text{global}}$ embodies a managerial or team-wide viewpoint, furnishing signals that improve joint strategy. Empirically and theoretically, the fusion of these critics produces tighter value bounds and reduces underestimation bias relative to single-critic temporal difference (TD) update schemes [1902.03079].

A key lemma shows that, for a fused critic $\hat V(s) = \max_i V^i(s)$, the expected squared error satisfies
\[
\mathbb{E}\bigl[\left(\hat V(s)-V^{\pi}(s)\right)^2\bigr] \leq \min_i \sigma_i^2 + C\,\mathbb{E}\left[\Delta^2\right],
\]
where $\Delta$ denotes the critic disagreement and $C$ a constant. Thus, if one critic is accurate, the fusion inherits its accuracy, and disagreements diminish as training progresses [1902.03079].

## 2. Hierarchical Critic Architectures

Several RLHC instantiations have emerged, tailored for diverse settings.

### 2.1 Classic Two-Level Critics

RLHC as presented in [1902.03079] maintains two critic types per agent:
- **Worker critic (local):**
  \[
  V^{\text{local},i}(s_t^{w_i};\,\theta_w) \approx \mathbb{E}[R_t | s_t^{w_i}, \pi_\theta]
  \]
  operating on local observations $s_t^{w_i}$.
- **Manager critic (global):**
  \[
  V^{\text{global}}(s_t^m;\,\theta_m) \approx \mathbb{E}[R_t | s_t^m, \pi_\theta]
  \]
  processing global state $s_t^m$.

The fused value function at each step is given by
\[
\hat V_t^\theta = \max \{ V^{\text{local},i}(s_t^{w_i};\theta), V^{\text{global}}(s_t^m;\theta) \}
\]
This max-fusion strategy incentivizes agents to select actions perceived as beneficial from either perspective.

### 2.2 Multi-Level, Multi-Agent Structures

The Hierarchical Lead Critic (HLC) framework [2602.21680] generalizes the two-level RLHC design to arbitrary numbers of hierarchy levels and agent groupings. Each agent retains its own local critic $Q^{\text{local},i}(o^i, a^i)$, while multiple lead/group critics $Q^{LC_G}(o^{G}, a^{G})$ are defined over subgroups $G \subseteq \{1,\ldots, N\}$. Critic architectures utilize Transformer encoders for group critics, and value fusion can occur sequentially through staged updates or by summing value contributions in policy loss gradients.

### 2.3 Temporally Abstract Critics

In hierarchical Backpropagated Adaptive Critics (BACs) [1512.02693], critics operate at different temporal resolutions. The high-level BAC updates at a slower rate, generating plans that guide lower-level controllers updating at every time step, with distinct value functions and Bellman equations for each level.

## 3. Algorithmic Framework

A typical RLHC update cycle, as formalized in [1902.03079], extends PPO-like actor–critic training as follows:

1. **Rollout:** For each agent, collect state-action-reward trajectories, recording both local and global observations.
2. **Critic Estimates:** For each time step $t$,
   - Compute local $V_t^{\text{local}}$ and global $V_t^{\text{global}}$ estimates.
   - Fuse estimates via $\hat V_t = \max \{V_t^{\text{local}}, V_t^{\text{global}}\}$.
   - Compute fused hierarchical advantage:
     \[
     \hat A_t = \sum_{l=0}^{K-1}\gamma^l r_{t+l} + \gamma^K \hat V_{t+K} - \hat V_t
     \]
3. **Critic Loss:**
   \[
   L_{\text{critic}} = \frac{1}{NT} \sum_{i=1}^N \sum_{t=0}^{T-1} \left( \hat V_t - R_t \right)^2
   \]
4. **Actor Loss (PPO):**
   \[
   L_{\text{actor}}(\theta) = \frac{1}{NT} \sum_{i=1}^N\sum_{t=0}^{T-1} \min \left[r_t(\theta) \hat A_t, \; \mathrm{clip}(r_t(\theta),1-\epsilon,1+\epsilon) \hat A_t \right]
   \]
   where $r_t(\theta) = \frac{\pi_\theta(a_t|s_t^{w_i})}{\pi_{\theta_\text{old}}(a_t|s_t^{w_i})}$.

In HLC [2602.21680], actor and critic updates may be staged: each lead critic is updated for the group it supervises, and actors are optimized both with respect to their local critics and all relevant lead critics. All updates employ CTDE (centralized training with decentralized execution) and exploit shared experience buffers. This approach mitigates destructive gradient interference and improves sample efficiency.

## 4. Empirical Evaluations

RLHC and its generalizations have been validated in synthetic and simulated multi-agent benchmarks:

| Study / Approach | Domains | Key Empirical Findings |
|------------------|---------|------------------------|
| [1902.03079] RLHC | Unity ML-Agents Tennis (2v2), Soccer (2v2) | RLHC achieves higher mean reward in ≈50k steps (vs. PPO’s 100k), episode length and value estimates stabilize faster. Disabling global critic reverts performance to PPO levels. |
| [2602.21680] HLC | MPE SimpleSpread, MOMAland Escort₃/₈, Surveillance₄ | HLC converges markedly faster and achieves higher team returns than single-hierarchy baselines. Remains robust under partial observability, scales to 8 agents. |
| [1512.02693] Hierarchical BAC | Cart–pole stabilization | Two-level BAC hierarchy significantly improves reliability and sample efficiency, especially when system presents separated time constants. |

Ablation studies confirm that the presence and effective fusion of both local and global (or group-level) critics are crucial for robustness and rapid learning. Max-fusion outperforms simple averaging and other fusion rules by providing a more reliable advantage signal [1902.03079].

## 5. Practical Implementations and Use Cases

RLHC techniques find application in environments demanding rapid, robust coordination, especially in multi-agent competitive or cooperative tasks. Unity ML-Agents-based 2v2 tennis and soccer environments are canonical testbeds [1902.03079]. HLC has been shown effective in environments with partial observability and no direct inter-agent communication, such as sophisticated formation-keeping and surveillance domains [2602.21680]. In control, hierarchical BACs have proven effective in high-frequency servo control of underactuated systems [1512.02693].

RLHC can be implemented with standard deep RL libraries, requiring extensions to the critic architecture and the advantage computation to accommodate multi-level signals.

## 6. Extensions, Limitations, and Future Directions

RLHC has been extended beyond two levels—introducing multiple hierarchical lead critics, arbitrary groupings (including graph-structured critics), and transformer-based group critics for handling heterogeneous and scalable agent systems [2602.21680]. Variants incorporating temporally abstract critics enable efficient credit assignment in time-scale separated or partially observed dynamical systems [1512.02693].

Limitations include increased computational overhead due to additional critics and nested training loops. Remedies suggested involve parameter or subnetwork sharing. Potential future directions identified are:
- Deeper hierarchies and learnable critic fusion rules [1902.03079].
- Generalized lead critics for multi-objective or structured task performance [2602.21680].
- Application to complex, real-world multi-robot or decentralized control systems.

A plausible implication is that further advances in critic fusion logic, temporal abstraction, and scalable architectures may generalize RLHC approaches for highly heterogeneous, partially observable, and dynamically coupled multi-agent systems.

## 7. Comparative Analysis and Significance

RLHC architectures systematically outperform their single-level critic analogues in multi-agent, competitive, and cooperative domains, both in terms of final episodic return and sample efficiency [1902.03079, 2602.21680, 1512.02693]. The key to these gains lies in multi-level value estimation, which enables richer and more stable policy gradients, more effective credit assignment in non-stationary or partially observed settings, and mitigation of myopic or suboptimal behaviors endemic to decentralized critics.

While classic actor–critic methods remain foundational for single-agent or stationary tasks, RLHC and its variants set the present standard for high-performance RL in challenging, team-based, and hierarchical control settings.

Source: https://www.emergentmind.com/topics/reinforcement-learning-from-hierarchical-critics-rlhc