---
title: 'Single-Policy REDI: Universal RL Approach'
url: https://www.emergentmind.com/topics/single-policy-redi
type: topic
---

# Single-Policy REDI: Universal RL Approach

Single-Policy REDI refers to reinforcement learning and control architectures where a single universal policy is optimized to perform robustly and efficiently across diverse domains or agents, under the Realizability and single-policy Concentrability (REDI) assumptions. This approach contrasts with traditional strategies that train separate policies per environment, morphology, or agent, or those requiring all-policy coverage or heavy domain randomization. Recent literature demonstrates the single-policy REDI paradigm's effectiveness in both continuous control for robot fleets and in offline RL for Markov Decision Processes (MDPs) and Markov games.

## 1. Conceptual Foundations: Realizability and Single-Policy Concentrability

Single-Policy REDI is grounded in two main assumptions:

- **Realizability**: The value function and (where applicable) density-ratio function classes are assumed to contain the true optimal solution. In formal terms, there exists $v^* \in V$ and $w^* \in W$ satisfying the saddle-point equations of the (regularized) linear program underlying the RL problem.

- **Single-Policy Concentrability (SPC)**: Instead of requiring that the offline data distribution $d^D$ covers all possible policies (all-policy concentrability), only the optimal policy's occupancy ratio relative to $d^D$ is assumed bounded: $\|d^{\pi^*}/d^D\|_\infty \leq B_w$. This assumption is both empirically weaker and more easily satisfied, and ensures that learning and generalization focus on the optimal (or target) policy rather than all possible behaviors [2202.04634].

These two foundations enable the derivation of sharp sample-complexity and generalization guarantees in both offline and online RL, under substantially milder conditions than prior frameworks.

## 2. Policy Architecture and Learning Frameworks

Recent work highlights two illustrative instantiations of Single-Policy REDI for very different RL settings:

### A. Universal Locomotion Policy for Quadrupedal Robots

In "ManyQuadrupeds: Learning a Single Locomotion Policy for Diverse Quadruped Robots," Shafiee et al. introduce a bio-inspired, hierarchical control pipeline, featuring:

- **Supraspinal Drive**: A multilayer perceptron (MLP) with layers [512, 256, 128] (ELU activations), inputting body and CPG state, outputting per-leg amplitude and frequency.
- **Central Pattern Generator (CPG)**: Each of four limbs uses an uncoupled nonlinear oscillator with amplitude $r_i$ and phase $\phi_i$, modulated by the MLP: 
  $$
  \dot r_i = \alpha(A_i - r_i), \quad \dot\phi_i = \omega_i
  $$
  with $A_i$ and $\omega_i$ in clipped ranges.
- **Pattern Formation (PF) Layer**: Transforms oscillator state to foot trajectories via robot-specific scaling parameters (stride, height), and applies inverse kinematics. PF parameters scale heuristically with robot size.

This design ensures a consistent observation–action space (task-space modulation) across 16 robot morphologies, relying on the PF layer alone for robot-dependent adaptation. No proprioceptive signals (e.g., joint angles/torques) enter the policy, enforcing strong morphology invariance [2310.10486].

### B. Offline RL under REDI: Primal-Dual Algorithmic Approach

For MDPs, "Offline Reinforcement Learning with Realizability and Single-Policy Concentrability" (PRO-RL) optimizes a regularized saddle-point objective with empirical losses:
- Empirical RL objective:
  $$
  \hat{L}_\alpha(v, w) = (1-\gamma)\bar{v}_{\mu_0} - \alpha \bar{f}(w) + \bar{w e_v}
  $$
  where $e_v$ are Bellman errors, $f$ is a strongly convex regularizer, $v$ is the primal variable (value), $w$ the density ratio, and expectations are taken over the static dataset.

The algorithm alternates minimizing in $v$ and maximizing in $w$, extracting the final policy as a modification of the data-generating behavior weighted by the learned $w$:
$$
\hat{\pi}(a|s) = \frac{\hat{w}(s,a)\pi_D(a|s)}{\sum_{a'} \hat{w}(s,a')\pi_D(a'|s)}
$$
This approach achieves polynomial sample complexity under REDI, dispensing with the need for all-policy coverage or Bellman-completeness [2202.04634].

## 3. Training, Reward Design, and Implementation Details

### Universal Locomotion Policy

- **Training**: Proximal Policy Optimization (PPO) in NVIDIA Isaac Gym with 16 parallel environments (1 per robot); policy evaluated and actuated at 100 Hz, CPG integrated at 1 kHz.
- **State**: Body orientation (roll, pitch, yaw), linear/angular velocities, four foot contact flags, four foot positions, previous action, CPG states (amplitude/phase, 8D).
- **Action**: 8-dimensional (per-leg amplitude and frequency).
- **Reward per timestep $t$**:
  $$
  R_t = 8.0\, \min(v_{x,t}, 1.5) - 0.25\, \|\boldsymbol{\theta}_{\text{base},t}\| - 10^{-5}\, \tau_t^\top \dot{q}_t
  $$
  promoting forward velocity, body orientation stability, and energy efficiency.

- **Domain Randomization**: Not used; training relies solely on natural morphologic diversity and task-space modulation.

### Offline RL Algorithm (PRO-RL)

- **Input**: Finite dataset, value function class $V$, density-ratio class $W$, strongly convex regularizer $f$, regularization parameter $\alpha$.
- **Optimization**: Empirical saddle-point solving, minimizing in $v$ and maximizing in $w$.
- **Policy Extraction**: Given learned $w$, extract a policy via normalized importance weighting over the data-generating behavior.

Both frameworks enforce a single invariant policy structure or exploratory strategy across all instances, with adaptation (if any) only permitted in non-centralized, modular components (e.g., pattern formation, post-processing layers).

## 4. Theoretical Guarantees and Sample Complexity

### Offline RL and REDI

The main theorem for PRO-RL establishes that, under realizability and SPC alone, with strong convexity of $f$ and boundedness of $V, W$, one achieves:
$$
J(d^*_\alpha) - J(\hat{\pi}) \leq \frac{4}{1-\gamma} \sqrt{\frac{E_{n_1, n_0, \alpha}}{\alpha M_f}}
$$
with $E_{n_1, n_0, \alpha} = O\left((1-\gamma) B_v \sqrt{\frac{\ln |V|}{n_0}} + (\alpha B_f + B_w B_e) \sqrt{\frac{\ln(|V||W|)}{n_1}}\right)$.
For sufficiently large $n_1, n_0 = \mathrm{poly}(1/\epsilon)$, $\hat{\pi}$ is $\epsilon$-optimal; only single-policy, not all-policy, concentrability is required [2202.04634].

### Parallel Exploration with a Single Policy

In reward-free RL for linear MDPs, using $P$ parallel agents under a shared single policy in each episode yields an almost-linear speedup. Total required samples is
$$
KP = \tilde{\Omega}(d^2 H^3 / \epsilon^2)
$$
matching minimax lower bounds up to logarithmic factors. All convergence and optimism-in-the-face-of-uncertainty lemmas hold with data collected via a single policy per episode, provided all data is pooled [2205.15891].

## 5. Empirical Results and Practical Implications

### Universal Quadruped Control

- **Diversity**: Demonstrated robust trotting across 16 quadruped robots (masses 2–200 kg, body heights 18–100 cm, various morphologies and DoF), with a single policy architecture [2310.10486].
- **Generalization**: Withheld morphologies (HYQ, Dog3, B1) not used during training still yielded stable locomotion across those unseen structures, indicating substantial zero-shot transfer.
- **Sim-to-Real Transfer**: Direct deployment to commercial robots (Unitree Go1, A1) achieved robust outdoor gaiting and unprecedented load-carrying, without any per-robot fine-tuning or retraining.
- **Efficiency**: All robots trained simultaneously in under two hours on a single GPU, with network [512, 256, 128], and no domain randomization.

### Parallel Single-Policy Exploration

Prior work on reward-free exploration in linear MDPs and Markov games confirms near-minimax optimality and almost-linear gains in sample complexity with a single-policy strategy, obviating the need for coordinated heterogeneous exploration [2205.15891].

## 6. Limitations and Open Directions

- **Policy Expressivity**: For locomotion, all PF parameters were hand-scaled per robot. No automated or learned mechanism for end-to-end PF adaptation was proposed [2310.10486].
- **Coverage**: SPC does not guarantee coverage for arbitrary policies; efficacy is limited to domains (or datasets) where the optimal policy's occupancy is adequately represented.
- **Task Diversity**: Locomotion policy currently supports only straight-line gaits. Extension to turning, omnidirectional, or terrain-adapted behaviors remains open.
- **Generalization**: Zero-shot adaptation to classes outside the training regime (e.g., hexapeds, extreme mass distributions) is untested in current studies.
- **Behavioral Inputs**: Inductive biases introduced by acting solely in task-space may limit optimality for highly specialized morphologies or control tasks.

A plausible implication is that automating robot-specific PF parameter determination or further reducing hand-engineered adaptation steps could substantially broaden the Single-Policy REDI paradigm.

---

**Key References:**
- "ManyQuadrupeds: Learning a Single Locomotion Policy for Diverse Quadruped Robots" [2310.10486]
- "Offline Reinforcement Learning with Realizability and Single-policy Concentrability" [2202.04634]
- "One Policy is Enough: Parallel Exploration with a Single Policy is Near-Optimal for Reward-Free Reinforcement Learning" [2205.15891]

Source: https://www.emergentmind.com/topics/single-policy-redi