Barrier-Inspired Reward Shaping in RL

• The paper introduces a barrier-inspired reward shaping framework that uses inverse barrier potentials to penalize unsafe actions while preserving policy optimality.
• It integrates seamlessly with both on- and off-policy RL algorithms, achieving 1.4–2.8× faster convergence and up to 50% reduction in actuation cost in various tasks.
• Empirical evaluations demonstrate successful sim-to-real transfers, smoother robotic motions, and elimination of manual constraint handling in complex environments.

Barrier-Inspired Reward Shaping is a safety-oriented framework for reinforcement learning (RL) that leverages potential-based shaping terms inspired by control-theoretic barrier functions. This approach delivers policy-invariant guidance, steering the agent away from constraint boundaries in high-dimensional and real-world RL tasks, while preserving optimality under nominal objectives. It has demonstrated substantial improvements in training convergence and energy efficiency, and enables direct sim-to-real deployment in complex robotic environments without manual constraint handling (Nilaksh et al., 2024).

1. Mathematical Framework

Barrier-inspired reward shaping introduces constraint-enforcing potentials into the RL reward via a barrier-function–derived term. For a continuous state $s\in\mathbb{R}^n$ subject to $M$ smooth inequality constraints $h_i(s)\equiv b_i - c_i(s) > 0$, the safe set is defined as:

$$S_\mathrm{safe} = \{ s\in\mathbb{R}^n\ |\ h_i(s) > 0\ \forall i\}.$$

The potential function adopts the inverse barrier formulation:

$$\varphi(s) = \sum_{i=1}^M \frac{\kappa_i}{h_i(s) + \varepsilon}$$

where $\kappa_i>0$ are user-chosen scaling coefficients and $\varepsilon>0$ is a numerical regularizer. As $s$ approaches any constraint boundary $h_i(s)\rightarrow 0^+$, $\varphi(s)\rightarrow+\infty$.

The potential-based shaping term at each transition $(s,a)\rightarrow s'$ is:

$$R_\mathrm{shape}(s, a) = \gamma \sum_{i=1}^M \frac{\kappa_i}{h_i(s') + \varepsilon} - \sum_{i=1}^M \frac{\kappa_i}{h_i(s) + \varepsilon}$$

with discount factor $\gamma$. The total reward signal becomes:

$$R_\mathrm{total}(s,a) = R_\mathrm{env}(s,a) + \alpha\, R_\mathrm{shape}(s,a)$$

where $R_\mathrm{env}(s,a)$ is the environment’s intrinsic reward and $\alpha\geq 0$ is a tunable shaping strength, typically $\alpha \in [0.5,\,2.0]$.

2. Safety Principles and Policy Invariance

The barrier potential is designed for safety by penalizing proximity to constraint boundaries. $\varphi(s)$ diverges if any $h_i(s)\rightarrow 0^+$, ensuring that the shaping term discourages transitions toward unsafe states. The shaping component is structured to guarantee policy invariance: the optimal policy for $R_\mathrm{env}$ remains optimal when using $R_\mathrm{total}$, assuming infinite-horizon Markov decision process and proper discounting. Conditions include continuously differentiable constraint functions and matching discount factors ($\gamma$) in both the environment and shaping terms. Exploration is restricted to $S_\mathrm{safe}$, with transitions violating constraints either clipped or assigned large negative rewards.

3. Algorithmic Incorporation

Barrier-inspired reward shaping is compatible with both on-policy and off-policy RL algorithms. The implementation integrates exclusively at the reward calculation step, requiring no changes to policy or value function architectures.

A prototypical integration with Proximal Policy Optimization (PPO) proceeds as follows:

1. Initialize policy $\pi_\theta$, value function $V_\phi$, and buffer $B$.
2. For each iteration:
   • Collect trajectories: for $t=0,\ldots,T-1$
     • Sample $a_t\sim\pi_\theta(\cdot|s_t)$; execute to obtain $s_{t+1}$
     • Compute $\varphi(s_t)$, $\varphi(s_{t+1})$
     • Compute $R_\mathrm{shape}$ and $R_\mathrm{total}$
     • Store transition $(s_t,a_t,R_\mathrm{total},s_{t+1},\mathrm{done})$ in $B$
   • Estimate advantages using $R_\mathrm{total}$ and $V_\phi$
   • Update policy and value parameters
   • Clear buffer

Essential hyperparameters include:

• $\alpha$: shaping weight ($\in[0.1,\,2.0]$)
• $\kappa_i$: per-constraint scaling (chosen so $\kappa_i/h_i$ is $\mathcal{O}(1)$ in typical states)
• $\varepsilon$: regularizer in $[10^{-3},\,10^{-2}]$
• $\gamma$: discount (to match environment)

4. Empirical Evaluation and Results

Evaluation spanned simulated and real-world environments, using CartPole-v1, Ant-v2, Humanoid-v2 (all with Proximal Policy Optimization or Soft Actor-Critic), and a Unitree Go1 quadruped in deployment.

Environment	State Dim	Action Dim	Key Constraints
CartPole-v1	4	2	$|\text{pole angle}| \leq 12^\circ$
Ant-v2 (Mujoco)	$\approx$111	8	Joint angle bounds, joint vel. $\leq 10$ rad/s
Humanoid-v2	$\approx$376	17	Torso, joint limits, height $\geq 0.8$ m
Unitree Go1 (real)	$\approx$50	$\approx$12	Foot clearance, base attitude, torque limits

Quantitative findings include:

• Convergence speed: Barrier-shaping delivered $1.4-2.8\times$ faster convergence (episodes to reach 90% of asymptotic return), e.g., CartPole: $120$ vs $210$ ($1.75\times$), Ant: $11$k vs $30$k ($2.7\times$), Humanoid: $25$k vs $60$k ($2.4\times$).
• Cumulative reward: 10–20% higher during initial ($<10$k) learning steps.
• Actuation effort: 50–60% of baseline for Ant/Humanoid; CartPole: 80% of baseline (measured as time-averaged $\|\mathrm{torque}\|^2$).
• All improvements are statistically significant (paired t-test, $p<0.01$, 5 random seeds).

5. Sim-to-Real Transfer and Practical Implications

The transfer of a PPO policy, trained in Isaac Gym with barrier shaping, from simulation to the physical Unitree Go1 required no neural fine-tuning. Minor tuning included increased $\varepsilon$ (from $10^{-3}$ to $5\times10^{-3}$) for sensor noise robustness and reduced $\kappa_i$ for joint constraints. Observed effects on the real robot:

• Zero falls in the first 1000 control steps (vs. four without shaping)
• Smoother locomotion: 30% reduction in IMU-measured body acceleration spikes
• Elimination of manual constraint-enforcement layers: the barrier-shaped reward alone kept trajectories within the safe operating envelope

6. Theoretical and Practical Significance

Barrier-inspired reward shaping adapts a safety-critical control paradigm to reward shaping in RL. Its inverse barrier potential provides:

• Automatic penalization as trajectories approach dangerous boundaries
• Policy invariance under classic RL theoretical guarantees
• Significant improvements in sample efficiency and actuation cost, as confirmed on both low- and high-dimensional continuous-control tasks

No additional value-function–dependent terms are required, thereby circumventing scalability issues prominent in previous shaping approaches.

7. Limitations and Assumptions

Critical assumptions are that all constraints are continuously differentiable (class $C^1$) and enforcement is tractable within the RL episode rollout. The domain must be effectively restricted to $S_\mathrm{safe}$ at all times. Barrier parameter tuning and regularization may require environment-specific considerations, especially in sim-to-real deployments where sensor imperfections impact constraint evaluation.

Barrier-inspired reward shaping offers a mechanism to combine model-free RL with provably safe exploration, demonstrated across simulated and hardware platforms, and is effective under both on- and off-policy update schemes (Nilaksh et al., 2024).