Sample-efficient Neuro-symbolic Proximal Policy Optimization

Abstract: Deep Reinforcement Learning (DRL) algorithms often require a large amount of data and struggle in sparse-reward domains with long planning horizons and multiple sub-goals. In this paper, we propose a neuro-symbolic extension of Proximal Policy Optimization (PPO) that transfers partial logical policy specifications learned in easier instances to guide learning in more challenging settings. We introduce two integrations of symbolic guidance: (i) H-PPO-Product, which biases the action distribution at sampling time, and (ii) H-PPO-SymLoss, which augments the PPO loss with a symbolic regularization term. We evaluate our methods on three benchmarks (OfficeWorld, WaterWorld, and DoorKey), showing consistently faster learning and higher return at convergence than PPO and a Reward Machine baseline, also under imperfect symbolic knowledge.

• The paper introduces a neuro-symbolic RL framework that integrates logical guidance into PPO to significantly improve sample efficiency.
• It presents two methods—H-PPO-Product and H-PPO-SymLoss—that bias action sampling and augment policy loss to overcome sparse-reward challenges.
• Experimental results on gridworld benchmarks demonstrate robust performance gains over traditional PPO and Reward Machines, underscoring scalable, interpretable policy optimization.

Sample-efficient Neuro-symbolic Proximal Policy Optimization: An Expert Analysis

Introduction

The paper "Sample-efficient Neuro-symbolic Proximal Policy Optimization" (2604.25534) introduces a neuro-symbolic reinforcement learning (RL) framework that leverages logical policy abstractions to improve sample efficiency for Proximal Policy Optimization (PPO) in sparse-reward, long-horizon environments. The methodology integrates action-level symbolic guidance into RL in two forms: modifying action sampling (H-PPO-Product) and augmenting the policy loss with a symbolic regularizer (H-PPO-SymLoss). Empirical validation across three gridworld benchmarks—DoorKey, OfficeWorld, and WaterWorld—demonstrates significant improvements in sample complexity, convergence, and final return compared to vanilla PPO and Reward Machine (RM) baselines, notably without hyperparameter retuning and robust to imperfect symbolic knowledge.

Neuro-symbolic Policy Guidance: Methodology

The core premise is the explicit transfer of partial logical rules, learned in simpler task instances, to downstream harder MDP configurations. Symbolic policies are specified as sets of Horn clauses over high-level state predicates and action terms, inducing a mapping from symbolic state abstraction to recommended actions.

H-PPO-Product biases action selection during sampling by upweighting actions entailed by the symbolic policy, with confidence regulated by an annealed coefficient. The action distribution at time $t$ is multiplied by a per-action mask, incremented for symbolically preferred choices, and normalized. A linear annealing schedule ensures that symbolic bias decays over the course of training, permitting fallback to the neural policy and thus mitigating the risk of misguidance in the case of incomplete or noisy rules.

H-PPO-SymLoss integrates logical guidance at the optimization stage. A symbolic policy-derived reference distribution is constructed via per-action weights ($\eta$ for symbolically entailed actions, $1-\eta$ otherwise, then renormalized). The standard PPO surrogate objective is augmented with a regularization term enforcing proximity between the learned policy and this reference. The strength of symbolic guidance is controlled by a weight $\Theta$, optionally decayed during training to balance exploration vs. exploitation, as highlighted in ablation studies.

This integration circumvents the limitations of reward shaping (as in RM), which is critically sensitive to heuristic design and can introduce degenerate behaviors when the reward signal poorly reflects task structure.

Experimental Evaluation Across Benchmarks

The paper conducts extensive experiments in three paradigmatic RL benchmarks, each featuring long-horizon, sparse-reward structures and possible sequences of sub-goals. Symbolic policies are always imperfect and learned from simpler settings.

DoorKey

In DoorKey tasks, both grid size and number of distractor keys are increased post-symbolic policy induction to test generalization and exploration. The results show substantial return gains:

Figure 1: H-PPO-Product and H-PPO-SymLoss accelerate and stabilize learning, outperforming PPO and Reward Machines as grid size and number of distractors increase.

H-PPO-Product drives early exploration most effectively, achieving rapid convergence in large and multi-key configurations. RMs fail to solve settings with multiple distractors, while H-PPO-SymLoss catches up in optimality but with slower onset.

OfficeWorld

OfficeWorld encompasses both simple (DeliverCoffee, PatrolAB) and complex (DeliverCoffeeAndMail, PatrolABC) variants requiring sequencing and hazard avoidance.

Figure 2: In OfficeWorld, H-PPO-Product dominates exploration and early convergence, while H-PPO-SymLoss yields the highest returns in complex compositions.

H-PPO-Product is superior in initial exploration, particularly in multi-room patrols, while H-PPO-SymLoss overtakes in end performance on complex sequences, demonstrating the utility of direct policy regularization in tasks requiring structured plan adherence.

WaterWorld

WaterWorld exposes the most challenging exploration bottleneck with continuous state/action spaces and long color-target sequences.

Figure 3: H-PPO-Product is the only method achieving meaningful reward in the hardest WaterWorld sequence, clearly outperforming both PPO and symbolic reward shaping.

While all methods solve the simplest RedGreen sequence, only H-PPO-Product maintains exploration and reward acquisition in the extended RedGreenAndBlueCyanAndMagentaYellow sequence. H-PPO-SymLoss provides insufficient exploration pressure, confirming that its symbolic regularization accelerates stabilization but does not itself promote optimal search in highly sparse domains.

Ablation Studies

Ablations on the symbolic guidance strength (via $\Theta$ in H-PPO-SymLoss and $\varepsilon_f$ in H-PPO-Product) demonstrate key effects:

• Decaying $\Theta$ in H-PPO-SymLoss balances initial exploration and eventual policy regularization. Fixed high values suppress exploration; low or decayed weights retain high final return with adequate exploration.

Figure 4: Ablation over $\Theta$ in DoorKey shows decayed symbolic penalty optimally balances exploration and regularization.

• Final symbolic bias $\varepsilon_f$ in H-PPO-Product: Maintaining a nonzero final bias can benefit performance specifically in the hardest, longest-horizon tasks by keeping symbolic guidance persistent.

Implications, Strengths, and Limitations

Empirical results underscore that action-level logical priors can be harnessed in high-performance model-free deep RL to drastically reduce sample requirements and enhance final policy fidelity, even when carry-over rules are partial or noisy.

• H-PPO-Product robustly overcomes exploration bottlenecks without hand-crafted reward shaping, excelling in domains with significant reward sparsity and long contingencies.
• H-PPO-SymLoss systematically improves policy generalization and final returns in tasks characterized by compositional structure and sequential sub-goals, conditional on encountering requisite events through exploration.
• Both mechanisms obviate repeated hyperparameter tuning when scaling task complexity, a nontrivial benefit for practical applications.

However,

• Symbolic policy quality remains critical: completely incorrect or adversarial rules may degrade performance, and overly strong regularization suppresses stochasticity necessary for long-horizon discovery.
• Symbolic extraction and abstraction remain nontrivial in non-symbolic domains; results rely on logical policies that are at least partially meaningful.

Future Directions

The paper signals several avenues for future work:

• Extending the framework to continuous action spaces, where integration of symbolic priors poses additional representational challenges.
• Exploration of zero-shot transfer, where symbolic priors could be induced in one domain and applied across structurally similar but perceptually distinct environments.
• Application to real-world robotics, particularly where interpretability and robust constraint adherence are crucial.

Conclusion

This work establishes a formal, empirical, and algorithmic foundation for neuro-symbolic RL at the intersection of interpretability, sample efficiency, and policy expressiveness. By integrating action-level logical specifications directly into state-of-the-art PPO, the methods presented enable consistent performance improvements over both neural and reward-based baselines, robust to suboptimal symbolic priors and scalable to challenging task instances. These contributions advance the synthesis of symbolic reasoning and gradient-based RL toward more reliable and explainable autonomous agents.