Neurosymbolic Reinforcement Learning

• Neurosymbolic RL is the integration of neural networks and symbolic reasoning within a reinforcement learning loop to enhance perception, decision-making, and interpretability.
• Key design patterns include using neural models for feature extraction and symbolic modules for state abstraction, reward shaping, and action constraints.
• The field reveals trade-offs between differentiability, scalability, and symbolic fidelity, driving research on automated grounding, efficient optimization, and formal safety.

Neurosymbolic Reinforcement Learning (Neurosymbolic RL) is the family of methods that tightly integrate neural learning and symbolic reasoning within a standard reinforcement learning loop, typically to combine high-dimensional perception and statistical optimization with rules, logic, automata, constraints, or other explicit symbolic structures (Acharya et al., 2023). In contemporary formulations, the symbolic component may define state abstractions, reward functions, action constraints, temporal task structure, safety certificates, or directly interpretable policies, while the neural component supplies perception, function approximation, and end-to-end differentiability (Grandien et al., 2024). The resulting systems are studied for data-efficiency and generalization, explainability and verifiability, and robustness in sequential decision-making, but the literature also emphasizes persistent trade-offs between differentiability, symbolic fidelity, and scalability (Graf et al., 2024).

1. Taxonomies and design patterns

The survey literature organizes Neurosymbolic RL into three taxonomies based on the role played by neural and symbolic components in reinforcement learning: Learning for Reasoning, Reasoning for Learning, and Learning–Reasoning (Acharya et al., 2023). In Learning for Reasoning, neural networks preprocess raw or unstructured inputs and extract symbols or subgoals, after which a downstream symbolic planner or logic engine performs high-level reasoning or policy extraction. In Reasoning for Learning, a symbolic module encodes high-level priors such as reward shaping, safety constraints, or subtask structure, and a neural network learns or fine-tunes the policy under these symbolic constraints. In Learning–Reasoning, neural and symbolic components interleave bidirectionally, often in planner–controller–meta-controller architectures (Acharya et al., 2023).

A complementary formulation appears in work on interpretable model and policy networks, which identifies three pathways: model-free reinforcement learning with differentiable policies, model-based reinforcement learning via Logical Neural Networks plus classical planning, and Differentiable Predictive Control with LNN policies (Graf et al., 2024). In that account, a differentiable decision tree can serve as the policy network in a standard actor–critic pipeline; an LNN can learn a symbolic world model and export exact symbolic rules for PDDL; and a differentiable dynamics model can be chained with an LNN policy for end-to-end optimization through time (Graf et al., 2024).

These categorizations show that Neurosymbolic RL is not a single architecture. The symbolic layer may be a rule set, an automaton, a program sketch, a decision tree, a reward machine, a behavior tree, a logic tensor network, or a propositional model compiled into a PSDD. The neural layer may be a CNN, RNN, GRU, LSTM, MLP, VAE-style object encoder, or policy/value network. A plausible implication is that the field is best understood as a design space defined by where symbolic structure enters the RL loop: before policy learning, inside policy learning, or as a post-training distillation target.

2. Symbol grounding, object-centric state construction, and relational abstraction

A central problem in Neurosymbolic RL is symbol grounding: mapping raw, often high-dimensional observations into symbolic or object-centric states that a reasoning module can use. This problem is explicit in Neural Reward Machines, which target non-symbolic non-markovian RL domains without any knowledge of the symbol grounding function. An NRM is defined as a probabilistic Moore machine

$\mathcal M = (Q,\Sigma,\delta_\theta,r_\phi),$

with a learned soft grounding $\sigma_\theta(o_t)\in\Delta(\Sigma)$ and a belief update

$$b_t=\sum_{p\in\Sigma}\sigma_\theta(o_t)[p]\;(b_{t-1}T_\theta(p)),$$

so that reward prediction and automaton-state inference remain differentiable end-to-end (Umili et al., 2024). The same work introduces semisupervised symbol grounding supervised only by agreement between predicted and observed reward sequences, and a groundability analysis algorithm that achieves a $\approx 10^3$-fold speedup over a naive $\mathcal O(|\Sigma|^{|\Sigma|})$ brute force (Umili et al., 2024).

Object-centric grounding is developed more explicitly in SCoBots. There, the standard deep-RL pipeline is decomposed into three sequential stages: an object extractor $\omega_{\theta_1}$ mapping raw frames $s_t$ to object slots $\Omega_t$, a relation extractor $\mu_F$ mapping slots to a relational feature vector $\Gamma_t$, and an action selector $\sigma_\theta(o_t)\in\Delta(\Sigma)$0 mapping relational features to actions (Grandien et al., 2024). The object-centric encoder is a VAE-style encoder trained with

$\sigma_\theta(o_t)\in\Delta(\Sigma)$1

and the relational layer applies a fixed set of concepts $\sigma_\theta(o_t)\in\Delta(\Sigma)$2, such as distances between detected objects, to produce a fixed-dimensional symbolic vector (Grandien et al., 2024). In the reported Atari setting, the object extractor uses input sequences of $\sigma_\theta(o_t)\in\Delta(\Sigma)$3 frames, downscaled to $\sigma_\theta(o_t)\in\Delta(\Sigma)$4, with a SPACE grid of $\sigma_\theta(o_t)\in\Delta(\Sigma)$5 cells, batch size $\sigma_\theta(o_t)\in\Delta(\Sigma)$6, 5 k training steps, learning rates $\sigma_\theta(o_t)\in\Delta(\Sigma)$7, and MOC weights $\sigma_\theta(o_t)\in\Delta(\Sigma)$8, $\sigma_\theta(o_t)\in\Delta(\Sigma)$9 (Grandien et al., 2024).

Other frameworks instantiate symbol grounding through logic-aware latent models rather than explicit object slots. NSAM compiles a global propositional constraint $$b_t=\sum_{p\in\Sigma}\sigma_\theta(o_t)[p]\;(b_{t-1}T_\theta(p)),$$0 into a PSDD and learns a state-conditioned distribution over symbolic models $$b_t=\sum_{p\in\Sigma}\sigma_\theta(o_t)[p]\;(b_{t-1}T_\theta(p)),$$1 via a gating network $$b_t=\sum_{p\in\Sigma}\sigma_\theta(o_t)[p]\;(b_{t-1}T_\theta(p)),$$2 (Han et al., 11 Feb 2026). The learned symbolic model is then used to infer a MAP interpretation

$$b_t=\sum_{p\in\Sigma}\sigma_\theta(o_t)[p]\;(b_{t-1}T_\theta(p)),$$3

for downstream action masking (Han et al., 11 Feb 2026). INSIGHT follows yet another route: it distills off-line vision-foundation models, specifically FastSAM and DeAot, into a small convolutional perception module that predicts object existence indicators, normalized coordinates, and normalized box sizes from four-frame Atari stacks, pretraining on approximately $$b_t=\sum_{p\in\Sigma}\sigma_\theta(o_t)[p]\;(b_{t-1}T_\theta(p)),$$4 labeled frames and then refining the perception network during policy learning (Luo et al., 2024).

Across these systems, symbol grounding is neither a fixed preprocessing step nor a purely supervised perception task. The literature repeatedly treats it as part of the RL problem itself: reward signals can refine perception, symbolic validity can regularize grounding, and temporal reward structure can supervise the emergence of latent symbols (Luo et al., 2024, Umili et al., 2024, Han et al., 11 Feb 2026).

3. Symbolic policies, rule extraction, and interpretable control

One of the most visible strands of Neurosymbolic RL seeks policies that are directly interpretable or can be extracted into explicit rule sets. SCoBots exemplify a pipeline in which a PPO policy over relational features is distilled into IF–THEN rules with ECLAIRE after convergence (Grandien et al., 2024). The neural policy is trained with PPO on $$b_t=\sum_{p\in\Sigma}\sigma_\theta(o_t)[p]\;(b_{t-1}T_\theta(p)),$$5, after which ECLAIRE extracts rules of the form

$$b_t=\sum_{p\in\Sigma}\sigma_\theta(o_t)[p]\;(b_{t-1}T_\theta(p)),$$6

For Pong, the paper reports representative rules stating that if $$b_t=\sum_{p\in\Sigma}\sigma_\theta(o_t)[p]\;(b_{t-1}T_\theta(p)),$$7 and the ball is above the player then the action is “MoveUp,” while the complementary condition selects “MoveDown” (Grandien et al., 2024). Quantitatively, the paper reports for Pong (max=21): Neural GT (2-layer) $$b_t=\sum_{p\in\Sigma}\sigma_\theta(o_t)[p]\;(b_{t-1}T_\theta(p)),$$8, Rule-set GT (pruned) $$b_t=\sum_{p\in\Sigma}\sigma_\theta(o_t)[p]\;(b_{t-1}T_\theta(p)),$$9, Neural SPACE+MOC (2-layer) $\approx 10^3$0, Rule-set SPACE+MOC (pruned) $\approx 10^3$1, and Pixel-based PPO $\approx 10^3$2; for Boxing (max=100): Neural GT (2-layer) $\approx 10^3$3, Rule-set GT (pruned) $\approx 10^3$4, Neural SPACE+MOC $\approx 10^3$5, Rule-set SPACE+MOC $\approx 10^3$6, and Pixel PPO $\approx 10^3$7 (Grandien et al., 2024).

Other frameworks place symbolic logic directly inside the policy network. NSRL for interpretable deep RL represents predicates as adjacency matrices and performs differentiable multi-hop reasoning through soft attention over predicates and chain lengths,

$\approx 10^3$8

then feeds the resulting reasoning tensor to per-action MLP heads (Ma et al., 2021). After training, rule confidences can be read from the attention weights and converted into chain-like Horn clauses (Ma et al., 2021). DERRL likewise learns first-order Horn clauses through a neural rule-generator and fuzzy forward chaining with the Lukasiewicz $\approx 10^3$9-norm, then extracts fully symbolic policies by thresholding learned rule probabilities (Hazra et al., 2023). In its reported experiments, DERRL generalizes from a 5-agent to an 8-agent Traffic grid and from 4-block to 5-block Blocks World settings, while yielding clauses such as

$\mathcal O(|\Sigma|^{|\Sigma|})$0

after extraction (Hazra et al., 2023).

Text-based environments have driven analogous developments. FOL-LNN converts observations into first-order logic facts, groups them by type using ConceptNet, and trains an LNN with a DQN-style temporal-difference loss plus a contradiction loss enforcing logical consistency (Kimura et al., 2021). On TextWorld Coin-Collector, the reported epoch-100 results include reward $\mathcal O(|\Sigma|^{|\Sigma|})$1 and $\mathcal O(|\Sigma|^{|\Sigma|})$2 steps on easy games, reward $\mathcal O(|\Sigma|^{|\Sigma|})$3 and $\mathcal O(|\Sigma|^{|\Sigma|})$4 steps on medium games, and reward $\mathcal O(|\Sigma|^{|\Sigma|})$5 and $\mathcal O(|\Sigma|^{|\Sigma|})$6 steps on hard games, outperforming the listed symbolic and neural baselines (Kimura et al., 2021). NESTA uses a pre-trained AMR parser, converts AMR graphs into first-order facts, and learns differentiable Horn-clause policies with LNNs, reporting on TextWorld-CommonSense that NESTA reaches normalized score $\mathcal O(|\Sigma|^{|\Sigma|})$7 on easy and medium out-of-distribution splits and $\mathcal O(|\Sigma|^{|\Sigma|})$8 on hard, while using fewer steps than the listed baseline (Chaudhury et al., 2023).

A different policy-form perspective appears in S-REINFORCE, which trains a neural policy gradient agent together with a symbolic regressor that fits explicit expressions $\mathcal O(|\Sigma|^{|\Sigma|})$9 and then reuses the symbolic proxy through importance sampling (Dutta et al., 2023). The paper reports symbolic policies such as $\omega_{\theta_1}$0 for CartPole and analogous explicit expressions in Acrobot and LunarLanderContinuous (Dutta et al., 2023). This suggests that interpretability in Neurosymbolic RL is not limited to discrete rules: it also includes sparse algebraic policies, differentiable decision trees, and logic-constrained networks.

4. Symbolic rewards, automata, and temporally extended task structure

A second major line of work uses symbolic structure to define or learn reward functions and long-horizon temporal abstractions. LISR represents intrinsic rewards as symbolic trees of depth $\omega_{\theta_1}$1, where leaves are components of an observation vector and internal nodes are arithmetic, logic, or gating operators (Sheikh et al., 2020). The intrinsic reward has the form

$\omega_{\theta_1}$2

and the framework optimizes a bilevel objective in which policy parameters $\omega_{\theta_1}$3 maximize return under the symbolic reward while tree structures $\omega_{\theta_1}$4 are evolved to maximize true extrinsic performance (Sheikh et al., 2020). The reported findings state that LISR (EA + SR) outperforms all baselines in 6/8 continuous and discrete tasks, that even the pure SR learner solves PixelCopter and Catcher using only its discovered symbolic reward, and that discovered trees use $\omega_{\theta_1}$5 layers and approximately $\omega_{\theta_1}$6–$\omega_{\theta_1}$7 operations versus approximately $\omega_{\theta_1}$8 K parameters in a neural ICM module (Sheikh et al., 2020).

Neural Reward Machines address temporally extended tasks more explicitly. The framework starts from classic symbolic reward machines or LTLf-derived automata, but replaces fixed symbolic transition and reward functions with differentiable neural parameterizations (Umili et al., 2024). In the reported experiments on grid-world “map” and “image” environments with tasks such as Visit, SeqVisit, and GlobalAvoid, NRM + A2C achieves success rate $\omega_{\theta_1}$9, cumulative reward $s_t$0, and $s_t$1 episodes to $s_t$2 success, compared with $s_t$3, $s_t$4, and $s_t$5 for RNN + A2C, nearly matching the RM + A2C upper bound of $s_t$6, $s_t$7, and $s_t$8 (Umili et al., 2024). The same paper reports that training without known $s_t$9 and $\Omega_t$0 degrades sample efficiency by approximately $\Omega_t$1, while adding $\Omega_t$2 label noise to grounding increases RL episodes by approximately $\Omega_t$3 (Umili et al., 2024).

The more recent SR-DQN also exploits symbolic temporal or task priors, though in a different form. It transfers partial policies from simple domain instances into harder settings by encoding them as logical rules, then performs online reasoning to bias exploration and rescale Q-values during exploitation (Veronese et al., 6 Jan 2026). In OfficeWorld and DoorKey, the paper reports that SR-DQN converges in roughly half the number of environment steps needed by plain DQN and attains approximately $\Omega_t$4 higher final return than either DQN or DQN+RewardMachines on larger-scale variants; the table excerpt gives, for example, OfficeWorld DeliverCoffeeAndMail: DQN $\Omega_t$5, RM-DQN $\Omega_t$6, SR-DQN $\Omega_t$7, and DoorKey $\Omega_t$8, 2 keys: DQN $\Omega_t$9, RM-DQN $\mu_F$0, SR-DQN $\mu_F$1 (Veronese et al., 6 Jan 2026).

These approaches differ in where symbolic structure is inserted. In LISR, symbolic programs define intrinsic rewards. In NRMs, symbolic task structure is represented by a latent automaton belief state. In SR-DQN, symbolic knowledge appears as partial policies that guide both exploration and exploitation. The shared theme is that symbolic abstractions are used to reshape the credit-assignment problem rather than merely to explain a learned policy.

5. Logic-guided learning, action masking, and formally safe exploration

A substantial part of the literature uses symbolic knowledge to constrain action selection, rank candidate actions, or certify safety during learning. SymDQN augments DuelDQN with Logic Tensor Network modules for shape recognition, reward prediction, and reasoning consistency (Amador et al., 3 Apr 2025). Its ActionReasoner axiom aligns rankings of predicted immediate rewards and learned Q-values,

$\mu_F$2

while an ActionFilter used at test time prunes any action whose predicted reward is lower than $\mu_F$3 (Amador et al., 3 Apr 2025). On the 5×5 grid navigation task, the reported score at 250 epochs is $\mu_F$4 for SymDQN(AF), $\mu_F$5 for SymDQN(AR,AF), $\mu_F$6 for SymDQN(AR), $\mu_F$7 for SymDQN, and $\mu_F$8 for DuelDQN; the corresponding precision table reports $\mu_F$9 negative-reward consumption for SymDQN(AF) and SymDQN(AR,AF) at 250 epochs (Amador et al., 3 Apr 2025).

NSAM pushes constraints further by learning symbolic models from high-dimensional states in a minimally supervised way and then masking infeasible actions through propositional preconditions (Han et al., 11 Feb 2026). Given a learned symbolic model, the mask

$\Gamma_t$0

is used to renormalize the policy,

$\Gamma_t$1

The paper reports that across all 16 tasks, NSAM converges faster and to higher reward than Rainbow, PPO, PPO-Lagrangian, KCAC, RC-PPO, and PLPG, reducing violation rates from $\Gamma_t$2–$\Gamma_t$3 down to approximately $\Gamma_t$4–$\Gamma_t$5; in Visual Sudoku 5×5, it reports 0.8 average reward with 2.5% violation versus $\Gamma_t$6 for the best competitor (Han et al., 11 Feb 2026).

Safe exploration under worst-case disturbances is addressed by Revel, which alternates between a restricted symbolic policy class that allows efficient verification and a more expressive neurosymbolic policy class that safely embeds neural controllers (Anderson et al., 2020). A neurosymbolic policy has the form

$\Gamma_t$7

where $\Gamma_t$8 is a safe symbolic policy, $\Gamma_t$9 is a differentiable neural policy, and $\sigma_\theta(o_t)\in\Delta(\Sigma)$00 is an invariant (Anderson et al., 2020). The paper reports zero worst-case safety violations in all runs for Revel across ten continuous-control benchmarks, while Ddpg and/or CPO incur nonzero worst-case safety violations per episode in 8/10 domains (Anderson et al., 2020).

NUMERLA applies symbolic safety constraints to online meta-reinforcement learning in CARLA driving scenarios (Lei et al., 2023). Its symbolic safety-constraint module maps latent environment modes to safe parameter subsets, and the lookahead update

$\sigma_\theta(o_t)\in\Delta(\Sigma)$01

is constrained so that $\sigma_\theta(o_t)\in\Delta(\Sigma)$02 (Lei et al., 2023). In Well-Behaved walking and Jaywalking scenarios, the paper reports collision rates of approximately $\sigma_\theta(o_t)\in\Delta(\Sigma)$03–$\sigma_\theta(o_t)\in\Delta(\Sigma)$04 for RL, approximately $\sigma_\theta(o_t)\in\Delta(\Sigma)$05–$\sigma_\theta(o_t)\in\Delta(\Sigma)$06 for COLA, and approximately $\sigma_\theta(o_t)\in\Delta(\Sigma)$07–$\sigma_\theta(o_t)\in\Delta(\Sigma)$08 for NUMERLA in the first scenario; and approximately $\sigma_\theta(o_t)\in\Delta(\Sigma)$09–$\sigma_\theta(o_t)\in\Delta(\Sigma)$10, approximately $\sigma_\theta(o_t)\in\Delta(\Sigma)$11–$\sigma_\theta(o_t)\in\Delta(\Sigma)$12, and approximately $\sigma_\theta(o_t)\in\Delta(\Sigma)$13–$\sigma_\theta(o_t)\in\Delta(\Sigma)$14 respectively in the second (Lei et al., 2023).

These works counter a common misconception that symbolic knowledge in RL is primarily explanatory. In this literature, symbolic modules frequently act as operational control mechanisms: they prune infeasible actions, regularize Q-value orderings, restrict online adaptation, and enable formal verification of exploration policies.

6. Interpretability, generalization, evaluation, and unresolved tensions

The literature consistently presents interpretability and generalization as central motivations, but it does so with different operational meanings. For SCoBots, interpretability is “intrinsic interpretability by design,” because decisions flow from object slots to relations to rules (Grandien et al., 2024). For NSRL, DERRL, FOL-LNN, and NESTA, interpretability means human-readable Horn clauses, first-order rules, or extracted relational chains (Ma et al., 2021, Hazra et al., 2023, Kimura et al., 2021, Chaudhury et al., 2023). For S-REINFORCE, it means analytic symbolic expressions over state variables (Dutta et al., 2023). For Revel and NUMERLA, interpretability is closely tied to safety certificates, invariants, and symbolic safe sets (Anderson et al., 2020, Lei et al., 2023). For INSIGHT, accessibility itself becomes an explicit problem: the framework therefore prompts GPT-4 to generate textual explanations of symbolic policies and individual decisions, while still jointly learning structured states and symbolic policies end-to-end (Luo et al., 2024).

Generalization is likewise instantiated in several ways. DERRL reports transfer to different configurations and contexts in Countdown, Blocks World, Gridworld, and Traffic (Hazra et al., 2023). FOL-LNN reports reliable solution of unseen larger-level games in TextWorld within 200 epochs when other listed methods fail to generalize (Kimura et al., 2021). NESTA reports stronger out-of-distribution performance on unseen text games than deep RL-based baselines, with fewer interactions (Chaudhury et al., 2023). SCoBots were motivated by the claim that deep RL agents rely on shortcut learning and do not generalize to slightly different environments, and the framework was proposed as a fairer object-centric comparison because it learns object-centric representations directly from raw pixel-based states (Grandien et al., 2024). Sample efficiency is emphasized in NRMs, SR-DQN, NSAM, and the 2026 sample-efficient neurosymbolic DQN, each using symbolic structure to reduce the effective search space or inject priors (Umili et al., 2024, Veronese et al., 6 Jan 2026, Han et al., 11 Feb 2026).

At the same time, the field repeatedly documents limitations. The “Three Pathways” paper states that the relaxation to real-valued, differentiable representations presents a trade-off: the more learnable, the less interpretable (Graf et al., 2024). SCoBots note dependence on unsupervised sampling of all object appearances and reliable optical flow, limited attributes beyond bounding-box location and class, and the possibility of large, overlapping, or conflicting rule sets (Grandien et al., 2024). LISR reports that symbolic search is costly, with training approximately $\sigma_\theta(o_t)\in\Delta(\Sigma)$15–$\sigma_\theta(o_t)\in\Delta(\Sigma)$16 slower than off-the-shelf SAC on tasks with known dense rewards, and that limited operator dictionaries can bias the discovered reward functions (Sheikh et al., 2020). NSAM highlights the exponential growth of PSDD compilation in the number of propositions and the current restriction to hard propositional constraints (Han et al., 11 Feb 2026). The 2023 survey identifies automated symbolic knowledge generation, symbol grounding at scale, joint optimization algorithms, and full-policy verification as open research opportunities (Acharya et al., 2023).

Taken together, these results indicate that Neurosymbolic RL is better viewed as a set of engineering and scientific compromises than as a settled paradigm. The symbolic layer can improve transparency, sample efficiency, temporal abstraction, and safety, but it introduces its own burdens: grounding errors, search complexity, architectural rigidity, and nontrivial mappings from raw signals to predicates. The recurring research question is therefore not whether to combine neural and symbolic methods, but which symbolic abstraction is appropriate for a given RL regime, how tightly it should be coupled to learning, and what form of interpretability or guarantee is actually required.