---
title: Neurosymbolic Reinforcement Learning
url: https://www.emergentmind.com/topics/neurosymbolic-reinforcement-learning
type: topic
---

# Neurosymbolic Reinforcement Learning

Neurosymbolic Reinforcement Learning (Neurosymbolic RL) is the family of methods that tightly integrate neural learning and symbolic reasoning within a standard reinforcement learning loop, typically to combine high-dimensional perception and statistical optimization with rules, logic, automata, constraints, or other explicit symbolic structures [2309.01038]. In contemporary formulations, the symbolic component may define state abstractions, reward functions, action constraints, temporal task structure, safety certificates, or directly interpretable policies, while the neural component supplies perception, function approximation, and end-to-end differentiability [2410.14371]. The resulting systems are studied for data-efficiency and generalization, explainability and verifiability, and robustness in sequential decision-making, but the literature also emphasizes persistent trade-offs between differentiability, symbolic fidelity, and scalability [2402.05307].

## 1. Taxonomies and design patterns

The survey literature organizes Neurosymbolic RL into three taxonomies based on the role played by neural and symbolic components in reinforcement learning: **Learning for Reasoning**, **Reasoning for Learning**, and **Learning–Reasoning** [2309.01038]. In **Learning for Reasoning**, neural networks preprocess raw or unstructured inputs and extract symbols or subgoals, after which a downstream symbolic planner or logic engine performs high-level reasoning or policy extraction. In **Reasoning for Learning**, a symbolic module encodes high-level priors such as reward shaping, safety constraints, or subtask structure, and a neural network learns or fine-tunes the policy under these symbolic constraints. In **Learning–Reasoning**, neural and symbolic components interleave bidirectionally, often in planner–controller–meta-controller architectures [2309.01038].

A complementary formulation appears in work on interpretable model and policy networks, which identifies three pathways: model-free reinforcement learning with differentiable policies, model-based reinforcement learning via Logical Neural Networks plus classical planning, and Differentiable Predictive Control with LNN policies [2402.05307]. In that account, a differentiable decision tree can serve as the policy network in a standard actor–critic pipeline; an LNN can learn a symbolic world model and export exact symbolic rules for PDDL; and a differentiable dynamics model can be chained with an LNN policy for end-to-end optimization through time [2402.05307].

These categorizations show that Neurosymbolic RL is not a single architecture. The symbolic layer may be a rule set, an automaton, a program sketch, a decision tree, a reward machine, a behavior tree, a logic tensor network, or a propositional model compiled into a PSDD. The neural layer may be a CNN, RNN, GRU, LSTM, MLP, VAE-style object encoder, or policy/value network. A plausible implication is that the field is best understood as a design space defined by where symbolic structure enters the RL loop: before policy learning, inside policy learning, or as a post-training distillation target.

## 2. Symbol grounding, object-centric state construction, and relational abstraction

A central problem in Neurosymbolic RL is symbol grounding: mapping raw, often high-dimensional observations into symbolic or object-centric states that a reasoning module can use. This problem is explicit in Neural Reward Machines, which target non-symbolic non-markovian RL domains without any knowledge of the symbol grounding function. An NRM is defined as a probabilistic Moore machine
$$
\mathcal M = (Q,\Sigma,\delta_\theta,r_\phi),
$$
with a learned soft grounding $\sigma_\theta(o_t)\in\Delta(\Sigma)$ and a belief update
$$
b_t=\sum_{p\in\Sigma}\sigma_\theta(o_t)[p]\;(b_{t-1}T_\theta(p)),
$$
so that reward prediction and automaton-state inference remain differentiable end-to-end [2408.08677]. The same work introduces semisupervised symbol grounding supervised only by agreement between predicted and observed reward sequences, and a groundability analysis algorithm that achieves a $\approx 10^3$-fold speedup over a naive $\mathcal O(|\Sigma|^{|\Sigma|})$ brute force [2408.08677].

Object-centric grounding is developed more explicitly in SCoBots. There, the standard deep-RL pipeline is decomposed into three sequential stages: an object extractor $\omega_{\theta_1}$ mapping raw frames $s_t$ to object slots $\Omega_t$, a relation extractor $\mu_F$ mapping slots to a relational feature vector $\Gamma_t$, and an action selector $\rho_{\theta_2}$ mapping relational features to actions [2410.14371]. The object-centric encoder is a VAE-style encoder trained with
$$
L_{obj}=L_{rec}+\lambda_M L_{motion}+\lambda_{OC}L_{cont},
$$
and the relational layer applies a fixed set of concepts $F=\{f_1,\ldots,f_D\}$, such as distances between detected objects, to produce a fixed-dimensional symbolic vector [2410.14371]. In the reported Atari setting, the object extractor uses input sequences of $n=4$ frames, downscaled to $128\times128$, with a SPACE grid of $16\times16$ cells, batch size $16$, 5 k training steps, learning rates $\{3e{-}5,1e{-}3\}$, and MOC weights $\lambda_M=100$, $\lambda_{OC}=10$ [2410.14371].

Other frameworks instantiate symbol grounding through logic-aware latent models rather than explicit object slots. NSAM compiles a global propositional constraint $\phi$ into a PSDD and learns a state-conditioned distribution over symbolic models $\Pr(\mathbf m\mid s)$ via a gating network $g_\psi(s)$ [2602.10598]. The learned symbolic model is then used to infer a MAP interpretation
$$
\hat{\mathbf m}(s)=\arg\max_{\mathbf m\in|[\phi]|}\Pr(\mathbf m\mid s)
$$
for downstream action masking [2602.10598]. INSIGHT follows yet another route: it distills off-line vision-foundation models, specifically FastSAM and DeAot, into a small convolutional perception module that predicts object existence indicators, normalized coordinates, and normalized box sizes from four-frame Atari stacks, pretraining on approximately $10^4$ labeled frames and then refining the perception network during policy learning [2403.12451].

Across these systems, symbol grounding is neither a fixed preprocessing step nor a purely supervised perception task. The literature repeatedly treats it as part of the RL problem itself: reward signals can refine perception, symbolic validity can regularize grounding, and temporal reward structure can supervise the emergence of latent symbols [2403.12451; 2408.08677; 2602.10598].

## 3. Symbolic policies, rule extraction, and interpretable control

One of the most visible strands of Neurosymbolic RL seeks policies that are directly interpretable or can be extracted into explicit rule sets. SCoBots exemplify a pipeline in which a PPO policy over relational features is distilled into IF–THEN rules with ECLAIRE after convergence [2410.14371]. The neural policy is trained with PPO on $\Gamma_t$, after which ECLAIRE extracts rules of the form
$$
\text{IF }(g_i>v_i\wedge g_j\le v_j\wedge\cdots)\text{ THEN }a=k.
$$
For Pong, the paper reports representative rules stating that if $\mathrm{Dist}(\mathrm{Ball}_1,\mathrm{Player}_1)<0.15$ and the ball is above the player then the action is “MoveUp,” while the complementary condition selects “MoveDown” [2410.14371]. Quantitatively, the paper reports for Pong (max=21): Neural GT (2-layer) $17.4\pm1.6$, Rule-set GT (pruned) $15.0\pm3.0$, Neural SPACE+MOC (2-layer) $16.8\pm1.5$, Rule-set SPACE+MOC (pruned) $14.4\pm2.6$, and Pixel-based PPO $16.4$; for Boxing (max=100): Neural GT (2-layer) $93.0\pm3.9$, Rule-set GT (pruned) $67.2\pm6.7$, Neural SPACE+MOC $65.0\pm11.3$, Rule-set SPACE+MOC $51.8\pm8.2$, and Pixel PPO $90.3$ [2410.14371].

Other frameworks place symbolic logic directly inside the policy network. NSRL for interpretable deep RL represents predicates as adjacency matrices and performs differentiable multi-hop reasoning through soft attention over predicates and chain lengths,
$$
\kappa(S_\psi,S_\varphi)=\sum_{t'=1}^{T}s_\psi^{(t')}\Bigl(\prod_{t=1}^{t'}\sum_{k=1}^{N}s_{\varphi,k}^{(t)}\mathbf M_k\Bigr),
$$
then feeds the resulting reasoning tensor to per-action MLP heads [2103.08228]. After training, rule confidences can be read from the attention weights and converted into chain-like Horn clauses [2103.08228]. DERRL likewise learns first-order Horn clauses through a neural rule-generator and fuzzy forward chaining with the Lukasiewicz $t$-norm, then extracts fully symbolic policies by thresholding learned rule probabilities [2304.08349]. In its reported experiments, DERRL generalizes from a 5-agent to an 8-agent Traffic grid and from 4-block to 5-block Blocks World settings, while yielding clauses such as
$$
move(X,Y)\leftarrow top(X),\;on(X,Z),\;isFloor(Y)
$$
after extraction [2304.08349].

Text-based environments have driven analogous developments. FOL-LNN converts observations into first-order logic facts, groups them by type using ConceptNet, and trains an LNN with a DQN-style temporal-difference loss plus a contradiction loss enforcing logical consistency [2110.10963]. On TextWorld Coin-Collector, the reported epoch-100 results include reward $\approx 0.95$ and $\approx 19$ steps on easy games, reward $\approx 0.94$ and $\approx 33$ steps on medium games, and reward $\approx 0.95$ and $\approx 45$ steps on hard games, outperforming the listed symbolic and neural baselines [2110.10963]. NESTA uses a pre-trained AMR parser, converts AMR graphs into first-order facts, and learns differentiable Horn-clause policies with LNNs, reporting on TextWorld-CommonSense that NESTA reaches normalized score $1.00\pm0.00$ on easy and medium out-of-distribution splits and $0.94\pm0.04$ on hard, while using fewer steps than the listed baseline [2307.02689].

A different policy-form perspective appears in S-REINFORCE, which trains a neural policy gradient agent together with a symbolic regressor that fits explicit expressions $\tilde\pi_\phi(a\mid s)=f_\phi(s,a)$ and then reuses the symbolic proxy through importance sampling [2305.07367]. The paper reports symbolic policies such as $\pi(a=\text{left}\mid s)\approx 0.52-2s_2-0.595s_3$ for CartPole and analogous explicit expressions in Acrobot and LunarLanderContinuous [2305.07367]. This suggests that interpretability in Neurosymbolic RL is not limited to discrete rules: it also includes sparse algebraic policies, differentiable decision trees, and logic-constrained networks.

## 4. Symbolic rewards, automata, and temporally extended task structure

A second major line of work uses symbolic structure to define or learn reward functions and long-horizon temporal abstractions. LISR represents intrinsic rewards as symbolic trees of depth $\le 3$, where leaves are components of an observation vector and internal nodes are arithmetic, logic, or gating operators [2010.03694]. The intrinsic reward has the form
$$
R_\phi(s_t,a_t)=T_\phi(o_t),
$$
and the framework optimizes a bilevel objective in which policy parameters $\theta$ maximize return under the symbolic reward while tree structures $\phi$ are evolved to maximize true extrinsic performance [2010.03694]. The reported findings state that LISR (EA + SR) outperforms all baselines in 6/8 continuous and discrete tasks, that even the pure SR learner solves PixelCopter and Catcher using only its discovered symbolic reward, and that discovered trees use $\le 3$ layers and approximately $10$–$30$ operations versus approximately $5$ K parameters in a neural ICM module [2010.03694].

Neural Reward Machines address temporally extended tasks more explicitly. The framework starts from classic symbolic reward machines or LTLf-derived automata, but replaces fixed symbolic transition and reward functions with differentiable neural parameterizations [2408.08677]. In the reported experiments on grid-world “map” and “image” environments with tasks such as Visit, SeqVisit, and GlobalAvoid, NRM + A2C achieves success rate $92\%\pm 4$, cumulative reward $88\pm 5$, and $1200\pm 300$ episodes to $80\%$ success, compared with $60\%\pm 10$, $45\pm 12$, and $3000\pm 800$ for RNN + A2C, nearly matching the RM + A2C upper bound of $95\%\pm 3$, $91\pm 4$, and $1000\pm 200$ [2408.08677]. The same paper reports that training without known $T$ and $r$ degrades sample efficiency by approximately $2\times$, while adding $10\%$ label noise to grounding increases RL episodes by approximately $30\%$ [2408.08677].

The more recent SR-DQN also exploits symbolic temporal or task priors, though in a different form. It transfers partial policies from simple domain instances into harder settings by encoding them as logical rules, then performs online reasoning to bias exploration and rescale Q-values during exploitation [2601.02850]. In OfficeWorld and DoorKey, the paper reports that SR-DQN converges in roughly half the number of environment steps needed by plain DQN and attains approximately $2\times$ higher final return than either DQN or DQN+RewardMachines on larger-scale variants; the table excerpt gives, for example, OfficeWorld DeliverCoffeeAndMail: DQN $0.45\pm0.12$, RM-DQN $0.52\pm0.10$, SR-DQN $0.78\pm0.05$, and DoorKey $16\times16$, 2 keys: DQN $0.05\pm0.03$, RM-DQN $0.08\pm0.04$, SR-DQN $0.32\pm0.06$ [2601.02850].

These approaches differ in where symbolic structure is inserted. In LISR, symbolic programs define intrinsic rewards. In NRMs, symbolic task structure is represented by a latent automaton belief state. In SR-DQN, symbolic knowledge appears as partial policies that guide both exploration and exploitation. The shared theme is that symbolic abstractions are used to reshape the credit-assignment problem rather than merely to explain a learned policy.

## 5. Logic-guided learning, action masking, and formally safe exploration

A substantial part of the literature uses symbolic knowledge to constrain action selection, rank candidate actions, or certify safety during learning. SymDQN augments DuelDQN with Logic Tensor Network modules for shape recognition, reward prediction, and reasoning consistency [2504.02654]. Its ActionReasoner axiom aligns rankings of predicted immediate rewards and learned Q-values,
$$
\forall\,\mathrm{Diag}((r_1,q_1),(r_2,q_2))\,(r_1>r_2):(q_1>q_2),
$$
while an ActionFilter used at test time prunes any action whose predicted reward is lower than $\max_{a'}\hat r(a')-0.5$ [2504.02654]. On the 5×5 grid navigation task, the reported score at 250 epochs is $0.71$ for SymDQN(AF), $0.59$ for SymDQN(AR,AF), $0.53$ for SymDQN(AR), $0.66$ for SymDQN, and $0.64$ for DuelDQN; the corresponding precision table reports $0.00$ negative-reward consumption for SymDQN(AF) and SymDQN(AR,AF) at 250 epochs [2504.02654].

NSAM pushes constraints further by learning symbolic models from high-dimensional states in a minimally supervised way and then masking infeasible actions through propositional preconditions [2602.10598]. Given a learned symbolic model, the mask
$$
m(s,a)=C(\hat{\mathbf m}(s),a)
$$
is used to renormalize the policy,
$$
\pi^+_\theta(a\mid s)=\frac{\pi_\theta(a\mid s)m(s,a)}{\sum_{a'}\pi_\theta(a'\mid s)m(s,a')}.
$$
The paper reports that across all 16 tasks, NSAM converges faster and to higher reward than Rainbow, PPO, PPO-Lagrangian, KCAC, RC-PPO, and PLPG, reducing violation rates from \(>50\)–\(100\%\) down to approximately \(0\)–\(5\%\); in Visual Sudoku 5×5, it reports 0.8 average reward with 2.5% violation versus \(>50\%\) for the best competitor [2602.10598].

Safe exploration under worst-case disturbances is addressed by Revel, which alternates between a restricted symbolic policy class that allows efficient verification and a more expressive neurosymbolic policy class that safely embeds neural controllers [2009.12612]. A neurosymbolic policy has the form
$$
h(s)=
\begin{cases}
f(s), & \text{if }P^\#(s,f(s))\subseteq\phi,\\
g(s), & \text{otherwise,}
\end{cases}
$$
where $g$ is a safe symbolic policy, $f$ is a differentiable neural policy, and $\phi$ is an invariant [2009.12612]. The paper reports zero worst-case safety violations in all runs for Revel across ten continuous-control benchmarks, while Ddpg and/or CPO incur nonzero worst-case safety violations per episode in 8/10 domains [2009.12612].

NUMERLA applies symbolic safety constraints to online meta-reinforcement learning in CARLA driving scenarios [2309.02328]. Its symbolic safety-constraint module maps latent environment modes to safe parameter subsets, and the lookahead update
$$
\theta'=\theta-\alpha\nabla_\theta L(\theta)-\beta\nabla_\theta L_{lookahead}(\theta)
$$
is constrained so that $\theta'\in f_t(z_t)$ [2309.02328]. In Well-Behaved walking and Jaywalking scenarios, the paper reports collision rates of approximately $25$–$72\%$ for RL, approximately $9$–$20\%$ for COLA, and approximately $0$–$0.04\%$ for NUMERLA in the first scenario; and approximately $35$–$44\%$, approximately $15$–$19\%$, and approximately $0$–$0.03\%$ respectively in the second [2309.02328].

These works counter a common misconception that symbolic knowledge in RL is primarily explanatory. In this literature, symbolic modules frequently act as operational control mechanisms: they prune infeasible actions, regularize Q-value orderings, restrict online adaptation, and enable formal verification of exploration policies.

## 6. Interpretability, generalization, evaluation, and unresolved tensions

The literature consistently presents interpretability and generalization as central motivations, but it does so with different operational meanings. For SCoBots, interpretability is “intrinsic interpretability by design,” because decisions flow from object slots to relations to rules [2410.14371]. For NSRL, DERRL, FOL-LNN, and NESTA, interpretability means human-readable Horn clauses, first-order rules, or extracted relational chains [2103.08228; 2304.08349; 2110.10963; 2307.02689]. For S-REINFORCE, it means analytic symbolic expressions over state variables [2305.07367]. For Revel and NUMERLA, interpretability is closely tied to safety certificates, invariants, and symbolic safe sets [2009.12612; 2309.02328]. For INSIGHT, accessibility itself becomes an explicit problem: the framework therefore prompts GPT-4 to generate textual explanations of symbolic policies and individual decisions, while still jointly learning structured states and symbolic policies end-to-end [2403.12451].

Generalization is likewise instantiated in several ways. DERRL reports transfer to different configurations and contexts in Countdown, Blocks World, Gridworld, and Traffic [2304.08349]. FOL-LNN reports reliable solution of unseen larger-level games in TextWorld within 200 epochs when other listed methods fail to generalize [2110.10963]. NESTA reports stronger out-of-distribution performance on unseen text games than deep RL-based baselines, with fewer interactions [2307.02689]. SCoBots were motivated by the claim that deep RL agents rely on shortcut learning and do not generalize to slightly different environments, and the framework was proposed as a fairer object-centric comparison because it learns object-centric representations directly from raw pixel-based states [2410.14371]. Sample efficiency is emphasized in NRMs, SR-DQN, NSAM, and the 2026 sample-efficient neurosymbolic DQN, each using symbolic structure to reduce the effective search space or inject priors [2408.08677; 2601.02850; 2602.10598].

At the same time, the field repeatedly documents limitations. The “Three Pathways” paper states that the relaxation to real-valued, differentiable representations presents a trade-off: the more learnable, the less interpretable [2402.05307]. SCoBots note dependence on unsupervised sampling of all object appearances and reliable optical flow, limited attributes beyond bounding-box location and class, and the possibility of large, overlapping, or conflicting rule sets [2410.14371]. LISR reports that symbolic search is costly, with training approximately \(2\)–\(4\times\) slower than off-the-shelf SAC on tasks with known dense rewards, and that limited operator dictionaries can bias the discovered reward functions [2010.03694]. NSAM highlights the exponential growth of PSDD compilation in the number of propositions and the current restriction to hard propositional constraints [2602.10598]. The 2023 survey identifies automated symbolic knowledge generation, symbol grounding at scale, joint optimization algorithms, and full-policy verification as open research opportunities [2309.01038].

Taken together, these results indicate that Neurosymbolic RL is better viewed as a set of engineering and scientific compromises than as a settled paradigm. The symbolic layer can improve transparency, sample efficiency, temporal abstraction, and safety, but it introduces its own burdens: grounding errors, search complexity, architectural rigidity, and nontrivial mappings from raw signals to predicates. The recurring research question is therefore not whether to combine neural and symbolic methods, but which symbolic abstraction is appropriate for a given RL regime, how tightly it should be coupled to learning, and what form of interpretability or guarantee is actually required.

Source: https://www.emergentmind.com/topics/neurosymbolic-reinforcement-learning