Deep Symbolic Reinforcement Learning

• Deep Symbolic Reinforcement Learning (DSRL) is a framework that combines deep network-based learning with symbolic representations, logic, and planning to address data inefficiency and opacity.
• It integrates neural perception with symbolic abstraction through sub-states, rule extraction, and planning modules, enhancing sample efficiency and transferability.
• Empirical studies in domains like Atari, gridworlds, and autonomous driving highlight DSRL’s benefits while also revealing challenges in grounding, expressiveness, and trade-offs between symbolic fidelity and flexibility.

Deep Symbolic Reinforcement Learning (DSRL) denotes a family of reinforcement-learning architectures that combine deep function approximation with symbolic representations, logic, planning models, automata, or explicit rule structures. Across the literature, the common objective is to move policy learning away from purely opaque latent mappings and toward objects, relations, subgoals, symbolic transitions, or symbolic expressions that are more sample-efficient, more transferable, and more interpretable. In its early formulation, DSRL was presented as an end-to-end architecture with a neural back end for grounding symbols from perception and a symbolic front end for reinforcement learning over structured interactions (Garnelo et al., 2016). Subsequent work broadened the design space to include symbolic planning over options, symbolic reward shaping, action shielding and masking, rule extraction, automaton-based transfer, exact symbolic reasoning environments, and differentiable symbolic modules trained jointly with deep RL (Lyu et al., 2018, Ye et al., 16 May 2025, Han et al., 11 Feb 2026).

1. Historical formation and research motivation

The original DSRL program was motivated by three limitations of conventional deep RL: data inefficiency, lack of abstraction, and opacity. The 2016 formulation proposed a system in which a convolutional autoencoder grounded objects and types from raw frames, after which a symbolic frontend learned over object interactions and relative relations rather than over pixels directly (Garnelo et al., 2016). This line of argument remained stable in later work: symbolic structure was repeatedly introduced to improve transfer learning, abstraction or generalization, and interpretability (Garcez et al., 2018).

By 2018, the literature had already split into two distinct but related trajectories. One trajectory retained the original symbolic-state view and investigated how symbolic sub-states should receive credit and how their Q-values should be aggregated. The SRL+CS variant restricted non-zero reward updates to the symbolic sub-state where agent-object interaction actually occurred and weighted action selection by squared distance to nearby objects, reporting that in the hardest deterministic-to-random transfer setting it achieved nearly 100% average accuracy compared to DSRL’s 70% and DQN’s 50% (Garcez et al., 2018). The other trajectory elevated the symbolic component from state abstraction to explicit planning. SDRL introduced a planner–controller–meta-controller architecture in which symbolic plans produced subtasks, deep RL learned low-level execution from high-dimensional sensory inputs, and a meta-controller re-evaluated subtasks through extrinsic reward estimates (Lyu et al., 2018).

Later work preserved the same motivation but diversified the mechanism. Dylan explicitly framed the problem as enabling RL agents to benefit from “human priors” by encoding symbolic subgoals and rules that decompose sparse-reward tasks into meaningful intermediate states, while newer neuro-symbolic systems used symbolic logic to filter actions, learn exact algebraic transformations, distill rule sets from object-centric concepts, or transfer high-level task structure across environments (Ye et al., 16 May 2025, Sharifi et al., 2023, Dabelow et al., 2024, Grandien et al., 2024). This suggests that DSRL is best understood not as a single algorithm, but as a research program organized around symbolic task structure as a complement to deep policy or value learning.

2. Symbolic representations and grounding strategies

A central design choice in DSRL is the symbolic substrate. The earliest systems used object types, persistent identities, and relative positions. In the prototype DSRL architecture, the symbolic state was constructed from detected object positions and types, then enriched with spatio-temporal relations built from frame differences and local relative positions (Garnelo et al., 2016). SRL and SRL+CS further simplified the state to symbolic sub-states of the form

$s^{k} = (x^m - x^n,\; y^m - y^n),$

which encode the relative position between the agent and an object (Garcez et al., 2018).

Other work adopted explicit predicate logic. NSRL assumed a symbolic state represented as a tensor or matrix of predicate truth values and learned attention over predicates and reasoning paths of bounded length, yielding chain-like first-order rules rather than purely numeric policies (Ma et al., 2021). Dylan encoded domain knowledge as definite clauses and transformed them into STRIPS-style symbolic transitions such as

$$\mathtt{move(go\_blue\_key, initial, get\_blue\_key)}.$$

The planner then reasoned over actions, preconditions, and postconditions, using symbolic clauses both as reward-model structure and as a high-level planner (Ye et al., 16 May 2025).

A separate strand used learned abstractions rather than hand-written predicates. GRL represented relational MDP states as sets of true facts and converted them into fixed-size abstract state vectors using automatically generated Description Logic features, thereby supporting zero-shot transfer across instances with different object names and object counts (Karia et al., 2022). The SCoBots instantiation for Atari transformed raw pixel stacks into object-centric representations via SPACE trained with MOC, extracted predefined relational concepts such as distance and speed, and finally distilled PPO policies into explicit IF–THEN rules with ECLAIRE (Grandien et al., 2024).

Symbolic grounding also expanded beyond vision in the narrow sense. NESTA used a generic semantic parser based on Abstract Meaning Representation, converting text observations into symbolic triples and then learning lifted action rules with Logical Neural Networks (Chaudhury et al., 2023). For autonomous emergency braking, a Vector Semantic Representation fused a hyperdimensional symbolic scene descriptor with deep spatial features derived from a ResNet18-UNet semantic segmentation backbone; the symbolic component encoded semantic class, spatial relations, and shape/location through HRR hypervectors (Asodia et al., 4 Feb 2026). In exact symbolic mathematics, the environment itself was symbolic: a deep RL agent manipulated equations with a symbolic stack calculator, and correctness was guaranteed because every action was an exact symbolic operation checked by SymPy (Dabelow et al., 2024).

3. Modes of coupling symbolic structure with deep RL

The symbolic component in DSRL can enter the learning loop at several different points. In the earliest architecture, symbolic structure defined the state space over which tabular Q-learning operated. Separate Q-functions were learned for interactions between object types $i$ and $j$,

$$Q^{ij}(s_{t}^{ij}, a_{t}) \leftarrow Q^{ij}(s_{t}^{ij}, a_{t}) + \alpha \left[ r_{t+1} + \gamma \max_{a}Q^{ij}(s_{t+1}^{ij},a) - Q^{ij}(s_{t}^{ij},a_{t}) \right],$$

and actions were chosen by summing Q-values across symbolic sub-problems (Garnelo et al., 2016). SRL+CS retained this decomposition but changed both reward assignment and action aggregation to privilege the sub-state that actually caused reward and to emphasize nearby objects (Garcez et al., 2018).

A second integration mode is symbolic planning over learned or given skills. SDRL mapped symbolic transitions to options with explicit initiation and termination conditions and used a planner–controller–meta-controller loop to schedule subtasks and replan according to learned plan quality (Lyu et al., 2018). SORL extended this pattern by learning action models, hierarchical task network models, and symbolic options automatically from trajectories, then using planning with the learned symbolic models to guide low-level deep RL (Zhuo et al., 2021). Dylan occupies the same part of the design space but makes the planner differentiable and deploys it in two roles: as a reward model that shapes deep RL rewards and as a high-level planner that composes primitive policies (Ye et al., 16 May 2025).

A third mode uses symbolic reasoning to alter action selection directly. DRLSL computed a safe action set $\mathcal{A}^{safe}_t$ from Prolog rules and restricted $\epsilon$-greedy DQN exploration and exploitation to that set, effectively acting as a symbolic action shield (Sharifi et al., 2023). SR-DQN performed online Answer Set Programming reasoning over partial symbolic policies learned on easier tasks, then used the entailed actions both to bias exploratory sampling and to rescale Q-values during exploitation (Veronese et al., 6 Jan 2026). NSAM learned symbolic grounding from high-dimensional states under domain constraints using PSDDs and then constructed action masks by evaluating symbolic preconditions under the MAP symbolic model, renormalizing PPO’s policy distribution after infeasible actions were removed (Han et al., 11 Feb 2026).

A fourth mode uses symbolic structure as the object being learned. NSRL learned attention over reasoning paths and extracted symbolic chain rules as intrinsic explanations of policy behavior (Ma et al., 2021). ESPL replaced opaque policies with differentiable symbolic expressions trained from scratch by gradient descent in an off-policy SAC-style framework (Guo et al., 2023). LISR searched over symbolic trees that map observations to scalar rewards, then used the discovered intrinsic symbolic reward to supervise a deep policy learner (Sheikh et al., 2020). Automaton distillation transferred task knowledge by compressing teacher Q-values into values on automaton transitions and then mixing those symbolic values into the student’s DQN targets in a different environment (Singireddy et al., 2023).

4. Differentiable symbolic reasoning and symbolic policy learning

Recent DSRL work increasingly attempts to preserve symbolic semantics while making the symbolic module trainable end-to-end. Dylan is a canonical example. It encodes each planning rule $C_i$ as a tensor $\mathbf{I}_i \in \mathbb{N}^{G \times S \times L}$, learns soft rule-selection weights through a matrix $\mathbf{W}$, and performs differentiable forward reasoning with multiplication for conjunction and a temperature-controlled $$\mathtt{move(go\_blue\_key, initial, get\_blue\_key)}.$$0 for disjunction (Ye et al., 16 May 2025). The same symbolic module can then define shaped rewards. In the reward-model setting, the RL objective becomes

$$\mathtt{move(go\_blue\_key, initial, get\_blue\_key)}.$$1

The explicit aim is to provide dense feedback at intermediate symbolic milestones while preserving task semantics.

NSRL adopts a different differentiable route. It represents each predicate as a binary matrix, uses a hierarchical stack of transformers to produce predicate attention and path attention, and composes a soft multi-hop reasoning operator $$\mathtt{move(go\_blue\_key, initial, get\_blue\_key)}.$$2 that feeds a policy module over action predicates (Ma et al., 2021). The extracted rule space is restricted to chain-like paths of bounded length, which yields readable but structurally limited first-order rules such as moving toward a key when the agent is without the key, or toward a door when the key has been acquired.

End-to-end symbolic policy learning goes further by making the policy itself a symbolic expression. ESPL defines a symbolic network whose operators are arithmetic and analytic primitives such as $$\mathtt{move(go\_blue\_key, initial, get\_blue\_key)}.$$3, $$\mathtt{move(go\_blue\_key, initial, get\_blue\_key)}.$$4, $$\mathtt{move(go\_blue\_key, initial, get\_blue\_key)}.$$5, $$\mathtt{move(go\_blue\_key, initial, get\_blue\_key)}.$$6, $$\mathtt{move(go\_blue\_key, initial, get\_blue\_key)}.$$7, $$\mathtt{move(go\_blue\_key, initial, get\_blue\_key)}.$$8, $$\mathtt{move(go\_blue\_key, initial, get\_blue\_key)}.$$9, and $i$0, then uses Bernoulli masks with a Gumbel-Sigmoid relaxation and a straight-through trick to select a compact symbolic subexpression during training (Guo et al., 2023). The deployed controller is therefore a symbolic formula rather than a neural network. LISR makes an analogous move on the reward side: instead of a neural intrinsic reward module, it evolves low-dimensional symbolic trees of bounded depth three using mutation and crossover, and trains deep actors and critics from the resulting symbolic reward signal (Sheikh et al., 2020).

Not all DSRL work seeks differentiable logic over physical environments. In symbolic equation solving, the state, action space, and transition function are already symbolic. A deep feedforward ReLU network approximates $i$1, but the environment uses exact symbolic transformations on equations and stack terms, so failure is possible without hallucination; success corresponds to reaching a formally solved equation (Dabelow et al., 2024). This widens the meaning of DSRL from “deep control with symbolic structure” to “deep RL over exact symbolic reasoning environments.”

5. Empirical domains and reported results

Empirical evaluation in DSRL spans small symbolic games, sparse-reward Atari tasks, gridworld planning benchmarks, autonomous driving, text-based games, constrained combinatorial domains, and symbolic mathematics. In the original proof-of-concept video game, the symbolic agent improved to roughly 70% positive-object collection on randomized layouts, whereas DQN fluctuated around 50%, and grid-trained DQN again stayed near chance when tested on the random variant (Garnelo et al., 2016). SRL+CS later reported nearly 100% average accuracy in the hardest deterministic-training/random-testing transfer condition, against about 70% for DSRL and 50% for DQN (Garcez et al., 2018).

Planning-based DSRL systems reported stronger sample efficiency on long-horizon tasks. On Montezuma’s Revenge, SDRL reached the maximal cumulative reward of 400 at about 1.5M samples, while hDQN reached the same score later, around 2.5M samples (Lyu et al., 2018). SORL reported a larger gain in the same game, reaching the maximum reward of +400 in about 0.7M samples, whereas both SDRL and HRL required more than 1.5M samples (Zhuo et al., 2021). In MiniGrid-DoorKey, Dylan improved learning speed and convergence for both PPO and A2C, with the strongest gains in the $i$2 environment where vanilla PPO failed to converge but PPO + Dylan succeeded; in the multitask setting, Dylan achieved 100% success on key retrieval, red-door reaching, and goal reaching, and 98.2% on safe goal reaching (Ye et al., 16 May 2025).

Safety-oriented systems provide a different empirical profile. In autonomous driving with the highD dataset, DRLSL converged after about 500 episodes, compared with around 970 episodes for vanilla DQN, and recorded 0 collision penalty and 0 off-road penalty during training, whereas DQN incurred about $i$3 collision penalty per episode and about $i$4 off-road penalty (Sharifi et al., 2023). In human-centered emergency braking in CARLA, the $i$5 fusion strategy achieved the highest mean reward in training, around $i$6, and in testing maintained 99% success under both partial and full occlusion in low density, 97%/95% success in medium density, and 97%/95% success in high density, while the full complete-efficiency/complete-safety rule set yielded the strongest overall reward performance (Asodia et al., 4 Feb 2026).

DSRL has also produced strong results in nontraditional RL environments. NESTA achieved 1.00 normalized score on easy in-distribution and easy out-of-distribution text games, 0.98 on medium OOD, and 0.85 on hard OOD, while also outperforming a deep text agent with up to 5× fewer training interactions (Chaudhury et al., 2023). The symbolic equation-solving agent eventually reached about 99.8% success on integer and rational linear equations, 98.9%+ across complex-valued test sets, 86.5%+ simultaneous success across numerical and symbolic test classes, and 94.2%+ across all test sets when adversarial pretraining was followed by fixed-distribution training (Dabelow et al., 2024). In constrained decision problems such as Sudoku, N-Queens, graph coloring, and Visual Sudoku, NSAM achieved near-zero or very low violation rates while matching or exceeding baseline reward, for example 0.1% violation on Sudoku $i$7, 1.1% on N-Queens $i$8, and 0.2%–0.7% across graph-coloring tasks (Han et al., 11 Feb 2026). These results do not establish a single universal advantage, but they do show that symbolic coupling has been empirically effective in domains where sparse reward, long horizons, safety constraints, exactness, or out-of-distribution generalization are central.

6. Limitations, controversies, and open directions

Despite the breadth of reported successes, DSRL remains constrained by several recurring bottlenecks. Symbol grounding is the most persistent one. Many systems assume an oracle or a manually designed mapping from raw states to symbolic predicates, including NSRL and SDRL, and even when grounding is learned from pixels, perception errors can dominate downstream performance (Ma et al., 2021, Lyu et al., 2018, Grandien et al., 2024). The SCoBots results make this explicit: Pong tolerated the learned object representation reasonably well, whereas Boxing exposed error accumulation in the modular pipeline, and Skiing suffered severe object-classification failures (Grandien et al., 2024).

A second limitation is the restricted expressive scope of many symbolic modules. The original DSRL prototype relied on locality assumptions that could induce locally optimal but globally suboptimal policies (Garnelo et al., 2016). SRL+CS acknowledged that symbolic abstraction must be chosen well, that Q-learning over state-action pairs can still overfit to layouts, and that its distance-weighted aggregation rule is heuristic rather than universal (Garcez et al., 2018). NSRL can learn only chain-like rules, not tree-like or junction-like structures, and its rule extraction may still require human inspection (Ma et al., 2021). Even ostensibly interpretable systems can produce rule sets that are too large, too long, or internally conflicting to be readily understood (Grandien et al., 2024).

A third tension concerns how much prior symbolic knowledge should be supplied. Some methods deliberately avoid human-biasing tricks such as supervised pretraining, intermediate rewards, or hand-selected inverse rules in order to study reward-driven symbolic discovery (Dabelow et al., 2024). Others depend centrally on symbolic priors: partial ASP policies in SR-DQN, explicit Prolog safety rules in DRLSL, domain constraints compiled into SDDs or PSDDs in NSAM, or hand-defined relation sets in object-centric Atari agents (Veronese et al., 6 Jan 2026, Sharifi et al., 2023, Han et al., 11 Feb 2026, Grandien et al., 2024). A plausible implication is that current DSRL research is organized around a trade-off between symbolic fidelity and symbolic burden: stronger prior structure can accelerate learning or improve safety, but it may reduce portability and increase the cost of system specification.

Transfer is likewise nontrivial. Automaton distillation showed that symbolic abstractions can bootstrap target learning even when source and target layouts differ, but it also showed that static automaton transfer can become misleading when shortest automaton traces do not correspond to shortest real trajectories, and that negative transfer remains possible (Singireddy et al., 2023). SR-DQN softened the problem by modulating trust in symbolic guidance with $i$9 and an $j$0-decay schedule, explicitly treating symbolic rules as partial and potentially inaccurate (Veronese et al., 6 Jan 2026). This suggests an active research direction toward symbolic modules that are neither hard constraints nor brittle heuristics, but calibrated sources of structure whose influence can be learned or annealed.

The most visible open directions in the current literature are richer learned grounding, more expressive rule spaces, broader multimodal environments, and tighter integration between symbolic reasoning and policy optimization. Recent work already points toward object extraction alternatives such as YOLO, Slot Attention, and CutLER; more informative object properties such as orientation; tree-like rather than chain-like rule forms; 3D environments; and symbolic reward mechanisms that continuously rebalance safety and efficiency according to context (Grandien et al., 2024, Ma et al., 2021, Asodia et al., 4 Feb 2026). In that sense, DSRL remains a moving target: its unifying idea is stable, but the symbolic object of interest may be a state abstraction, a planner, a safety filter, a reward function, an automaton, a policy expression, or an exact symbolic environment.