Interpretable RL Architectures

• Interpretable RL architectures are machine learning frameworks that use explicit symbolic, rule-based, and neuro-symbolic methods to produce transparent decision-making.
• They enable auditability by representing policies with human-readable rule sets, logic programs, and attention maps, balancing performance with clarity.
• These methods enhance data efficiency, robustness to noise, and facilitate formal verification, supporting trustworthy autonomous systems in varied applications.

Interpretable Reinforcement Learning (RL) architectures are machine learning frameworks explicitly designed to make the decision-making processes of RL agents transparent, auditable, and understandable to expert users. Unlike traditional deep RL systems that use opaque, high-capacity neural networks, interpretable RL relies on symbolic, rule-based, fuzzy, modular, or attention-driven methods to produce models where reasoning, structure, and even individual policy decisions can be inspected and analyzed at multiple system levels.

1. Symbolic and Logic-Based Policy Synthesis

A central vein of interpretable RL research constructs policies as explicit symbolic programs, rule sets, or decision lists, often using methods inspired by symbolic AI and logic programming. Notable approaches include:

• Inductive Logic Programming (ILP) for Hierarchical RL: Xu & Fekri (Xu et al., 2021) propose a hierarchical RL architecture comprising (i) a symbolic high-level agent operating over abstract states defined by Boolean predicate valuations, (ii) a set of low-level options (subtasks) targeting subgoal predicates, and (iii) a symbolic transition model $\tilde{T}_H$ learned via differentiable ILP. The transition model predicts symbolic state changes, serving as a fully inspectable "simulator" at the abstract level. The ILP process infers human-readable Horn clauses encoding symbolic preconditions and effects for high-level decision making, with errors triggering clause refinement.
• Programmatically Interpretable RL (PIRL): Verma et al. (Verma et al., 2018) introduce a domain-specific policy language and an iterative synthesis method called Neurally Directed Program Search (NDPS). NDPS searches for compact programmatic policies matching a neural "oracle," allowing for verification and formal analysis, such as smoothness guarantees using SMT solvers.
• Fuzzy Rule-Based and Equation-Based Policies: Schwarz et al. (Hein et al., 2020, Hein et al., 2016) develop model-based population search (particle swarm or genetic programming) over fuzzy rule sets or algebraic formulae. Policies are constrained to low rule counts and/or algebraic trees, producing concise rules (e.g., two Gaussian-antecedent fuzzy rules for CartPole, or a 4-term linear control law) which can be directly audited and deployed.

In these paradigms, interpretability is achieved by (a) formal policy representation using rules, logic, or algebra; (b) explicit complexity regularization (Pareto-optimal trade-offs between performance and policy size); (c) human-readability of the learned models.

2. Hybrid Neuro-Symbolic and Distillation Approaches

Interpretable RL can also be realized by bridging high-performing neural policies and interpretable surrogate models:

• S-REINFORCE (Dutta et al., 2023): This framework co-trains (1) a deep neural policy $\pi_\theta$, and (2) a symbolic regressor $\tilde{\pi}_\mathrm{sym}$ fitted via genetic programming on the current policy's outputs. Periodically, $\tilde{\pi}_\mathrm{sym}$ provides symbolic expressions for action probabilities, which can be used for importance sampling during gradient estimation and as transparent fallback policies.
• Neurally-Guided Differentiable Logic (NUDGE) (Delfosse et al., 2023): NUDGE distills a neural policy oracle into a small set of weighted logic rules via a beam search guided by neural policy agreement, then fine-tunes rule weights using a differentiable forward-reasoning network. The outcome is a compact symbolic policy with gradient-based local feature attributions and direct human-readability.
• BASIL (Shahnazari et al., 31 May 2025): BASIL evolves compact, rule-based policies online using quality-diversity evolutionary search, synthesizing ordered lists of symbolic predicates that map directly to actions. Complexity constraints enforce interpretability, and the QD archive provides diverse, performant, transparent controllers.

These systems enable a trade-off between neural-network performance and the clarity of symbolic representations, providing pathways for policy audit, explanation, and modification.

3. Interpretable Model Learning and Feature Attribution

Several works convert black-box RL policies into interpretable forms using feature attribution and model explanation techniques:

• Shapley-Value-Based Model Extraction (Li et al., 16 Jan 2025, Qian et al., 22 Oct 2025): The "SILVER" method computes global Shapley value attributions for each state in the latent feature space of a trained RL agent. By clustering Shapley vectors and projecting decision boundaries back into state space, simple surrogate models (e.g., linear regressions, decision trees, logistic regressors) are fit to approximate the policy selections. SILVER's RL-guided extension (Qian et al., 22 Oct 2025) adds an RL-policy labeling mechanism and scales to high-dimensional, multi-action environments, verifying surrogate fidelity both quantitatively (agreement scores) and via human-subject studies.
• SMART Feature Engineering (Bouadi et al., 3 Oct 2024): For RL-driven feature generation, interpretability is enforced both via semantic knowledge graphs (Description Logics and SWRL rules) and by constraining the RL action space to semantic transformations. The resulting Decomposition Graph provides a provenance-based explanation for each engineered feature.

These model-level extraction approaches produce policies or feature sets that can be directly interpreted in terms of original state variables or domain concepts.

4. Modular and Queryable Architectures for Transparent Reasoning

Beyond direct policy transparency, some RL architectures deliver interpretability by modularizing learning and exposing internal knowledge:

• Query-Conditioned Deterministic Inference Networks (QDIN) (Zakershahrak, 11 Nov 2025): QDIN decomposes an RL agent into query-specialized modules—policy, reachability, path, and comparison functions—each architected for a different class of logical or control question. Query inputs are fused with shared state representations through cross-attention, and the architecture is optimized for joint answerability of diverse queries. Experiments reveal that inference modules can achieve near-perfect environmental models independent of control performance, exposing a fundamental decoupling between world modeling and optimal policy.
• Brain-Inspired Modular RL (Fang et al., 2023): This modular design employs an encoder (visual cortex analog), predictive model (hippocampal analog), and value network (striatum analog), each equipped with auxiliary objectives guiding the emergence of clearly separated function and representation. Quantitative analysis confirms that auxiliary predictive modeling drives both interpretability and robust transfer properties.

Such systems frame the agent as an interpretable knowledge engine, enabling introspection, formal verification, and compositional query-answering.

5. Interpretable Representations via Object, Relation, or Attention Bottlenecks

A distinct strategy inserts explicit interpretable bottlenecks in the perception-to-action pathway:

• Neurosymbolic Agents and Object-Centric RL (Grandien et al., 18 Oct 2024): The SCoBots framework extracts object representations via a SPACE+MOC module, computes symbolic relational vectors, and distills learned policies into rule sets (e.g., IF-THEN relational logic), yielding full transparency from pixels to actions. Each module is independently evaluated for both detection fidelity and conceptual clarity.
• SSINet: Self-Supervised Interpretable Networks (Shi et al., 2020): SSINet attaches a U-Net style mask decoder to a frozen RL agent, training the decoder to produce sparse, pixel-level masks that retain only the information supporting the agent's (already learned) decision. The resulting masks are both human-auditable and strongly predictive of task transfer robustness.
• Attention and Vision Transformer Architectures (George, 2023, Pandey, 6 May 2025): Soft and self-attention layers (e.g., Mott-style spatial attention, Transformer-XL, Vision Transformers) yield interpretable attention overlays. These overlays and spatiotemporal saliency maps (also quantifiable via entropy- and attention-shift metrics) identify where and when agents focus for decision making; they also track perceptual adaptation and abstraction in the presence of intrinsic motivation or generalization pressures.

By constraining or exposing intermediate object, relation, or focus features, these architectures reveal the causal structure of agent perception and control.

6. Quantitative Interpretability Metrics and Evaluation

Interpretable RL approaches report and optimize explicit interpretability metrics, including:

• Structural Complexity: Rule count, predicate count, or algebraic tree depth (as in BASIL, FPSRL, GPRL), Pareto-frontier selection for transparency-performance trade-off (Shahnazari et al., 31 May 2025, Hein et al., 2020).
• Shapley-Vector Boundary Simplicity: Linear boundary dimensionality, decision tree depth, and logistic regression coefficient sparsity; fidelity scores measuring surrogate vs. original policy agreement (Li et al., 16 Jan 2025, Qian et al., 22 Oct 2025).
• Saliency and Mask Metrics: Feature-overlap rate (FOR), Background-elimination rate (BER), spatial/temporal attention diversity; correlation with test-time returns under domain shifts (Shi et al., 2020, Pandey, 6 May 2025).
• Formal Verification Properties: Action-boundedness, smoothness, logical consistency with safety constraints (SMT-based verification for programs (Verma et al., 2018)).
• Human-Subject Studies: Policy comprehension accuracy, response time, and trust via Likert scales (Qian et al., 22 Oct 2025).

These metrics facilitate comparative, domain-agnostic assessment of interpretability and support deployment in regulated or safety-critical settings.

7. Practical Significance, Limitations, and Future Directions

Interpretable RL architectures have demonstrated:

• Data Efficiency and Generalization: Symbolic and modular architectures attain 30–40% reduction in environment steps on structured tasks, and generalize better under distribution shifts (Xu et al., 2021, Mu et al., 2022).
• Robustness to Input Noise and Task Variation: Rule-mining and modular methods outperform black-box RL models on noisy graphs and out-of-distribution settings (Mu et al., 2022, Fang et al., 2023).
• Auditability and Correctability: Symbolic policies can be verified, debugged, or edited directly by domain experts, a property unavailable to end-to-end neural architectures (Xu et al., 2021, Shahnazari et al., 31 May 2025, Verma et al., 2018).

Current limitations include challenges in scaling symbolic and programmatic methods to high-dimensional sensory domains, extension to continuous/control tasks, and automating the selection of policy language or rule sets. Promising research trajectories target: hybrid neuro-symbolic models, modular oracle architectures (QDIN), automated abstraction discovery, improved interpretability metrics, and further integration with formal methods and human-in-the-loop supervisory workflows (Zakershahrak, 11 Nov 2025, Grandien et al., 18 Oct 2024).

Taken together, interpretable RL architectures provide a rigorous, multifaceted roadmap for transparent, trustworthy, and high-performing autonomous decision-making systems.