Rule-Based Reinforcement Learning

• Rule-based reinforcement learning is a paradigm where explicit, interpretable rules are integrated into the RL framework to compute rewards, filter actions, or distill policies.
• This approach enhances verification, safety, and generalization by utilizing hand-specified rules, temporal logic, and domain-specific guidelines across various applications.
• Practical implementations span logic puzzles, visual recognition tasks, and safety-critical control, highlighting a balance between deterministic design and adaptive learning.

Searching arXiv for papers on rule-based reinforcement learning to ground the article in published work. Rule-based reinforcement learning denotes a family of reinforcement-learning designs in which explicit rules enter the learning system through deterministic reward computation, action filtering, persistent advice, symbolic intermediate structures, or post-hoc policy distillation. Across the literature, the rules may be hand-specified from task structure, derived from behavioral science, compiled from safety specifications, expressed as temporal logic, or supplied by known physical laws; what unifies the area is that learning is shaped by interpretable, externally specified structure rather than by a learned reward model alone (Wang et al., 29 May 2025, Kou et al., 14 Nov 2025, Nikonova et al., 2022, Bignold et al., 2021).

1. Definitions, scope, and recurring design patterns

The term is used broadly. In some papers, it means that the reward is computed by exact rules over correctness and output format, as in logic puzzles, multimodal math, visual jigsaws, document classification, safety alignment, and grammatical error correction (Xie et al., 20 Feb 2025, Meng et al., 10 Mar 2025, Wang et al., 29 May 2025, Jungo et al., 26 Sep 2025, Mu et al., 2024, Li et al., 26 Aug 2025). In other papers, it means that rules intervene at action selection time, for example by filtering unsafe actions, storing reusable human advice, or providing a deterministic high-level advisor above a learned low-level controller (Nikonova et al., 2022, Bignold et al., 2021, Ramezani et al., 29 Apr 2026). A third usage treats rules as the main symbolic object to be generated, selected, or extracted, rather than as reward terms: natural-language rule bottlenecks for resource allocation, temporal logic rule search inside point-process learning, and post-hoc distillation of black-box RL policies into executable rule lists all fall into that category (Tec et al., 15 Feb 2025, Yang et al., 2023, Coppens et al., 2021). A related but distinct strand studies environments with hidden latent rules that the agent must infer online, rather than rules being injected into the learner directly (Mathew et al., 7 Sep 2025).

Pattern	How rules enter	Representative papers
Reward-level integration	Deterministic or fixed reward terms	(Xie et al., 20 Feb 2025, Wang et al., 29 May 2025, Kou et al., 14 Nov 2025)
Behavior-policy guidance	Action filters, advice, or coaching	(Nikonova et al., 2022, Bignold et al., 2021, Ramezani et al., 29 Apr 2026)
Symbolic bottlenecks	Rules mediate action and explanation	(Tec et al., 15 Feb 2025)
Rule discovery or extraction	RL searches for or distills rules	(Yang et al., 2023, Coppens et al., 2021)
Latent-rule environments	Agent infers hidden governing rules	(Mathew et al., 7 Sep 2025)

This breadth also explains why the area cannot be reduced to symbolic policy learning. RLSLM is explicitly rule-based because a fixed Social Locomotion Model shapes reward, even though the learned policy is neural and stochastic (Kou et al., 14 Nov 2025). Jigsaw-R1 defines rule-based visual RL through algorithmically verifiable rewards over jigsaw correctness and output format, not through symbolic action rules (Wang et al., 29 May 2025). An earlier scientific-computing strand used governing equations, boundary conditions, and initial conditions as the rule-based critic for solving nonlinear ODEs and PDEs, again without a learned reward model (Wei et al., 2018).

2. Deterministic rewards and verifiable feedback

The most visible contemporary form of rule-based RL is post-training with deterministic, machine-checkable rewards. Logic-RL uses procedurally generated Knights-and-Knaves problems, a strict > ...<answer>...</answer> format, a binary format score $S_{\text{format}}\in\{1,-1\}$, and an answer score $S_{\text{answer}}\in\{2,-1.5,-2\}$. On its main logic benchmark, average accuracy rises from 0.19 for Qwen2.5-7B-Instruct-1M to 0.89 after Logic-RL, and the model also shows an overall improvement of 125% on AIME and 38% on AMC (Xie et al., 20 Feb 2025). MM-EUREKA applies the same verifiable-reward logic to multimodal mathematics, using binary accuracy and format rewards with RLOO-style leave-one-out advantages and PPO-style clipping; MM-Eureka-8B reaches an average of 33.0 with 54k filtered samples, compared with 25.5 for SFT on 54k and 30.0 for MPO on 1M, while MM-Eureka-Zero-38B reaches 44.7 using about 9.3K K12 samples (Meng et al., 10 Mar 2025). CPGD is motivated by the observation that GRPO, REINFORCE++, and RLOO can become unstable under such verifiable-reward training, and proposes clipped policy gradient optimization with policy drift as a stabilizing alternative for language-model post-training (Liu et al., 18 May 2025).

Reward-level rule integration is not confined to reasoning benchmarks. RLSLM inserts a fixed Social Locomotion Model into the A2C reward by combining mechanical energy, goal progress, and a socially grounded comfort field; the framework reports a mean comfort rating of 4.21/5 in immersive VR, outperforming the best rule-based baseline by $\Delta \text{rating}=1.12$ with Bonferroni-corrected $P<0.001$ (Kou et al., 14 Nov 2025). RRLL augments a frozen medical predictor with a reinforcement-learning relabeler whose reward depends on both label correctness and whether the chosen transition violates a small set of impossible physiological transitions; FC-Attention + RRLL reaches kappa 0.88 on SHHS sleep staging, and DCRNN + RRLL reaches accuracy $0.981 \pm 0.01$ on CHB-MIT seizure detection (Zhu et al., 31 Jan 2025). RBR for language-model safety composes rule-derived features such as apology, judgment, disallowed content, and illogical continuation into an additive PPO reward, and reports human-evaluation F1 97.1 versus 91.7 for a human-feedback baseline (Mu et al., 2024). Rule-Based RL for Chinese grammatical error correction uses a format reward plus a five-case correctness reward that explicitly favors preserving already-correct input and penalizes overcorrection; its best FCGEC result is 58.74 $F_{0.5}$ with 16-vote inference (Li et al., 26 Aug 2025).

A notable commonality across these systems is that the reward is computable from known task structure. Jigsaw-R1 defines accuracy and format rewards from the known original arrangement of shuffled image patches (Wang et al., 29 May 2025). Logic-RL and MM-EUREKA rely on exact answer verification for logic and multimodal mathematics (Xie et al., 20 Feb 2025, Meng et al., 10 Mar 2025). RBR uses LLM-graded propositions, but the propositions themselves are hand-authored behavioral rules, and the total reward is the sum of a helpful-only reward-model score and a linear rule-based term (Mu et al., 2024). This suggests that the decisive bottleneck is often not RL machinery but whether a domain affords robust symbolic or algorithmic verification.

3. Action filtering, coaching, and human-guided control

A second major lineage uses rules to shape the behavior policy directly. “Don’t do it: Safer Reinforcement Learning With Rule-based Guidance” defines safety rules as conjunctions of qualitative spatial relations and forbidden actions, then modifies $\epsilon$-greedy exploration through Guided Exploration and Full Guidance. The method reduces deaths and collisions during training across Crossroad, Freeway, Frostbite, Ms-Pacman, and Space Invaders, and after 25,000 episodes DQN+FG scores 30.5 versus 27.3 on Freeway and 3820 versus 2780 on Ms-Pacman (Nikonova et al., 2022). The rules function as an action-filtering or override mechanism rather than as a reward term.

Persistent Rule-based Interactive Reinforcement Learning places rules in the human-advice channel. Advice is represented as reusable “IF condition THEN recommendation” rules stored in a ripple-down rules tree, and probabilistic policy reuse starts with an 80% chance of following retained advice and decays by 5% per episode. In Mountain Car, persistent rule-based informative advice achieves similar performance to persistent state-based advice but requires far fewer interactions: FULL needs 2 interactions instead of 254, HALF 3 instead of 227, QUARTER 3 instead of 139, and MIDDLE 3 instead of 45 (Bignold et al., 2021). Here the rules are neither rewards nor hard constraints; they are persistent, generalized policy fragments layered over tabular Q-learning.

Rule-based high-level coaching for UAV search-and-rescue missions extends this pattern to hierarchical control. A fixed high-level advisor compiled from a structured task specification outputs recommended actions, avoided actions, and regime-dependent arbitration weights, while a goal-conditioned low-level controller learns online with prioritized replay augmented by rule-derived metadata. In both battery-aware multi-goal delivery and moving-target delivery, the method improves early safety and sample efficiency primarily by reducing collision terminations while preserving online adaptation (Ramezani et al., 29 Apr 2026).

The power-grid topology-control literature shows a practically important hybrid form. In the L2RPN study, a strong rule-based “Tutor” uses simulation-based action screening, an N-1 strategy, and topology reversion to the original grid; a PPO-based “Senior” is then trained from that rule-structured pipeline. The improved rule-based controller rises from 38.44 to 48.90, statistically indistinguishable from Senior at 49.12, while the RL agent retains a clear computational advantage (Lehna et al., 2023). This makes the paper a case study in how expert rules can define the action set, generate demonstrations, and remain in the runtime loop even when the final controller is partly learned.

4. Rules as symbolic bottlenecks, discovered objects, and extracted policies

Some work makes rules the central symbolic object of optimization. Rule-Bottleneck Reinforcement Learning generates candidate natural-language rules with an LLM, embeds them, selects among them with an attention-based RL policy, and then conditions both the environment action and the explanation on the selected rule. The rule reward is an average of three LLM-judged criteria—whether the rule predicts the action, whether it is state-relevant, and whether the explanation is compatible—and the overall optimization target combines environment reward with this explainability reward (Tec et al., 15 Feb 2025). In this design, rules are not post-hoc rationalizations; they are explicit intermediate decisions.

A different form of rule optimization appears in temporal point processes. “Reinforcement Logic Rule Learning for Temporal Point Processes” alternates between a convex restricted master problem, which reweights the current temporal logic rules, and a combinatorial subproblem, which searches for a new rule that most improves likelihood. The subproblem is handled by a neural search policy trained with a risk-seeking policy gradient, where the reward is the reduced-cost-like gain of the generated rule (Yang et al., 2023). This is not RL for control, but RL for symbolic rule discovery inside a probabilistic temporal model.

Rule extraction works in the opposite direction: instead of using rules to guide RL, it turns a learned RL policy into a symbolic controller. Set-valued inductive rule learning extends CN2 so that each state may be labeled by a set of acceptable near-optimal actions rather than a single greedy action, with $\tau=90\%$ used to define near-optimality. In Mario, the initial extracted policy contains 10 rules, and after refinement the rule policy attains episodic return similar to BDPI while remaining executable and human-readable (Coppens et al., 2021). The contribution is not end-to-end symbolic RL but interpretable distillation informed by RL-specific meta-information such as action probabilities and Q-values.

A boundary case is the GOHR work on hidden-rule environments. There the active rule is not supplied to the learner; it is a latent property of the environment that must be inferred under partial observability. The comparison between Feature-Centric and Object-Centric representations shows that hidden-rule inference is strongly representation-dependent and that training success does not automatically imply abstract rule learning (Mathew et al., 7 Sep 2025). This line broadens the concept of rule-based RL from rule injection to rule-governed environments.

5. Generalization, reasoning, and the controversy over explicit thinking

A recurring empirical theme is that rule-based RL often improves generalization when the reward is verifiable. Jigsaw-R1 trains multimodal LLMs on 2x1 jigsaw puzzles using rule-based accuracy and format rewards, and reports that Qwen2.5-VL-3B rises on 2x1 pair puzzles from 48.50% to 96.80% in thinking mode and from 52.20% to 98.80% in non-thinking mode, while RL generalizes better than SFT and an SFT cold start can hinder later RL (Wang et al., 29 May 2025). In document image classification, GRPO with format and class rewards improves unseen-class generalization dramatically: when trained on 10 seen classes and evaluated on 6 unseen classes with the 6-class prompt, RL scores 78.65 versus 43.23 for SFT, although RL underperforms SFT on cross-modality transfer (Jungo et al., 26 Sep 2025). In Chinese GEC, rule-based RL improves both precision and recall over reasoning-based SFT and is notably more robust on NaCGEC (Li et al., 26 Aug 2025).

The most sustained controversy concerns explicit visible reasoning. In perception-heavy multimodal tasks, explicit chain-of-thought is often unnecessary and may even be behaviorally decoupled from the final answer. Jigsaw-R1 finds that open-source MLLMs often perform better in non-thinking mode, that the reasoning trace can contradict the final answer while the answer is correct, and that complex reasoning patterns appear to be pre-existing rather than emergent (Wang et al., 29 May 2025). “Think or Not Think” reports that No-Thinking-RL usually outperforms thinking-based CLS-RL across 11 image-classification datasets and is far more efficient: on ImageNet, No-Thinking-CLS-RL reaches 92.31 accuracy with 94 min training time, whereas CLS-RL reaches 92.24 with 1587 min (Li et al., 20 Mar 2025). By contrast, MM-EUREKA reports increasing response length and “visual aha moments” on multimodal mathematics, indicating that long-form reasoning can still be useful when answers are reliably machine-checkable and the task genuinely requires multi-step symbolic reasoning over images (Meng et al., 10 Mar 2025).

These results suggest that “reasoning” in rule-based RL is format-sensitive and domain-sensitive rather than uniform. Visual classification and jigsaw solving often reward direct answering (Li et al., 20 Mar 2025, Wang et al., 29 May 2025). Multimodal mathematical reasoning may benefit from extended rollouts and reflection (Meng et al., 10 Mar 2025). Logic-RL, which trains entirely on synthetic logic puzzles, reports gradual emergence of verification, checking, backtracking, and summarization rather than a sudden phase transition (Xie et al., 20 Feb 2025). A plausible implication is that explicit chain-of-thought is not a defining property of rule-based RL; verifiability of the target and alignment between reward and task structure are more fundamental.

6. Limitations, open problems, and enduring research questions

The literature is explicit about its limitations. Verifiability is the enabling condition for many of the strongest results, but it is also the main bottleneck. Jigsaw-R1 emphasizes that many real-world visual tasks are less neatly verifiable than jigsaw puzzles (Wang et al., 29 May 2025). MM-EUREKA excludes proof problems and hard-to-parse multiple-choice problems because robust machine checking is required for reward stability (Meng et al., 10 Mar 2025). RBR argues that explicit rule-based rewards work best when desired behaviors can be separated into easy-to-judge propositions, and is less confident about subjective domains such as essay quality (Mu et al., 2024).

Hand-crafted rules can also be brittle or ambiguous. RLSLM notes a sign ambiguity in the printed reward equation, even though the intended interpretation is to penalize socially uncomfortable trajectories (Kou et al., 14 Nov 2025). Safe $\epsilon$-greedy does not provide complete safety guarantees and can still fail when only short-horizon local rules are available (Nikonova et al., 2022). RRLL depends on the correctness of the impossible-transition set and remains a post-hoc correction layer rather than a jointly optimized predictor (Zhu et al., 31 Jan 2025). The power-grid study shows that strong performance can come from carefully engineered rule systems alone, which cautions against assuming that RL automatically dominates in safety-critical control (Lehna et al., 2023).

Optimization stability remains another central issue. CPGD is motivated by collapse modes in GRPO, REINFORCE++, and RLOO under verifiable rewards, including unstable policy-ratio dynamics and degenerate response-length behavior (Liu et al., 18 May 2025). Hidden-rule environments add a different difficulty: GOHR shows that rule inference can be highly sensitive to representation and that partial observability encourages memorization unless the state encoding matches the latent rule family (Mathew et al., 7 Sep 2025). Even the early rule-based self-learning work on differential equations reports failure in the chaotic Lorenz regime, underscoring that rule-based critics do not automatically solve hard optimization landscapes (Wei et al., 2018).

The broad significance of rule-based reinforcement learning lies in this tension between explicit structure and adaptive learning. Rules can specify what matters—correctness, safety, social comfort, physiological plausibility, minimal editing, mission sequencing, or explainability—with a transparency that learned reward models often lack (Kou et al., 14 Nov 2025, Mu et al., 2024, Li et al., 26 Aug 2025). Reinforcement learning then determines how to optimize behavior under that structure. Whether the rules are rewards, action filters, advice, bottlenecks, extracted surrogates, or latent environmental regularities, the field’s central question remains the same: how much of intelligent behavior can be driven by externally specified, interpretable structure, and where must that structure give way to learned adaptation (Tec et al., 15 Feb 2025, Coppens et al., 2021, Mathew et al., 7 Sep 2025).