Knowledge-Informed Residual Reinforcement Learning

• Knowledge-informed model-based residual RL integrates expert-provided logical rules with empirical transition models to rapidly adapt and enhance decision making.
• It leverages a dynamic reasoning engine that fuses declarative knowledge with probabilistic learning, resulting in improved sample efficiency and robust performance.
• Experimental validations in robotics, dialog management, and delivery tasks demonstrate significant gains in task success rates and reduced uncertainties.

Knowledge-informed model-based residual reinforcement learning is a class of methods that explicitly injects external, often symbolic or human-understandable, knowledge into the learning and decision process of reinforcement learning (RL) agents, while simultaneously leveraging data-driven, model-based RL and the concept of residual learning. In these frameworks, domain knowledge is represented via logic-based or expert-crafted models, which are combined with empirical models and residual policies learned from interaction data. This approach aims to harness the robustness, data efficiency, and interpretability of knowledge-based modeling with the adaptivity and optimality of RL, enabling fast adaptation, generalization to new tasks, and improved performance in environments characterized by partial knowledge, uncertainty, or complex dynamics.

1. Integration of Knowledge Representation and Reasoning with Model-Based RL

A defining principle of knowledge-informed model-based residual RL is the unification of declarative, often logical-probabilistic, knowledge with stochastic world dynamics learned through RL. Representations such as P-log (a probabilistic extension of logic programming) can encode domain rules, relationships between variables (e.g., temporal context, spatial arrangement), and exogenous influences. Simultaneously, the model-based RL component—exemplified by algorithms such as R-Max—extracts empirical transition probabilities and rewards through environmental interaction.

These components are merged within a reasoning engine that–given a particular task–constructs a task-specific Markov Decision Process (MDP) by dynamically fusing logical constraints/rules and learned probabilistic transition models. This process can be formalized as:

$$\mathcal{M}(v' | a, v) \leftarrow \text{Reason with }\Pi^L\text{ (logical rules) and }\Pi^P\text{ (probabilistic rules)}$$

Here, $\Pi^L$ denotes expert-provided symbolic rules, while $\Pi^P$ denotes learned probabilistic rules. The reasoning process dynamically instantiates the transition model $\mathcal{M}$ by accounting for both rigorously encoded prior knowledge and empirically learned uncertainties (Lu et al., 2018).

2. Mechanism of Knowledge Fusion and Task-Specific Model Construction

In this framework, knowledge is categorized into two main sources:

• Human-provided declarative rules: These define variable segmentations (endogenous vs. exogenous), context-sensitive transition restrictions, or high-level planning guidelines.
• Empirically learned transition and reward models: Captured via model-based RL, these describe dynamics such as navigation success rates, action costs, or stochastic effects of environmental changes.

At run-time, when a new task $\tau$ is specified, guidance functions $f_V(\mathcal{V},\tau)$ and $f_A(\tau)$—engineered by domain experts—determine which variables and actions are relevant. The reasoning system then composes a partial world model by combining endogenous variables (task-relevant, to be actively controlled) and exogenous variables (contextual, to be reasoned over) along with learned probabilities.

In structured tasks such as delivery where outcomes depend on dialog, navigation, and external context, the system can formalize transition probabilities for complex composite actions:

$$T^D(s^r, a^d, s^t) = \begin{cases} P^N(s^p, s^g) & \text{if alignment condition holds} \ P^N(s^p, s^m) \times P^N(s^m, s^p) \times P^N(s^p, s^g) & \text{otherwise} \end{cases}$$

This explicit knowledge fusion allows transfer of empirical skills (e.g., navigation in one context) to higher-level tasks (e.g., delivery that requires both navigation and dialog) (Lu et al., 2018).

3. Experimental Systems: Robotic Dialog, Navigation, and Delivery

Knowledge-informed model-based residual RL has been validated on integrated service robot platforms handling multiple high-level tasks:

• Dialog management: Deployed as partially observable MDPs (POMDPs); the robot uses dialog actions to resolve ambiguities in user requests under uncertainty.
• Navigation: Executed in occupancy-grid-based environments (30×30, 50×50 cells), including stochastic obstacles and dynamic blocks (e.g., human interference).
• Delivery: Spans multimodal tasks combining dialog (for request resolution) and navigation (for task execution).

Metrics for evaluation include total reward, fulfiLLMent rates, question-asking (QA) costs, and navigation success, with direct comparison between systems with and without knowledge-informed model construction. For example, in delivery tasks, knowledge extraction yielded higher rewards and fulfiLLMent, and lower QA costs under varying blocking rates (e.g., reward of 206.21 vs. 182.07 at a blocking rate of 0.1) (Lu et al., 2018).

4. Empirical Impact and Comparative Performance

Empirical results demonstrate several core benefits of this approach:

• Sample Efficiency and Convergence: Integrating prior knowledge with residual learning accelerates convergence. In navigation, systems leveraging knowledge extraction converge more rapidly than those solely reliant on model-based RL.
• Generalization and Robustness: The approach shows strong generalization, for instance achieving an 83.8% navigation success rate in unfamiliar conditions, compared to 26.8% for a baseline policy-selection agent.
• Composite Task Performance: In tasks where skills are transferred across domains (e.g., navigation knowledge used for delivery actions), higher-level task performance is consistently improved. The combination of declarative rules and empirical learning allows for the emergence of adaptive, context-sensitive behaviors.

Experimental results are statistically rigorous, with significance established (e.g., p-values < 0.05) (Lu et al., 2018).

5. Application Domains

The hybridization of knowledge-informed reasoning and model-based RL is broadly applicable, with demonstrated and plausible relevance in:

• Robotics: Autonomous indoor service robots operating in dynamic human environments, requiring fast adaptation and robust task execution.
• Human–Robot Interaction: Agents adapting dialog strategies based on uncertainties in speech recognition and environmental context.
• Autonomous Delivery: Systems needing both robust navigation and adaptive task planning under partial observability.
• General Symbolic–Subsymbolic Systems: Scenarios involving high-level symbolic planning over uncertain, stochastic, or only partially modeled physical domains (e.g., smart homes, industrial automation) (Lu et al., 2018).

6. Scalability, Limitations, and Open Challenges

Current challenges and limitations include:

• Scalability: The system must balance state space completeness (capturing all relevant exogenous variables) with computational tractability. Excessively rich exogenous modeling may become intractable.
• Curriculum Learning and Transfer: Determining the optimal sequence of source tasks to maximize transfer for a target task is nontrivial and remains an open research area.
• Exploration vs. Exploitation: The architecture focuses on safe, efficient task execution rather than long-term exploration, and new algorithms are needed to optimize for both.
• Generalization to More Complex Domains: The primary demonstrations are in indoor office settings. Scaling to highly dynamic, open-world environments with richer forms of prior knowledge is an open direction (Lu et al., 2018).

7. Significance and Prospects

Knowledge-informed model-based residual RL provides a template for integrating expressive, interpretable domain knowledge with statistically learned dynamics; empirical evidence points to faster learning, improved generalization, and robustness to ambiguity or environmental change. The formulation supports online adaptation and cross-task skill transfer, making it attractive for long-lived, autonomous agents in real-world domains. Ongoing research aims to enhance scalability, expand application reach, and systematically balance the integration of declarative knowledge and empirical adaptation.