Model-Reference Reinforcement Learning (MRRL)

• Model-Reference Reinforcement Learning (MRRL) integrates MRAC with RL to improve control performance in uncertain environments.
• MRRL utilizes adaptive control to maintain reference tracking while RL optimizes actions, enhancing stability and sample efficiency.
• Applications include robotics, autonomous vehicles, and aerospace, benefiting from MRRL's robust performance under real-world discrepancies.

Model-Reference Reinforcement Learning (MRRL) is a class of control algorithms that synergistically integrate model-reference adaptive control (MRAC) and reinforcement learning (RL), targeting robust performance under model uncertainties by explicitly closing the gap between simulation-trained policies and real-world system dynamics. The core conceptual structure involves using a nominal or reference model to define desired system trajectories or behavior—potentially realized in simulation—and then employing both an adaptive controller and an RL agent to ensure the real plant replicates this reference behavior even in the presence of unmodeled dynamics and parameter variations (Guha et al., 2020, Zhang et al., 2020, Zhang et al., 2020, Abouheaf et al., 2023). MRRL achieves stability, adaptability, and improved sample efficiency by combining the structure and guarantees of traditional control with the flexibility of modern RL techniques.

1. Theoretical Foundations and Terminology

The MRRL paradigm operates on the canonical model following problem, where the true system dynamics are of the form

$$\dot{x}(t) = f\bigl(x(t), u(t), \phi\bigr),\quad x\in\mathbb{R}^n,\, u\in\mathbb{R}^m$$

with unknown or uncertain parameters $\phi$. A reference model with nominal parameters $\phi_r$ generates trajectories via

$$\dot{x}_r(t) = f_r\bigl(x_r(t), u_r(t), \phi_r\bigr)$$

or (in discrete time) $x_r(k+1) = x_r(k) + \Delta f_r(x_r, u_r, \phi_r)$, establishing a performance benchmark for the plant (Guha et al., 2020).

The associated control objective is to find a policy $u(t) = \pi(x(t))$ minimizing a cost, such as

$J = \int_0^T c\bigl(x(t), u(t), t\bigr)\,dt$

where $c$ encapsulates both reference tracking and control effort.

MRRL employs adaptive control (typically MRAC) in an inner loop to drive the plant state $x$ to track the reference $x_r$, and an RL-based policy (outer loop) to set high-level actions or reference inputs. The composite control architecture typically assumes the plant is subject to disturbances and uncertainties not modeled in the simulation environment, necessitating robust online adaptation (Guha et al., 2020, Abouheaf et al., 2023).

2. Core Methodologies and Algorithmic Structures

MRRL implementations follow a modular inner–outer control architecture. Two principal algorithmic families arise in the literature:

Inner-outer Loop MRAC-RL

The MRAC-RL framework (Guha et al., 2020) deploys:

• Inner-loop adaptive controller: Synthesized via MRAC theory, it dynamically identifies parametric variations and forces the plant state $\phi$0 to track the simulated reference state $\phi$1. For linear systems, parameter adaptation laws are derived using Lyapunov methods:

$\phi$2

with $\phi$3, $\phi$4, where $\phi$5 and $\phi$6 is an augmented input.

• Outer-loop reinforcement learning policy: The RL agent (e.g., PPO/SAC/DDPG) is trained in simulation and outputs $\phi$7. The MRAC compensates for modeling error at a higher control rate, enabling closed-loop stability and robust tracking even under significant parametric deviations.

Composite Law on Baseline Controller

Several works (Zhang et al., 2020, Zhang et al., 2020) implement MRRL via superposition: $\phi$8 where $\phi$9 is a baseline controller designed for a nominal system (via backstepping, linear feedback, etc.); $\phi_r$0 is a learned compensator realized with RL (e.g., soft-actor-critic). The RL agent observes a state vector $\phi_r$1 including plant state, reference state, and baseline action, and outputs corrections accounting for modeling uncertainties, disturbances, or collision avoidance requirements.

Distinct data-driven approaches (Abouheaf et al., 2023) directly solve the Hamilton-Jacobi-Bellman equation in a model-free setting using integral temporal-difference (ITD) learning, leveraging only reference outputs and plant measurements, decoupling controller design from explicit dynamics identification.

3. Stability and Performance Guarantees

A principal advantage of MRRL over standalone RL is the ability to guarantee closed-loop stability using Lyapunov-based constructions inherited from the model-based baseline (Guha et al., 2020, Zhang et al., 2020, Zhang et al., 2020).

• The MRAC/RL composite is constructed so that the tracking error $\phi_r$2 converges to zero asymptotically, even when the policy $\phi_r$3 is trained offline in inaccurate simulators.
• In MRRL with baseline controllers, stability is maintained by ensuring each RL policy update is Lyapunov-admissible, with theoretical results demonstrating negative definite drift outside small error bounds. For instance, if

$\phi_r$4

then $\phi_r$5 corrections do not destabilize the closed-loop system (Zhang et al., 2020).

• The entropy-augmented Bellman operator in the RL component is contractive, and monotonic policy improvement can be established (see Theorems and Lemmas in (Zhang et al., 2020)).
• Data-driven ITD-based MRRL (Abouheaf et al., 2023) provides Lyapunov-based stability even in the absence of explicit model knowledge.

4. Empirical Results and Comparative Analyses

Representative works validate MRRL on diverse benchmark tasks:

• Inverted pendulum tracking under severe parameter mismatches: MRAC-RL maintains plant–reference trajectory tracking where direct deployment of simulation-trained RL results in large errors; the inner-loop adaptation yields dramatic reductions in $\phi_r$6 (e.g., from 78 to 1.67 in linear models) without policy retraining (Guha et al., 2020).
• Autonomous surface vehicles (ASV) with dynamic uncertainties and obstacles: MRRL consistently achieves smaller position error and avoids collisions where pure RL or baseline-only controls fail. Sample efficiency is improved by a factor of two or more due to leveraging nominal model structure (Zhang et al., 2020, Zhang et al., 2020).
• Aircraft control with unknown/dynamic parameter variations: Data-driven MRRL achieves tight output tracking (tracking error decays to zero in $\phi_r$7 s), robust convergence of control policies, and resilience against unmodeled disturbances without recourse to system identification (Abouheaf et al., 2023).

Key comparative findings are summarized below:

Approach	Tracking Error	Collision Avoidance	Sample Efficiency	Stability Guarantee
Model-based baseline only	Finite, large	Not addressed	High	Yes
Pure RL	High/unstable	Fails in real-world	Low/slow	No (typically)
MRRL (MRAC-RL, SAC+baseline)	Low	Achieved	High (x2-x10 faster)	Yes (Lyapunov-based)

All MRRL schemes reported in the cited literature maintained smooth, non-oscillatory control inputs throughout test episodes, demonstrating that residual RL corrections did not degrade actuation quality (Zhang et al., 2020, Zhang et al., 2020).

5. Practical Design Considerations and Tuning

Successful MRRL implementation depends crucially on selecting appropriate model-based architectures and RL hyperparameters:

• Adaptation Gains: In MRAC-RL, $\phi_r$8, $\phi_r$9 must be tuned to balance adaptation speed against sensitivity to measurement noise (Guha et al., 2020).
• Lyapunov Parameters: Diagonal matrices $$\dot{x}_r(t) = f_r\bigl(x_r(t), u_r(t), \phi_r\bigr)$$0, $$\dot{x}_r(t) = f_r\bigl(x_r(t), u_r(t), \phi_r\bigr)$$1, and solution of $$\dot{x}_r(t) = f_r\bigl(x_r(t), u_r(t), \phi_r\bigr)$$2 are chosen to ensure $$\dot{x}_r(t) = f_r\bigl(x_r(t), u_r(t), \phi_r\bigr)$$3 is well-damped.
• Loop Rates: Inner-loop adaptive updates are typically executed at rates $$\dot{x}_r(t) = f_r\bigl(x_r(t), u_r(t), \phi_r\bigr)$$4 higher than outer-loop RL policy updates, to ensure fast compensation for plant–model mismatch.
• RL Hyperparameters: Optimizer settings, value function network architectures, and experience replay buffer sizes directly affect sample efficiency and convergence (Zhang et al., 2020, Zhang et al., 2020).
• Window Length in ITD Methods: Length of error history in the ITD-based approach trades off computational burden versus convergence properties (Abouheaf et al., 2023).

Addressing practical limitations, full data-driven MRRL requires sufficient history storage and appropriate window sizes, and the quadratic Lyapunov form may be conservative for highly nonlinear systems (Abouheaf et al., 2023). Extension to multi-output or highly nonlinear plants might require richer function approximators and further theoretical developments.

6. Broader Impact and Research Outlook

MRRL offers a structured avenue for robust real-world control under significant uncertainties by merging classical adaptive control with the generality of RL. Its strengths include:

• Stability and performance preservation under sim-to-real mismatch and unmodeled dynamics.
• Enhanced sample efficiency due to leveraging nominal models and baseline stabilizers.
• General applicability to a wide range of linear and nonlinear plants.
• Realization of both parametric adaptation and task-level intelligence, such as obstacle avoidance and disturbance rejection.

Challenges remain, including generalization to highly nonlinear/high-dimensional dynamics, minimization of computational overhead, and theoretical characterization of performance levels in deep RL settings. Active research focuses on data-efficient implementations, improved integration of model-based and model-free learning, and rigorous guarantees for complex, safety-critical systems (Guha et al., 2020, Zhang et al., 2020, Abouheaf et al., 2023, Zhang et al., 2020).