AMLAS-RL: Assurance Framework for Safe RL

• AMLAS-RL is a safety assurance framework that adapts AMLAS to meet the unique challenges of reinforcement learning within cyber-physical systems.
• It employs a six-stage lifecycle integrating data management, simulation testing, model verification, and runtime monitoring for comprehensive safety assurance.
• The framework addresses RL-specific issues such as trajectory-based safety constraints, reward hacking, and the simulation-reality gap through iterative evidence generation.

Searching arXiv for the cited paper to ground the article in current literature. AMLAS-RL is a framework for assuring the safety of reinforcement learning components in cyber-physical systems by adapting AMLAS, the “Assurance of Machine Learning in Autonomous Systems” methodology, to the distinctive requirements of RL. It is presented as a repeatable, end-to-end assurance framework integrated into the RL development lifecycle, with emphasis on allocation and decomposition of system-level safety requirements, data and training planning for state-action coverage, model learning with safe-RL adaptations, evidence generation through simulation tests, formal model checking, and statistical validation, and synthesis of a Goal Structuring Notation-style safety argument (Imrie et al., 8 Jul 2025).

1. Rationale for adapting AMLAS to reinforcement learning

AMLAS is described as a structured, stage-wise methodology for developing safety assurance arguments and collecting supporting evidence for ML components, but it was originally focused on supervised learning. AMLAS-RL arises from the claim that RL poses distinct assurance problems: sequential decision-making under uncertainty formalized as an MDP, an interactive and dynamic environment in which policies evolve through trial-and-error, the requirement that safety be guaranteed during training and exploration as well as deployment, and exposure to reward hacking and the simulation-reality gap (Imrie et al., 8 Jul 2025).

The adaptation therefore extends AMLAS in four directions. First, it requires specification of safety constraints over entire trajectories rather than only pointwise predictions. Second, it incorporates safe-RL techniques such as reward shaping, constrained Markov decision processes, and shielding. Third, it introduces formal verification over abstracted Markov models. Fourth, it makes the assurance process iterative, with feedback between system safety analysis and RL development. A plausible implication is that AMLAS-RL treats safety assurance not as a final certification step but as a lifecycle property of RL-enabled CPS.

The stated scope is the assurance of an RL controller’s safety properties in CPS. The methodology is oriented toward the derivation of RL-specific safety requirements from system-level hazards, the production of evidence that the learned controller satisfies those requirements, and the construction of an explicit assurance case. In that sense, AMLAS-RL addresses a common misconception in safe-RL practice: incorporating a safety mechanism into learning does not by itself provide systematic assurance across the RL lifecycle.

2. Six-stage lifecycle and assurance artefacts

AMLAS-RL comprises six iterative stages. At each stage, specific artefacts are produced and used by distinct roles, including system safety engineer, ML safety engineer, RL engineer, RL developer, test engineer, verification engineer, independent verification engineer, system integrator, and safety engineer (Imrie et al., 8 Jul 2025).

Stage	Principal activities	Outputs
1. RL Safety Assurance Scoping	Allocate system-level safety requirements from HAZOP/HAZID, environment, architecture, and high-level RL description to the RL component	E. Safety requirements allocated to RL component
2. RL Safety Requirements Assurance	Translate allocated system safety requirements into RL-specific safety requirements that can be measured or enforced	H. RL safety requirements
3. Data Management Assurance	Define data requirements for state-action exploration; produce training, internal test, and verification plans	N. Training plan; O. Internal test plan; P. Verification plan
4. Model Learning Assurance	Develop RL models, train according to plan, conduct internal testing, and select the best model compliant with H	U. RL model development log; V. Trained RL model; X. Internal test results
5. Model Verification Assurance	Execute verification plan through statistical trials and formal model checking on an abstract DTMC	Z. RL verification results; AA. Verification log
6. Model Deployment Assurance	Integrate the RL model into the CPS, run integration tests under operational scenarios, and apply runtime monitors or shields if necessary	DD. Erroneous behaviour log; FF. Integration testing results

The framework is organized around artefact traceability. Stage 1 yields the allocated safety requirements $E$; Stage 2 turns these into RL safety requirements $H$; Stage 3 prepares the training, testing, and verification basis through $N$, $O$, and $P$; Stage 4 documents development in $U$, produces the trained model $V$, and collects internal results $X$; Stage 5 produces independent verification evidence $Z$ and $AA$; Stage 6 records deployment behaviour in $H$0 and $H$1. This suggests that AMLAS-RL is designed to preserve a chain of evidence from hazard analysis to deployment monitoring.

3. Assurance argument and evidence structure

AMLAS-RL uses a GSN-style claim-argument-evidence structure. Its top-level claim is stated as: “RL component is acceptably safe under all defined operational scenarios.” That claim is decomposed into five subclaims covering requirements derivation, adequacy of training and simulation data, internal model satisfaction of safety constraints, independent verification, and preservation of safety during integration into the full CPS (Imrie et al., 8 Jul 2025).

The first subclaim, $H$2, asserts that RL safety requirements $H$3 are soundly derived from system requirements $H$4. The corresponding argument, Arg1, relies on a traceability matrix $H$5 and a requirements specification document. The second subclaim, $H$6, asserts that training data and simulations suffice to explore the critical state-action space. Arg2 is supported by data requirement justification and by evidence $H$7 together with training plan $H$8.

The third subclaim, $H$9, concerns satisfaction of safety constraints under internal tests. Arg3 is tied to internal test results $N$0 showing three thresholded properties: $N$1, $N$2, and $N$3. The fourth subclaim, $N$4, states that independent verification confirms safety under general and targeted tests plus formal proofs; this is divided into statistical verification $N$5 and formal model checking results. The fifth subclaim, $N$6, concerns deployment, with Arg5 grounded in integration test results $N$7 and erroneous behaviour log $N$8.

This structure is significant because it separates development evidence from independent verification evidence and from deployment evidence. A plausible implication is that AMLAS-RL aims to avoid conflating successful training performance with an assurance case.

4. Formalization of RL, safety constraints, and verification targets

The framework adopts the standard MDP tuple $N$9, with transition $O$0 and reward $O$1, and policy $O$2. It also states the usual value functions: $O$3

$O$4

For safe-RL, it introduces a constrained MDP formulation that minimizes $O$5 subject to $O$6 (Imrie et al., 8 Jul 2025).

The running example is governed by three system-level safety constraints. $O$7 requires reaching the goal before energy depletion with probability at least $O$8: $O$9 $P$0 requires total time in the unsafe zone $P$1 timesteps. $P$2 requires collision probability $P$3.

For abstraction, the paper defines distance

$P$4

The abstract state is

$P$5

where $P$6 is coded as $P$7 for travelling safely, $P$8 for in unsafe zone, $P$9 for collision, and $U$0 for goal reached. The episode is terminal if $U$1 or $U$2.

Formal verification is expressed in PCTL. Two stated properties are

$U$3

and

$U$4

The paper also includes a property $U$5 associated with unsafe occupancy. Taken together, these definitions anchor the assurance argument in explicit probabilistic and temporal constraints rather than informal safety desiderata.

5. Evidence generation mechanisms

AMLAS-RL combines safe-RL techniques, simulation-based testing, formal verification, statistical validation, and runtime monitoring. The safe-RL techniques explicitly named are reward shaping with penalty cost $U$6 for unsafe states or collisions, CMDP formulation with a separate cost function, and shielding to override unsafe actions at runtime (Imrie et al., 8 Jul 2025).

Simulation-based testing uses Safety-Gymnasium to emulate $U$7-ray LIDAR sensors, obstacles, and unsafe zones. The training protocol specified for the running example uses 500 episodes with randomized initial placements, while tracking reward, successes, collisions, and unsafe time. Internal testing uses 1000 trials to obtain empirical estimates for $U$8, $U$9, and $V$0.

Formal verification proceeds by abstracting the learned controller and environment interaction to a discrete-time Markov chain via state aggregation, extracting transition probabilities from 5000 simulation runs, and applying model checking with PRISM against the PCTL properties. Statistical validation estimates collision probability $V$1 over $V$2 trials, constructs a 95% confidence interval, and compares the interval bounds against the $V$3 threshold.

Runtime monitoring is represented by the erroneous behaviour log $V$4, which collects any sequence of state-action pairs leading to violations of the safety requirements during deployment. The paper states that this enables post-hoc analysis for reward hacking or out-of-distribution inputs. A plausible implication is that AMLAS-RL reserves part of the safety case for operational evidence rather than assuming that pre-deployment evidence is complete.

6. Running example: wheeled vehicle and reported results

The demonstration system is a wheeled vehicle tasked with reaching a target goal without collision. The vehicle state is $V$5, the goal is $V$6, and the action is $V$7. The observation space is 48-dimensional, obtained from three sets of 16 range sensors $V$8. The system also includes energy $V$9, which depletes at 1 unit per timestep, eight unsafe zones $X$0, and one obstacle $X$1 (Imrie et al., 8 Jul 2025).

The six AMLAS-RL stages are instantiated directly in this example. Stage 1 defines the three safety requirements $X$2–$X$3. Stage 2 translates them into RL safety requirements identical to $X$4–$X$5. Stage 3 produces a training plan $X$6 with 500 episodes, random object placement, and an episode cut-off at 500 timesteps; an internal test plan $X$7 with 1000 random trials; and a verification plan $X$8 comprising 500 general trials, 250 obstacle-proximal trials, and formal DTMC abstraction.

In Stage 4, the controller is a DDPG actor-critic with reward

$X$9

The internal test results are reported as 60.2% probability of reaching the goal with $Z$0, average time in unsafe zones of 15.94, and collision rate of 7.56%. Stage 5 verification reports statistical results of 60.8% success on general trials, 57.6% success on targeted trials, and collision at most 7.6%; model checking is reported to satisfy the PCTL properties with 60.4% success and 17.52 unsafe steps. Stage 6 deployment includes operational scenarios such as post-mission goods integrity, records hazardous sequences in the erroneous log, and considers post-shielding if failure is detected.

These results matter in the logic of AMLAS-RL not because they are high or low in isolation, but because they are aligned to pre-specified assurance thresholds. The example therefore functions as a worked safety case rather than merely a benchmark result.

7. Limitations, challenges, and practitioner guidance

The paper identifies several recurring challenges in RL assurance and associates them with corresponding AMLAS-RL responses. For safety defined over trajectories, it recommends using CMDP and cost signals in the reward. For reward hacking, it recommends the development log $Z$1, runtime logs $Z$2, and iterative refinement. For the simulation-reality gap, it recommends physical fine-tuning and extension of the verification plan $Z$3 to cover real-world uncertainties. For interpretability, it states that formal abstraction encourages deeper system understanding (Imrie et al., 8 Jul 2025).

The practitioner checklist is organized by stage. Stage 1 should ensure traceability from system HAZOP to RL allocations. Stage 2 should express each safety requirement as a measurable probabilistic or temporal constraint. Stage 3 should justify simulator fidelity and state-action coverage while producing distinct test and verification plans. Stage 4 should log model architecture, reward and cost functions, and convergence metrics. Stage 5 should include broad statistical tests, targeted worst-case scenarios, and formal methods. Stage 6 should record erroneous sequences, deploy runtime monitors or shields, and plan for incremental updates.

A common misunderstanding is that assurance for RL can be reduced either to safe exploration during training or to verification of a final trained policy. AMLAS-RL rejects both reductions. Its central claim is procedural: assurance requires traceable requirements, staged evidence generation, independent verification, and deployment monitoring within a unified, iterative lifecycle.