PyMOP: Python Monitor for Runtime Verification

• PyMOP is a runtime verification framework in Python that automatically synthesizes monitors from formal specifications and supports multiple logical formalisms.
• It integrates with pytest and employs diverse instrumentation strategies, including monkey patching and AST transformation, to track API usage and detect violations efficiently.
• Its scalable design leverages parametric trace slicing and multiple monitoring algorithms to handle large codebases and diverse test suites.

Searching arXiv for PyMOP and adjacent Python MOO/RV ecosystem papers. PyMOP is a generic, extensible, and efficient runtime verification (RV) system for Python, positioned as the first Python instance of Monitoring-Oriented Programming (MOP). Its defining objective is to bring to Python a logic-parametric, automatically synthesized, parametric-monitoring workflow analogous to the JavaMOP/RV-Monitor lineage, while adapting to Python’s instrumentation model, runtime costs, and testing workflows. In the reported evaluation, PyMOP supports five logics, implements five existing monitoring algorithms, ships with 73 API specs of Python and widely-used libraries, supports three instrumentation strategies, and was evaluated on 290,133 unit tests in 1,463 GitHub projects (Shen et al., 8 Sep 2025).

1. Definition and problem setting

PyMOP addresses a specific gap in Python runtime verification: prior Python systems were described as too narrow, too specialized, too offline, too hard-wired to one logic, or too slow to support large-scale test-time monitoring (Shen et al., 8 Sep 2025). The system therefore combines multiple specification logics, automatic monitor synthesis, parametric trace-slicing monitoring algorithms, multiple instrumentation strategies, and a practical pytest-based workflow for checking Python programs, libraries, and parts of the Python runtime.

The paper frames this against several limitations in earlier Python work. Some systems support only one formalism, despite the observation that no single logic expresses all useful RV properties. Tools based on handwritten checkers do not exploit automatic monitor synthesis from formal specifications. Prior Python tools also lacked the parametric trace-slicing algorithms that made MOP-style monitoring scale in Java, and some were limited to offline log analysis rather than online monitoring during tests or CI. The paper characterizes PythonRV as having few primitives for specs, supporting only a fraction of Python syntax, and not evaluating overheads; LogScope and PyContract as offline log analyzers; Hat-RV and VyPR2 as domain-specific; and DynaPyt and DyLin as dynamic analysis frameworks that can emulate RV via handwritten checkers but are not generic MOP-style RV frameworks (Shen et al., 8 Sep 2025).

A central misconception is to confuse PyMOP with Python multi-objective optimization libraries that use similar naming conventions. The adjacent systems PymooLab and LibMOON occupy the Python multi-objective optimization ecosystem, not the runtime-verification space. PymooLab is built on top of pymoo and focuses on visual analytics, reproducibility, LLM-based code generation, and MCDM in multi-objective optimization workflows (Santos et al., 2 Mar 2026). LibMOON is a PyTorch-first library for gradient-based multi-objective optimization in machine learning and is explicitly described as not the same library as PyMOP (Zhang et al., 2024). This suggests that PyMOP’s identity is best understood through the MOP/RV tradition rather than through the Python optimization ecosystem.

2. Formal model and MOP semantics

PyMOP follows the Monitoring-Oriented Programming paradigm: developers write formal specifications of behavioral properties, the system automatically synthesizes executable monitors from those specs, the monitored program is instrumented to emit events, and those events drive runtime monitor instances (Shen et al., 8 Sep 2025). The paper identifies two inherited MOP ideas as especially important: automatic monitor synthesis and parametric trace slicing.

Parametricity is essential because many Python API properties depend not only on event order but also on which runtime object instance the events refer to. The paper’s motivating example is a TOCTOU pattern involving os.access and open: a non-parametric analysis could incorrectly report a violation when the access and open refer to different files, whereas PyMOP uses parametric trace slicing to prevent that class of false alarm (Shen et al., 8 Sep 2025).

The formalization is given in terms of parameter instances, compatibility, trace slicing, and monitor state evolution. A parameter instance is a partial function from parameter names or types to concrete runtime objects. Two parameter instances $\theta$ and $\theta'$ are compatible if they agree wherever both are defined; if compatible, they can be combined using $\sqcup$:

$(\theta \sqcup \theta')(x) = \left\{ \begin{array}{ll} \theta(x) & \mbox{when } \theta(x) \mbox{ is defined} \ \theta'(x) & \mbox{when } \theta'(x) \mbox{ is defined} \ \mbox{undefined} & \mbox{otherwise} \end{array} \right.$

The paper also defines a partial order written $\theta' \sqsubseteq \theta$, interpreted as “less informative than”: $\theta$ binds at least the parameters that $\theta'$ does, consistently. For a parametric trace $\tau$ and parameter instance $\theta$, the $\theta$-slice is defined recursively as

$\theta'$0

and

$\theta'$1

so that a parametric execution is projected onto object-specific non-parametric traces (Shen et al., 8 Sep 2025).

The monitor abstraction is the standard tuple

$\theta'$2

where $\theta'$3 is the set of states, $\theta'$4 the non-parametric events, $\theta'$5 the categories or verdicts, $\theta'$6 the initial state, $\theta'$7 the transition function, and $\theta'$8 the state categorization (Shen et al., 8 Sep 2025). Once slicing has projected a parametric trace into a non-parametric slice, the corresponding monitor instance behaves as a state machine or equivalent logic-specific monitor.

3. Architecture, specification frontends, and execution workflow

PyMOP has three main architectural components: the Monitor Synthesizer, the Instrumenter, and the Monitoring Engine (Shen et al., 8 Sep 2025). It supports both online and offline RV. The monitor synthesizer uses logic plugins to compile user specifications into monitor templates. The instrumenter rewrites the code under test, third-party libraries, and parts of the Python runtime so that relevant before or after events are emitted. The monitoring engine creates monitor instances, dispatches events, tracks verdict categories, invokes violation handlers, and garbage-collects dead monitors.

The specification workflow is explicitly not tied to a separate DSL. PyMOP supports two frontends: specifications can be written directly in Python or in a JSON-like frontend. The JSON-like frontend includes fields such as Description, Variables, Formalism, Formula, Creation_Events, Events, Event_Actions, and Handlers (Shen et al., 8 Sep 2025). The TOCTOU example illustrates several design decisions at once: the formalism is explicit per specification, the formula is encoded in the syntax of the chosen logic, event mappings bind abstract events such as check and use to concrete Python APIs such as os.access and builtins.open, event actions can update monitor-local state or filter events, and handlers execute arbitrary Python code on violations.

This architecture gives PyMOP a broader coverage profile than many competing Python systems. The paper states that PyMOP monitors the code under test, third-party libraries, and Python runtime sources (Shen et al., 8 Sep 2025). A practical consequence is that utility and overhead are tightly coupled: monitoring more code expands the space of observable violations, but also amplifies instrumentation and dispatch costs.

The runtime workflow is correspondingly direct. The user supplies specs; the monitor synthesizer compiles each spec into a monitor template; the instrumenter rewrites target code, tests, libraries, or runtime points to emit named events; during execution, wrappers signal parametric events $\theta'$9; and the monitoring engine identifies affected parameter-instance slices, creates monitors as needed, updates their states using the selected algorithm, categorizes states, invokes handlers on violations, and garbage-collects monitors when safe (Shen et al., 8 Sep 2025). The paper states that PyMOP monitors multiple specs simultaneously.

PyMOP is implemented as a pytest plugin and invoked as:

$\theta'$0

This provides immediate integration with Python testing workflows, but it also creates a startup limitation: because monitoring begins only after pytest loads the plugin, PyMOP can miss early events occurring before plugin initialization. The paper identifies sitecustomize.py as a future direction to start earlier and become framework-independent (Shen et al., 8 Sep 2025).

4. Supported logics and monitoring algorithms

PyMOP ships with five logic plugins: future-time LTL, past-time LTL, ERE, FSM, and CFG (Shen et al., 8 Sep 2025). For ERE, FSM, and LTL, the generated monitor template is an FSM; for CFG, the template is a parser-derivative monitor. The system reuses JavaMOP’s mature synthesis plugins for ERE, FSM, past-time LTL, and future-time LTL, and adds a new CFG plugin as proof of extensibility.

FSM specifications are explicit state-transition systems over event names. The TOCTOU property is given as

$\sqcup$0

which formalizes the pattern that after a check, a subsequent use moves into a violation state (Shen et al., 8 Sep 2025). ERE specifications capture regular event languages with extended regex constructs. The paper’s UnsafeDictIterator example uses

$\sqcup$1

to detect next on a dictionary iterator after the dictionary was modified. CFG support is emphasized because it allows context-free patterns, which strictly exceed regular languages in expressive power; TornadoNoAdditionalOutput is the representative CFG example (Shen et al., 8 Sep 2025).

PyMOP implements five trace-slicing-based monitoring algorithms: $\sqcup$2, $\sqcup$3, $\sqcup$4, $\sqcup$5, and $\sqcup$6 (Shen et al., 8 Sep 2025). Algorithm $\sqcup$7 is an offline slicing baseline. It maintains $\sqcup$8, a map from parameter instances to trace slices, with $\sqcup$9 and $(\theta \sqcup \theta')(x) = \left\{ \begin{array}{ll} \theta(x) & \mbox{when } \theta(x) \mbox{ is defined} \ \theta'(x) & \mbox{when } \theta'(x) \mbox{ is defined} \ \mbox{undefined} & \mbox{otherwise} \end{array} \right.$0, and for each event updates compatible parameter-instance combinations. Its stated complexity is

$(\theta \sqcup \theta')(x) = \left\{ \begin{array}{ll} \theta(x) & \mbox{when } \theta(x) \mbox{ is defined} \ \theta'(x) & \mbox{when } \theta'(x) \mbox{ is defined} \ \mbox{undefined} & \mbox{otherwise} \end{array} \right.$1

where $(\theta \sqcup \theta')(x) = \left\{ \begin{array}{ll} \theta(x) & \mbox{when } \theta(x) \mbox{ is defined} \ \theta'(x) & \mbox{when } \theta'(x) \mbox{ is defined} \ \mbox{undefined} & \mbox{otherwise} \end{array} \right.$2 is trace length and $(\theta \sqcup \theta')(x) = \left\{ \begin{array}{ll} \theta(x) & \mbox{when } \theta(x) \mbox{ is defined} \ \theta'(x) & \mbox{when } \theta'(x) \mbox{ is defined} \ \mbox{undefined} & \mbox{otherwise} \end{array} \right.$3 is the number of possible parameter combinations. The paper presents $(\theta \sqcup \theta')(x) = \left\{ \begin{array}{ll} \theta(x) & \mbox{when } \theta(x) \mbox{ is defined} \ \theta'(x) & \mbox{when } \theta'(x) \mbox{ is defined} \ \mbox{undefined} & \mbox{otherwise} \end{array} \right.$4 as conceptually simple and useful for offline monitoring, but expensive in time and space (Shen et al., 8 Sep 2025).

Algorithm $(\theta \sqcup \theta')(x) = \left\{ \begin{array}{ll} \theta(x) & \mbox{when } \theta(x) \mbox{ is defined} \ \theta'(x) & \mbox{when } \theta'(x) \mbox{ is defined} \ \mbox{undefined} & \mbox{otherwise} \end{array} \right.$5 performs straightforward online slicing and monitoring without storing full traces. It maintains $(\theta \sqcup \theta')(x) = \left\{ \begin{array}{ll} \theta(x) & \mbox{when } \theta(x) \mbox{ is defined} \ \theta'(x) & \mbox{when } \theta'(x) \mbox{ is defined} \ \mbox{undefined} & \mbox{otherwise} \end{array} \right.$6 for monitor states, $(\theta \sqcup \theta')(x) = \left\{ \begin{array}{ll} \theta(x) & \mbox{when } \theta(x) \mbox{ is defined} \ \theta'(x) & \mbox{when } \theta'(x) \mbox{ is defined} \ \mbox{undefined} & \mbox{otherwise} \end{array} \right.$7 for verdict categories, and $(\theta \sqcup \theta')(x) = \left\{ \begin{array}{ll} \theta(x) & \mbox{when } \theta(x) \mbox{ is defined} \ \theta'(x) & \mbox{when } \theta'(x) \mbox{ is defined} \ \mbox{undefined} & \mbox{otherwise} \end{array} \right.$8 for seen parameter instances, updating state on each event according to

$(\theta \sqcup \theta')(x) = \left\{ \begin{array}{ll} \theta(x) & \mbox{when } \theta(x) \mbox{ is defined} \ \theta'(x) & \mbox{when } \theta'(x) \mbox{ is defined} \ \mbox{undefined} & \mbox{otherwise} \end{array} \right.$9

and

$\theta' \sqsubseteq \theta$0

Its weakness is that searching all of $\theta' \sqsubseteq \theta$1 for compatible parameter instances becomes expensive on long traces and large binding spaces (Shen et al., 8 Sep 2025).

Algorithm $\theta' \sqsubseteq \theta$2 introduces an auxiliary data structure $\theta' \sqsubseteq \theta$3 to avoid exhaustive compatibility search; $\theta' \sqsubseteq \theta$4 further adds a creation-event optimization so that monitors are created only after user-designated creation events occur; and $\theta' \sqsubseteq \theta$5 uses statically computed enable sets to avoid creating semantically useless monitors (Shen et al., 8 Sep 2025). The paper identifies $\theta' \sqsubseteq \theta$6 as the most sophisticated algorithm and the one used by JavaMOP/RV-Monitor, but one of its main findings is that in Python, unlike Java, $\theta' \sqsubseteq \theta$7 is not always the fastest in practice. Across projects where all online algorithms were comparable, $\theta' \sqsubseteq \theta$8 was fastest in 5 projects, $\theta' \sqsubseteq \theta$9 in 141, $\theta$0 in 295, and $\theta$1 in 996 (Shen et al., 8 Sep 2025). This suggests that Python’s runtime and instrumentation costs alter the tradeoff surface enough that Java’s default choice is not automatically dominant.

Monitor garbage collection is a further optimization. PyMOP uses coenable sets to determine when a monitor can no longer reach a category of interest. The reported evaluation found that on 1,363 projects with 69 specs, 645 projects were faster end-to-end with MGC and 718 were faster without it; with a 0.5 s tolerance, 1,290 projects showed no meaningful difference (Shen et al., 8 Sep 2025). A plausible implication is that MGC is workload-sensitive and matters mainly when many monitors for multi-object specifications are created.

5. Instrumentation, runtime behavior, and implementation strategy

PyMOP supports three instrumentation strategies through a common API: Python-level monkey patching only; Python-level monkey patching plus C-level monkey patching via forbiddenfruit / curses; and Python-level monkey patching plus AST transformation (Shen et al., 8 Sep 2025). The paper treats instrumentation as a first-order systems issue rather than a minor implementation detail, since instrumentation costs contribute heavily to end-to-end overhead.

Python-level monkey patching only replaces Python-level functions or methods with wrappers that emit before or after events. It is lightweight, easy to deploy, and requires no AST rewriting, but it has limited coverage for lower-level or built-in or C-implemented functionality and empirically finds the fewest violations (Shen et al., 8 Sep 2025). The combined monkey patching plus curses strategy extends interception to selected C-implemented builtins and runtime functions. The paper describes this as the preferred compromise. Python-level monkey patching plus AST transformation instruments code by parsing and rewriting the AST; it can yield efficient instrumented code at runtime, but incurs much higher instrumentation time, is more brittle, and failed on 239 projects in the instrumentation comparison (Shen et al., 8 Sep 2025).

The instrumentation comparison reported on 1,224 filtered projects shows the scale of these differences. Mean instrumentation time was 0.52 s for monkey patching, 0.47 s for monkey patching plus curses, and 7.52 s for AST-based instrumentation. End-to-end time sums were 29,024.62 s, 15,760.25 s, and 43,635.25 s respectively, while unique violations were 833, 11,332, and 5,043 (Shen et al., 8 Sep 2025). The paper therefore concludes that monkey patching plus curses is both much faster than AST instrumentation and detects more violations than either alternative.

Several implementation decisions are presented as relevant to efficiency. PyMOP reuses JavaMOP’s mature synthesis plugins rather than reimplementing all monitor compilation logic. The core runtime data structures include $\theta$2, $\theta$3, $\theta$4, $\theta$5, and $\theta$6. Unlike prior Java systems that often used code-generated monitoring code, PyMOP implements all algorithms as libraries, which the authors describe as easier to understand and extend, though likely slower than generated-code implementations (Shen et al., 8 Sep 2025). The paper explicitly suggests that moving from library implementations to generated-code monitoring backends is a future path to higher performance.

6. Built-in specifications, empirical evaluation, and bug-finding results

PyMOP ships with 73 API usage specifications: 55 Python specs and 18 library specs (Shen et al., 8 Sep 2025). These cover Python itself and widely used libraries including requests, TensorFlow, SciPy, NLTK, Flask, and Tornado; Pandas is mentioned as part of the generality claim. The 73 specs include 12 multi-object specs. Representative examples include UnsafeDictIterator (ERE; Python), UselessFileOpen (FSM; Python), ArraysSortBeforeBinarySearch (LTL; Python), RequestsPreparedInit (FSM; library), NLTKProbSum (ERE; library), TornadoNoAdditionalOutput (CFG; library), and the running TOCTOU example (Shen et al., 8 Sep 2025).

The evaluation scale is unusually large for Python RV. The paper reports 1,463 GitHub projects, 290,133 unit tests, 37,675,099 SLOC, total baseline runtime without RV of 25,072.6 s, 18,254,008 monitored events, and 5,686,846 monitors created (Shen et al., 8 Sep 2025). Per-project statistics include a mean of 226.0 tests and median of 17.5 tests, mean baseline runtime of 19.5 s and median of 1.1 s, mean SLOC of 29,342 and median of 2,317.5, and mean statement and branch coverage of 35.3% and 29.8% respectively. The experiments ran on an AMD EPYC 7763 64-Core Processor with 4 virtual CPUs, 15 GB RAM, Ubuntu 24.04.2 LTS, and Docker containers for reproducibility (Shen et al., 8 Sep 2025).

The abstract’s headline findings are threefold: the default monitoring algorithm for Java is often not the fastest for Python; PyMOP is up to 1,168.32x faster than DynaPyt-with-libraries in one comparison; and 44 of 121 bugs that PyMOP helped find so far were fixed by developers (Shen et al., 8 Sep 2025). In the comparison across 840 projects where all compared configurations succeeded, end-to-end time sums were 9,020 s for PyMOP, 38,385.43 s for DynaPyt, 36,217.28 s for DyLin, and 1,702,663.64 s for DynaPyt with libraries. Relative speedups reached a maximum of 412.25x versus DyLin, 419.97x versus DynaPyt, and 1,168.32x versus DynaPyt with libraries (Shen et al., 8 Sep 2025).

The bug-finding workflow consists of collecting violations, manually inspecting them, and reporting true bugs to developers through pull requests or GitHub issues. PyMOP reportedly helped find 121 true bugs so far, 93 of which were reported to developers, with 44 confirmed or fixed and 12 rejected (Shen et al., 8 Sep 2025). It also found 4 bugs in the Python interpreter itself: 1 fixed in a newer version, 1 open, and 2 confirmed although the PRs were ultimately rejected by maintainers. Manual inspection covered 240 unique violations occurring 366 times from 109 projects, with 215 true-positive occurrences, 135 false-positive occurrences, and 16 occurrences classified as difficult to inspect; the estimated manual effort was about 200 person hours (Shen et al., 8 Sep 2025).

The TOCTOU case study in mycli illustrates the practical style of these findings. PyMOP detected the pattern $\theta'$1 which yields

$\theta$7

and therefore violates the TOCTOU specification. The reported fix was to open directly and handle failure by exception: $\theta'$2 The fix was merged (Shen et al., 8 Sep 2025).

7. Extensibility, limitations, and ecosystem position

PyMOP is explicitly designed for extension in four dimensions: new logics, new monitoring algorithms, new specifications, and new instrumentation strategies (Shen et al., 8 Sep 2025). The CFG plugin is presented as proof that new logic backends can be added. The monitoring engine is modular enough that the algorithms $\theta$8 are exposed as libraries and can be selected from the command line, for example:

$\theta'$3

Specifications can be written either directly in Python or via the JSON-like frontend, and the instrumenter exposes an API for plugging in future instrumentation mechanisms (Shen et al., 8 Sep 2025). This modularity is one of the paper’s main claims: logic, instrumentation, and monitoring backends are independently replaceable.

The paper is also explicit about limitations. PyMOP was evaluated only during testing, not for deployed production monitoring. It currently supports five logics, leaving many more RV formalisms to be added. Even the best algorithm can incur very high overhead on some projects: for algorithm $\theta$9, the maximum relative overhead reported is 848.76x and the maximum absolute overhead 13,279.03 s. Monitoring starts too late because of the pytest plugin initialization model, and no user study has yet evaluated the ease of writing specifications in the Python or JSON-like frontend (Shen et al., 8 Sep 2025).

Open directions stated in the paper include better Python instrumentation infrastructure analogous to AspectJ/DiSL/BISM, generated-code monitoring backends, more logic plugins, evolution-aware monitoring for Python CI inspired by eMOP-style techniques, earlier startup via sitecustomize.py, user studies on specification authoring, and tighter integration of static analysis with runtime verification (Shen et al., 8 Sep 2025). The paper’s broader technical picture is that PyMOP’s novelty lies less in proposing a new logic or a single new algorithm than in integrating logic-parametric monitor synthesis, parametric trace slicing, multiple online and offline algorithms, multiple instrumentation mechanisms, practical API-usage specifications, and a large-scale empirical evaluation into one Python system.

Within the broader Python tooling landscape, PyMOP should therefore be distinguished sharply from optimization libraries with superficially similar names. PymooLab is a visual analytics framework for multi-objective optimization built on top of pymoo, centered on experiment orchestration, LLM-assisted formulation, and MCDM (Santos et al., 2 Mar 2026). LibMOON is a gradient-based multiobjective optimization library in PyTorch for machine learning, with support for finite-solution solvers, Pareto set learning, and multi-objective Bayesian optimization (Zhang et al., 2024). PyMOP, by contrast, belongs to the runtime-verification lineage: its technical center of gravity is formal behavioral specification, automatic monitor synthesis, parametric event slicing, and scalable test-time detection of API-usage and protocol violations (Shen et al., 8 Sep 2025).