ParamBench: Benchmark Suite for PTAs

• ParamBench is a benchmark library for parametric timed model checking, comprising diverse PTA models from academic examples to industrial case studies and unsolvable instances.
• It offers a standardized evaluation framework with benchmarks ranging from toy models for teaching to complex real-world scenarios to assess verification tools.
• The suite supports key verification problems such as reachability, safety, optimal and liveness synthesis, and enables cross-tool comparisons through multiple input formats.

Searching arXiv for ParamBench and closely related benchmark-library papers. ParamBench is a benchmark library for parametric timed model checking, presented as a common, publicly-available suite of benchmarks and later as a public library of extended Parametric Timed Automata (PTAs). Its stated purposes are to encourage fair comparison of new tools and techniques by providing a stable collection of models and properties, and to cover a spectrum from toy examples for pedagogical use up to real industrial case studies and “hard” or unsolvable instances. The library includes academic benchmarks, industrial case studies, and examples unsolvable using existing techniques; the later extension adds several dozens of new benchmarks emphasizing liveness properties and extensions of timed automata including stopwatches and multi-rate clocks (Étienne, 2018, André et al., 2021).

1. Scope, objectives, and evolution

The initial library was introduced to provide a common, publicly-available suite of benchmarks for parametric timed model checking, to support fair comparison of tools and algorithms, and to serve academic researchers, tool developers, and industrial practitioners working on scheduling, communication protocols, and pattern matching. In its first version, it contained 34 benchmarks, 80 models, and 122 properties. The later extended library, referred to as ParamBench 2.0, contains 56 named benchmarks, 119 PTA models, and 216 $(\text{model},\text{property})$ pairs (Étienne, 2018, André et al., 2021).

The benchmark families are explicitly heterogeneous. The library covers well-known academic examples, real-world industrial case studies, and deliberately hard or unsolvable examples. This breadth is central to its role as an evaluation corpus: toy models support pedagogical and algorithmic sanity checks, industrial models stress realism and feature coverage, and unsolvable instances expose the limits of current parameter-synthesis technology. The later extension broadens the scope further by adding liveness properties, stopwatches, multi-rate clocks, and unsolvable toy benchmarks intended to emphasize the limits of state-of-the-art techniques (André et al., 2021).

A common misconception is that PTA benchmark suites are primarily collections of small academic protocols. ParamBench explicitly includes real-time scheduling, train-gate controllers, automotive pattern-matching logs, Ariane guidance, avionics scheduling, and scaled families such as Gear and Fischer, alongside small examples designed to be beyond current automated techniques (Étienne, 2018, André et al., 2021).

2. Formal basis: PTAs and benchmarked verification problems

The formal core of ParamBench is the Parametric Timed Automaton. A PTA is given as

$A = (L, l_0, X, P, Inv, E),$

where $L$ is a finite set of locations, $l_0 \in L$ is the initial location, $X=\{x_1,\dots,x_n\}$ is a finite set of real-valued clocks, $P=\{p_1,\dots,p_k\}$ is a finite set of timing parameters, $Inv : L \to \Phi(X,P)$ assigns a conjunctive clock constraint to each location, and

$$E \subseteq L \times \Phi(X,P) \times Act \times 2^X \times L$$

is a finite set of edges of the form $(\ell,g,a,r,\ell')$. Atomic constraints in $\Phi(X,P)$ have the form

$A = (L, l_0, X, P, Inv, E),$0

with $A = (L, l_0, X, P, Inv, E),$1 (or $A = (L, l_0, X, P, Inv, E),$2) and $A = (L, l_0, X, P, Inv, E),$3; for example, the guard $A = (L, l_0, X, P, Inv, E),$4 requires that the difference between clocks $A = (L, l_0, X, P, Inv, E),$5 and $A = (L, l_0, X, P, Inv, E),$6 does not exceed parameter $A = (L, l_0, X, P, Inv, E),$7 (Étienne, 2018).

ParamBench organizes benchmarks around several parametric model-checking problems. These include reachability or safety in the form of EF-synthesis, where one computes the set of parameter valuations $A = (L, l_0, X, P, Inv, E),$8 such that a target location $A = (L, l_0, X, P, Inv, E),$9 is reachable, or never reachable, in the instantiated automaton $L$0. It also includes optimal reachability, where one seeks valuations for which $L$1 is reachable while minimizing or maximizing a selected parameter; unavoidability in the form of AG-synthesis, where all runs of $L$2 eventually visit $L$3; robustness, where a region $L$4 is synthesized so that $L$5 has the same discrete behavior as a reference valuation $L$6 for all $L$7; and parametric timed pattern matching, where intervals in a timed log and parameter valuations are synthesized so that a parametrized property holds (Étienne, 2018).

The extended library broadens the modeling basis beyond pure PTAs. Its added benchmarks include liveness properties such as CycleThrough, as well as automata features including stopwatches and multi-rate clocks. In the “Researcher” model, for example, one clock has rate $L$8 in one location and rate $L$9 in another, while a discrete variable $l_0 \in L$0 counts coffees; in the “Cycle” example, the expected synthesis result for CycleThrough(\ell_1) is $l_0 \in L$1 (André et al., 2021).

3. Corpus organization and benchmark families

The first library description distinguishes three benchmark classes: academic examples, industrial case studies, and unsolvable instances, and notes a fourth “educational” category of very simple models for teaching. Academic examples include Fischer mutual exclusion, CSMA/CD, and the And-Or circuit. Industrial case studies include real-time scheduling from the FMTV challenge, train-gate controllers, and parametric pattern matching from automotive logs. Unsolvable instances include toy PTAs such as the example whose exact reachability set is $l_0 \in L$2 (Étienne, 2018).

ParamBench 2.0 reorganizes the corpus as 56 named benchmarks, 119 PTA models, and 216 $l_0 \in L$3 pairs, with approximately $l_0 \in L$4 of models from academic case studies, approximately $l_0 \in L$5 from industrial case studies, and approximately $l_0 \in L$6 from toy and deliberately unsolvable examples. Named benchmarks include Fischer, BRP, Accel, Gear, FMTV1, ArianeGP, AvioSched, Researcher, MonitorRW, unsolvable_N, unsolvable_InvN, unsolvable_pN+q, and unsolvable_Rplus (André et al., 2021).

For each model, the library records structural and semantic metadata. The first description states that each model is annotated with $l_0 \in L$7, $l_0 \in L$8, $l_0 \in L$9, $X=\{x_1,\dots,x_n\}$0, a flag indicating whether it is an L/U-PTA or U-PTA, the presence of location invariants and stopwatches, and the property type, chosen among Safety, Reachability, Optimal Reachability, Unavoidability, Robustness, PTPM, or Misc. The later description adds that one or more models may be provided for a benchmark family and that one or more properties may be associated with each model, including reachability, minimal-time reachability, liveness or cycle, and deadlock-free properties (Étienne, 2018, André et al., 2021).

Scalability is built into several entries. The first version already notes parameterized scalability dimensions such as Fischer_N processes and Flip-flop with 2, 5, or 12 parameters. The later version gives the example of Gear, which comes in 10 size scales: Gear:1000, Gear:2000, through Gear:10000 (Étienne, 2018, André et al., 2021).

4. Specification formats, repository structure, and execution workflow

The primary syntax of the original library is the IMITATOR input language in plain text, and benchmarks are distributed in a public GitHub repository at https://github.com/imitator-model-checker/imitator/tree/master/benchmarks. When available, the original model-checker syntax is also provided, with examples including HyTech and UPPAAL. The extended library is retrieved by cloning https://gitlab.inria.fr/andre/imitator-benchmarks-2.0.git, and its directory layout stores, for each benchmark, .imi models, .imiprop property files, .jani model-interchange files, .result outputs, and .expres files for expected unsolvability annotations (Étienne, 2018, André et al., 2021).

In ParamBench 2.0, .imi denotes IMITATOR 3.0 native input, .imiprop denotes IMITATOR property files such as reachability, EFpmin, or Cycle, .jani denotes a model-interchange format that can be consumed by Storm, Prism2JANI, and related tooling, .result stores completed IMITATOR output, and .expres stores human-annotated expected results for unsolvable models. The later description also states that the library includes an automated IMITATOR→JANI translator so that other tools can at least run a fully instantiated version of any model (André et al., 2021).

The usage workflow described for integrating ParamBench into tools is operationally simple but feature-sensitive. It consists of cloning or downloading the benchmarks directory, adapting the front-matter if a tool requires a different header, ensuring support for the PTA features actually used by the model, and invoking the tool’s syntactic front-end, optionally with pre-processing to eliminate unsupported constructs or to translate parameters into an internal representation. This suggests that cross-tool comparison is feasible, but only under explicit control of syntactic and semantic compatibility (Étienne, 2018).

An illustrative IMITATOR specification for the 1/n toy PTA is given as follows (Étienne, 2018):

$(\ell,g,a,r,\ell')$0

The later library description also provides sample command lines for reachability synthesis, extraction of all parameter constraints, and cycle synthesis using IMITATOR’s command-line interface (André et al., 2021).

5. Representative benchmarks and empirical profile

The Fischer mutual exclusion benchmark is one of the canonical academic entries. In the three-process version, the model consists of three identical timed automata modeling entry and exit sections. It uses 1 or 2 clocks per process and parameters $X=\{x_1,\dots,x_n\}$1 controlling delay bounds. The associated property is safety: no two processes are in the critical section simultaneously. The recorded size metrics are 3 automata, 2 clocks each, 4 parameters, no invariants, and property type Safety. The reported typical solving time for IMITATOR EF-synthesis is $X=\{x_1,\dots,x_n\}$2 s for 3 processes (Étienne, 2018).

The FMTV real-time scheduling challenge benchmark exemplifies industrial usage. Instance 1A3 models tasks with uncertain execution periods as parameters and an idle-time scheduler modeled as a PTA. Its property is optimal reachability, described as maximizing resource utilization under schedulability constraints. The recorded metrics are 3 automata, 3 clocks, 3 timing parameters plus 7 integer variables, with invariants but no stopwatches; the typical solving time reported for this instance is $X=\{x_1,\dots,x_n\}$3 s (Étienne, 2018).

The 1/n toy PTA is the standard unsolvable example in the first library description. It consists of a single automaton with two clocks $X=\{x_1,\dots,x_n\}$4 and one parameter $X=\{x_1,\dots,x_n\}$5, location invariant $X=\{x_1,\dots,x_n\}$6 at $X=\{x_1,\dots,x_n\}$7, a self-loop on $X=\{x_1,\dots,x_n\}$8 guarded by $X=\{x_1,\dots,x_n\}$9 that resets $P=\{p_1,\dots,p_k\}$0, and a transition $P=\{p_1,\dots,p_k\}$1 guarded by $P=\{p_1,\dots,p_k\}$2. The property is reachability of $P=\{p_1,\dots,p_k\}$3, and the exact solution set is

$P=\{p_1,\dots,p_k\}$4

an infinite, nonconvex set of rational points (Étienne, 2018).

The extended library adds benchmarks that stress features absent from standard PTA corpora. Researcher uses stopwatches and multi-rate clocks together with guards such as $P=\{p_1,\dots,p_k\}$5 and an update $P=\{p_1,\dots,p_k\}$6. Cycle is a compact liveness benchmark whose expected result is $P=\{p_1,\dots,p_k\}$7. BRP and Cycle are used to exercise the CycleThrough algorithm, while Researcher and AvioSched stress flows different from $P=\{p_1,\dots,p_k\}$8 or equal to $P=\{p_1,\dots,p_k\}$9 (André et al., 2021).

The reported empirical profile reflects both easy and difficult cases. The first library description gives the following selected times: Fischer(3) Safety, $Inv : L \to \Phi(X,P)$0 s; Flip-flop with 12 parameters, $Inv : L \to \Phi(X,P)$1 s; idle-time-sched with 5 tasks, $Inv : L \to \Phi(X,P)$2 s; FMTV-1A3 optimal reachability, $Inv : L \to \Phi(X,P)$3 s; and parametric pattern matching on accel:10 intervals, $Inv : L \to \Phi(X,P)$4 s. The extended library summarizes 157 successful runs on an Intel Xeon @ 2.2 GHz with 96 GiB RAM, with average total computation time $Inv : L \to \Phi(X,P)$5 s and median $Inv : L \to \Phi(X,P)$6 s, average number of symbolic states $Inv : L \to \Phi(X,P)$7 and median $Inv : L \to \Phi(X,P)$8, and average number of computed states $Inv : L \to \Phi(X,P)$9 and median $$E \subseteq L \times \Phi(X,P) \times Act \times 2^X \times L$$0 (Étienne, 2018, André et al., 2021).

The benchmark suite is explicitly designed to expose algorithmic stress points. Examples listed in the later description include Fischer models with up to 20 processes and up to approximately 10 000 locations, MonitorRW benchmarks with 1000+ clocks, nonlinear parameter constraints such as $$E \subseteq L \times \Phi(X,P) \times Act \times 2^X \times L$$1, liveness synthesis through CycleThrough, and stopwatch or multi-rate handling in Researcher and AvioSched (André et al., 2021).

6. Limits of current techniques and continuing development

ParamBench devotes significant attention to benchmarks that are hard or unsolvable for current tools. The first description states that no existing tool can compute symbolically the exact solution set of the 1/n example, and it remarks that tools typically produce convex polyhedral regions only. It also notes that Fischer for arbitrary $$E \subseteq L \times \Phi(X,P) \times Act \times 2^X \times L$$2 admits a human-derivable closed form for any $$E \subseteq L \times \Phi(X,P) \times Act \times 2^X \times L$$3, yet tools do not scale symbolically beyond small $$E \subseteq L \times \Phi(X,P) \times Act \times 2^X \times L$$4 because of explosion of parametric constraints, and that some industrial cases such as large protocol BRP time out after 300 s because of combinatorial blow-up in state and parameter space (Étienne, 2018).

The later extension sharpens these limitations. It states that unsolvable examples typically require countably-infinite or non-convex constraints such as

$$E \subseteq L \times \Phi(X,P) \times Act \times 2^X \times L$$5

and that current parametric DBM-based techniques cannot represent infinitely many disjoint points or periodic arithmetic progressions. It further notes that liveness synthesis can blow up when the Büchi-automaton product has many locations, that nonlinear guards and updates such as multiplication by constants less than $$E \subseteq L \times \Phi(X,P) \times Act \times 2^X \times L$$6 or assignments of the form $$E \subseteq L \times \Phi(X,P) \times Act \times 2^X \times L$$7 fall outside the classical L/U-PTA decidable fragment, and that stopwatches and multi-rate clocks push the library beyond pure PTAs into multi-rate automata, where decidability is fragile and often requires syntactic restrictions (André et al., 2021).

The research directions named in the two descriptions are correspondingly concrete. They include handling non-convex solution sets, managing stopwatches and shared rational variables while scaling to larger parameter counts, combining SMT-based corner analyses with traditional zone enumeration, distributed or multicore synthesis for large industrial models, integrating an SMT-based “infinite union of points” theory to capture solutions such as $$E \subseteq L \times \Phi(X,P) \times Act \times 2^X \times L$$8, extending L/U-PTA algorithms to allow simple fractional updates such as $$E \subseteq L \times \Phi(X,P) \times Act \times 2^X \times L$$9, developing parametric partial-order reductions for liveness properties, and exploring abstraction-refinement loops mixing parameter sampling with symbolic sweeping (Étienne, 2018, André et al., 2021).

The library is actively maintained at https://www.imitator.fr/library.html. The first description states that contributions of new benchmarks, models in other syntaxes such as UPPAAL XML and HyTech, or enhancements of existing entries are very welcome, while the later description states that ParamBench is under CC BY 4.0 and supports contributions via pull requests (Étienne, 2018, André et al., 2021).