Reproducibility Maturity Model (RMM)

• RMM is a structured framework that evaluates reproducibility as a graded, multi-dimensional continuum rather than a simple binary outcome.
• It builds on previous reproducibility rubrics by integrating assessments for artifact accessibility, environment specification, versioning rigor, execution fidelity, and legal openness.
• Empirical studies reveal that binary badge systems are insufficient, motivating a tiered approach to address reproducibility failures in computational research.

The Reproducibility Maturity Model (RMM) denotes a maturity-oriented approach to assessing reproducibility as a graded, multi-dimensional property rather than a binary outcome. In the literature, the term is explicitly formulated as a five-axis, four-tier framework for LLM-for-SE research, designed to evaluate the durability, rigor, legal openness, and long-term verifiability of artifacts and experimental pipelines; more broadly, closely related work in computational science, systems biology, research software engineering, high-performance computing, FAIR-oriented workflow design, and machine learning provides the conceptual and operational foundations for treating reproducibility as an assessable progression of capabilities rather than a simple presence-or-absence condition (Siddiq et al., 29 Nov 2025).

1. Conceptual foundations

A central precursor to RMM is the argument that reproducibility in computational science should be treated as a technical specification. In that formulation, a result is reproducible if the source code can be rebuilt and run on a new machine and the published behavior can be observed again without manual intervention from the original author. The proposed specification requires a service that can demonstrate that the source of an algorithm or tool can be compiled, run and behave as described, allow new benchmarks to be added by users other than the developer, and store and link specific artefacts with their linked publications or other publicly-accessible datasets. This line of work frames reproducibility as unambiguous, measurable, and enforceable, and closely associates it with de novo compilation, automated testing, benchmark validation, commit-level traceability, and archival linkage (Crick et al., 2015).

The maturity concept also depends on terminological distinctions. One systems-biology account defines repeatability as the ability of a third party to generate the same results using the same data and the same computational software, ideally also the same or similar hardware and operating system, whereas reproducibility requires that a third party recreate some or all of the analysis de novo (Sauro, 2021). A high-performance computing survey emphasizes the ACM’s post-2020 terminology in which repeatability is “Same team, same experimental setup,” reproducibility is “Different team, same experimental setup,” and replicability is “Different team, different experimental setup” (Antunes et al., 2024). A model-centric analysis further separates reliability, auditability, replicability, and reproducibility, and formalizes an idealized experiment as $\xi = (M_\theta, D, S, K)$, with data decomposed into observed values and structural aspects, and methods decomposed into pre- and post-analysis components (Baumgaertner et al., 2018).

These distinctions matter because RMM presupposes that reproducibility is not a single thing. One recent predictive framework therefore models reproducibility as a spectrum rather than a binary label and distinguishes an author-centric spectrum from an external-agent spectrum. That framing is not a full maturity model in the classic process sense, but it reinforces the idea that reproducibility status is ordered, partial, and dependent on both what authors provide and what independent evaluators can achieve (Akella et al., 2024).

2. Maturity-oriented antecedents

Before the explicit RMM formulation, several papers already encoded reproducibility as progressive capability. A practical guide for computational models proposed a 10-point repeatability rubric in which score 0 is reserved for non-repeatable work, and the remaining points are distributed as follows: Code and Data = 3, Documentation = 2, Testing = 2, DOI = 1, Version Control = 1, and Continuous Integration = 1. The paper also describes an ordered progression: 0 for irreproducible or not repeatable work, then 3 for code and data, 5 after documentation, 7 after testing, 8 with DOI, 9 with version control, and 10 with continuous integration. Although framed as a repeatability score rather than a named maturity model, it functions as a staged path from non-repeatable work to more robust, preserved, tested, and continuously integrated computational practice (Sauro, 2021).

A FAIR-oriented framework for computational experiments proposed five reproducibility levels: Repeatable, Re-runnable, Portable, Extendable, and Modifiable. These levels are defined by what must be preserved and what may change across executions. Repeatability preserves all entities; re-runnability fixes script, functions, hardware, OS, OS packages, and module packages; portability requires preservation of input data, script, functions, parameters, and module packages; extendability and modifiability increasingly relax environmental constraints while preserving access to script and functions. This is not presented as a numeric rubric, but it is explicitly level-based and artifact-centered (Mondelli et al., 2019).

A further maturity-like development appears in the dual-spectrum framework that introduces three author-centric labels—$A_{PWA}$, $A_{PUNX}$, and $A_{PAX}$—and four external-agent labels—$E_{NR}$, $E_{AR}$, $E_{Re}$, and $E_{R}$. In that scheme, $A_{PAX}$ is the highest author-centric standard because it denotes validated artifacts that are permanently archived, and $E_{R}$ is the highest external-agent status, associated with reproduced work aligned with archived-artifact conditions. This arrangement resembles a maturity ladder at the paper level, even though it is described as a status taxonomy plus predictive classifier rather than a process maturity model (Akella et al., 2024).

Collectively, these strands suggest that RMM emerged from a convergence of technical specification, artifact preservation, staged capability, and independent validation. The explicit name is recent, but the underlying logic was already present in earlier reproducibility rubrics, service specifications, and level-based frameworks.

3. Canonical structure of the named RMM

The explicit Reproducibility Maturity Model is introduced as a multi-dimensional, tiered framework for LLM-for-SE research. Its purpose is to move beyond binary artifact certification and assess reproducibility as a continuum of maturity. The model evaluates five orthogonal axes: Accessibility, Environment Specification, Versioning Rigor, Execution Fidelity, and Legal Openness. It defines four maturity tiers: RMM-0, RMM-1, RMM-2, and RMM-3 (Siddiq et al., 29 Nov 2025).

Tier	Description	Defining features
RMM-0	Minimal Reproducibility	Missing or fragmentary artifacts, vague environment, unspecified versions, legal ambiguity
RMM-1	Operational Reproducibility	Artifacts available and possibly functional at publication time, but brittle and manually reconstructed
RMM-2	Durable Reproducibility	Fully runnable, containerized or equivalent, fully versioned, legally open, end-to-end reproducible
RMM-3	Independently Verified Reproducibility	All RMM-2 criteria plus successful re-execution by an external reviewer or independent research group

The axis-by-level mapping is equally explicit. For Accessibility, RMM-0 corresponds to no or fragmented artifacts, while RMM-2 requires versioned, persistently hosted artifacts clearly mapped to experiments. For Environment Specification, the progression runs from absent specification to a complete machine-readable environment specification including OS, libraries, and hardware. For Versioning Rigor, the model moves from floating or unspecified versions to pinned versions at all hierarchy levels with manifests or changelogs. For Execution Fidelity, the progression runs from prose or pseudocode without runnable scripts to end-to-end automated pipelines or containers. For Legal Openness, the ladder runs from missing or ambiguous licenses to explicit compatible licenses for all required artifacts. RMM-3 is defined as an evaluation-driven extension that assumes all RMM-2 criteria plus independent re-execution by a third party (Siddiq et al., 29 Nov 2025).

This explicit RMM is narrower than generic research software maturity models because it is centered on reproducibility failure modes observed in a specific corpus. Its analytical strength lies in treating durability, version pinning, execution automation, and legal reuse as coequal components of maturity, rather than collapsing them into a single badge or checklist item.

4. Empirical motivation and assessment logic

The named RMM is grounded in a large empirical study of reproducibility practices in LLM-for-SE research. The study manually annotated 640 papers and associated artifacts using a taxonomy of seven smell categories: Code and Execution, Data, Documentation, Environment and Tooling, Versioning, Model, and Access and Legal. The reported prevalence was Access and Legal in 35.9% of papers, Code and Execution in 35.5%, Versioning in 32.2%, Environment and Tooling in 21.1%, Model in 9.2%, Data in 8.0%, and Documentation in 3.6%; only 13.3% of papers had no reproducibility smells. The study also reports that smells are structurally coupled: among papers with Access and Legal smells, 43.5% also had Versioning smells and 34.8% also had Environment and Tooling smells, while Code and Execution smells co-occurred with Data smells in 22.5% and Environment and Tooling smells in 19.8%. The strongest pairwise associations were Documentation + Model, Documentation + Environment and Tooling, and Code and Execution + Data. The authors therefore characterize reproducibility debt as clustered and structural rather than isolated (Siddiq et al., 29 Nov 2025).

The same study motivates RMM by showing the limits of badge-based evaluation. It reports 63 badged papers, or 9.8% of the corpus, and notes that badge adoption increased over time, yet badged papers still exhibited reproducibility smells; in 2024, 41.7% of badged papers still had versioning smells. The study further states that current badge systems mostly evaluate momentary functionality rather than long-term reproducibility, and that over 40% of “functional” artifacts from 2024–2025 failed within months because of drift, missing versions, incomplete environments, or licensing issues. This is the direct rationale for replacing binary certification with a graded maturity model (Siddiq et al., 29 Nov 2025).

Operationally, the assessment logic of RMM aligns with earlier automation-centered proposals. One reproducibility service specification describes a workflow in which a developer pushes code, a server detects the new commit, pulls the code, downloads dependencies, compiles it, runs developer-defined basic tests, then runs benchmark models and stores results with the commit ID in a web-accessible database. That workflow distinguishes verification from validation, minimizes developer intervention, and treats undocumented build steps as a reproducibility failure; it also proposes a staged adoption plan of Year $A_{PWA}$0 optional use, Year $A_{PWA}$1 mandatory use without review consequences, and Year $A_{PWA}$2 mandatory use with results used in assessment (Crick et al., 2015). A complementary repeatability rubric recommends code and data release, documentation, formal testing, DOI assignment, version control, and continuous integration, thereby supplying a lightweight author-side pathway for climbing from low to high maturity (Sauro, 2021).

5. Relation to adjacent maturity frameworks

RMM sits within a broader ecosystem of maturity frameworks for research software, computational experiments, and ML systems. A prominent example is the Research Software focus area Maturity Model (RSMM), a focus area maturity model for research software project management. RSMM v1.0 is organized into 4 focus areas, 17 capabilities, and 79 best practices, with maturity assessed per focus area through the longest path of implemented practices. Reproducibility is distributed across capabilities rather than isolated in a separate axis and is operationalized through practices such as archive software for reproducibility, make code citable, enable indexing of project metadata, publish in a research software directory, provide executable tests, and execute tests in a public workflow. The model uses practice codes such as 2.3.5 and structures “when implemented” criteria with MoSCoW prioritization. Its case studies report maturity vectors of 4-3-6-7 for GGIR and 5-4-8-8 for ESMValTool (Deekshitha et al., 2024).

An industrial ML maturity framework extends the same general idea into governance. It defines seven quality characteristics—Utility, Economy, Robustness, Productionizability, Modifiability, Comprehensibility, and Responsibility—and 30 sub-characteristics, including Repeatability, Monitoring, Traceability, Discoverability, Readability, and Ownership. It formalizes quality scoring through

$A_{PWA}$3

with gap values 0, 1, and 2 for no gap, small gap, and large gap, and couples the score to five maturity levels and business criticality targets. In that framework, maturity is not a linear rescaling of quality score, because some sub-characteristics are gating requirements for advancing to higher levels (Castelli et al., 12 Feb 2025).

Other adjacent frameworks emphasize the artifact and environment dimensions that RMM later codifies. FAIR-oriented work on computational experiments uses dependency management, virtual machines, provenance capture, and repository publication to move experiments across the levels Repeatable, Re-runnable, Portable, Extendable, and Modifiable; in its case study, local execution without the framework failed on three machines, whereas VM-based execution succeeded on all three (Mondelli et al., 2019). A high-performance computing survey, while not proposing a named maturity model, identifies the assessment dimensions that an HPC-oriented RMM would need to score: artifact availability, documentation quality, environment capture, workflow capture, versioning and archiving, dependency management, PRNG control, numerical stability, hardware disclosure, handling nondeterminism, and silent-error resilience (Antunes et al., 2024).

Taken together, these models indicate that RMM is best understood as a specialized member of a larger family of maturity-oriented frameworks. What distinguishes the named RMM is its concentration on sustainable reproducibility of artifacts and pipelines, especially under rapid model, dependency, and legal drift.

6. Limits, misconceptions, and unresolved issues

A recurrent misconception in reproducibility discourse is that reproducibility can be reduced to artifact presence. The literature repeatedly rejects that simplification. The dual-spectrum predictive framework argues that a paper may still be hard or impossible to reproduce even if code and data are available, for example because of missing libraries, and the named RMM makes the same point by separating accessibility from environment specification, versioning rigor, execution fidelity, and legal openness (Akella et al., 2024). Another misconception is that badge acquisition implies durable reproducibility; the empirical evidence behind RMM shows that it often signals artifact presence or momentary review-time functionality instead of long-term verifiability (Siddiq et al., 29 Nov 2025).

A second misconception is that reproducibility is equivalent to truth, or that open data alone guarantees it. The model-centric account explicitly states that reproducible results may be false and true results may be irreproducible. It also argues that open access to raw observed values is necessary for auditability, but not necessarily for reproducibility of a specific inferential result; by contrast, structural data, model specification, and method–inference compatibility may be the decisive factors (Baumgaertner et al., 2018). This complicates any simple maturity scale that treats “more openness” as automatically “more reproducible.”

A further limitation is domain specificity. In HPC, reproducibility is threatened by parallel computing nondeterminism, floating-point non-associativity, PRNG control, compiler optimizations such as -O3, FMA and AVX, hardware heterogeneity, silent errors, and the probabilistic character of current quantum hardware. The same survey notes that VMs are usually too heavy for HPC, Docker is often unsuitable in HPC because it requires privileges and can have security concerns, and Apptainer/Singularity is better suited for HPC reproducibility, while Guix is described as the best dependency-management solution for transparent environment capture (Antunes et al., 2024). Earlier work on automated reproducibility services likewise notes that performance benchmarking in the cloud is difficult because VM scheduling and infrastructure noise make raw timing less reliable, and that running arbitrary code introduces security concerns requiring sandboxing and privilege restriction (Crick et al., 2015).

Current work also identifies unresolved policy and methodological questions. One practical guide argues that journals should require repeatability and ideally reproducibility, and that universities, government agencies, and research institutions should modify incentives so reproducible work is valued in hiring, promotion, tenure, and funding decisions (Sauro, 2021). FAIR-oriented work proposes future efforts to evaluate and quantify FAIR metrics, map FAIR metrics to reproducibility levels, and assess what reproducibility level is achieved from those metrics (Mondelli et al., 2019). Predictive modeling of reproducibility offers decision-support rather than replacement of human judgment and warns that automated scoring could encode unfair biases, for example by penalizing papers because of language style or institution (Akella et al., 2024). These lines of work suggest that mature reproducibility assessment will likely combine technical verification, artifact governance, independent re-execution, and policy enforcement rather than relying on any single checklist or badge.