Experimental Rigor Engine Overview

• Experimental Rigor Engine is a structured framework defining rigorous, context-aware methodologies for evaluating simulation software in experimental research.
• It employs quantitative validation techniques like χ² goodness-of-fit and contingency table analyses to objectively compare simulation outputs with experimental data.
• The engine integrates robust software quality controls, including automated testing and modular design, to mitigate performance degradation during code evolution.

An Experimental Rigor Engine refers to a structured set of concepts, methodologies, and protocols designed to ensure the objective, robust, and context-sensitive evaluation of scientific experiments—particularly in computational and simulation-heavy research. As articulated in "Negative Improvements, Relative Validity and Elusive Goodness" (Batic et al., 2013), this engine encompasses not only quantitative validation procedures but also context-aware appraisal, comprehensive software quality assurance, and systematic approaches to mitigate the risk of quality degradation in evolving research software.

1. Context-Dependent Software Validity

The notion of software validity in scientific simulations is fundamentally relative rather than absolute. The paper emphasizes that the demonstration of validity for a physics model implemented in a Monte Carlo code (such as Geant4) depends strongly on the specifics of the experimental environment, including detector geometry and characteristics. Identical configurations in code can produce substantially different compatibilities with observational data solely due to differences in detector segmentation or sensitive volume definition. For example, experimental setups based on the Sandia SAND79 detector with fine longitudinal segmentation (see Fig. 1, Fig. 3) show distinct simulation agreement compared to a bulk sensitive volume arrangement as in SAND80 (see Fig. 5, Fig. 7). The experimental rigor engine thus must always specify the full context—detector properties, geometric configuration, and operational parameters—against which code validity is assessed.

2. Quantitative Validation of Simulation Models

A central tenet of the rigor engine is the use of statistical, quantitative techniques for validation—superseding informal visual or qualitative assessments. Model appraisal proceeds in two major steps:

• Goodness-of-fit testing: The χ² test is recommended for determining whether the simulation's output is statistically compatible with experimental data, using

$\chi^2 = \sum_{i} \frac{(O_i - E_i)^2}{E_i}$

where $O_i$ are observed and $E_i$ are expected outcomes.

• Model comparison via contingency tables: Tests such as Fisher’s exact test, Barnard’s test, and a further χ² analysis are used to statistically distinguish model performance, for instance demonstrating that a specific model (such as Bote and Salvat, 2008) yields significantly superior results to EEDL-based alternatives at p-values below 0.01.

These practices ensure that model selection and claims of improvement are not anecdotal or circumstantial, but grounded in formal significance testing. The approach aids both in selecting optimal physical models and in monitoring their performance over software updates and reconfigurations.

3. Software Quality Mitigation Strategies

The preservation of simulation software quality as it evolves requires rigorous engineering controls across multiple development disciplines:

• Automated and agile testing: Robust unit test suites are advocated to promptly detect regressions in physical performance, contingent on modular and non-overly-complex codebases.
• Domain decomposition and clear, modular design: Complexity in class responsibilities and excessive inheritance (visualized in the paper's UML diagrams, Figs. 2 and 3) are flagged as impediments to effective testing and future-proof maintenance.
• Elimination of code duplication: The existence of multiple, near-identical classes (e.g., for Compton scattering) is identified as a problem for maintainability and introduces risk of inconsistency; more modular design is suggested for improved manageability.

These strategies indicate that experimental rigor is inseparable from sound software engineering and lifecycle management, with close collaboration needed between developers, quality assurance teams, and end-users.

4. Real-World Experimental Case Study

A detailed case paper demonstrates the contextual dependency of simulation validity. Energy deposition of electrons in differently configured detectors was simulated:

• Fine longitudinal segmentation versus bulk volume configurations led to substantial differences in simulation–experiment agreement (see Figs. 1 and 5).
• The test case, referencing [tns_dress2], shows unequivocally that even with unchanged physics models, physical layout and measurement protocols in the real experiment alter validation outcomes.

This confirms that the experimental rigor engine must not extrapolate model validity across substantially different experimental setups without dedicated re-validation.

5. Evolution of Functional Quality in Simulation Software

The paper provides evidence that software updates do not guarantee monotonic improvements. Trends observed over Geant4 versions indicate occasional degradation of compatibility with experiment. This underscores several best practices:

• Maintain continuous, quantitative validation across software generations.
• Early detection of performance regression is facilitated by rigorous design (modularity, minimized dependencies, no code duplication).
• Multidisciplinary vigilance (combining software design, testing, and governance) is required to prevent and respond to quality erosion.

The rigor engine thus integrates not only upfront validation but also persistent monitoring and responsive process controls as software evolves.

6. Summary Table of Core Practices

Rigor Component	Mechanism/Method	Contextual Dependency
Software validity	Empirical context, detector	Strong
Model validation	χ² goodness-of-fit, contingency tables	Moderate–Strong
Quality assurance	Unit testing, modular design	Moderate
Model comparison	Statistical significance	Moderate
Evolution monitoring	Iterative quantitative checks	Strong

7. Broader Implications

The experimental rigor engine, as formulated, implies that:

• Claims of model superiority must always be context-bound and statistically substantiated.
• Rigor is not static: changes in software, detector, or experimental protocol necessitate renewed validation.
• Software engineering practices are intrinsic to scientific rigor: technical debt, code duplication, and fragile architectures undermine the reliability of simulation outcomes.
• Quantitative monitoring tools are essential for both ensuring and documenting the functional quality throughout software lifecycle and across model versions.

This approach provides a robust, technically grounded justification for the implementation and continuous refinement of experimental rigor engines in scientific simulation workflows, enabling objective, reproducible, and incrementally improvable model evaluation in complex experimental environments.