Rigor in Research

• Rigor is defined as the multidimensional standard ensuring research integrity, including epistemic, normative, conceptual, and methodological facets.
• Methodological rigor is enforced through systematic frameworks, pre-specified workflows, and robust reporting that validate theoretical and empirical claims.
• Evidential rigor spans statistical inference and AI benchmarks, emphasizing reproducibility, error control, and the aggregation of sound evidence.

Rigor denotes the standards by which research claims are made trustworthy, but contemporary scholarship treats it neither as a single timeless property nor as a synonym for formal correctness alone. In education it is described as “often sought after but ill-defined,” historically variable, context dependent, and multidimensional rather than binary; in the history of mathematics it is likewise analyzed as “not a unitary phenomenon” (Garver, 2020, Toffoli et al., 2022). Recent work in AI and visualization extends this pluralization, framing rigor through six facets—epistemic, normative, conceptual, methodological, reporting, and interpretative—or through conceptual, epistemic, and operational rigor, while interpretivist design-study research defines it through the criteria INFORMED, REFLEXIVE, ABUNDANT, PLAUSIBLE, RESONANT, and TRANSPARENT (Olteanu et al., 17 Jun 2025, Nguyen, 19 May 2026, Meyer et al., 2019).

1. Conceptual structure and major frameworks

Across disciplines, rigor is increasingly formalized as a structured combination of dimensions rather than a unitary threshold.

Framework	Core dimensions or criteria	Source
Science education	content depth; cognitive demand	(Garver, 2020)
Classical Italian algebraic geometry	small-scale rigor; large-scale rigor; formal rigor; substantial rigor	(Toffoli et al., 2022)
Responsible-AI-informed framework	epistemic; normative; conceptual; methodological; reporting; interpretative rigor	(Olteanu et al., 17 Jun 2025)
Philosophy of AI	conceptual rigor; epistemic rigor; operational rigor	(Nguyen, 19 May 2026)
Visualization design study	INFORMED; REFLEXIVE; ABUNDANT; PLAUSIBLE; RESONANT; TRANSPARENT	(Meyer et al., 2019)

These frameworks differ in vocabulary but converge on several points. First, rigor is treated as relative to an “object of concern”: constructs, methods, evidence, reporting, interpretation, or social context. Second, rigor is not reducible to local procedural correctness. The Italian-school analysis distinguishes “small-scale” from “large-scale” rigor and “formal” from “substantial” rigor, thereby separating stepwise derivation from structural intelligibility and faithfulness to the mathematical object (Toffoli et al., 2022). The broader AI framework makes an analogous move by arguing that technically correct methods cannot rescue work built on unsound assumptions, unclear constructs, poor reporting, or unsupported inferences (Olteanu et al., 17 Jun 2025).

This suggests that rigor functions as a layered constraint on inquiry. Some layers concern local validity—whether a proof step, estimator, or benchmark protocol is correctly executed. Others concern upstream framing and downstream interpretation—what is being studied, why it matters, what assumptions are being imported, and what claims the evidence actually warrants.

2. Statistical and evidential rigor

In statistical inference, rigor is often recast as alignment between formal analysis and scientific meaning. A prominent example is the second-generation $p$-value, which replaces the point null with a pre-specified interval null $H_0:[\theta_0-\delta,\theta_0+\delta]$ representing scientifically negligible effects, and defines

$$p_\delta=\frac{|I\cap H_0|}{|I|}\times \max\left\{\frac{|I|}{2|H_0|},1\right\}.$$

Here $I$ is an interval estimate, usually an unadjusted $95\%$ confidence interval, and $p_\delta$ is the proportion of data-supported hypotheses that are also null hypotheses (Blume et al., 2017). Its interpretation is explicitly three-way: $p_\delta=0$ means compatibility only with meaningful alternatives, $p_\delta=1$ means compatibility only with null hypotheses, and $0<p_\delta<1$ denotes inconclusiveness. The method is described as a descriptive proportion rather than a posterior probability, and its rigor comes from forcing the interval null to be specified a priori, thereby preventing post hoc redefinition of what counts as meaningful (Blume et al., 2017).

The same paper argues that this structure changes error control. If $I$ is a $H_0:[\theta_0-\delta,\theta_0+\delta]$0 confidence interval, then under any null hypothesis inside the interval null,

$H_0:[\theta_0-\delta,\theta_0+\delta]$1

and if the true null lies in the interior of the null interval, this probability goes to zero with increasing sample size, a property the authors call inferential consistency (Blume et al., 2017). The broader claim is that many classical Type I errors arise from effects that are statistically distinguishable from a point null but scientifically indistinguishable from it.

A related conception appears in causal inference. The Causal Roadmap requires that the research question, causal model, causal effect, observed data structure, identifiability assumptions, statistical estimand, estimator, and interpretation be linked in a pre-specified workflow. To prevent ad hoc estimator choice, the paper advocates realistic, application-specific plasmode simulations conducted after data collection but before effect estimation, so that the Statistical Analysis Plan can pre-specify the exact estimation and inference algorithm on the basis of finite-sample performance rather than convenience or software defaults (Nance et al., 2023).

At meta-analytic scale, evidential rigor is formulated directly as a Bayesian estimand. A corpus-scale workflow defines rigor as a joint model-family Bayes-factor summary for the better-supported clean resolved branch of a bias-aware ensemble, where “clean” means absence of an explicit publication-selection or small-study-effect component. Under this definition, evidence for no effect can be highly rigorous, whereas weak or bias-dependent evidence scores poorly (Hester, 31 May 2026). This extends the statistical notion of rigor from estimator choice to the structure of evidence aggregation itself.

3. Rigor in AI and machine learning

In contemporary AI, rigor is frequently discussed as a response to benchmark inflation, hidden supervision, leakage, and overinterpretation. In unsupervised cross-lingual learning, one position paper argues that the field needs much stronger rigor in motivation, definition, and methodology. It challenges the practical narrative that many languages have abundant monolingual data but no usable cross-lingual signal, distinguishes monolingual from cross-lingual training signals, and states that “any systematic use of parallel data should not be considered UCL” (Artetxe et al., 2020). Its methodological recommendations include development on one language pair, minimizing target-pair parallel data, preferring unsupervised validation criteria where possible, and keeping the test set blind (Artetxe et al., 2020).

A parallel critique appears in Generative AI evaluation. The claim there is that static benchmark logic inherited from IID supervised learning does not match novelty-based generalization, and that leakage and contamination are now the primary threats to empirical validity. Public benchmarks are treated as effectively leaked once shared online; in response, the paper argues that AI competitions, with hidden test data, anti-cheating controls, expert-authored fresh tasks, prospective ground truth, and post-deadline data collection, now provide the field’s “gold standard” for empirical rigor (Sculley et al., 1 May 2025).

Instruction-tuning research exposes a different failure mode: confounding dataset-quality claims with arbitrary training choices. A study of LIMA and a 1,000-example Alpaca subset shows that different plausible hyperparameter settings can reverse the conclusion about which dataset is better, even with the same base model and the same evaluation benchmarks. The paper’s summary phrase is that arbitrary hyperparameter decisions can make “any arbitrary conclusion,” and it therefore argues that data-quality claims are credible only if the training setup is itself rigorously justified or locally optimized (Moon et al., 4 Mar 2025).

Medical LLM evaluation adds a domain-specific version of the same critique. Widely used datasets such as MedQA, MedMCQA, PubMedQA, and MMLU are argued to lack clinical realism, transparency, and robust validation, while public challenge questions are limited by small size, narrow scope, and exposure to training corpora. The paper therefore calls for secure, comprehensive, and representative datasets, collaborative governance, and hidden or access-controlled testing environments rather than reliance on public exam-style question answering as evidence of clinical competence (Alwakeel et al., 11 Jul 2025).

At the level of automated discovery, rigor is treated as something that must be structurally enforced. A functional architecture for AI-driven discovery introduces a Haskell eDSL in which sequential hypothesis testing and online FDR control are embedded in a Research monad, while “Declarative Scaffolding” generates rigid harnesses that prevent methodological errors such as exploration/validation leakage in LLM-generated imperative code (Sargsyan, 10 Nov 2025). The underlying claim is that in high-throughput automated science, prompt-level advice is insufficient; statistical and methodological invariants must be enforced by architecture.

4. Reflexive, interpretive, and design-centered rigor

Qualitative and design-oriented research reformulates rigor away from reliability-style convergence and toward reflexivity, provenance, and productive disagreement. In Reflexive Thematic Analysis, rigor is described as deliberate, reflexive, transparent, and dialogic interpretive work rather than speed, consensus, or inter-coder reliability. The system Reflexis was built around three design goals—Systematic In-the-Moment Reflexivity Support, Automated Code Evolution Tracking, and Collaborative Positionality Integration—and evaluated in a paired-analyst study with $H_0:[\theta_0-\delta,\theta_0+\delta]$2 (Ye et al., 21 Jan 2026). Reported findings include that $H_0:[\theta_0-\delta,\theta_0+\delta]$3 participants agreed the analysis history helped them see code changes over time and increased transparency, $H_0:[\theta_0-\delta,\theta_0+\delta]$4 agreed that discussion focus helped them notice meaningful disagreements and made discussion easier, and all displayed LLM outputs in the post hoc review were judged relevant rather than “correct” or “incorrect” (Ye et al., 21 Jan 2026).

Visualization design-study research develops a parallel framework. “Criteria for Rigor in Visualization Design Study” argues that design study should be understood as an interpretivist mode of inquiry capable of producing multiple, subjective, and socially constructed contributions, not merely novel software systems. From that stance it derives the six criteria INFORMED, REFLEXIVE, ABUNDANT, PLAUSIBLE, RESONANT, and TRANSPARENT (Meyer et al., 2019). A later EvoBio collaboration operationalizes three of these—ABUNDANT, REFLEXIVE, and TRANSPARENT—through immersive field study, reflexive memoing, reflective transcription, meticulous curation of artifacts, and a web-based “trrrace” that links claims directly to notes, sketches, and design materials (Rogers et al., 2020).

These works share a distinctive rejection: rigor is not equated with faster completion, maximal agreement, or the suppression of interpretation. Instead, disagreement can be methodologically productive, reflexive notes can be evidential artifacts, and transparency may require making the analytical journey inspectable rather than presenting only final themes or final systems. This suggests a conception of rigor in which friction is not noise but part of the epistemic work.

5. Historical variability and disciplinary forms of rigor

Historical studies show that rigor changes with what a field takes itself to be doing. In twentieth-century U.S. science education, rigor is modeled through two dimensions—content depth and cognitive demand—with the analogy

$H_0:[\theta_0-\delta,\theta_0+\delta]$5

where $H_0:[\theta_0-\delta,\theta_0+\delta]$6 is content depth, $H_0:[\theta_0-\delta,\theta_0+\delta]$7 is cognitive demand, and $H_0:[\theta_0-\delta,\theta_0+\delta]$8 is overall rigor (Garver, 2020). On this account, rigor is a continuum rather than a binary state, and “content depth is not simply ‘more content’, but rather generalizations, multiple perspectives and connections between different areas of content knowledge” (Garver, 2020). Garver’s historical argument is that when science education was framed as a social endeavor, it often achieved high cognitive demand but lower content depth, whereas the Cold War and Space Race era pushed science education toward the combination of deep content knowledge and high cognitive demand, and therefore “tended toward a higher level of rigor than the decades directly preceding or following it” (Garver, 2020).

A different historical lesson appears in classical Italian algebraic geometry. The paper on Castelnuovo, Enriques, and Severi argues that these mathematicians did value rigor, but not as a unitary notion. Enriques distinguished small-scale from large-scale rigor, while Severi distinguished formal from substantial rigor (Toffoli et al., 2022). Small-scale or formal rigor concerns explicit, local inferential articulation; large-scale or substantial rigor concerns grasp of the “built edifice,” the organic connections of the theory, and faithfulness to the mathematical facts. The paper further links these notions to two conceptions of objectivity: objectivity as faithfulness to facts and objectivity as intersubjectivity, that is, shareable public checkability (Toffoli et al., 2022).

The common implication is that rigor is historically produced by the aims of inquiry. Educational systems shift with sociopolitical motivations; mathematical communities shift with conceptions of proof, intuition, and communication. What counts as rigorous in one period may later appear insufficiently explicit, while still having captured something that a later formalism must reconstruct.

6. Infrastructure, governance, and the material conditions of rigor

Rigor is also infrastructural. In computational biomedicine, a policy paper argues that reproducibility, transparency, and rigor require open and discoverable data, open-source software, usable and documented tools, archivally stable repositories, reproducible workflows, living figures and papers, and institutional incentives that reward maintainable resources rather than novelty alone (Brito et al., 2020). Its recommendations span teaching, development and distribution, implementation, and incentives, and include SRA, GEO, Software Heritage Archive, Zenodo, Conda, Bioconda, Galaxy, CWL, Snakemake, Jupyter Notebook, MyBinder, Stencila, FAIR data principles, RRIDs, and OSI-approved software licenses (Brito et al., 2020).

Automated assessment tools reveal both the promise and the limit of procedural enforcement. A comparison of 11 tools across 9 rigor-related criteria—registration, inclusion/exclusion, blinding, randomization, power calculations, software use, open code, problematic cell lines, and baseline tables—shows that some tasks have clear winners, such as ODDPub for open code, SoftCite for software, SciScore for blinding, and a dedicated baseline-table extractor for baseline tables, while others benefit from combinations of tools, as in inclusion/exclusion and randomization (Eckmann et al., 23 Jul 2025). The paper’s key conceptual caution is that these systems mostly detect reported signals of rigor or transparency, not the underlying methodological rigor itself (Eckmann et al., 23 Jul 2025).

In recommender systems, WarpRec makes the same infrastructural point at industrial scale. It presents a backend-agnostic framework with 55 algorithms, 40 metrics, and 19 filtering and splitting strategies, designed so that local experimentation and distributed execution share the same logical pipeline (Avolio et al., 19 Feb 2026). Its evaluation module includes significance testing and both Bonferroni and Benjamini–Hochberg correction, while CodeCarbon is integrated for real-time energy tracking. The framework thereby treats scalability, reproducibility, and ecological responsibility as compatible requirements rather than trade-offs (Avolio et al., 19 Feb 2026).

Taken together, these works indicate that rigor depends not only on concepts and proofs, but also on repositories, identifiers, containers, workflows, hidden test sets, logging systems, governance structures, and sustainable computational practice. A plausible implication is that rigor is increasingly inseparable from the systems that preserve, expose, and constrain research behavior across the full lifecycle of knowledge production.