RigorBench: AI Coding Process Discipline

• RigorBench is a benchmark that measures the process discipline of autonomous AI coding agents, focusing on planning, verification, recovery, abstention, and atomic transitions.
• It evaluates agents across 30 tasks in five stress categories, demonstrating how disciplined engineering practices lead to improved outcome accuracy and token efficiency.
• Its composite RigorScore aggregates weighted metrics from five pillars, providing quantitative evidence that robust process quality is crucial for reliable code generation.

RigorBench is a benchmark for evaluating engineering process discipline in autonomous AI coding agents. Introduced in "RigorBench: Benchmarking Engineering Process Discipline in Autonomous AI Coding Agents," it is presented as the first benchmark designed to measure process discipline rather than treating software-engineering performance as an outcome-only question of whether generated code passes tests or resolves issues. The benchmark evaluates agentic coding harnesses across five pillars—Planning Fidelity, Verification Coverage, Recovery Efficiency, Abstention Quality, and Atomic Transition Integrity—and aggregates them into a composite RigorScore via a weighted sum over scores in $[0,1]$. It comprises 30 tasks spanning five categories and is evaluated in a controlled with/without design against baseline coding assistants. A distinct and unrelated usage of the same name appears in a benchmark for interval computation libraries (Madiraju et al., 21 Jun 2026, Tang et al., 2021).

1. Scope and motivating problem

RigorBench for autonomous coding agents is motivated by a specific critique of prevailing evaluation practice: existing benchmarks assess these agents almost exclusively on outcome correctness, such as whether the generated patch passes tests or whether an issue is resolved. The benchmark’s central claim is that this lens is insufficient, because an agent that reaches a correct answer through reckless trial-and-error, without planning, verification, or graceful recovery, is less reliable than one that follows sound engineering discipline. In this formulation, process quality is not treated as incidental metadata but as a first-class evaluation target (Madiraju et al., 21 Jun 2026).

The benchmark is situated in the context of agentic coding harnesses that augment underlying LLMs for real-world software-engineering tasks. The paper specifically names Agent-Skills, Superpowers, and Agent-Rigor as such harnesses. RigorBench is therefore not a benchmark of base-model capability in isolation; it is a benchmark of the engineering process discipline induced by higher-level orchestration, workflow scaffolding, and recovery behavior.

A second motivation is methodological. Traditional code-generation benchmarks reward end-products, whereas the benchmark’s framing invokes a broader software-engineering concern with robustness, maintainability, and cost. The benchmark is designed to make the hidden “how” of AI-powered coding visible and measurable. A plausible implication is that RigorBench is intended to bridge evaluation criteria used in offline benchmark settings and the reliability demands of deployment settings, but the benchmark itself operationalizes this idea only through the specified process pillars and downstream outcome measurements.

2. Measurement model and RigorScore

RigorBench decomposes process quality into five orthogonal dimensions. Each pillar is scored on $[0,1]$ using automated and LLM-judge–assessed signals, and the final score is a weighted sum (Madiraju et al., 21 Jun 2026).

Pillar	Weight	Key sub-metrics
Planning Fidelity (PF)	0.20	PAC, DQ, PEA
Verification Coverage (VC)	0.25	TCR, $\Delta C$, RT
Recovery Efficiency (RE)	0.25	$f(\mathrm{RAC})$, SD, $(1-\mathrm{TWR})$
Abstention Quality (AQ)	0.15	Correct Abstention, False Confidence Penalty, Clarification Seeking
Atomic Transition Integrity (ATI)	0.15	BH, TS, CH

Planning Fidelity measures whether the agent externalizes a plan, whether that plan is decomposed at an appropriate granularity, and whether execution follows the planned order. Its sub-metrics are Plan Artifact Creation (PAC), Decomposition Quality (DQ), and Plan–Execution Alignment (PEA). PAC is binary, with value $1$ if an explicit plan document or TODO list appears before coding and $0.5$ for inline reasoning. DQ is scored by a 4-point rubric from “no decomposition” to “fine-grained atomic steps.” PEA is Kendall’s $\tau$ rank-correlation between planned steps and the actual code-change sequence. The pillar is defined as

$$\mathrm{PF} = 0.30\cdot \mathrm{PAC} + 0.35\cdot \mathrm{DQ} + 0.35\cdot \mathrm{PEA}.$$

Verification Coverage measures whether the agent validates what it implements and whether tests are traceable to requirements. Its sub-metrics are Test Creation Rate (TCR), Coverage Delta ($\Delta C$), and Requirements Traceability (RT). TCR is the fraction of implemented features or functions covered by at least one new test; $[0,1]$0 is the increase in line or branch coverage from agent-written tests; RT is the fraction of task requirements each linked to at least one test case. The pillar is

$[0,1]$1

Recovery Efficiency targets error handling under failure. Its sub-metrics are Recovery Attempt Count (RAC), Strategy Diversity (SD), and Token Waste Ratio (TWR). RAC counts error-fix cycles and is mapped by a decreasing function $[0,1]$2 so that fewer attempts score higher. SD counts distinct recovery strategies. TWR is the tokens spent on failed recoveries divided by total tokens, so lower is better. The pillar is

$[0,1]$3

Abstention Quality is evaluated only on six “impossible or ambiguous” tasks. It rewards correct abstention when a task is provably impossible, assigns a score of $[0,1]$4 to bogus confident proposals, and also rewards clarification seeking when inputs are genuinely ambiguous. Unlike the other pillars, no closed-form linear combination is given in the provided specification.

Atomic Transition Integrity measures whether intermediate states remain healthy while the agent is modifying the codebase. Its sub-metrics are Build Health (BH), Test-Suite Stability (TS), and Commit Hygiene (CH). BH is the fraction of intermediate snapshots that compile or build; TS is the fraction in which no previously passing test regressed; CH evaluates whether commits are atomic and have descriptive messages using a rubric. The pillar is

$[0,1]$5

The composite score is defined formally as

$[0,1]$6

In the paper’s abstract notation, with $[0,1]$7,

$[0,1]$8

3. Task suite and stress categories

The benchmark contains 30 realistic engineering tasks, grouped into five categories with six tasks each. Every pillar is scored on every task, but each category is constructed to stress one or two dimensions more heavily. This design makes RigorBench a structured stress suite rather than a flat collection of unrelated issues (Madiraju et al., 21 Jun 2026).

Plan-Then-Build targets Planning Fidelity and Atomic Transition Integrity at Medium–Hard difficulty. The example given is “Multi-Service API Gateway” in Node.js/Express, where the agent must first write a plan decomposing routing, rate limiting, logging, circuit-breaking, and health-checks, and then implement each feature with build-passing commits.

Verify-Or-Die targets Verification Coverage and Recovery Efficiency at Medium difficulty. The example is “Off-By-One Calendar” in Python, involving subtle leap-year and month-end bugs. The agent is expected to write failing edge-case tests before patching.

Doom-Loop Gauntlet concentrates on Recovery Efficiency at Hard difficulty. The example is “Cryptographic Hash Mismatch” in Rust, described as an intermittent encoding bug that defeats successive naive fixes. The agent must diversify recovery strategies and converge efficiently.

Know When to Fold targets Abstention Quality with Variable difficulty. The example task, “Conflicting Requirements” for sorting, requires stable, in-place, $[0,1]$9 worst-case behavior with $\Delta C$0 space and is described as provably impossible. The expected response is to abstain, explain the theoretical trade-offs, or propose the closest feasible alternative.

Don’t Break the Build emphasizes Atomic Transition Integrity and Planning Fidelity at Hard difficulty. The example is “Database Migration Refactor” in Django, where the agent must migrate raw-SQL queries into the ORM across multiple modules while preserving all existing tests at each step and producing atomic, descriptive commits.

This suite structure encodes a particular conception of coding-agent competence: planning, verification, recovery, abstention, and maintenance of healthy intermediate states are all treated as measurable properties of execution trajectories rather than as latent qualities inferred from final correctness alone.

4. Experimental protocol and evaluated harnesses

The experimental design compares four harnesses, all using the same underlying LLM described as “June 2025 state-of-the-art,” and all executed in isolated Docker sandboxes. The harnesses are: a Baseline ReAct loop with no process scaffolding; Agent-Skills, described as tool-based skill modules; Superpowers, described as extended context and prompt management; and Agent-Rigor, described as a markdown-driven six-phase lifecycle enforcing explicit plans, tests, and commits (Madiraju et al., 21 Jun 2026).

Each harness tackled all 30 tasks, yielding 120 total runs. The benchmark logged full trajectories, including plans, code diffs, test and build invocations, token counts, and timestamps. This logging design is integral to the benchmark: because RigorBench measures process discipline, the trace itself is part of the evaluand rather than merely a debugging artifact.

Two broad families of metrics were collected. Process quality was measured through the five pillars and aggregated into RigorScore. Outcome quality was measured through task-specific correctness, including test-pass rate, feature completeness, and regression absence. The paper characterizes the setup as a controlled with/without design against baseline coding assistants. In effect, the benchmark is structured to compare disciplined harnesses against less scaffolded execution loops while holding the underlying LLM constant.

The released scoring stack combines heuristics with an LLM-as-judge ensemble using predefined rubrics. This is a notable methodological choice because some properties being scored—such as decomposition quality or commit hygiene—are not reducible to purely syntactic checks, while others—such as build health or coverage deltas—are directly instrumentable.

5. Quantitative findings and their interpretation

The central quantitative result is that process discipline, as measured by RigorScore, increased by 41% on average under disciplined harnesses versus the baseline, while downstream outcome correctness rose by 17% when agents followed disciplined processes. The paper also reports a strong process–outcome correlation, with Pearson $\Delta C$1 and $\Delta C$2, together with the linear fit

$\Delta C$3

These results are presented as the first quantitative evidence that how agents code matters as much as what they produce (Madiraju et al., 21 Jun 2026).

At the pillar level, Planning Fidelity improved by $\Delta C$4 mean absolute, from approximately $\Delta C$5 for the baseline to approximately $\Delta C$6 under disciplined harnesses. Abstention Quality improved by $\Delta C$7 mean absolute, but the benchmark notes that this change came from a near-zero baseline on impossible tasks to approximately $\Delta C$8. Recovery Efficiency and Atomic Transition Integrity are described as having smaller but meaningful improvements.

The benchmark also reports a token-efficiency result that runs against a common assumption that added planning and testing necessarily increase overhead: disciplined agents used 12% fewer total tokens despite the extra planning and testing steps. This suggests that better process discipline reduced wasteful retries and failed recovery effort more than it increased up-front deliberation and verification cost.

These findings do not eliminate the importance of outcome correctness; rather, they reframe it. In RigorBench, outcome quality remains a measured variable, but it is analyzed jointly with process quality. The benchmark thereby treats planning, verification, recovery, abstention, and atomic state transitions as explanatory dimensions of reliability rather than as secondary implementation details.

6. Open-source release and adoption workflow

RigorBench is released under an open-source license and includes the full benchmark plus auxiliary tooling required to reproduce the evaluation pipeline. The release contains the full task repository with 30 starter projects, detailed task specifications, and hidden test suites; an automated trajectory logger and parser that captures code diffs, commands, tokens, and time; a scoring pipeline combining heuristics with an LLM-as-judge ensemble and predefined rubrics; comprehensive scoring rubrics for every sub-metric and pillar; and example analyses together with baseline harness implementations (Madiraju et al., 21 Jun 2026).

The benchmark’s adoption procedure is specified as a four-step workflow. First, an agent run is instrumented to emit the same trajectory logs, including stdout and stderr, file diffs, test and build commands, and token counts. Second, those logs are fed into the RigorBench parser to extract signals such as plans, tests added, errors and recoveries, build states, commits, and abstentions. Third, the pillar scorers compute PF, VC, RE, AQ, and ATI using the published rubrics and the LLM-judge ensemble. Fourth, these scores are aggregated into a RigorScore and compared against published baselines or against a local “with/without discipline” ablation.

This release strategy makes the benchmark not only an evaluation dataset but also a trajectory-analysis framework. A plausible implication is that RigorBench can function as a regression-testing apparatus for harness design: because it logs and scores intermediate behavior, changes to scaffolding, prompting, or tool orchestration can be evaluated for process effects even when headline pass rates move only modestly.

7. Distinct benchmark using the same name

The name RigorBench has also been used for a separate benchmark in rigorous numerics: "A Cross-Platform Benchmark for Interval Computation Libraries" (Tang et al., 2021). In that work, RigorBench refers to a benchmark for interval arithmetic implementations in C and C++, not to autonomous coding agents.

That interval-computation RigorBench is described as the first benchmark for interval computations, coupled with reference solutions computed with exact arithmetic, and compares popular libraries over different architectures, operating systems, and compilers. Its stated goals include measuring correctness, performance, interval-widths, consistency, and portability. The benchmark contains 132 problems in total, including basic arithmetic operations, transcendental functions, composite expressions, and 104 real-world floating-point benchmarks imported from FPBench. Reference solutions are obtained through exact rational arithmetic with GMP and containment queries evaluated in Mathematica.

The coexistence of these two usages is primarily terminological rather than methodological. The 2026 benchmark addresses engineering process discipline in agentic coding, whereas the 2021 benchmark addresses correctness and portability of interval computation libraries across platforms. The shared name does not indicate a shared evaluation framework, task domain, or metric design.