ProdCodeBench: Production AI Coding Benchmark

• ProdCodeBench is a production-derived benchmark that pairs authentic developer prompts with landed code diffs and execution-based fail-to-pass tests, ensuring realistic evaluation.
• Its methodology features a curation pipeline with multi-run stability checks, test relevance validation, and automated task filtering to maintain execution reliability.
• Empirical findings reveal that robust IDE tooling and active validation tool use significantly enhance solve rates in complex monorepo environments.

Searching arXiv for the specified paper and a few closely related benchmarks to ground the article with current identifiers. Tool call: arxiv_search(query="ProdCodeBench A Production-Derived Benchmark for Evaluating AI Coding Agents", max_results=5) ProdCodeBench is a production-derived benchmark for evaluating AI coding agents against the realities of enterprise software development. It is built from real sessions with a production AI coding assistant and pairs verbatim developer prompts with their corresponding landed code changes and execution-based fail-to-pass tests within a large industrial monorepo. The benchmark is intended to provide fast, reproducible, and production-relevant evaluation signals in settings where online A/B tests are slow and resource-intensive and shadow deployments suffer from non-determinism. Its central methodological contribution is a curation pipeline that preserves prompt realism while enforcing execution-based reliability through test relevance validation, multi-run stability checks, and diff backout during evaluation (Jha et al., 2 Apr 2026).

1. Motivation and benchmark positioning

ProdCodeBench is motivated by a mismatch between industrial coding-assistant deployment needs and the properties of existing public offline benchmarks. Organizations require evaluation signals that are reproducible, closely aligned with production workloads, and sensitive to model, configuration, and infrastructure changes. Existing benchmarks are reproducible, but the paper argues that they diverge from real usage in programming language distribution, prompt style, and codebase structure (Jha et al., 2 Apr 2026).

Several benchmark families are contrasted with this production-derived approach. HumanEval and MBPP are described as single-function code synthesis in Python from docstrings, with minimal repository context, minimal tool interaction, and no polyglot distribution or monorepo issues. APPS is similarly characterized as focusing on algorithmic problems rather than integration into large codebases. SWE-bench is repository-level and based on GitHub issues paired with pull requests, but is predominantly Python, uses issue descriptions written for human collaborators rather than prompts intended for an AI agent, and evaluates public multi-repo projects rather than large-scale monorepos with distributed tooling. RepoBench emphasizes repository-level code completion via retrieval and generation rather than mapping real prompts to landed diffs or fail-to-pass tests. Defects4J provides strong repair signals, especially in Java, but lacks realistic assistant prompts and multi-language monorepo constraints, and frames tasks as bug repair without preserving natural prompts from production sessions (Jha et al., 2 Apr 2026).

Benchmark	Task framing	Mismatch highlighted by ProdCodeBench
HumanEval / MBPP	Single-function synthesis from docstrings	Minimal repository context; no monorepo or polyglot realism
APPS	Algorithmic problems	Not integrated into large codebases
SWE-bench / SWE-bench Verified	GitHub issues paired with PRs	Human-oriented issue prose; public multi-repo setting; predominantly Python
RepoBench	Retrieval + generation for repository completion	Completion-centric rather than prompt-to-diff evaluation
Defects4J	Curated bug-fix corpus with tests	No realistic assistant prompts; no multi-language monorepo constraints

The distinguishing characteristics of ProdCodeBench are therefore not merely its use of tests, but the specific combination of verbatim IDE prompts, landed production diffs, and monorepo-grounded execution signals. Prompt style is a central differentiator: the prompts are single-turn requests typed into an IDE-integrated assistant by professional developers and often contain enterprise jargon and implicit context. This suggests that benchmark difficulty derives not only from code synthesis or repair, but also from recovering unstated project conventions and repository-local assumptions.

2. Construction methodology

The dataset is sourced from authentic IDE-integrated assistant conversations and focuses on single-turn sessions in which a developer supplied one natural-language prompt and the agent produced a change that later landed on the main branch. Instrumentation logs keystrokes and AI provenance, defined as the fraction of characters in the landed diff originating from accepted AI suggestions. These logs are used to establish a one-to-one mapping between a prompt and the landed code change and to quantify AI contribution. Diffs tied to multiple conversations are excluded to preserve clear intent-change alignment (Jha et al., 2 Apr 2026).

A key design choice is the backout mechanism. During evaluation, the environment removes the landed changes and runs agents on the backed-out commit. This is intended to prevent leakage of the ground-truth solution. Diffs that cannot be cleanly reverted because of conflicts are excluded. The paper also emphasizes “time travel” constraints in industrial monorepos: tooling and indices are ephemeral, services index only recent code, toolchain binaries expire, and CI logs and contextual signals are transient. For that reason, the benchmark is designed as a rolling benchmark whose samples are periodically refreshed to remain executable, avoid contamination, and stay in-distribution (Jha et al., 2 Apr 2026).

Task filtering is partly automated through LLM-based task classification. The label taxonomy begins with a binary distinction between testable and non-testable tasks, supplemented by finer task-type labels such as bug_fix, feature_request, refactor, and debug_investigation. Testable tasks are defined as changes that can be validated via automated tests, including bug fixes, feature additions, and refactorings. Non-testable tasks include documentation-only edits, UI-only tweaks, test generation requests, and investigative prompts without an intended code change. Manual validation by two experienced annotators showed 96.67% agreement between the classifier and human consensus on testability over 30 prompts, with the sole disagreement being an investigative prompt labeled testable by the classifier (Jha et al., 2 Apr 2026).

Test selection proceeds through a separate relevance-validation pipeline. A probabilistic retrieval step proposes candidate tests while explicitly excluding flaky, failing, or disabled tests and ensuring executability on the current master. A Test Relevance Agent, equipped with a diff-fetch tool and a code search subagent, checks whether each candidate test exercises the changed code directly or indirectly, or resides in the same module or suite. The agent does not execute tests. Remaining tests are then executed on both pre-change and post-change versions multiple times in order to classify them and filter out spurious failures. Inconsistent tests are removed. A no-op validation of the overall pipeline yields a 0.0% solve rate, which the paper treats as evidence of signal integrity. On a sample of 15 (diff, test) pairs, human validation found two false negatives and one false positive relative to the agentic relevance classifier, for overall agreement of about 80%; the disagreements were at the boundary of module-level versus function-level relevance (Jha et al., 2 Apr 2026).

3. Task representation and monorepo evaluation design

Each curated ProdCodeBench task contains three elements: the verbatim developer prompt from a single-turn IDE session, the landed code change recorded in version control but hidden during evaluation, and a set of executable tests. The tests are partitioned into fail-to-pass (F2P) and pass-to-pass (P2P) categories. F2P tests fail on the pre-change version and pass on the post-change version; they define correctness for evaluation. P2P tests pass on both versions and serve as a regression oracle to ensure that generated changes do not break existing functionality (Jha et al., 2 Apr 2026).

The paper is explicit that “Fail-to-Pass” characterizes test behavior rather than task semantics. An F2P test may correspond to a bug fix, but it may also reflect newly added behavior. This matters because ProdCodeBench is not restricted to bug-repair benchmarking. Its task mix includes bug fixes, feature requests, and refactorings, and its evaluation signal is anchored in how tests behave across pre-change and post-change states rather than in a narrower notion of defect correction.

The monorepo setting shapes both context exposure and isolation. Agents operate with repository navigation tools such as file read and search, along with localized diagnostics, so that evaluation does not rely on global or stale indices. Test retrieval prioritizes tests executable on the current master and affected by the change, thereby avoiding incidental failures. Isolation is enforced by backing out the diff and auditing tools that might reveal the backout patch. The paper notes that tools exposing local diffs must be guarded carefully to avoid leaking the solution (Jha et al., 2 Apr 2026).

Two harnesses are compared. Agent-Basic provides a lightweight toolset with file read and search plus limited validation. Agent-IDE exposes a richer IDE-like toolset, described as having approximately three times more tools, including code navigation, local diagnostics, formatting, knowledge search, validation tools such as “Validate Changes,” and a “Todo Write” tool. The tasks are harness-agnostic and can support both internal agents and external systems such as Claude Code. Codebase-specific verification mechanisms, including unit test frameworks, linters, and static analyzers, are exposed as tools to the agent, while tests are automatically executed at the end of a trajectory (Jha et al., 2 Apr 2026).

4. Dataset composition and evaluation protocol

ProdCodeBench spans seven programming languages chosen to mirror enterprise distribution, although the paper does not disclose the language names or proportions. All tasks are situated in a large monorepo, and samples are refreshed in a rolling design to maintain executability and distributional relevance. The task mix covers bug fixes, feature requests, and refactorings, but the manuscript does not provide a numeric table for the distribution. Likewise, it does not provide public train, validation, and test splits; the benchmark is curated for evaluation and is not publicly released (Jha et al., 2 Apr 2026).

Test composition is partially quantified. Approximately 75% of tasks have at least one F2P test, while the remaining roughly 25% rely solely on P2P tests for regression validation. Unreliable tests are excluded. The paper does not disclose sample counts, prompt lengths, commit sizes, files touched, or tests per sample in numeric form.

The interaction protocol is single-turn. Agents receive the prompt and may use tools to navigate and edit the codebase. At the end of the trajectory, tests are executed to determine success. No LLM judges are used. Depending on the harness, agents may read files, search repositories, run shell commands, perform edits or structured replacements, request validation through static checks and tests, and, in Agent-IDE, write todo notes and use formatting and knowledge-search tools. The paper notes that GPT 5.1 Codex often exhausts maximum iterations on Agent-Basic because of protracted search, although exact iteration or time budgets are not specified (Jha et al., 2 Apr 2026).

The principal metric is solve rate, defined as binary success over tasks for which all relevant F2P tests pass after the agent’s patch is applied. There is no partial credit:

$$S = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}[\text{task}_i\ \text{solved}]$$

The paper also defines pass@k for settings with multiple attempts:

$$\mathrm{pass@k} = 1 - \frac{\binom{n-m}{k}}{\binom{n}{k}}$$

where $n$ is the total number of attempts and $m$ is the number of successful attempts. Confidence intervals are reported as 95% CIs over proportions using a Wilson interval:

$$\mathrm{CI} = \frac{ \left(\hat{p} + \frac{z^2}{2N}\right) \pm z\sqrt{\frac{\hat{p}(1-\hat{p})}{N} + \frac{z^2}{4N^2}} }{ 1 + \frac{z^2}{N} }$$

Each model is evaluated three times to quantify variance. Test suites are rerun on pre-change and post-change versions multiple times to filter flakiness and setup or caching artifacts, but the exact rerun counts and thresholds are not disclosed (Jha et al., 2 Apr 2026).

5. Empirical findings

The paper reports a systematic analysis of four foundation models and states that solve rates range from approximately 53.2% to 72.2%, with Claude Opus 4.5 highest overall. In the frontier model comparison, Claude Opus attains the highest solve rate, followed by Claude Sonnet, then Claude Haiku, with GPT Codex trailing when all are run on the same harness. Exact per-model percentages and per-language or per-task-type breakdowns are redacted in the manuscript version described in the source material, but the ordering and the overall range are explicit (Jha et al., 2 Apr 2026).

A central empirical finding concerns tool use. Claude models exhibit higher rates of validation and test execution than GPT Codex. The paper reports the distribution of tool calls for GPT 5.1 Codex versus Opus 4.6. Under Agent-Basic, Read File accounts for 28.9% versus 36.8% of tool calls; Search Files 48.5% versus 31.4%; Shell 8.6% versus 17.0%; and Validate Changes 1.5% versus 4.0%. Under Agent-IDE, Read File is 27.8% versus 26.5%; Search Files 11.6% versus 13.4%; Shell 5.8% versus 6.1%; and Validate Changes 5.8% versus 6.3%. Agent-IDE-only tools include Todo Write at 14.6% versus 13.7% and Str Replace Edit at 8.7% versus 9.4% (Jha et al., 2 Apr 2026).

These usage patterns are tied to harness effects. Agent-Basic shows Opus 4.6 significantly outperforming GPT 5.1 Codex, and GPT is reported to often exhaust iterations due to long searches. Under Agent-IDE, both models perform similarly and outperform their Agent-Basic variants. The paper attributes this to stronger navigation, which reduces search calls and raises solve rates. It also reports that supplying developer-authored markdown “Context Files” improves solve rates on Agent-Basic, especially for company-specific languages and conventions, with smaller gains in more standardized language ecosystems (Jha et al., 2 Apr 2026).

The benchmark’s interpretive claim is that iterative verification matters. Models that more frequently invoke validation tools such as static checks and tests achieve higher solve rates. A plausible implication is that, in proprietary monorepos unfamiliar to externally trained models, runtime and static feedback serve as a compensating mechanism for incomplete prior knowledge. The paper therefore treats validation behavior not as an incidental tool preference but as a meaningful determinant of agent effectiveness.

6. Reliability, limitations, and broader implications

ProdCodeBench emphasizes construct validity through multi-stage curation, but the paper also identifies several limitations. The resulting benchmark contains only a few hundred high-quality instances, which limits its suitability for reinforcement learning workflows that typically require tens of thousands of trajectories. The benchmark is also single-turn only, excluding multi-step iterative tasks that are common in real development and potentially making the benchmark easier than broader production usage (Jha et al., 2 Apr 2026).

Internal-validity concerns arise from both tooling and data provenance. Any tool that exposes local diffs could inadvertently reveal the backed-out patch, so harnesses must be audited to prevent solution leakage. The paper also notes an AI provenance confound: roughly 65% of retained diffs have 100% AI provenance, meaning every character originated from accepted AI suggestions. These instances function partly as self-consistency checks rather than fully independent evaluations, since measured solve rates may reflect whether the agent can reproduce prior AI-generated solutions.

External validity is constrained in two ways. First, frontier models reach around 70% solve rate, which the paper identifies as a potential ceiling effect that reduces discriminative power among top models. Second, the language distribution and development conventions are organization-specific, so results may not generalize directly to other environments. Reliability is strengthened by multi-run filtering of flaky tests and three-run evaluation per model, but the rolling-refresh design that preserves executability and avoids contamination also complicates direct longitudinal comparison over time (Jha et al., 2 Apr 2026).

The benchmark is not publicly released. Instead, the paper presents a replicable methodology that organizations can apply to their own production sessions. The documented pipeline begins with collecting single-turn IDE assistant conversations with landed diffs and AI provenance; filtering prompts that reference diff IDs or system/template messages; running taskability classification with manual spot checks; discovering candidate tests; applying the Test Relevance Agent over (diff, test) pairs; executing tests on pre-change and post-change states multiple times to classify F2P and P2P tests and remove inconsistent ones; backing out the diff from current master; preparing the agent harness; and computing solve rate over the F2P subset. The required environment includes access to the monorepo, the ability to back out landed diffs cleanly, test runners and static analyzers, instrumented code-search tools, and snapshot or restore capabilities for evaluation environments (Jha et al., 2 Apr 2026).

Taken together, ProdCodeBench formalizes a benchmark design in which realism comes from preserving production prompts and production diffs, while reliability comes from execution-based validation in a backed-out monorepo state. Its reported findings support two practical conclusions: stronger IDE-like tooling can narrow apparent model gaps, and exposing codebase-specific verification mechanisms can materially improve agent behavior in unfamiliar repositories.