- The paper introduces Harness-IF, a benchmark that evaluates 642 atomic constraints individually across system prompts, tools, skills, project files, and user turns rather than reducing agent performance to task success.
- The paper finds that plain accuracy overstates compliance with against-prior rules by 3.6–7.4 percentage points across 12 frontier models, while AP-Acc ranges from 66.1% to 78.6% and changes several adjacent rankings.
- The paper shows that measurement reliability limits fine-grained leaderboard claims: judge replacement produced only 62.1% agreement, and round-to-round variation exceeded the full 13.7-point leaderboard spread, supporting robust pattern comparisons over exact rankings.
Harness-IF is a benchmark that reframes instruction following in coding agents as a rule-level, surface-aware measurement problem. Rather than collapsing a long agentic trajectory into a single task-success signal, it scores each applicable operational rule individually and asks where that rule was delivered. The central methodological contribution is a prior-control metric, Against-Prior Accuracy (AP-Acc), which separates genuine compliance from behavior the model would have exhibited anyway. Across 12 frontier models, aggregate accuracy systematically overstates compliance on rules that oppose unprompted defaults, by a model-specific margin of 3.6 to 7.4 points.
Motivation and positioning
Deployed coding agents read instructions from multiple sources: system prompts, tool descriptions, skill descriptions, project files such as CLAUDE.md, user turns, and harness defaults. Existing instruction-following benchmarks (IFEval, FollowBench, InfoBench, ComplexBench) concentrate constraints in the user turn; coding-agent benchmarks (SWE-bench family, Terminal-Bench) emphasize final task success; instruction-hierarchy work such as IHEval tests compliance with a prescribed message hierarchy over synthetic conflicts. Harness-IF differs on two axes: it treats the delivery surface as an evaluation dimension while holding rule semantics fixed, and it scores one verdict per applicable rule per run rather than per task. The paper is explicit that surface placement and instruction hierarchy are distinct constructs—hierarchy concerns which privileged source prevails under conflict, whereas Harness-IF measures distributed constraints across executable workspaces.
Benchmark construction
The benchmark rests on a library of 642 atomic constraints tagged along eight axes: rule family (seven families including output control, workflow, code style, conditional logic, tool use, quantitative limits, professional writing), logical modality (require, forbid, conditional-require, limit-max/min, prefer, allow), behavioral prior (align/against/neutral), observability, verifiability, universality, surface fit, and pre-authored surface variants. Constraints were seeded from a hash-pinned manual survey of public GitHub project-instruction documents (CLAUDE.md, AGENTS.md, CONTRIBUTING.md, tool schemas), atomized into project-authored rules rather than copied text; LLM-assisted proposals with human review filled under-covered taxonomy cells.
The evaluated coding panel comprises 60 multi-turn items (retained from an 80-item working set after quality review), each injecting 25–35 rules of which 10–27 are scorable. Across the panel, 302 distinct library rules are instantiated and 256 receive at least one verdict. Prior labels are deliberately mixed—44% against-prior—so defaults alone cannot pass the panel. The paper concedes two selection issues plainly: item selection used observed discriminativeness, introducing selection optimism, and the panel instantiates fewer than half the library's rules, so claims are scoped to this panel.
Metrics
Rule verdicts come from six scoring methods: regex, AST, cross-file, command-output checks (13.3% of eligible verdicts), hybrid deterministic-plus-judge refinement, and rubric-based GPT-5.2 judging with three-vote majority at temperature 0.3 (86.8% of rows involve the judge). Four metrics are reported over the same eligible verdicts: plain accuracy, filtered accuracy (F-Acc, dropping non-discriminating item–rule pairs), discrimination-weighted accuracy (DW-Acc), and AP-Acc, restricted to against-prior rules. A cascade-dedup pass prevents one missing artifact from multiplying dependent failures, and a cascade-fairness audit promotes untestable-design-gap rules out of denominators.
Main results
The main panel evaluates 12 frontier builds over 60 items × 3 rounds, yielding 2,160 runs and 40,104 verdict rows (37,616 eligible binary verdicts, 19,449 against-prior). Accuracy spans 72.1% (StepFun-3.5) to 85.9% (Claude-Opus-4.7); AP-Acc spans 66.1% to 78.6%. Every model shows a positive Acc−AP-Acc gap, mean +5.81 points, and the gap survives a common-support analysis retaining 72.7% of observations with item-clustered bootstrap intervals positive for all 12 models (lower bounds +1.06 to +4.83 points). Because the inflation varies twofold across models, prior control leaves the top build unchanged but exchanges three adjacent rank pairs (2–3, 4–5, 11–12). Models largely agree on which rules are hard: per-rule pass-rate vectors correlate with the cohort mean at 0.57–0.89 (mean 0.80).
| Model |
Acc |
AP-Acc |
Δ |
| Claude-Opus-4.7 |
85.9 |
78.6 |
+7.3 |
| GPT-5.5 |
83.1 |
77.0 |
+6.1 |
| Claude-Sonnet-4.6 |
82.5 |
78.5 |
+4.0 |
| GLM-5.1 |
78.8 |
75.2 |
+3.6 |
| Seed-2.0-Pro |
73.6 |
66.1 |
+7.4 |
| StepFun-3.5 |
72.1 |
66.2 |
+5.9 |
Failure decomposition, computed from rule modality alone without free-text classification, shows shortfall rules (requiring an action or minimum) absorb 77.1% of the 8,440 failures versus 20.8% for overstep rules—but the paper correctly notes this reflects exposure (27,306 shortfall instances vs. 8,443 overstep instances) since failure rates are similar (23.8% vs. 20.8%). By family, output control combines the largest failure share (27.6%) with the lowest pass rate (70.9%, lowest for 11 of 12 builds); workflow's large share follows from panel size. By modality, commanding rules are hardest (76.0%) against preferences (90.6%), consistent with default alignment.
Surface precedence (E0)
A separate counterbalanced conflict pilot, E0, uses nine older builds, four synthetic conflict pairs, 916 runs, and deterministic-only scoring. Pooled Bradley–Terry ranks place system prompts, project files, and user instructions in an exact three-way tie (mean rank 2.22 each), ahead of tool descriptions (3.78) and skill descriptions (4.56). This ordering contradicts simple prompt-depth accounts, since the last-placed user turn only ties for first. The result survives direction-specific fits, leave-one-out analyses, equal-cell weighting, and four conservative error assignments; a crossed model-and-pair bootstrap reproduces the full ordering in 9,652/10,000 resamples. However, only 6/9 individual-build fits reproduce it exactly, and the three leading surfaces are not statistically distinguished, so the authors interpret E0 as a pooled cross-build tendency, not a universal hierarchy. Notably, E0 places skill descriptions last under conflict while the main panel finds them easier than system prompts in isolation—a surface can be easy to comply with alone yet lose a direct conflict.
Reliability
The paper is unusually candid about measurement uncertainty. Test–retest ICC is 0.725 across agents but 0.599 across agent–item cells, with a mean per-cell round-to-round range of 15.6 points exceeding the entire 13.7-point leaderboard spread—so single-round adjacent comparisons are uninformative. Judge-swap sensitivity is the dominant concern: swapping GPT-5.2 for Claude Opus 4.7 yields only 62.1% agreement (κ=0.163) on 116 paired clean verdicts, with per-agent pass-rate deltas up to ±40 points. Since 86.8% of verdicts involve the judge, absolute levels are instrument-specific; the paper therefore rests its claims on cross-model patterns sharing the instrument and treats displayed rankings as point estimates. Human-reference calibration exists only as qualified historical evidence (69.0% agreement, κ=0.515, n=919, under a different five-vote configuration).
Prior-label lineage receives similar scrutiny. Of the 19,449 against-prior verdicts, 44.1% carry labels from a zero-injection consensus whose nine-build probe cohort overlaps five evaluated models; seven evaluated models contributed no probe votes yet still show a positive gap (+5.63 points mean vs. +6.06 for overlapping builds). The paper reports this overlap as a bound rather than evidence of independence, noting no zero-injection label is fully independent of the scored cohort. Threshold sweeps (4/9 through 7/9 consensus) leave the gap direction stable (+10.2 to +11.8 points on the relabeled subset).
An exploratory non-coding extension (40 cases, five domains, case-macro metric) transfers the instrument beyond code, spanning 65.8–84.8% NC-Macro, led by GPT-5.5 rather than the coding leader; it is explicitly not pooled with coding results, and raw trajectories no longer survive, making it exploratory.
Limitations and open questions
The paper scopes its claims carefully. The 60 items instantiate 302 of 642 rules and score 256; professional-writing rules appear only outside the coding panel. Selection optimism may favor items that separated pilot models. AP-Acc is a behavioral stratification, not a training-provenance claim. Adjacent-rank resolution is unavailable: all adjacent common-support intervals include zero, producing one tie group containing all 12 models. The main panel's surface placements are admissibility-assigned rather than counterbalanced, so surface-by-family cells are descriptive strata, not controlled effects; a held-out historical conflict analysis (Kendall's Ï„=0.91) lacks retained row-level artifacts and is treated as descriptive only. Open questions include whether the E0 precedence ordering generalizes beyond synthetic style conflicts and older builds, whether judge-independent scoring can preserve the against-prior contrast, and how surface effects behave under controlled relocation at scale in the main panel.
Conclusion
Harness-IF demonstrates that operational instruction following can be measured at rule granularity across delivery surfaces, and that doing so changes conclusions: every evaluated frontier model complies less with rules opposing its defaults, by margins large enough to reorder adjacent leaderboard positions, and surface precedence under conflict does not follow prompt depth. Its reliability analyses—particularly the judge-swap fragility affecting most verdicts—define the appropriate reading of its numbers as robust cross-model patterns rather than fine-grained rankings.