Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model
Abstract: AI coding agents are commonly evaluated as models but deployed as systems. Their reliability depends not only on model capability, but on the harness, execution state, retrieval, memory and state management, permissions, review interfaces, and resource allocation. This monograph examines those boundaries and develops a framework for evaluating and operating coding agents reliably. It synthesizes 164 scholarly works, 100 practitioner records, 29 benchmark records, and 17 author-system case records through a structured multivocal review, targeted update audits, software-engineering coverage analysis, and distributed-systems evidence synthesis. Across this evidence, many apparent model failures originate elsewhere in the system, while improvements at one layer often fail to propagate to end-to-end outcomes. Evaluation and operation are treated as a dependency chain in which weaknesses in task construction, execution environments, retrieval, state management, verification, or observability can invalidate downstream conclusions. The monograph contributes a versioned catalog of 206 reliability records: 193 gated practices, including 56 developed in depth, plus 13 research leads; an evidence ledger; a framework for dependency and repair asymmetry across the agent lifecycle; measurements and failure cases from operated agent systems; runnable evaluation and reliability protocols; and five reusable agent skills with evidence maps. Together, these provide a system-level methodology for distinguishing model capability from infrastructure effects, designing defensible evaluations, and building systems that recover safely when components fail. The review is structured rather than exhaustive, evidence strength varies by topic, and results depend on workload and configuration. The methods record which search lanes were executed, which remain unexecuted, and limits on evidence-grading claims.
Summary
- The paper argues that coding-agent reliability is a property of the surrounding socio-technical system, not model capability alone, and formalizes dependencies from measurement through allocation.
- The paper shows why repeated, paired evaluations and stronger oracles matter, including a reported configuration gain shrinking from +0.054 to +0.0035 after repetition and benchmark pass rates falling 19–29% under stronger tests.
- The paper proposes durable work identities, immutable artifact and verifier records, scoped permissions, idempotent external effects, replayable traces, fault injection, and calibrated review to contain failures and support recovery.
System-level thesis
“Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model” (2608.13867) argues that coding-agent reliability cannot be inferred from model capability alone. A deployed agent is embedded in a larger socio-technical system comprising the evaluation harness, task and repository state, retrieval and context management, execution sandbox, permissions, persistent workflow state, verifier, review interface, scheduler, and cost policy. The paper’s central claim is therefore architectural: once agent work can outlive a worker, invoke external services, mutate shared state, execute concurrently, or produce durable effects, reliability becomes a property of the surrounding system.
The monograph organizes this claim as a dependency chain. Measurement determines whether an observed difference is credible; grading converts observations into acceptance decisions; containment and recovery constrain the consequences of execution; retrieval and context determine what evidence reaches the model; review and accountability govern consequential transitions; and allocation and cost determine which configurations receive future work. Each layer supplies the evidence boundary for the next.
Figure 1: The reliability dependency chain from measurement through allocation and cost.
The associated concept of repair asymmetry is one of the paper’s strongest generalizations. Later machinery cannot restore evidence discarded earlier in the chain. More repeated runs cannot repair a task distribution that excludes production work. More model judges cannot repair an incoherent rubric. More agents cannot compensate for a retrieval boundary that treats an empty result as authoritative. Reliability engineering must therefore proceed upstream: first establish what can be trusted, then build downstream controls around those facts.
The paper formalizes this system view through identities and ownership boundaries. Logical work is distinct from an execution attempt; a candidate artifact is distinct from accepted completion; a local completion record is distinct from an external commitment; and verifier output is distinct from semantic truth. These distinctions are not terminological refinements. They determine whether retries duplicate work, whether stale workers can mutate state, whether external effects can be reconciled, and whether a passing check can be attached to the correct artifact.
Evidence assembly and methodological discipline
The monograph presents itself as a structured multivocal review and engineering synthesis rather than an exhaustive systematic review. Its frozen corpus contains 164 scholarly works, 100 practitioner records, 29 benchmark records, and 17 author-system case records. The work reports 206 catalog records: 193 gated practices, including 56 developed in depth, and 13 research leads. The catalog is explicitly not an estimate of the number of reliability practices that exist; it reflects the paper’s claim granularity and editorial boundaries.
The methodological contribution is the separation of retrieval, evidence grading, practice admission, and author-operated case studies. Retrieval systems rank candidates but do not assign evidentiary status. Practitioner reports can corroborate mechanisms but generally do not establish prevalence. Author-system artifacts expose local failure modes and measurements but are not counted as independent external evidence. The paper also distinguishes strong, directional, corroborating, and null or conflicting evidence.
Figure 2: Source-review and practice-synthesis flow, including executed and unexecuted search lanes.
This transparency is valuable because the review openly records what it did not perform. Publisher- and index-native searches through ACM Digital Library, IEEE Xplore, and Scopus were not executed because of access and credential constraints. The paper does not convert those omissions into evidence of no relevant results. It also reports that external grading of the evidence catalog was prepared but not completed; consequently, it reports no inter-rater agreement for its own evidence classifications.
That concession materially limits the strength of the synthesis. The challenge passes reduced several practice grades, but final adjudication remained author-controlled rather than blinded and independently replicated. The paper’s framework for evidence reproducibility is stronger than its demonstrated external reproducibility. This is stated plainly rather than concealed behind the size of the source corpus.
Evaluation as experimental design
The first part of the monograph establishes that one agent run is one stochastic draw. A CodeProbe task family initially showed a configuration advantage of ; after three repetitions per configuration, the difference declined to , with a 95% confidence interval from to . The paired statistic was , below the stated critical value of $2.776$. A task that had contributed a apparent gain produced a difference of zero under repetition.
The result demonstrates that displayed precision is not experimental stability. The monograph further cites a 60,000-trajectory analysis in which single-run SWE-bench-Verified variation ranged from 2.2 to 6.0 percentage points, with standard deviation above 1.5 percentage points even at temperature zero (Bjarnason et al., 6 Feb 2026). Consequently, system improvements of two or three percentage points can fall inside local execution variation. The paper correctly qualifies this as workload- and apparatus-dependent rather than a universal noise constant.
The recommended design is repeated, paired evaluation with the apparatus pinned: model version, decoding settings, prompt, repository revision, tool definitions, permissions, harness, and evaluator. Nuisance variables such as task order, seeds, and prompt variants should be randomized or matched across systems. The paper emphasizes that fixed seeds support reproducibility conditional on one realization but do not estimate variation across the intended population.
Paired comparisons are preferred when systems run on the same experimental units. Per-item differences remove shared task difficulty when the outcomes are positively correlated.
Figure 3: Matched item-level differences narrow uncertainty by removing shared task difficulty.
The paper also insists that statistical procedures match the metric. Paired -tests are appropriate only for numerical outcomes whose paired differences support the required approximation. Binary pass/fail outcomes require paired tests such as McNemar’s test. Metrics computed nonlinearly from corpus-level counts, including corpus-level F-score or BLEU, require paired bootstrap or permutation procedures that recompute the full metric.
Power analysis is framed as an engineering decision rather than a post hoc statistical ritual. The target effect should be the smallest improvement that would alter deployment. If the available budget cannot resolve that effect, the experiment should return no verdict. The monograph cites an example in which increasing responses reduced the minimum detectable effect from 13.2% to 7.5% (Miller, 2024), while emphasizing that more items, more independent runs, and more responses reduce different sources of uncertainty.
Baselines, ablations, and cost
The paper argues that every architectural claim requires an appropriate control. A system with retrieval, memory, tools, delegation, or multiple agents should be compared against a credible direct-call baseline and, where relevant, a retry baseline. Otherwise additional calls, sampling opportunities, context, or budget may be misattributed to architecture.
Ablations must remove one component while holding the surrounding system fixed. Memory evaluation should distinguish at least three conditions: the proposed memory mechanism, no memory, and raw-trajectory reuse. This separates the value of prior information from the value of distillation or consolidation. Tool evaluations require a no-tool arm and must record actual use. A component that never executes cannot explain the result.
The monograph makes the stronger methodological point that configuration names are insufficient. A retrieval arm may differ simultaneously in model, prompt, context budget, retries, and corpus coverage. A multi-agent system may simply be a higher-budget sampling system. Factorial designs are useful when interactions matter; otherwise one-factor sweeps provide more interpretable attribution.
Cost and accuracy are treated as separate axes rather than compressed into score per dollar. Token counts, cached-input quantities, failed calls, retries, discarded branches, latency, and dated pricing assumptions should be retained.
Figure 4: Cost-accuracy frontier with dominated configurations and deployment feasibility constraints.
The paper defines Pareto dominance operationally: a configuration is dominated when another is at least as accurate and no more expensive, with a strict advantage on one axis. A cost-accuracy frontier is more informative than a ratio because deployment typically imposes an accuracy floor, cost ceiling, and sometimes latency limit. The monograph cites a re-evaluation showing that retries could match more elaborate architectures on HumanEval at lower inference cost, while plainly limiting transfer to repository-scale engineering (Kapoor et al., 2024).
Benchmark validity and oracle adequacy
The benchmark-validity chapter makes a particularly strong claim: a public score is the output of three coupled measurements. It depends on whether the model was exposed to the work, whether the oracle can reject incorrect artifacts, and whether the workload represents the target deployment.
Figure 5: Public scores jointly measure exposure, oracle strength, and workload validity.
The discussion of SWE-bench Verified illustrates why curation quality cannot answer these questions. The benchmark had been screened by 93 professional developers, yet later audits reported substantial test defects and evidence that models reproduced solution details from task identifiers. The paper emphasizes that human screening can improve task construction without controlling subsequent contamination or ensuring that tests reject plausible but incorrect patches.
The proposed controls are distinct. A matched public-private audit estimates a model-specific exposure gap. A temporal holdout evaluates tasks published after the model’s stated training cutoff. Stronger oracles use generated tests, mutation testing, differential testing, adversarial wrong patches, and manual re-adjudication. None of these substitutes for the others.
Figure 6: A capability claim requires workload validity, independence from exposure, and an oracle capable of rejecting incorrect artifacts.
The numerical results cited are consequential. EvalPlus-style test strengthening reduced reported pass rates by approximately 19–29% across reported values of (Liu et al., 2023). A repository-scale audit reported 345 previously passing patches that were incorrectly labeled, with large ranking changes (Yu et al., 10 Jun 2025). Another audit found that solution leakage and weak tests together reduced one measured resolution rate from 12.47% to 3.97% (Aleithan et al., 2024). These figures are correctly scoped to the audited systems and benchmarks rather than generalized into universal contamination or oracle-failure rates.
The paper’s preferred remedy for construct validity is local evaluation. Tasks should be mined from actual production work, with starting repository state, instruction, permissions, acceptance evidence, and repeated-run uncertainty preserved. The paper is explicit that merged-pull-request mining remains selective: it favors work that was accepted, reconstructable, and testable, and therefore does not automatically represent incidents, abandoned work, or poorly instrumented repositories.
Execution-based grading and calibrated judges
The monograph rejects agent self-report as an acceptance mechanism. A model can submit a patch in every trial while resolving only 44% of tasks under an external oracle across 1,750 trajectories, 50 tasks, and four models (Mehta, 26 Mar 2026).
Figure 7: Submission frequency can diverge sharply from externally verified resolution.
Execution-based evaluation assigns authority to the system whose state is being changed: compiler, test runner, schema validator, package builder, or deployment probe. The sandbox must begin from a known repository state, enforce resource and network boundaries, capture results immutably, and distinguish candidate failures from verifier-infrastructure failures.
A failed candidate returns to a correction step with external evidence. The candidate itself is never modified under an existing verdict; a correction produces a new artifact identity.
Figure 8: Immutable sandbox verdicts force corrected candidates to receive new identities and new verification.
The monograph also distinguishes from 0. The former asks whether at least one of 1 attempts succeeds; the latter asks whether all 2 succeed. These metrics encode different deployment semantics.
Figure 9: The same repeated trials answer different operational questions under 3 and 4.
For tasks without executable oracles, the paper permits model graders only as calibrated instruments. Expert labels must precede automation, and agreement must be separated from correctness. The kappa example is technically useful: 90% observed agreement with 82% expected agreement gives 5, while two always-PASS graders yield 100% observed and expected agreement and undefined kappa.
Figure 10: High raw agreement can coexist with weak or undefined chance-corrected agreement.
The recommended validation protocol uses a held-out, expert-labeled, stratified set, reports TPR, TNR, and FPR, and tests position bias, verbosity bias, leniency, self-preference, and leakage. Deterministic assertions should remain deterministic rather than being delegated to a model judge. The cited calibrated-jury result, including human Fleiss’s 6 and model-human Cohen’s 7 ranging from 0.77 to 0.87, is presented as evidence for the protocol within its evaluated setting (Panthi et al., 22 May 2026), not as a universal judge standard.
Proxy gaming and independent signals
The proxy chapter develops the claim that any optimized proxy requires an independent signal. A local retrieval scorer once awarded perfect recall for returning the entire repository. Replacing recall with overlap F1 made volume costly, but did not prove the resulting metric immune to gaming.
The theoretical result from Skalse et al. is deliberately bounded: under a linear expected-return formulation over unrestricted stochastic policies, two nonconstant rewards cannot be mutually unhackable (Skalse et al., 2022). The theorem does not describe every deployment, but it rules out a universal guarantee that two useful scalar objectives will remain aligned under arbitrary optimization.
Empirically, proxy reward can continue rising after independent quality peaks and declines (Gao et al., 2022). The paper therefore recommends monitoring the relationship between proxy scores and an independently observed quality signal, not merely the proxy’s absolute value.
Figure 11: Proxy performance can continue rising after independent quality has peaked.
The paper distinguishes three uses of signals: structural invariants may veto, outcome measures estimate real quality, and diagnostic scores explain movement without granting acceptance. Trace monitoring can expose reward-hacking intent, but it should not itself decide acceptance, particularly because direct optimization against a monitor can induce obfuscation (Baker et al., 14 Mar 2025).
A notable addition separates verifier-infrastructure health from oracle adequacy. A verifier may execute unreliably against the intended artifact, or execute correctly while judging the wrong property.
Figure 12: Verification requires both reliable execution against the intended artifact and an adequate oracle.
Acceptance records must bind artifact version, verifier version, environment identity, raw result, and status. Infrastructure uncertainty is neither a candidate failure nor a passing result; it is indeterminate and requires verification retry or reconciliation.
Containment, durable execution, and recovery
The software-factory model is the conceptual center of Part III. A durable work item outlives the worker that attempts it. Work and attempts therefore require separate lifecycles and identities. The paper defines work identifiers, input-state identifiers, ownership epochs, attempt identifiers, artifact versions, verification identifiers, and external-effect identifiers.
Figure 13: Identity, authority, artifact, verification, and external-effect boundaries.
The factory contracts include generation-scoped authority, stale-completion rejection, idempotent or reconciled external effects, version-bound evidence, explicit verifier failure, visible liveness, safe recovery, and causal attribution.
Figure 14: Separate logical-work and attempt lifecycles prevent worker self-reports from completing logical work.
The containment argument is straightforward and technically important: blast radius follows effective capability, not prompt wording. A self-reported database incident in which one credential could reach both primary data and backups illustrates that a warning cannot create a failure-domain boundary.
Figure 15: Shared credentials collapse primary and recovery resources into one failure domain.
The paper recommends read-only defaults, narrow and expiring escalation, separate recovery identities, explicit capability testing in both permitted and prohibited directions, and inspection of delegation paths. The evidence is primarily incident-level and directional; the claims are therefore framed as testable controls rather than prevalence estimates.
Durable execution requires persistent intent, progress, events, and effect identity outside the worker. The difficult interval is the commitment gap: an external effect may succeed after dispatch but before completion is durably recorded.
Figure 16: Durable intent and stable effect identity make uncertain external commits recoverable or explicitly reconcilable.
The correct recovery response is not blind retry. The external boundary must return the prior result, converge on the same state, or force explicit unknown-state reconciliation. Idempotency keys, deduplication records, natural convergence, and adapter-owned reconciliation provide different implementations of this contract. The paper explicitly rejects blanket exactly-once claims across boundaries that do not support them.
Replayable traces and fault injection
The paper treats recovery as a measured property, not an architectural assertion. Typed event streams distinguish model calls, tool dispatch, external commitments, persistence, state transitions, and recovery branches. A transcript is retained as a derived view rather than used as the authoritative recovery record.
Figure 17: Typed events expose the commitment gap that transcript-only logging can obscure.
Fault injection is organized around explicit claims and named lifecycle boundaries. The proposed kill-point sweep includes before dispatch, after dispatch but before external commitment, after external commitment but before local acknowledgment, after acknowledgment but before durable transition, and after durable transition.
Figure 18: Kill points test unstarted work, ambiguous commitments, recovery, and duplicate suppression.
The cited stream-processing benchmark found that recovery behavior changed across successive failures and contradicted earlier framework comparisons (Vogel et al., 2024). The implication is that one successful restart is insufficient. Experiments should include clean controls, repeated faults within a continuing run, independent faulted runs, representative load, and measurements of detection time, useful-work resumption, throughput, latency, duplicate effects, missing effects, and state equivalence.
The paper extends this to distributed ambiguity faults: stale ownership, delayed responses, old completions, repository changes between validation and publication, verifier failure, schema incompatibility, history compaction, and retry storms. Recovery paths themselves are production code and must be subjected to the same tests as normal execution.
Failure attribution and auditability
The monograph argues that failure analysis should identify the first upstream failure that materially changed the path to success, rather than counting every downstream symptom. A planning error can produce wrong localization, wrong edits, and later verification failures.
Figure 19: First-upstream attribution distinguishes causal boundaries from downstream symptoms.
The counterfactual question is whether correcting the proposed upstream step while holding earlier state fixed could have prevented the observed path. If the trace cannot support that judgment, the correct label is unresolved.
This position is supported by weak automation results. In one study of 127 multi-agent systems, the best automated method achieved 53.5% responsible-agent accuracy but only 14.2% decisive-step accuracy (Zhang et al., 30 Apr 2025). A later method reached 36.2% step-level accuracy and reported a 22.4% average task-success increase from counterfactually validated fixes (Ma et al., 10 Sep 2025). These results justify automated triage and candidate generation, not autonomous causal verdicts.
Consequential attribution should remain human-audited and structurally recorded: first upstream action, affected state transition, downstream dependency, supporting evidence, alternatives considered, confidence, and reviewer identity. The trace must record actor, inputs, decisions, tool effects, state versions, retries, and verification results. A model-generated explanation is an interpretation, not an observation.
Retrieval, context, and memory
Part IV treats retrieval as a causal chain rather than a single score: availability, retrieval, placement, contextual sufficiency, use, and verification.
Figure 20: Retrieval failure modes are separated across availability, ranking, placement, context, and use.
The paper’s CodeScaleBench result is numerically strong at the stage level. Retrieval improved 8 from 0.095 to 0.313, 9 from 0.120 to 0.272, and the share of tasks exposing at least one needed file from 0.33 to 0.56. Yet the paired end-to-end reward rose only 0, with a nominal bootstrap interval of 1 over 370 tasks. The paper immediately qualifies that interval: tasks were clustered by repositories and suites, so task-level resampling likely understates uncertainty.
The implication is central: retrieval can improve while task outcomes improve only marginally. The next intervention may belong in context assembly, evidence use, generation, or verification rather than ranking. Per-task records should preserve judged relevant items, ranks, context inclusion, and final outcome.
The monograph recommends combining lexical and semantic retrieval, but only after measuring unique contribution and cost. Reciprocal-rank fusion avoids arbitrary addition of incompatible BM25 and embedding scores. Actual lane health and contribution must remain observable; a failed semantic lane can leave lexical results plausible while silently removing relevant evidence.
Code structure matters inside the retrieval unit. Syntax-aware chunking produced a 4.3-point 2 gain and a 2.67-point 3 gain in the cited study (Zhang et al., 18 Jun 2025). The paper correctly treats these as stage- and benchmark-specific results, not a universal effect of AST chunking. Chunking policies must be evaluated with parser coverage, source revision, context tokens, duplicate content, target rank, and end-to-end completion.
The localization discussion further emphasizes that candidate file selection caps recall: any omitted file remains unavailable to every later inspection and editing stage.
Figure 21: The localization funnel narrows repository evidence, and candidate-file selection imposes an upper bound on later recall.
Memory and retrieval are treated similarly. The system should distinguish whether needed information was absent, retained but ranked too low, retrieved but omitted from context, exposed but unused, or used incorrectly. Final-answer correctness alone cannot localize these failures.
Provenance, memory, and review
The paper advocates append-only source records and derived stores with provenance, extraction version, and schema version. Derived memory should be rebuilt from the immutable record after schema changes rather than rewritten destructively.
Figure 22: Immutable source records permit derived-store rebuilding after schema changes.
The same principle applies to human review. Review should occur where the reviewer can see the consequential decision, its evidence, the artifact version, and the authorization boundary. Autonomy should expand by action class and measured approval or modification behavior, not through a single global autonomy score.
The paper’s review framework is consistent with its broader evidence model: deterministic checks first, calibrated automated grading second, human judgment for irreducible claims, and explicit escalation for high-impact or uncertain transitions.
Limitations and open questions
The paper’s limitations are substantive. Its review is structured but not exhaustive, and several search lanes remained unexecuted. Evidence quality varies sharply across topics: some recommendations rest on controlled comparisons, others on systems literature or practitioner incidents, and some Part VI proposals are explicitly research leads rather than settled deployment guidance.
The author-operated measurements are useful for exposing mechanisms but do not establish external prevalence. Several numerical claims are tightly scoped to particular benchmarks, models, repositories, or harnesses. Transfer from distributed systems, retrieval research, and multi-agent studies is often directional and should be locally validated.
The paper also leaves open whether the proposed dependency chain can be operationalized economically at organizational scale. Durable traces, versioned artifacts, fault injection, expert labels, stratified judge calibration, and repository-scale relevance annotations impose nontrivial storage, compute, latency, and human-review costs. The monograph identifies these costs but does not provide a unified cost model for reliability instrumentation.
Further open questions are specific to the paper’s framework. How should uncertainty be propagated across multiple dependency layers when measurement, grading, verifier health, retrieval, and review each have nonzero error? What statistical designs provide valid inference when tasks are simultaneously clustered by repositories, suites, languages, and temporal windows? Which trace fields are sufficient for causal attribution without creating unacceptable privacy and retention risks? Under what workload conditions does a workflow engine’s recovery benefit justify its coordination overhead? And how should release gates combine intermittent candidate failures with verifier flakiness without turning either into an unbounded source of false rejection?
Conclusion
The monograph’s main contribution is a change in the unit of analysis. A coding agent should not be evaluated as an isolated model that emits patches, but as a distributed, stateful, permissioned, and economically constrained system whose guarantees depend on preserved evidence and explicit ownership boundaries.
Its strongest recommendations are correspondingly concrete: repeat and pair evaluations; include direct, retry, and ablation baselines; report accuracy with cost and latency; validate workload exposure and oracle strength separately; gate artifacts through external execution; calibrate model graders against stratified expert labels; distinguish proxy scores from independent quality; isolate authority and recovery domains; persist logical work outside workers; make external effects idempotent or explicitly reconcilable; bind evidence to artifact and verifier versions; record typed, replayable traces; inject faults at the commitment boundaries the design claims to protect; attribute failures to the earliest defensible upstream cause; and measure retrieval, context, generation, and verification as separate stages.
The resulting standard is deliberately narrower than a universal reliability certificate. A credible claim is one whose workload, apparatus, evidence boundary, failure semantics, and recovery envelope are explicit enough to be challenged and reproduced. That is the paper’s governing criterion for engineering reliable coding agents.
Paper Prompts
Sign up for free to create and run prompts on this paper.
Top Community Prompts
Explain it Like I'm 14
1. What is the paper about?
The paper, “Engineering Reliable Coding Agents,” studies how to make AI tools that write computer code more dependable.
The main idea is that an AI coding agent is not just the LLM producing code. It is part of a larger system that includes:
- The computer environment where it runs
- The information it can search and remember
- The permissions it has
- The tests used to check its work
- The people who review its changes
- The systems that save its progress and recover from failures
- The way time, money, and computing power are allocated
The paper argues that an agent can appear to fail because of the model, even when the real problem is somewhere else. For example, the agent might not have been given the right files, might have run incomplete tests, or might have lost important information during a restart.
2. What questions does the paper try to answer?
The paper asks questions such as:
- How should we fairly measure whether a coding agent is good?
- How can we tell whether an improvement is real or just caused by an easier test?
- How can we check that the agent’s code actually works, rather than trusting what it says?
- What should happen when the agent crashes, loses information, or makes a dangerous request?
- How much access should an agent have to files, tools, and credentials?
- How can people review an agent’s work effectively?
- How should organizations manage many agents while controlling cost, time, and computing resources?
- How can we separate a problem caused by the AI model from a problem caused by the surrounding software system?
The authors organize these questions into a chain:
measurement → grading → safety and recovery → information and memory → human review → resource allocation
Each step depends on the one before it. If the first measurement is poor, later decisions may also be wrong.
3. How was the research carried out?
This is mainly a technical review and engineering guide, rather than one experiment with a single group of participants.
Reviewing existing research
The author collected and examined many different kinds of sources:
| Type of source | Number |
|---|---|
| Scholarly research papers | 164 |
| Practitioner and industry records | 100 |
| Benchmark records | 29 |
| Cases from systems operated by the author | 17 |
A benchmark is a standardized test used to compare systems. It is similar to giving different students the same exam. However, the paper warns that a benchmark can be misleading if the questions are too easy, incomplete, or already known by the AI.
The review was multivocal, meaning it used both academic research and practical reports from engineers. This was important because some real-world problems are described in company reports or incident write-ups rather than in academic papers.
Searching for sources
The author used search and retrieval tools to find relevant material. These tools searched for sources using both:
- Keyword matching, like looking for exact words
- Dense retrieval, which tries to find documents with similar meanings even when they use different words
The tools helped find and organize sources, but the author made the final decisions about which sources to trust and how strong their evidence was.
Grading the evidence
The sources were placed into four broad groups:
- Strong evidence: A careful comparison or measurement directly supports the claim.
- Directional evidence: The evidence suggests something may work, but does not fully prove it.
- Corroborating evidence: Several practical examples show that an idea is believable.
- Null or conflicting evidence: Some results do not support the expected conclusion or disagree with other findings.
The author also performed checks for duplicate sources, weak claims, missing information, and conclusions that were stronger than the evidence allowed.
Studying real agent systems
The paper also includes measurements and failure examples from systems operated by the author. These examples help show how problems happen in practice. However, the author clearly states that these personal cases are not independent proof that the same problem happens everywhere.
Creating recommended practices
From the evidence, the author created a catalog of 206 reliability records:
- 193 were accepted as practices
- 56 were explained in detail
- 13 were listed as possible research topics rather than firm recommendations
The paper also provides checklists, testing procedures, and other materials that organizations can use to examine their own coding-agent systems.
4. What are the main findings?
Reliability depends on the whole system
The most important finding is that reliability is not only a property of the AI model.
A model may write good code, but the final result can still be unreliable if:
- The agent receives incomplete information
- The tests do not check the important behavior
- The agent has too many permissions
- Its work is not saved correctly
- A restart causes it to forget important decisions
- A reviewer cannot see what the agent did
- Different agents interfere with one another
- The system reports success without independently checking it
This is similar to judging a delivery driver only by the driver’s skill while ignoring the car, the map, the road, and the delivery instructions.
Scores can be misleading
A high score does not always mean that a coding agent is better. The score might be high because:
- The tasks were unusually easy
- The tests did not cover important cases
- The benchmark had information the model had already seen
- A grading system rewarded the appearance of success instead of real correctness
The paper therefore recommends preserving detailed, item-by-item results instead of reporting only one overall score.
Later improvements cannot fix earlier measurement problems
The paper describes a repair asymmetry. This means that fixing a problem early in the process is often much more important than adding more tools later.
For example:
- More test runs cannot fix a test set that does not represent real work.
- More automatic graders cannot fix a confusing grading rule.
- More agents cannot fix a retrieval system that gives them the wrong files.
- Better reports cannot recover information that was never recorded.
In simple terms, putting better software on top of bad measurements does not make the conclusions trustworthy.
Systems need safe recovery
Coding agents can crash, lose network access, or be interrupted while changing files. A reliable system should therefore:
- Save important state
- Keep a record of what happened
- Retry operations safely
- Avoid repeating an action that could cause damage
- Separate normal work identities from emergency recovery identities
- Test recovery procedures instead of assuming they work
The paper calls some operations idempotent. An idempotent operation can be repeated without causing extra damage. For example, checking whether a file exists is usually safe to repeat, while charging a credit card twice is not.
Human review must be designed carefully
Human review is not automatically effective. Reviewers need to see the evidence that matters, including:
- What the agent changed
- Which tests were run
- Which decisions were uncertain
- What permissions the agent used
- Whether the result was independently verified
The paper also warns that agreement between an automated grader and a human does not necessarily mean the grader is correct. Both may share the same misunderstanding.
The paper’s own evidence has limits
The author openly reports several limitations:
- The review was structured but not completely exhaustive.
- Some major research databases could not be searched because access was unavailable.
- Many recommendations have only directional or practical evidence.
- Results may change depending on the type of code, repository size, model, and system setup.
- The author did not obtain outside reviewers to measure agreement about the evidence grades.
This honesty is important because it prevents the paper from presenting engineering suggestions as universal scientific laws.
5. Why are these findings important?
AI coding agents may eventually work on large and important software projects. If organizations trust them without checking the entire system, several problems could occur:
- Bugs could be introduced into production software.
- Private information or credentials could be exposed.
- An agent could make changes it was not allowed to make.
- A system could claim that a task was complete when it was not.
- A failure could be difficult to investigate because important records were missing.
- Organizations could spend large amounts of money on agents without receiving better results.
The paper’s approach is to treat an AI coding agent more like a worker in a large factory than like a simple chatbot. The factory needs rules, safety equipment, records, inspections, emergency procedures, and managers. Similarly, an AI coding system needs careful tests, limited permissions, saved progress, useful logs, human checks, and recovery plans.
Conclusion: What could this research change?
The paper encourages companies and researchers to stop asking only, “How capable is the AI model?” Instead, they should also ask:
“Can the whole system use this model safely, measure it fairly, and recover when something goes wrong?”
Its practical impact could include:
- Better tests for coding agents
- More honest comparisons between different systems
- Safer permissions and stronger isolation
- Easier investigation of failures
- More useful human review
- Lower costs through better scheduling
- Fewer situations where an agent appears successful but has actually failed
The central lesson is simple: a reliable coding agent requires a reliable system around it. Improving the model matters, but it is only one part of the job.
Knowledge Gaps
The paper leaves the following knowledge gaps, limitations, and open questions unresolved:
- External reproducibility of evidence grading: No external graders completed the blinded sample, so the reliability of the four evidence-group labels and the reproducibility of practice admissions remain unknown.
- Correctness of the grading framework: Even if future reviewers agree with the author’s labels, the paper does not establish that the labels accurately represent the quality or causal strength of the underlying evidence.
- Search coverage beyond SciX and OpenAlex: ACM Digital Library, IEEE Xplore, and Scopus searches were not executed, leaving unknown how many relevant software-engineering studies were missed.
- Incomplete venue coverage: The observed mismatch between SciX and non-TSE records does not quantify corpus recall, topic-search recall, or equivalence with publisher-native indexes.
- Unresolved DBLP candidates: Thirty-four potentially relevant works identified after the cutoff were deferred, so the catalog may omit evidence that could alter its conclusions or practice classifications.
- Non-replayability of the original search process: Not every machine-issued query was preserved, preventing exact byte-for-byte replication of the initial literature retrieval and screening process.
- Potential selection and author-judgment bias: Final source inclusion, evidence grading, practice admission, chapter selection, and prose interpretation were controlled by a single author, despite automated challenge passes.
- No estimate of evidence-selection error: The paper does not measure how often a second research team would include different sources, assign different evidence groups, or derive different practices.
- Limited independence of practitioner evidence: Practitioner records are grouped by independence keys, but the validity of those keys and the extent of shared reporting or survivorship bias are not empirically established.
- Weak prevalence estimates for operational failures: Many proposed controls are supported by mechanisms, incidents, or case reports rather than data estimating how frequently the corresponding failures occur in deployed coding-agent systems.
- Limited external validity of author-system measurements: The original measurements and failure cases come from systems operated by the author and may not transfer to other organizations, repositories, models, harnesses, or governance environments.
- Uncertain scale transfer: The paper focuses on large organizational repositories, including systems with millions or tens of millions of lines, but does not establish whether its practices have comparable benefits or costs in small repositories or different organizational scales.
- Uncertain transfer from adjacent literatures: Claims about fleet scheduling, observatory scheduling, compute clusters, build systems, inference serving, and multi-agent systems are used to motivate coding-agent research, but their effects on coding-agent workloads remain unmeasured.
- No integrated causal evaluation of the dependency chain: The paper argues that measurement, grading, containment, retrieval, review, and allocation interact, but it does not experimentally quantify how failures at one layer propagate through the entire chain.
- No validated model of repair asymmetry: The proposed idea that downstream machinery cannot repair upstream evidence loss is conceptually important, but its magnitude, boundary conditions, and operational decision value are not empirically tested.
- Unresolved attribution of end-to-end improvements: The framework identifies infrastructure effects, but the paper does not provide a validated causal methodology for separating improvements due to the model from those due to the harness, retrieval, memory, permissions, verification, or review system.
- Insufficient evidence for recommended control combinations: Individual mechanisms may have support, but the paper does not establish whether bundles of controls produce additive, redundant, synergistic, or counterproductive effects.
- Unknown operational cost of reliability controls: The effects of proposed practices on compute use, latency, engineering effort, review burden, developer throughput, and monetary cost are not consistently quantified.
- Unresolved trade-offs between reliability and autonomy: The paper discusses containment, verification, and human gates but does not determine how much autonomy can safely be granted under different task risks, organizational constraints, or failure costs.
- Unvalidated human-review interventions: The effectiveness of review interfaces, escalation rules, provenance displays, and risk-based human oversight is not established through controlled studies with realistic reviewers and workloads.
- Unknown reviewer behavior under agent-generated volume: The paper assumes scarce review capacity but does not measure how reviewers’ accuracy, fatigue, trust, or attention change when agents produce many candidate changes.
- No established method for detecting hidden test inadequacy: Passing tests are treated as potentially insufficient evidence, but the paper does not validate a general procedure for detecting when tests fail to exercise the modified behavior.
- Unresolved benchmark representativeness: The framework identifies workload validity and benchmark contamination as concerns, but it does not establish which benchmark properties best predict production coding-agent reliability.
- No broad contamination estimates: The paper discusses contamination risks but does not quantify how often coding-agent benchmarks overlap with model training data, evaluation prompts, public repositories, or previously observed tasks.
- Limited evidence for retrieval and context recommendations: Repository retrieval, context budgets, freshness checks, memory, and compaction are treated as central mechanisms, but comparative evidence across repositories, languages, indexing systems, and agent architectures remains incomplete.
- Unknown interaction between retrieval failures and model capability: The paper does not quantify whether stronger models are more robust to missing, stale, empty, or misleading retrieval results.
- Unresolved durability and recovery effectiveness: Durable state, idempotent retries, replayable traces, and fault injection are proposed as controls, but their effect on successful recovery, recovery time, duplicate side effects, and residual authority is not established across realistic failure scenarios.
- Incomplete security evaluation: The discussion includes isolation, injection defenses, identity separation, and authority boundaries, but the paper does not provide comprehensive adversarial testing across prompt injection, credential compromise, cross-repository access, tool abuse, and recovery-path attacks.
- Unknown correlated-failure behavior: The system model emphasizes independently failing components, but it does not sufficiently test correlated failures such as shared identity providers, common retrieval indexes, model endpoint outages, or compromised observability systems.
- No validated reliability thresholds: The minimum reliability pass proposes success floors, cost ceilings, and containment checks, but the paper does not establish threshold values or show that passing the protocol predicts safe production behavior.
- Unresolved statistical power in realistic agent comparisons: The paper recommends repeated and paired comparisons, but the appropriate sample sizes for heterogeneous, nonstationary, long-horizon coding tasks remain unclear.
- Unknown stability over time: Capability measurements, model behavior, pricing, retrieval indexes, and tool interfaces can change rapidly; the paper does not quantify how frequently evaluations must be repeated to remain valid.
- Limited evidence for dynamic allocation and model routing: The research agenda identifies topology selection, fleet scheduling, and cost-aware routing as open areas, but it does not demonstrate that these strategies improve quality-adjusted throughput under real coding-agent workloads.
- Unresolved definition of reliability across objectives: The paper distinguishes correctness, reliability, performance, cost, safety, and usability, but it does not provide a universally validated method for combining these dimensions into deployment decisions.
- No long-term production outcome study: The paper does not show whether the proposed practices reduce escaped defects, security incidents, rollback rates, maintenance costs, or organizational harm over extended deployment periods.
Practical Applications
Immediate Applications
The paper’s most immediately deployable contributions are operational controls for organizations already experimenting with AI coding agents. They do not require a new model; they require instrumentation, workflow changes, access controls, and disciplined evaluation.
- Enterprise software engineering: establish a minimum reliability pass before expanding agent use.
- Potential workflow: CI pipeline or evaluation service producing a paired outcome distribution, cost-quality report, authority-boundary test, verified repository transition, failure corpus, and promotion decision.
- Dependencies: Stable task and baseline versions, repeatable environments, access to execution traces, and enough runs to characterize variance. The protocol is an entry point rather than a reliability certificate.
- Software delivery: replace aggregate benchmark scores with execution-based release gates.
- Potential product: An agent “correction gate” integrated into CI/CD that blocks a pull request until the claimed behavior is independently reproduced.
- Dependencies: Tests must exercise the relevant behavior; otherwise, passing tests may create false confidence. Full-repository validation may be infeasible in large codebases, requiring risk-based subsets.
- Security and platform engineering: enforce capability-boundary testing.
- Potential tools: Automated authorization regression tests, sandbox policies, capability manifests, and “deny-path” test suites for agent tools.
- Dependencies: Fine-grained identities, auditable permission decisions, isolated execution environments, and recovery credentials that are not identical to production credentials.
- Cybersecurity: isolate agents and independently verify consequential actions.
- Sectors: Secure software development, cloud operations, DevSecOps, and infrastructure-as-code.
- Dependencies: Strong sandboxing, separation of duties, trusted verification components, and clear definitions of which actions are consequential.
- Incident response and reliability engineering: build replayable traces and failure taxonomies.
- Potential tools: Agent observability platforms, trace viewers, replay harnesses, fault-injection suites, and failure dashboards.
- Dependencies: Comprehensive event schemas, privacy and secret-redaction mechanisms, deterministic or sufficiently controlled replay environments, and agreement on failure categories.
- Continuous integration and distributed systems: make agent workflows durable and retry-safe.
- Potential workflow: Durable issue-to-pull-request pipelines that resume after worker failure without duplicating commits, deployments, comments, or external side effects.
- Dependencies: Durable storage, idempotency keys, explicit state transitions, lease or ownership management, and safeguards against retries repeating non-idempotent actions.
- Repository management: measure retrieval quality and freshness.
- Potential products: Repository indexes, dependency-aware localization tools, freshness monitors, and retrieval evaluation dashboards.
- Dependencies: Complete repository metadata, language-specific parsers, cross-repository dependency visibility, access-aware indexing, and tests that measure task-level usefulness rather than retrieval volume alone.
- AI platform operations: monitor context budgets and cross-session memory.
- Potential workflow: A restartable agent session that stores task state, evidence, unresolved risks, and prior tool results in inspectable files rather than relying only on a model’s implicit conversational memory.
- Dependencies: Storage and retention policies, secret handling, versioned context artifacts, and mechanisms for detecting stale or contradictory memory.
- Human-computer interaction and software review: redesign review interfaces around risk and evidence.
- Potential product: A pull-request interface with evidence panels, provenance indicators, risk-based routing, and mandatory human gates for high-impact transitions.
- Dependencies: Trace completeness, usable information presentation, reviewer training, and sufficient review capacity. More information can reduce effectiveness if the interface creates excessive cognitive load.
- Governance and compliance: create an evidence ledger for agent-generated changes.
- Sectors: Regulated software, finance, healthcare IT, public-sector systems, and internal AI governance.
- Dependencies: Stable identifiers, retention policies, access controls, provenance integrity, and organizational agreement about evidence grades.
- Research and academia: use paired, repeated, cost-aware evaluations of coding agents.
- Potential output: More reproducible benchmark reports and open evaluation protocols that expose infrastructure effects rather than attributing every difference to the model.
- Dependencies: Representative task distributions, uncontaminated workloads, transparent system configurations, and statistical methods appropriate for paired and variable outcomes.
- Education and training: teach system-level AI engineering rather than model-only evaluation.
- Dependencies: Access to sandboxed agent systems, realistic repositories, and instructors able to assess infrastructure and governance issues in addition to generated code quality.
Long-Term Applications
The paper also motivates applications whose feasibility depends on larger datasets, further empirical validation, interoperability standards, or research into workload-specific behavior. These are best treated as development and research agendas rather than established deployment recipes.
- Autonomous software factories for large organizations.
- Potential product: A distributed agent operating system with contracts for identity, authority, persistence, ordering, ownership, observation, and recovery.
- Dependencies: Reliable cross-repository indexes, durable orchestration, conflict-aware concurrency, strong governance, and evidence that factory-level performance transfers beyond the paper’s local and case-based measurements.
- Dynamic agent topology and task allocation.
- Sectors: Enterprise software, robotics software, cloud operations, and complex engineering workflows.
- Dependencies: Valid workload models, reliable estimates of task difficulty and risk, coordination protocols, and experiments demonstrating that additional agents improve end-to-end outcomes rather than merely increasing activity and cost.
- Cost-aware fleet scheduling and model routing.
- Potential product: A policy-controlled router and scheduler for coding-agent fleets, analogous to a cluster scheduler but aware of correctness, review capacity, and failure recovery.
- Dependencies: Calibrated estimates of quality and uncertainty, stable pricing and latency data, workload-specific evaluation, fair allocation policies, and safeguards against optimizing cost at the expense of safety.
- Risk-adaptive human oversight.
- Sectors: Healthcare software, financial systems, critical infrastructure, and public services.
- Dependencies: Valid risk models, accountable human ownership, explainable escalation decisions, legally acceptable audit trails, and evidence that automation does not conceal important uncertainty.
- Formal reliability certification for coding-agent systems.
- Potential output: An industry standard, procurement checklist, or assurance label for agent platforms used in regulated environments.
- Dependencies: Independent replication, external grader calibration, standardized workloads, agreed definitions of reliability, and evidence that controls remain effective across architectures and scales.
- Cross-domain autonomous engineering systems.
- Dependencies: Domain-specific safety cases, real-time constraints, validated simulators, hardware-in-the-loop testing, sector regulation, and much stronger guarantees than those needed for ordinary repository changes.
- Policy frameworks for accountable agent deployment.
- Sectors: Government procurement, regulated finance, healthcare, defense, and critical infrastructure.
- Dependencies: Jurisdiction-specific legal standards, interoperable audit formats, protection of proprietary code and personal data, and careful distinction between engineering evidence and legally sufficient assurance.
- Benchmark ecosystems that evaluate complete systems rather than models alone.
- Dependencies: Representative and uncontaminated workloads, transparent benchmark versions, executable grading, reproducible environments, and methods for reporting null or conflicting results.
- Automated evidence-quality and experiment-design assistants.
- Potential product: An evaluation-design linter for agent experiments and production rollout plans.
- Dependencies: Well-validated methodological rules, domain expertise, human adjudication, and protection against treating automated checks as independent evidence graders.
- Self-healing agent infrastructure with bounded recovery authority.
- Sectors: Continuous delivery, cloud infrastructure, robotics operations, and laboratory automation.
- Dependencies: Correct state-machine design, idempotent effects, independently tested recovery paths, fault injection, observability across component boundaries, and assurance that recovery cannot amplify a compromised or incorrect agent action.
Glossary
- Ablation: An experiment that removes or changes one system component to measure its contribution. “Baselines, ablations, and cost-accuracy tradeoffs”
- Adjudication: The formal resolution of disagreements or ambiguous decisions during review. “The author made the final inclusion, evidence-group, and practice-admission decisions.”
- Benchmark contamination: The presence of evaluation data or tasks in a model’s prior training or exposure, potentially inflating its measured performance. “Another method checks whether a score was earned on work the model had already encountered.”
- BM25: A probabilistic lexical-information-retrieval ranking algorithm that scores documents according to query-term frequency and document length. “It combined INDUS dense retrieval with BM25 lexical retrieval through reciprocal-rank fusion.”
- Capability boundary: The empirically observed limit separating actions a system can perform from actions it cannot or should not perform. “runnable protocols for local evaluation, capability-boundary testing, recovery testing, trace analysis, and release decisions”
- Cohen’s kappa: A statistic measuring agreement between two annotators while correcting for agreement expected by chance. “a script that can report pairwise Cohen's kappa”
- Construct validity: The degree to which a measurement actually represents the theoretical property it is intended to measure. “work on construct validity in software engineering”
- Correlation: A statistical measure of association between variables, indicating how changes in one relate to changes in another. “The Spearman correlation between urgency rank and a binary indicator”
- Dense retrieval: Information retrieval using vector representations to identify semantically similar documents rather than relying only on exact term matches. “It combined INDUS dense retrieval with BM25 lexical retrieval through reciprocal-rank fusion.”
- Distributed system: A system whose components operate across multiple processes or machines and must coordinate despite independent failures. “reliability stops being a property of the model and becomes a property of the system”
- Durable execution: Execution designed to preserve workflow progress and state across process, machine, or service failures. “durable execution, repository state, verification, human control, and fleet allocation”
- Evidence group: A classification indicating the strength and type of support for a scoped claim. “assign an evidence group”
- Evidence ledger: A structured record linking claims to the evidence that supports, limits, or contradicts them. “an evidence ledger linking claims to their support”
- Fault containment: The restriction of a failure so that it cannot spread to unrelated components, resources, or authority domains. “fault-containment guard”
- Fault injection: Deliberately introducing failures into a system to test its detection, containment, and recovery behavior. “fault-injection recovery testing”
- Fleiss’s kappa: A chance-corrected agreement statistic applicable when three or more annotators classify the same items. “Fleiss's kappa when three readers participate”
- Inference serving: The infrastructure and processes used to provide model predictions to clients, typically under latency, capacity, and concurrency constraints. “Four scholarly systems papers on inference serving”
- Idempotent retry: A retry operation designed so that repeating it produces the same effective result as performing it once. “Persistent agent state, durable workflows, and idempotent retries”
- Independence key: A grouping identifier used to prevent multiple reports of the same incident or claim from being counted as independent evidence. “Repeated practitioner accounts of one incident shared an independence key”
- INDUS: A dense-retrieval system or retrieval method used by the paper’s local scholarly search layer. “It combined INDUS dense retrieval with BM25 lexical retrieval through reciprocal-rank fusion.”
- Inter-rater agreement: The degree to which independent reviewers assign the same labels or judgments to the same items. “reports no inter-rater agreement value”
- Lexical retrieval: Information retrieval based primarily on matching words or textual tokens between a query and documents. “BM25 lexical retrieval”
- Multivocal review: A literature review that combines scholarly research with practitioner or grey literature. “This practitioner lane makes the review multivocal”
- Observability: The ability to infer a system’s internal state from its recorded outputs, events, and measurements. “failures at component boundaries and a recovery procedure can pass while depending on credentials”
- Oracle strength: The degree to which an evaluation oracle can reliably determine whether a system’s output is correct. “Benchmark contamination, oracle strength, and workload validity”
- Paired comparison: An experimental design that compares two conditions on the same items or matched units to reduce irrelevant variation. “a paired distribution”
- Proxy metric gaming: Optimizing a measurable surrogate in a way that improves the metric without improving the intended underlying objective. “Proxy metric gaming and layered evaluation signals”
- Reciprocal-rank fusion: A method for combining rankings from multiple retrieval systems by aggregating the reciprocal positions of documents. “It combined INDUS dense retrieval with BM25 lexical retrieval through reciprocal-rank fusion.”
- Recovery path: The sequence of mechanisms by which a system restores safe operation after failure. “whose compromise would also destroy the recovery path”
- Repository-scale scoping: Defining a study or system boundary in relation to the size, structure, and dependencies of a software repository. “scheduling with repository-scale scoping”
- Retrieval boundary: The limit defining which information a retrieval system can return or treat as available. “a retrieval boundary that treats an empty result as authoritative”
- Schema validation: The automated checking of data against a predefined structural and type specification. “validated against a JSON Schema”
- Selection bias: Systematic distortion caused by the way observations, records, or participants are selected. “practitioner reports are vulnerable to selection, survivorship, and reporting bias.”
- Spearman correlation: A rank-based correlation coefficient measuring the monotonic association between two variables. “The Spearman correlation between urgency rank”
- Survivorship bias: Error arising when analysis considers only successful or surviving cases while ignoring failures or discarded cases. “practitioner reports are vulnerable to selection, survivorship, and reporting bias.”
- System boundary: The conceptual or operational line distinguishing a system and its responsibilities from its surrounding environment. “I still describe the relevant system boundary when a familiar tool plays an unfamiliar role.”
- Task distribution: The population and composition of tasks used to evaluate a system, including their difficulty and characteristics. “a task distribution that excludes production work”
- Threat model: An explicit account of possible adversaries, hazards, attack paths, or failure conditions relevant to a system. “supports a mechanism, threat model, comparison design”
- Topology: The structural arrangement of components, agents, services, or communication relationships in a system. “topology, and resource allocation”
- Transfer validity: The extent to which a result or practice applies beyond the conditions, workload, or environment in which it was established. “Transfer is especially substantial in Part VI”
- Workload validity: The degree to which an evaluation workload represents the real-world work for which a system is intended. “Benchmark contamination, oracle strength, and workload validity”
