Agent Incompetence Bias Insights

• Agent incompetence bias is a family of biases causing misattribution of competence where observed outcomes distort evaluations despite constant agent behavior.
• It manifests in diverse settings such as human–robot interactions, coding agent evaluations, LLM safety protocols, and multi-agent decision systems.
• Mitigation strategies include unbiased feedback models, calibrated assessment thresholds, and tailored prompts to align perceived competence with true agent performance.

Searching arXiv for the cited papers to ground the article in recent literature. Agent incompetence bias is a label used in recent literature for several distinct but related phenomena in which actual agent performance becomes misaligned with judgment, intervention, or action policy. In human–agent interaction, it denotes post-hoc distortion of users’ trait attributions to an agent in the direction of their own outcomes while the agent’s behavior is held constant (Gurney et al., 2023). In coding-agent evaluation, the closely related “action bias” denotes a tendency to produce a non-empty patch even when no code change is needed (Gloaguen et al., 8 May 2026). In LLM-agent safety, it denotes training-induced degradation of step-1 tool-use competence on benign tasks (Li et al., 19 Mar 2026). In multi-agent decision support under delayed feedback, it appears as biased error attribution to the wrong advisor (Parakh et al., 24 Mar 2026). In principal–agent learning models, it names a dogmatic mis-belief about an agent’s innate ability that distorts inference about effort productivity and assessment (Echenique et al., 30 Nov 2025). The term therefore does not identify a single invariant construct; rather, it marks a family of competence-related biases at the levels of perception, control, evaluation, and autonomous execution.

1. Terminological scope and canonical formulations

Recent usage divides into at least five technically distinct formulations.

Usage	Core distortion	Canonical setting
Post-hoc trait attribution	User outcomes color judgments of ability, benevolence, and integrity	Human–robot reconnaissance missions
Action bias in coding agents	Agent edits code when abstention is correct	SWE-Bench-derived “Resolved” tasks
Defense-induced incompetence	Defense training reduces valid benign tool calls	AgentDojo multi-step tool-use tasks
Delayed-feedback misattribution	Humans “correct” the wrong AI advisor after a loss	Seven-step naval-battle allocation game
Misspecified ability belief	Society under- or overestimates innate ability and mislearns productivity	Principal–agent assessment and market inference

In the behavioral studies by Gurney, Pynadath, and Wang, agent incompetence bias is the “post-hoc distortion of users’ trait attributions to an agent—ability, benevolence, integrity—in the direction of their own observed outcomes, holding the agent’s actual behavior constant” (Gurney et al., 2023). In “Coding Agents Don’t Know When to Act,” the relevant failure is formalized by a repository $R$ in which the golden patch $X^*$ has already been applied and all golden tests $T^*$ pass; a competent agent should output a patch $\hat{s}$ with $|\hat{s}|=0$, yet many agents return $|\hat{s}|>0$ (Gloaguen et al., 8 May 2026). In “The Autonomy Tax,” agent incompetence bias is expressed as

$$P(f_{\theta'}(c_1)\in\mathcal{A}\mid c_1)\ll P(f_\theta(c_1)\in\mathcal{A}\mid c_1),$$

where $f_\theta$ is a base instruction-tuned model, $f_{\theta'}$ its defense-trained counterpart, $c_1=[s,u]$ the initial benign context, and $X^*$0 the set of syntactically correct tool actions (Li et al., 19 Mar 2026).

A plausible unifying interpretation is that each formulation isolates a failure of competence attribution or competence preservation under conditions where the ground truth is, by design, held fixed or recoverable. The literatures differ mainly in whether the distortion arises in the human evaluator, in the training pipeline, or in the agent’s own policy over actions.

2. Post-hoc trait attribution in human–agent interaction

The most explicit psychological formulation appears in the search-and-rescue experiments reported by Gurney, Pynadath, and Wang (Gurney et al., 2023). Participants interacted with a simulated robot that scouted locations using three potentially noisy sensors and issued binary recommendations, “gear up” versus “gear down.” Crucially, the robot’s sensor readings and recommendations were scripted, so variation in mission outcomes arose from participants’ own decisions to follow or ignore the advice rather than from differences in robot behavior.

Study 1 used 78 participants, each completing three missions of eight locations, under a $X^*$1 factorial crossing robot ability and explanation type. Study 2 used 159 participants in a single 45-location mission with one of three reinforcement-learning–based robots. In both studies, the key behavioral predictors were $X^*$2, the percentage of correct decisions, and $X^*$3, the percentage of times the participant followed the robot’s advice (Gurney et al., 2023).

After the mission, participants completed a modified Mayer and Davis three-factor trust inventory on a 7-point Likert scale. Ability was measured with 13 items, benevolence with 5 items, and integrity with 3 items, with one integrity item reverse-coded. For each trait $X^*$4, the full regression model added correct-decision percentage and follow-advice percentage to treatment controls, and the significance of $X^*$5 was tested both directly and by comparing the control and full models (Gurney et al., 2023).

The strongest and most consistent effect was on benevolence. In Study 1, $X^*$6, $X^*$7, $X^*$8, with $X^*$9, $T^*$0 for adding $T^*$1 to the control model. In Study 2, $T^*$2, $T^*$3, $T^*$4, with $T^*$5, $T^*$6 (Gurney et al., 2023). Ability showed a detectable effect in Study 2, $T^*$7, $T^*$8, $T^*$9, $\hat{s}$0, although the corresponding model-improvement test was marginal. Integrity showed a significant effect in Study 1, $\hat{s}$1, $\hat{s}$2, $\hat{s}$3, $\hat{s}$4, but no significant effect in Study 2 (Gurney et al., 2023).

The paper interprets the phenomenon as a close analog of the fundamental attribution error. The proposed mechanisms include affect-misattribution, outcome-driven causal inference, and mindless anthropomorphism. The corresponding mitigation proposals are to augment user-belief models with user performance and compliance features, embed debiasing prompts in the interface, present objective logs of the agent’s own accuracy and uncertainty estimates, and incorporate mood or outcome sensors so that agents can detect when emotional valence may color judgments (Gurney et al., 2023).

3. Action bias and abstention failure in coding agents

In coding-agent evaluation, the bias is operational rather than perceptual. FixedBench contains 200 human-verified tasks sampled from SWE-Bench Verified, each corresponding to a GitHub issue whose fix has already been merged, so that no further edits to production code are required (Gloaguen et al., 8 May 2026). Construction proceeds from a pre-patch repository $\hat{s}$5 and golden patch $\hat{s}$6; applying $\hat{s}$7 yields $\hat{s}$8, the “Resolved” setting. The agent receives the issue description and $\hat{s}$9 and is asked to resolve it. Any non-comment or non-test code change counts as failure (Gloaguen et al., 8 May 2026).

The benchmark separates three task types. In Novel tasks, only $|\hat{s}|=0$0 is provided, tests fail, and the agent should patch. In Partial tasks, an incorrect patch $|\hat{s}|=0$1 is pre-applied, tests still fail, and the agent should repair. In Resolved tasks, the golden $|\hat{s}|=0$2 has already been applied, tests pass, and the agent should abstain. It also varies between a Best scenario with full git history and a ready-to-run environment and a Worst scenario with no .git folder and no preinstalled dependencies (Gloaguen et al., 8 May 2026).

Evaluation used Sonnet-4.6, GPT-5.3 Codex, GPT-5.4 mini, Gemini-3 Pro, Qwen3.5-122B, and the SORCAR framework with GPT-5.3 Codex, across four agent harnesses. The principal metrics are abstention rate

$|\hat{s}|=0$3

failure rate $|\hat{s}|=0$4, and resolution rate $|\hat{s}|=0$5, the fraction of instances where $|\hat{s}|=0$6 makes all golden tests $|\hat{s}|=0$7 pass (Gloaguen et al., 8 May 2026).

On Resolved tasks under the Best+Issue prompt, abstention is far from reliable. Across models, $|\hat{s}|=0$8–$|\hat{s}|=0$9, so $|\hat{s}|>0$0–$|\hat{s}|>0$1. GPT-5.4 mini reached $|\hat{s}|>0$2, implying $|\hat{s}|>0$3, while Sonnet-4.6 reached $|\hat{s}|>0$4, implying $|\hat{s}|>0$5 (Gloaguen et al., 8 May 2026). Under the adverse “Edit” prompt, abstention falls further; for GPT-5.4 mini, $|\hat{s}|>0$6, so $|\hat{s}|>0$7. In the Worst scenario, baseline abstention drops by 10–15 percentage points (Gloaguen et al., 8 May 2026).

The failure modes include unrelated code modifications, no-op edits with nil net effect, and “refactoring” or defense-in-depth additions where none is needed. A targeted mitigation is the “Abstain or Fix” prompt: “First reproduce the issue. If it is not present, report that and make no changes. Otherwise, fix it.” Under this prompt, GPT-5.4 mini’s abstention rises from $|\hat{s}|>0$8 to $|\hat{s}|>0$9 on Resolved tasks, with $$P(f_{\theta'}(c_1)\in\mathcal{A}\mid c_1)\ll P(f_\theta(c_1)\in\mathcal{A}\mid c_1),$$0, cutting failure to $$P(f_{\theta'}(c_1)\in\mathcal{A}\mid c_1)\ll P(f_\theta(c_1)\in\mathcal{A}\mid c_1),$$1 (Gloaguen et al., 8 May 2026). However, the same intervention creates a new failure mode in Partial tasks: agents may discover that the partial patch has already fixed the reproduced symptom and then abstain even though further repair is required. Under “Abstain or Fix,” Sonnet-4.6 incorrectly abstains on over $$P(f_{\theta'}(c_1)\in\mathcal{A}\mid c_1)\ll P(f_\theta(c_1)\in\mathcal{A}\mid c_1),$$2 of Partial tasks, up from $$P(f_{\theta'}(c_1)\in\mathcal{A}\mid c_1)\ll P(f_\theta(c_1)\in\mathcal{A}\mid c_1),$$3 under the Issue prompt (Gloaguen et al., 8 May 2026).

The paper attributes the pattern to overreliance on human guidance. Training objectives, especially RLHF, overwhelmingly reward “doing something” rather than “doing nothing when appropriate,” producing task-level sycophancy toward the implicit assumption that a bug exists. The proposed formal criterion is to apply a patch $$P(f_{\theta'}(c_1)\in\mathcal{A}\mid c_1)\ll P(f_\theta(c_1)\in\mathcal{A}\mid c_1),$$4 only when

$$P(f_{\theta'}(c_1)\in\mathcal{A}\mid c_1)\ll P(f_\theta(c_1)\in\mathcal{A}\mid c_1),$$5

for some threshold $$P(f_{\theta'}(c_1)\in\mathcal{A}\mid c_1)\ll P(f_\theta(c_1)\in\mathcal{A}\mid c_1),$$6, with zero-line patches rewarded as positive examples during RL or supervised fine-tuning (Gloaguen et al., 8 May 2026).

4. Defense training and step-1 competence collapse in LLM agents

“The Autonomy Tax: Defense Training Breaks LLM Agents” studies agent incompetence bias as a capability-alignment paradox: safety training intended to improve resistance to prompt injection reduces the model’s ability to execute benign tasks and still fails against sophisticated attacks (Li et al., 19 Mar 2026). The evaluation spans 97 realistic multi-step AgentDojo tasks across workspace and file management, banking, travel, and Slack communication, with up to $$P(f_{\theta'}(c_1)\in\mathcal{A}\mid c_1)\ll P(f_\theta(c_1)\in\mathcal{A}\mid c_1),$$7 ReAct reasoning–action cycles, plus 1,000 adversarial prompts (Li et al., 19 Mar 2026).

The central metrics are Step-1 Valid Action Rate, Step-1 Refusal Rate, Step-1 Invalid-Output Rate, Completion Rate

$$P(f_{\theta'}(c_1)\in\mathcal{A}\mid c_1)\ll P(f_\theta(c_1)\in\mathcal{A}\mid c_1),$$8

and Cascade Failure Rate

$$P(f_{\theta'}(c_1)\in\mathcal{A}\mid c_1)\ll P(f_\theta(c_1)\in\mathcal{A}\mid c_1),$$9

The cascade amplification factor is $f_\theta$0 (Li et al., 19 Mar 2026).

The quantitative degradation at step 1 is severe. Base Llama-3-8B and Mistral-7B produce valid actions on $f_\theta$1–$f_\theta$2 of benign tasks, with approximately $f_\theta$3 refusal and approximately $f_\theta$4 invalid output. Under StruQ defense, valid actions drop to $f_\theta$5–$f_\theta$6, invalid outputs rise to approximately $f_\theta$7–$f_\theta$8, and refusals are approximately $f_\theta$9–$f_{\theta'}$0. Under SecAlign defense, valid actions are $f_{\theta'}$1–$f_{\theta'}$2, refusals are $f_{\theta'}$3–$f_{\theta'}$4, and invalid outputs are zero (Li et al., 19 Mar 2026).

Trajectory-level competence also collapses. Base Llama-3-8B reaches $f_{\theta'}$5 and $f_{\theta'}$6, while SecAlign Llama-3-8B falls to $f_{\theta'}$7 and rises to $f_{\theta'}$8, an amplification factor of approximately $f_{\theta'}$9. Base Mistral-7B reaches $c_1=[s,u]$0 and $c_1=[s,u]$1, whereas SecAlign Mistral-7B reaches only $c_1=[s,u]$2 and $c_1=[s,u]$3, an amplification factor of approximately $c_1=[s,u]$4 (Li et al., 19 Mar 2026). Defense models exhibit depth-stratified bimodality, concentrating failures at depth 10 as cascade timeouts rather than showing the gradual fall-off across depths 1–9 seen in base models.

The paper attributes these failures to shortcut learning. Defense training on benign–attack pairs pushes empirical risk minimization toward surface correlations such as trigger keywords, XML delimiters, or financial terms, rather than semantic threat understanding. The authors propose chain-of-thought defenses with explicit safety reasoning, multi-stage filtering, agent-aware training on full trajectories with cascade-penalizing loss such as

$c_1=[s,u]$5

and position-invariant architectures (Li et al., 19 Mar 2026). The design objective is to preserve tool execution competence while maintaining context-sensitive threat assessment.

5. Delayed feedback and misattribution in multi-agent human-AI systems

“Biased Error Attribution in Multi-Agent Human-AI Systems Under Delayed Feedback” studies a seven-step naval-battle resource-allocation game in which two AI advisors, DefenseAI and OffenseAI, recommend defensive and offensive allocations, while feedback is delayed until the end of the full sequence (Parakh et al., 24 Mar 2026). The final score is

$c_1=[s,u]$6

with scores $c_1=[s,u]$7 displayed as “0” and positive scores shown exactly (Parakh et al., 24 Mar 2026).

The paper defines an attribution error on trial $c_1=[s,u]$8 when the participant chooses to revise future behavior along dimension $c_1=[s,u]$9 even though the true causal contribution to the loss was dominated by the other dimension $X^*$00, so that

$X^*$01

Misattribution over loss trials is then

$X^*$02

Operationally, the study tracks strategy-level adjustments such as delaying offense or changing defense timing (Parakh et al., 24 Mar 2026).

Defense strategy is labeled by $X^*$03, where $X^*$04 is fully risk-seeking, $X^*$05 fully loss-averse, and $X^*$06 erratic. Within balanced strategies $X^*$07, offense timing is labeled $X^*$08 as early or late. The signed defense shift is

$X^*$09

with positive values indicating more loss-averse corrections. Transition counts are analyzed with Poisson regression, and distributions of $X^*$10 after large losses and large gains are compared using a Mann–Whitney $X^*$11 test (Parakh et al., 24 Mar 2026).

The main result is asymmetric adaptation following negative outcomes. After a bad outcome, switching away from risk-seeking $X^*$12 is more likely than switching away from loss-averse $X^*$13, with $X^*$14, $X^*$15, corresponding to a rate ratio of $X^*$16. The interaction $X^*$17 is $X^*$18, $X^*$19, indicating that switching away from $X^*$20 drops sharply after good outcomes. Under extreme outcomes with $X^*$21, shifts in $X^*$22 concentrate near large positive values after losses, with Mann–Whitney $X^*$23, $X^*$24, $X^*$25 (Parakh et al., 24 Mar 2026).

Offense timing shows a particularly clear misattribution pattern. The baseline intercept is $X^*$26, corresponding to $X^*$27 transitions. The late-offense effect is $X^*$28, $X^*$29, and the interaction $X^*$30 is $X^*$31, $X^*$32. In words, following a poor final score, participants switch from early to late offense at roughly five-to-one odds even when defense timing was the true lever for improvement (Parakh et al., 24 Mar 2026).

The paper recommends per-decision feedback, counterfactual explanations, adaptive blame debiasing, transparency about advisor interdependence, and explicit user training on temporal-credit-assignment heuristics. This suggests that, under delayed and aggregated feedback, agent incompetence bias becomes a credit-assignment failure: the wrong sub-agent is treated as incompetent because causal structure is not directly observable.

6. Misspecified ability beliefs in principal–agent learning

A distinct use of the term appears in “Implicit Incentive Provision with Misspecified Learning,” following Echenique–Li, where “agent incompetence bias” denotes a dogmatic mis-belief about an agent’s innate ability that feeds into distorted learning about effort productivity, effort incentives, and assessment (Echenique et al., 30 Nov 2025). The model posits innate ability $X^*$33, effort $X^*$34 at private cost $X^*$35, productivity parameter $X^*$36, and evaluator-chosen assessment intensity $X^*$37 at cost $X^*$38. The public outcome is

$X^*$39

equivalently

$X^*$40

so increasing $X^*$41 reduces the variance of outcome noise (Echenique et al., 30 Nov 2025).

Society, comprising evaluator and market, holds a dogmatic prior $X^*$42 with $X^*$43, while maintaining a nondegenerate prior over $X^*$44. Long-run updating concentrates beliefs on the $X^*$45 minimizing the discrepancy between the true and perceived mean outcomes:

$X^*$46

With $X^*$47, a Berk–Nash self-confirming equilibrium satisfies

$X^*$48

where $X^*$49 is the truncated map induced by the misspecified-learning problem (Echenique et al., 30 Nov 2025).

Theorem 1 yields directional distortions. If $X^*$50, meaning society underestimates ability, then $X^*$51 is strictly decreasing and has a unique stable fixed point with $X^*$52, so effort productivity is overestimated. If $X^*$53, meaning society overestimates ability, then $X^*$54 is strictly increasing and may have multiple fixed points, all with $X^*$55 (Echenique et al., 30 Nov 2025). In the linear–quadratic case, with $X^*$56, the evaluator’s optimal assessment obeys

$X^*$57

and the first-order condition

$X^*$58

Because generally $X^*$59, an underestimated $X^*$60 implies insufficient assessment, which further worsens learning about $X^*$61 (Echenique et al., 30 Nov 2025).

Theorem 2 states that in any stable equilibrium, the distortion $X^*$62 increases with $X^*$63 and decreases in $X^*$64; equivalently, lowering assessment cost or increasing incentives for assessment mitigates the distortion (Echenique et al., 30 Nov 2025). The applications to education and labor markets describe how stereotypes about “talent” can be reinforced through this feedback loop.

Across these literatures, agent incompetence bias is best understood as a family of systematic deviations in how competence is inferred, preserved, or operationalized. In some settings the bias lies in human judgment; in others it is induced by training objectives, delayed feedback, or misspecified beliefs. What remains constant is the technical structure of the error: observed outcomes, prompts, or priors are allowed to dominate the underlying evidence about what the agent actually did, should do, or can do.