---
title: Agent Incompetence Bias Insights
url: https://www.emergentmind.com/topics/agent-incompetence-bias
type: topic
---

# Agent Incompetence Bias Insights

Searching arXiv for the cited papers to ground the article in recent literature.
Agent incompetence bias is a label used in recent literature for several distinct but related phenomena in which actual agent performance becomes misaligned with judgment, intervention, or action policy. In human–agent interaction, it denotes post-hoc distortion of users’ trait attributions to an agent in the direction of their own outcomes while the agent’s behavior is held constant [2301.09011]. In coding-agent evaluation, the closely related “action bias” denotes a tendency to produce a non-empty patch even when no code change is needed [2605.07769]. In LLM-agent safety, it denotes training-induced degradation of step-1 tool-use competence on benign tasks [2603.19423]. In multi-agent decision support under delayed feedback, it appears as biased error attribution to the wrong advisor [2603.23419]. In principal–agent learning models, it names a dogmatic mis-belief about an agent’s innate ability that distorts inference about effort productivity and assessment [2512.01129]. The term therefore does not identify a single invariant construct; rather, it marks a family of competence-related biases at the levels of perception, control, evaluation, and autonomous execution.

## 1. Terminological scope and canonical formulations

Recent usage divides into at least five technically distinct formulations.

| Usage | Core distortion | Canonical setting |
|---|---|---|
| Post-hoc trait attribution | User outcomes color judgments of ability, benevolence, and integrity | Human–robot reconnaissance missions |
| Action bias in coding agents | Agent edits code when abstention is correct | SWE-Bench-derived “Resolved” tasks |
| Defense-induced incompetence | Defense training reduces valid benign tool calls | AgentDojo multi-step tool-use tasks |
| Delayed-feedback misattribution | Humans “correct” the wrong AI advisor after a loss | Seven-step naval-battle allocation game |
| Misspecified ability belief | Society under- or overestimates innate ability and mislearns productivity | Principal–agent assessment and market inference |

In the behavioral studies by Gurney, Pynadath, and Wang, agent incompetence bias is the “post-hoc distortion of users’ trait attributions to an agent—ability, benevolence, integrity—in the direction of their own observed outcomes, holding the agent’s actual behavior constant” [2301.09011]. In “Coding Agents Don’t Know When to Act,” the relevant failure is formalized by a repository $R$ in which the golden patch $X^*$ has already been applied and all golden tests $T^*$ pass; a competent agent should output a patch $\hat{s}$ with $|\hat{s}|=0$, yet many agents return $|\hat{s}|>0$ [2605.07769]. In “The Autonomy Tax,” agent incompetence bias is expressed as
$$
P(f_{\theta'}(c_1)\in\mathcal{A}\mid c_1)\ll P(f_\theta(c_1)\in\mathcal{A}\mid c_1),
$$
where $f_\theta$ is a base instruction-tuned model, $f_{\theta'}$ its defense-trained counterpart, $c_1=[s,u]$ the initial benign context, and $\mathcal{A}$ the set of syntactically correct tool actions [2603.19423].

A plausible unifying interpretation is that each formulation isolates a failure of competence attribution or competence preservation under conditions where the ground truth is, by design, held fixed or recoverable. The literatures differ mainly in whether the distortion arises in the human evaluator, in the training pipeline, or in the agent’s own policy over actions.

## 2. Post-hoc trait attribution in human–agent interaction

The most explicit psychological formulation appears in the search-and-rescue experiments reported by Gurney, Pynadath, and Wang [2301.09011]. Participants interacted with a simulated robot that scouted locations using three potentially noisy sensors and issued binary recommendations, “gear up” versus “gear down.” Crucially, the robot’s sensor readings and recommendations were scripted, so variation in mission outcomes arose from participants’ own decisions to follow or ignore the advice rather than from differences in robot behavior.

Study 1 used 78 participants, each completing three missions of eight locations, under a $2\times4$ factorial crossing robot ability and explanation type. Study 2 used 159 participants in a single 45-location mission with one of three reinforcement-learning–based robots. In both studies, the key behavioral predictors were $CP$, the percentage of correct decisions, and $FP$, the percentage of times the participant followed the robot’s advice [2301.09011].

After the mission, participants completed a modified Mayer and Davis three-factor trust inventory on a 7-point Likert scale. Ability was measured with 13 items, benevolence with 5 items, and integrity with 3 items, with one integrity item reverse-coded. For each trait $Y_i$, the full regression model added correct-decision percentage and follow-advice percentage to treatment controls, and the significance of $\beta_{\mathrm{CP}}$ was tested both directly and by comparing the control and full models [2301.09011].

The strongest and most consistent effect was on benevolence. In Study 1, $\beta_{\mathrm{CP}}=3.016$, $\mathrm{SE}=1.262$, $t\approx2.39$, with $F(6,192)=3.498$, $p<.01$ for adding $CP$ to the control model. In Study 2, $\beta_{\mathrm{CP}}=4.522$, $\mathrm{SE}=1.640$, $t\approx2.76$, with $F(4,154)=4.966$, $p<.001$ [2301.09011]. Ability showed a detectable effect in Study 2, $\beta_{\mathrm{CP}}=3.034$, $\mathrm{SE}=1.377$, $t\approx2.20$, $p<.05$, although the corresponding model-improvement test was marginal. Integrity showed a significant effect in Study 1, $\beta_{\mathrm{CP}}=1.913$, $\mathrm{SE}=0.894$, $t\approx2.14$, $p<.05$, but no significant effect in Study 2 [2301.09011].

The paper interprets the phenomenon as a close analog of the fundamental attribution error. The proposed mechanisms include affect-misattribution, outcome-driven causal inference, and mindless anthropomorphism. The corresponding mitigation proposals are to augment user-belief models with user performance and compliance features, embed debiasing prompts in the interface, present objective logs of the agent’s own accuracy and uncertainty estimates, and incorporate mood or outcome sensors so that agents can detect when emotional valence may color judgments [2301.09011].

## 3. Action bias and abstention failure in coding agents

In coding-agent evaluation, the bias is operational rather than perceptual. FixedBench contains 200 human-verified tasks sampled from SWE-Bench Verified, each corresponding to a GitHub issue whose fix has already been merged, so that no further edits to production code are required [2605.07769]. Construction proceeds from a pre-patch repository $R$ and golden patch $X^*$; applying $X^*$ yields $R\circ X^*$, the “Resolved” setting. The agent receives the issue description and $R\circ X^*$ and is asked to resolve it. Any non-comment or non-test code change counts as failure [2605.07769].

The benchmark separates three task types. In Novel tasks, only $R$ is provided, tests fail, and the agent should patch. In Partial tasks, an incorrect patch $X^P$ is pre-applied, tests still fail, and the agent should repair. In Resolved tasks, the golden $X^*$ has already been applied, tests pass, and the agent should abstain. It also varies between a Best scenario with full git history and a ready-to-run environment and a Worst scenario with no .git folder and no preinstalled dependencies [2605.07769].

Evaluation used Sonnet-4.6, GPT-5.3 Codex, GPT-5.4 mini, Gemini-3 Pro, Qwen3.5-122B, and the SORCAR framework with GPT-5.3 Codex, across four agent harnesses. The principal metrics are abstention rate
$$
A=\text{fraction of instances where }|\hat{s}|=0,
$$
failure rate $F=1-A$, and resolution rate $R$, the fraction of instances where $\hat{s}$ makes all golden tests $T^*$ pass [2605.07769].

On Resolved tasks under the Best+Issue prompt, abstention is far from reliable. Across models, $A\approx35$–$65\%$, so $F\approx35$–$65\%$. GPT-5.4 mini reached $A=60.5\%\pm6.8\%$, implying $F=39.5\%\pm6.8\%$, while Sonnet-4.6 reached $A=65.0\%\pm6.6\%$, implying $F=35.0\%\pm6.6\%$ [2605.07769]. Under the adverse “Edit” prompt, abstention falls further; for GPT-5.4 mini, $A=36.5\%$, so $F=63.5\%$. In the Worst scenario, baseline abstention drops by 10–15 percentage points [2605.07769].

The failure modes include unrelated code modifications, no-op edits with nil net effect, and “refactoring” or defense-in-depth additions where none is needed. A targeted mitigation is the “Abstain or Fix” prompt: “First reproduce the issue. If it is not present, report that and make no changes. Otherwise, fix it.” Under this prompt, GPT-5.4 mini’s abstention rises from $60.5\%$ to $88.5\%$ on Resolved tasks, with $p<10^{-8}$, cutting failure to $11.5\%$ [2605.07769]. However, the same intervention creates a new failure mode in Partial tasks: agents may discover that the partial patch has already fixed the reproduced symptom and then abstain even though further repair is required. Under “Abstain or Fix,” Sonnet-4.6 incorrectly abstains on over $81.3\%$ of Partial tasks, up from $29.3\%$ under the Issue prompt [2605.07769].

The paper attributes the pattern to overreliance on human guidance. Training objectives, especially RLHF, overwhelmingly reward “doing something” rather than “doing nothing when appropriate,” producing task-level sycophancy toward the implicit assumption that a bug exists. The proposed formal criterion is to apply a patch $\hat{s}$ only when
$$
P[\text{bug present}\mid \text{evidence}]>\tau,
$$
for some threshold $\tau$, with zero-line patches rewarded as positive examples during RL or supervised fine-tuning [2605.07769].

## 4. Defense training and step-1 competence collapse in LLM agents

“The Autonomy Tax: Defense Training Breaks LLM Agents” studies agent incompetence bias as a capability-alignment paradox: safety training intended to improve resistance to prompt injection reduces the model’s ability to execute benign tasks and still fails against sophisticated attacks [2603.19423]. The evaluation spans 97 realistic multi-step AgentDojo tasks across workspace and file management, banking, travel, and Slack communication, with up to $T_{\max}=10$ ReAct reasoning–action cycles, plus 1,000 adversarial prompts [2603.19423].

The central metrics are Step-1 Valid Action Rate, Step-1 Refusal Rate, Step-1 Invalid-Output Rate, Completion Rate
$$
CR=\frac{|\{i:\tau_i\text{ finishes within }T_{\max}\text{ with a correct “finish” action}\}|}{97},
$$
and Cascade Failure Rate
$$
CFR(f_{\theta'})=P(\forall t\le T_{\max}: f_{\theta'}(c_t)\neq \text{“finish”}\mid u\ \text{benign}).
$$
The cascade amplification factor is $CFR(f_{\theta'})/CFR(f_\theta)$ [2603.19423].

The quantitative degradation at step 1 is severe. Base Llama-3-8B and Mistral-7B produce valid actions on $96.9$–$97.9\%$ of benign tasks, with approximately $1\%$ refusal and approximately $2\%$ invalid output. Under StruQ defense, valid actions drop to $22.7$–$23.7\%$, invalid outputs rise to approximately $69$–$71\%$, and refusals are approximately $5$–$8\%$. Under SecAlign defense, valid actions are $52.6$–$53.6\%$, refusals are $46$–$47\%$, and invalid outputs are zero [2603.19423].

Trajectory-level competence also collapses. Base Llama-3-8B reaches $CR=86.6\%$ and $CFR=13.4\%$, while SecAlign Llama-3-8B falls to $CR=63.9\%$ and rises to $CFR=36.1\%$, an amplification factor of approximately $2.7\times$. Base Mistral-7B reaches $CR=50.5\%$ and $CFR=49.5\%$, whereas SecAlign Mistral-7B reaches only $CR=1.0\%$ and $CFR=99.0\%$, an amplification factor of approximately $2.0\times$ [2603.19423]. Defense models exhibit depth-stratified bimodality, concentrating failures at depth 10 as cascade timeouts rather than showing the gradual fall-off across depths 1–9 seen in base models.

The paper attributes these failures to shortcut learning. Defense training on benign–attack pairs pushes empirical risk minimization toward surface correlations such as trigger keywords, XML delimiters, or financial terms, rather than semantic threat understanding. The authors propose chain-of-thought defenses with explicit safety reasoning, multi-stage filtering, agent-aware training on full trajectories with cascade-penalizing loss such as
$$
\mathcal{L}_{\text{cascade}}=\sum_t \alpha^{T-t}\ell(a_t,y_t),\quad \alpha>1,
$$
and position-invariant architectures [2603.19423]. The design objective is to preserve tool execution competence while maintaining context-sensitive threat assessment.

## 5. Delayed feedback and misattribution in multi-agent human-AI systems

“Biased Error Attribution in Multi-Agent Human-AI Systems Under Delayed Feedback” studies a seven-step naval-battle resource-allocation game in which two AI advisors, DefenseAI and OffenseAI, recommend defensive and offensive allocations, while feedback is delayed until the end of the full sequence [2603.23419]. The final score is
$$
G=H_{\rm red\,carrier\,damage}-H_{\rm blue\,carrier\,damage},
$$
with scores $\le 0$ displayed as “0” and positive scores shown exactly [2603.23419].

The paper defines an attribution error on trial $t$ when the participant chooses to revise future behavior along dimension $a_t\in\{\mathrm{def},\mathrm{off}\}$ even though the true causal contribution to the loss was dominated by the other dimension $c_t\in\{\mathrm{def},\mathrm{off}\}$, so that
$$
\mathbb{I}\{a_t\neq c_t\}=1.
$$
Misattribution over loss trials is then
$$
\mathrm{Misattr}=\frac{1}{N}\sum_{t:G_t\le 0}\mathbb{I}\{a_t\neq c_t\}.
$$
Operationally, the study tracks strategy-level adjustments such as delaying offense or changing defense timing [2603.23419].

Defense strategy is labeled by $S\in\{0,1,2,3,4,5\}$, where $S=0$ is fully risk-seeking, $S=4$ fully loss-averse, and $S=5$ erratic. Within balanced strategies $S\in\{1,2,3\}$, offense timing is labeled $O\in\{1,2\}$ as early or late. The signed defense shift is
$$
\Delta S_t=S_{t+1}-S_t,
$$
with positive values indicating more loss-averse corrections. Transition counts are analyzed with Poisson regression, and distributions of $\Delta S$ after large losses and large gains are compared using a Mann–Whitney $U$ test [2603.23419].

The main result is asymmetric adaptation following negative outcomes. After a bad outcome, switching away from risk-seeking $S=0$ is more likely than switching away from loss-averse $S=4$, with $\beta_{S0}=1.10$, $p=.045$, corresponding to a rate ratio of $\exp(1.10)\approx3.0$. The interaction $S=0\times\mathrm{Good}$ is $\delta_{S0}=-1.79$, $p=.016$, indicating that switching away from $S=0$ drops sharply after good outcomes. Under extreme outcomes with $|G|\ge4$, shifts in $\Delta S$ concentrate near large positive values after losses, with Mann–Whitney $U=547$, $p=0.043$, $r=-0.30$ [2603.23419].

Offense timing shows a particularly clear misattribution pattern. The baseline intercept is $\alpha'=1.70$, corresponding to $\exp(1.70)\approx5.5$ transitions. The late-offense effect is $\beta'=-1.70$, $p=.027$, and the interaction $O=2\times\mathrm{Good}$ is $\delta'=2.62$, $p=.007$. In words, following a poor final score, participants switch from early to late offense at roughly five-to-one odds even when defense timing was the true lever for improvement [2603.23419].

The paper recommends per-decision feedback, counterfactual explanations, adaptive blame debiasing, transparency about advisor interdependence, and explicit user training on temporal-credit-assignment heuristics. This suggests that, under delayed and aggregated feedback, agent incompetence bias becomes a credit-assignment failure: the wrong sub-agent is treated as incompetent because causal structure is not directly observable.

## 6. Misspecified ability beliefs in principal–agent learning

A distinct use of the term appears in “Implicit Incentive Provision with Misspecified Learning,” following Echenique–Li, where “agent incompetence bias” denotes a dogmatic mis-belief about an agent’s innate ability that feeds into distorted learning about effort productivity, effort incentives, and assessment [2512.01129]. The model posits innate ability $\eta\sim N(\mu^*,1)$, effort $a\ge0$ at private cost $c(a)$, productivity parameter $\beta>0$, and evaluator-chosen assessment intensity $h\in[0,1]$ at cost $\kappa(h)$. The public outcome is
$$
X=\eta+r(a,\beta)+\epsilon,\qquad \epsilon\sim N(0,h^{-1}-1),
$$
equivalently
$$
X\sim N(\mu^*+r(a,\beta),h^{-1}),
$$
so increasing $h$ reduces the variance of outcome noise [2512.01129].

Society, comprising evaluator and market, holds a dogmatic prior $\Pr[\mu=\hat{\mu}]=1$ with $\hat{\mu}\neq\mu^*$, while maintaining a nondegenerate prior over $\beta$. Long-run updating concentrates beliefs on the $\beta$ minimizing the discrepancy between the true and perceived mean outcomes:
$$
\min_{\beta\in[\underline{\beta},\bar{\beta}]} \big[\hat{\mu}+R(h,\beta)-(\mu^*+R(h,\beta^*))\big]^2.
$$
With $\Delta_\mu=\hat{\mu}-\mu^*$, a Berk–Nash self-confirming equilibrium satisfies
$$
\hat{\beta}=\tilde{\psi}(\hat{\beta}),\qquad \hat{h}=h(\hat{\beta}),
$$
where $\tilde{\psi}$ is the truncated map induced by the misspecified-learning problem [2512.01129].

Theorem 1 yields directional distortions. If $\Delta_\mu<0$, meaning society underestimates ability, then $\tilde{\psi}$ is strictly decreasing and has a unique stable fixed point with $\hat{\beta}>\beta^*$, so effort productivity is overestimated. If $\Delta_\mu>0$, meaning society overestimates ability, then $\tilde{\psi}$ is strictly increasing and may have multiple fixed points, all with $\hat{\beta}<\beta^*$ [2512.01129]. In the linear–quadratic case, with $r(a,\beta)=\beta a$, the evaluator’s optimal assessment obeys
$$
a(h,\beta)=\frac{h\beta}{c},\qquad R(h,\beta)=\frac{h\beta^2}{c},
$$
and the first-order condition
$$
\frac{\lambda_1\beta^2}{c}=\kappa'(h).
$$
Because generally $h'(\beta)>0$, an underestimated $\beta$ implies insufficient assessment, which further worsens learning about $\beta$ [2512.01129].

Theorem 2 states that in any stable equilibrium, the distortion $|\hat{\beta}-\beta^*|$ increases with $|\Delta_\mu|$ and decreases in $h$; equivalently, lowering assessment cost or increasing incentives for assessment mitigates the distortion [2512.01129]. The applications to education and labor markets describe how stereotypes about “talent” can be reinforced through this feedback loop.

Across these literatures, agent incompetence bias is best understood as a family of systematic deviations in how competence is inferred, preserved, or operationalized. In some settings the bias lies in human judgment; in others it is induced by training objectives, delayed feedback, or misspecified beliefs. What remains constant is the technical structure of the error: observed outcomes, prompts, or priors are allowed to dominate the underlying evidence about what the agent actually did, should do, or can do.

Source: https://www.emergentmind.com/topics/agent-incompetence-bias