The Inattentional Gap: Task-Conditioned Language and Vision Models Omit the Safety-Critical Signals They Can Otherwise Report

Abstract: AI safety is evaluated by how reliably a model detects the hazards it is told to find, yet accidents often arise from the hazard no one specified. We show that conditioning a language or vision model on a narrow task suppresses its reporting of co-present, safety-critical signals it can otherwise report, a machine analogue of human inattentional blindness arising from a different mechanism. Across radiology and driving text scenarios and chest-radiograph vision tasks, suppression appeared in every model tested, did not diminish with scale, persisted in a reasoning model, and varied more by model family than by size, while the same models reported these signals at substantially higher rates when unconstrained. We name this dissociation the Inattentional Gap and argue that it decouples measured benchmark safety from real-world safety: a system can score near-perfectly on the hazards an evaluation specifies while remaining blind to those that cause harm.

• The paper demonstrates a measurable Inattentional Gap: models that report co-present hazards under open instructions often omit them under narrow tasks, including 0.00 versus about 1.00 reporting in radiology scenarios.
• The experiments show that suppression occurs across language and vision systems, is driven more by output scope than task-load or model scale, and varies substantially by model family.
• The findings suggest deployment safety requires task-conditioned omission tests and external critic models, since a second open-ended review process restored suppressed safety reports from 0.00 to 1.00 in one probe.

Task-conditioning is the default deployment mode for large language and vision-LLMs in safety-critical settings: a radiology assistant is asked about a nodule, a driving system about a lead vehicle, a screening model about a specified threat. The paper under review demonstrates that this scoping carries a measurable cost that standard safety evaluation does not capture. Across language and vision tasks, the same model that reports a co-present, life-threatening signal when given an open instruction omits that signal almost entirely when instructed to perform a narrow task. The author formalizes this within-item dissociation as the Inattentional Gap and argues that it decouples benchmark safety from deployment safety (2606.26529).

Construct and design

The construct is defined behaviorally. For a model $m$ and item $i$, let $R^{\text{open}}_{m,i}$ and $R^{\text{task}}_{m,i}$ denote whether the safety-critical signal is reported under an unconstrained and a task-conditioned instruction. The item-level gap is $$IG_{m,i} = R^{\text{open}}_{m,i} - R^{\text{task}}_{m,i}$$, and the model-level gap is estimated only over items where the model reports the signal in the open condition. This within-item reportability control is the paper's key methodological move: it separates task-induced omission from mere capability failure, because an item whose open condition does not surface the signal cannot support a suppression claim and is excluded. The author anticipates the objection that omission under a narrowing instruction is simply instruction-following, and answers that instruction is precisely the channel through which an attentional set is imposed in humans as well; the open-condition recovery is the machine analogue of the full-attention control.

The empirical program comprises six experiments: 100 procedurally composed text scenarios (64 radiology, 36 driving) tested on four models under focused, strict, and open instructions; 48 composited chest radiographs tested under a rib-counting versus open-description instruction; a dose study varying sub-question count from zero to three; a cross-vendor validation on three flagship models; and two dual-process probes. Adjudication used two independent language-model judges rather than keyword matching, with Cohen $\kappa$ between 0.895 and 0.98 across studies, and cluster bootstrap over templates confirmed robustness to the procedural composition of stimuli.

Suppression in language and vision

The strict-condition results are the strongest in the paper. Under an instruction to report only the designated target, every model reported the co-present critical signal at a rate of 0.00 in radiology, against approximately 1.00 under the open instruction (McNemar $p = 1.1 \times 10^{-19}$, $N = 64$); driving showed 0.08–0.33 versus 1.00 ($p < 1.3 \times 10^{-7}$). Under the high-load focused instruction, suppression was substantial but heterogeneous: radiology report rates of 0.08–0.53 against open-condition ceilings, with gaps of 0.47–0.92 (all $p < 2 \times 10^{-9}$). Notably, the frontier gpt-4o suppressed as strongly as the small gpt-4o-mini (0.08 vs 0.09), an early indication that scale does not resolve the effect.

The vision study transplants the invisible-gorilla paradigm: a subtle translucent disc or yellow star composited onto real chest radiographs, presented under a rib-counting task. Every model suppressed report of the object under the counting task. For the star, gpt-4o and gpt-4o-mini omitted it in every counting item while reporting it at rates of 0.96 and 0.79 under open description (exact McNemar $i$0 for gpt-4o, $i$1, $i$2). Inspection of responses confirmed task-induced omission rather than refusal: models performed the count earnestly and simply did not mention the object, then flagged "an artificial object" on the same image under the open instruction. Open-condition report was below ceiling for the subtlest stimuli, but the within-item design conditions on each model's own open-condition reportability, so this does not confound the estimate.

Mechanism: output scope, not load

The dose study yields the paper's most distinctive mechanistic claim. Report of the critical finding collapsed from 1.00 at zero sub-questions to near zero at one, and adding further questions did not monotonically worsen suppression: the Anthropic models partially recovered (haiku from 0.00 to 0.88 across one to three questions) while the OpenAI models stayed at zero. Response-length analysis resolves this: the single narrow question produced the shortest outputs (mean 189 characters, versus 435 and 1,046 for two and three questions, and 1,770 open), and longer outputs re-admitted the critical finding. The proximal trigger is therefore the breadth of committed output—output scope—rather than question count or a monotonic load gradient. This distinguishes the machine effect from human inattentional blindness, which is classically attributed to a capacity-limited attentional bottleneck that transformers do not instantiate. The behavioral parity is real; the mechanism is report-conditioning, not perceptual load.

Scale, reasoning, and model family

Cross-vendor validation on three flaggships sharpened the heterogeneity into a family-level pattern. GPT-5, a reasoning model, showed complete suppression in valid responses (0.00 focused and strict versus 1.00 open; strict McNemar $i$3 on 21 paired items after reasoning-budget attrition), refuting the hypothesis that inference-time reasoning alone supplies boundary monitoring. Gemini 2.5 Pro suppressed nearly as strongly (0.00–0.09). Opus 4.8, by contrast, reported the co-present finding in 0.83–0.96 of task-conditioned items, including 0.96 under an adversarial-suppression instruction that explicitly forbade safety caveats and scored such mentions as failure—indicating a robust model-family safety-reporting disposition rather than a fragile prompt reflex. Across all studies, model family was a stronger correlate of the gap than model scale. The implication is that the highest-risk deployment is a high-load narrow task over a low-salience but safety-critical signal, handled by a model family that compresses output to the requested target.

The dual-process probes complete the picture. Routing the focused reports of a fully suppressed task model (gpt-4o-mini, 0.00 on its own) to an independent open-ended critic raised the pipeline report rate to 1.00 (McNemar $i$4), reproduced exactly across repeated runs. In dual-process terms, the conditioned model exhibits System-1-style task capture; where monitoring appeared in the sampled systems, it took the form of either a model-specific reporting disposition or an explicit second process, never an architecture-general behavior.

Evaluation implications

The paper's central evaluative claim is that conventional safety benchmarks measure performance on the specified target set while deployment harm often turns on the unspecified set, so benchmark gains need not translate into deployment safety. Because exhaustive, unscoped analysis on every input is infeasible in cost and certification terms, real deployment is necessarily task-scoped, and the same scoping that makes AI tractable creates the condition under which unrequested critical signals are omitted. The paper also connects this to the human-factors literature on the ironies of automation: a model's reporting gate and an operator's automation complacency form two attention gates in series, compounding the risk that an omitted finding is never independently recovered.

Limitations

The paper concedes several constraints plainly. Scale is moderate (100 textual and 48 visual scenarios), and adjudication relied on language-model judges; blinded human and domain-expert adjudication remains to be added. The visual stimuli used composited non-anatomical objects, which isolates the attentional-set mechanism at the cost of clinical realism; a pilot on real radiographs using each model's own open-condition description as reference qualitatively reproduced the suppression (e.g., a model reporting cardiomegaly openly omitted it under rib counting), but reliable quantification was limited by judge disagreement on open-ended clinical findings and by vision refusals. The dual-process probes use only 24 scenarios and are demonstrations rather than benchmarks. The construct is established behaviorally but not yet causally decomposed into the separate contributions of task load, salience, output scope, and family-specific dispositions. Whether the Opus 4.8 override generalizes beyond these items is an open empirical question.

Conclusion

The paper demonstrates a robust, statistically strong, within-item dissociation between what task-conditioned models report and what the same models can report, across modalities, domains, and seven models spanning three vendors. The effect is insensitive to scale, survives inference-time reasoning, is triggered by output scope rather than perceptual load, and varies more by model family than by capability. Its principal contribution is to shift the safety-evaluation question from whether a model can identify a specified hazard to whether task-conditioning suppresses reporting of hazards it can otherwise report—and to show that where intrinsic monitoring is absent, an explicit external critic can close the gap.