Claude 4.1 Opus: Performance and Challenges

• Claude 4.1 Opus is a large-scale, multimodal vision-language model designed for complex tasks like radiological diagnosis and automated grading.
• Independent evaluations reveal its diagnostic accuracy in radiology is near chance and its reproducibility is extremely poor compared to human experts and other models.
• In programming education, the model grades more strictly than human instructors, exhibiting high internal consistency among similar systems but poor agreement with human standards.

Claude 4.1 Opus, commonly referred to as "claude-opus-4-1," is a large-scale, general-purpose vision-LLM (VLM) developed by Anthropic and released as a frontier multimodal AI system. The model has attracted significant academic scrutiny due to its evaluation in high-stakes professional contexts, notably complex radiological diagnosis and automated programming assignment assessment. Two recent, independent, large-scale benchmarking studies highlight its performance profile, practical limitations, comparative reliability, error taxonomies, and implications for deployment in clinical and educational settings (Datta et al., 29 Sep 2025, Jukiewicz, 30 Sep 2025).

1. Diagnostic Performance in Expert-Level Radiology

In the RadLE v1 benchmark, which presents 50 expert-level "spot diagnosis" medical imaging cases, Claude Opus 4.1 achieved a mean diagnostic accuracy of 0.01 (1%), with a Wilson 95% confidence interval of 0%–3%. This performance is statistically indistinguishable from random guessing and is dramatically lower than all other evaluated cohorts: board-certified radiologists (0.83, 95% CI 0.75–0.90), radiology trainees (0.45, 95% CI 0.39–0.52), and peer commercial VLMs such as GPT-5 (0.30), Gemini 2.5 Pro (0.29), and OpenAI o3 (0.23). See summary:

Group	Mean Accuracy	95% CI
Board-certified radiologists	0.83	0.75–0.90
Radiology trainees	0.45	0.39–0.52
GPT-5	0.30	0.20–0.42
Gemini 2.5 Pro	0.29	0.19–0.39
OpenAI o3	0.23	0.14–0.33
Grok-4	0.12	0.06–0.19
Claude Opus 4.1	0.01	0.00–0.03

Statistical analysis using the Friedman test ($\chi^{2} = 336.0$, $p < 1 \times 10^{-64}$; Kendall’s $W = 0.56$) confirmed that Claude Opus 4.1 underperformed every comparator (Holm-adjusted Wilcoxon $p \leq 0.010$ against trainees) (Datta et al., 29 Sep 2025).

2. Reproducibility and Reliability Analysis

Claude Opus 4.1 exhibited essentially no reproducibility across three independent runs of RadLE v1. Pairwise quadratic‐weighted Cohen’s kappa ($\kappa$) spanned –0.02 to 0.00 (mean $\kappa \approx -0.01$), corresponding to "poor" reliability. The two-way random-effects intraclass correlation coefficient, ICC(2,1), was near zero (≈ 0.00, 95% CI –0.15–0.17), signifying that nearly all performance variance was unexplained noise rather than systematic skill.

By comparison, Grok-4 (the next-worst model) achieved $\kappa$ = 0.27–0.62 (ICC = 0.41), and top models (GPT-5, o3) reached "substantial" agreement ($\kappa \approx 0.64$, ICC $\approx 0.64$). The mathematical formalism for these statistics was directly reported, reflecting rigorous evaluation and a severe lack of internal consistency for Claude Opus 4.1 (Datta et al., 29 Sep 2025).

3. Error Taxonomy: Nature of Visual and Reasoning Failures

The failure modes of Claude Opus 4.1 were classified using a structured taxonomy:

• Perceptual errors: under-detection (omission of subtle findings), over-detection (hallucinated pathologies), mislocalization (labeling on incorrect anatomical sides).
• Interpretive errors: misattribution (incorrect inference to disease), premature closure (ceasing reasoning after identifying an implausible diagnosis).
• Communication errors: discordance between summarized findings and the final diagnostic conclusion.
• Cognitive bias modifiers: anchoring/confirmation bias, availability bias, inattentional and framing effects.

Claude Opus 4.1 exhibited these errors in exaggerated form, with under-detection and interpretive errors dominating. Pathologies were routinely missed (e.g., subtle CT nodules, small fractures), and misclassification was frequent. Over-detection and communication discordance occurred regularly, wherein findings listed contradicted conclusions. Cognitive biases led to diagnostic rigidity or a bias towards common pathologies (Datta et al., 29 Sep 2025).

4. Underlying Causes of Poor Performance

Multiple factors contributed to Claude Opus 4.1's degradation on clinical vision-language reasoning tasks:

• Insufficient domain specialization: The model was fine-tuned on broad Internet data and lacked the domain priors and vocabulary necessary for subtle radiological task performance.
• Deficient visual feature extraction: Systematic inability to identify low-contrast or small abnormalities pointed to architectural limitations in the vision encoder.
• Non-deterministic interface behavior: Stochastic inference, potentially caused by ambiguous web interface sampling and A/B testing regimes, increased output variance and eliminated reproducibility.
• Failure of advanced "reasoning" modes: Despite the "Opus 4.1 Reasoning Mode" designation, no empirical performance improvement from structured or elaborate prompts was observed; accuracy remained at chance with variable latency and no meaningful increase in reasoning depth (Datta et al., 29 Sep 2025).

5. Implications for Clinical and Educational Deployment

RadLE v1 benchmark findings demonstrate that unsupervised deployment of Claude Opus 4.1 in radiological workflows is contraindicated: 1% diagnostic accuracy and essentially zero reproducibility render it unsuitable for independent clinical use. Robust human expert oversight is mandated. The authors advocate:

• Domain-specific fine-tuning: Genuine improvement for high-stakes vision-language analytics requires tailored training on curated radiology datasets, with attention to subtle and rare findings.
• Transparent sampling control: Platforms must expose inferential parameters and enable reproducibility logging to meet clinical audit and regulatory standards.
• Hybrid AI–human workflows: The model may at best contribute to low-level flagging, not final decision-making.
• Error-aware user interfaces: Interfaces should highlight low-confidence outputs and annotate predictable failure modes for downstream users (Datta et al., 29 Sep 2025).

6. Automated Grading in Programming Education: Quantitative and Comparative Findings

A large-scale evaluation of Claude Opus 4.1 in the grading of 6,081 introductory programming assignments revealed the following profile (Jukiewicz, 30 Sep 2025):

• Grade distributions: 46.7% of submissions graded 0 (incorrect), 23.0% at 0.5 (almost correct), and 30.3% at 1 (correct).
• Central tendency: Sample mean of 0.418, standard deviation 0.431, substantially lower than the human instructor mean of 0.726 (SD 0.391). The model exhibits stricter grading than human teachers, awarding full marks less frequently.
• Reliability: ICC(2,1) with the reference grader was 0.382 ("poor to fair" reliability, sub-threshold for "good" reliability, defined as ICC ≥ 0.75). The closest model, claude-haiku-3.5, reached 0.470.
• Model consensus: Claude Opus 4.1 demonstrated high agreement with the average LLM consensus (ICC = 0.888), but this internal consistency among models diverged from human-grading standards.
• Cluster analysis: The model grouped closely with other Anthropic models ("Claude cluster") and demonstrated high internal consistency within this group (Spearman ρ ≈ 0.78–0.83).
• Statistical comparisons: No significant mean-score differences between Claude Opus 4.1 and both gemini-2.5-flash and gpt-5 (post hoc pairwise Conover test, Holm/Holm–Sidak corrections, p ≥ 0.05).

Pedagogical recommendations emphasize the necessity of aligning model selection with grading philosophy, ongoing human oversight, and vigilant monitoring for grading fairness and bias. The persistent gap to human performance, even for the best-aligned LLMs, precludes direct substitution for high-stakes evaluation scenarios (Jukiewicz, 30 Sep 2025).

7. Synthesis and Outlook

Claude Opus 4.1, while exhibiting strong internal consistency within vendor clusters and consensus with other large-scale LLMs, substantially underperforms on expert-level clinical reasoning tasks and demonstrates a grading philosophy more stringent than human instructors in educational settings. Its practical deployment in safety-critical or high-value assessment scenarios is limited by near-chance task performance, poor reproducibility, and notable misalignment with human normative standards. Research highlights the vital need for domain adaptation, transparency in sampling, and rigorous evaluation frameworks to close the gap between generalist multimodal models and specialized human expertise (Datta et al., 29 Sep 2025, Jukiewicz, 30 Sep 2025).