Performance and failure modes of AI chatbots on a novel concept inventory on relativity in classical mechanics

Abstract: AI chatbots are increasingly used by students as study tools in physics, raising practical questions about their reliability on conceptual tasks. Existing evaluations of LLMs on physics concept inventories rely almost exclusively on instruments that have been publicly available for years and likely appear in model training data, making it difficult to disentangle physics competence from familiarity with the test items themselves. We address this issue by evaluating three frontier LLMs (GPT-5.2, Gemini 3 Pro, Gemini 3 Flash) on the Classical Relativity Concept Inventory (CRCI), a recently developed and validated 21-item instrument on Galilean relativity that was not publicly available at the time of testing. Each item was administered 30 times per model, and all 1890 responses were qualitatively coded along three dimensions: visual interpretation, physics reasoning, and coordination. Mean accuracy was 97% for Gemini 3 Flash, 89% for Gemini 3 Pro, and 73% for GPT-5.2, compared to 62% for the student sample (N = 267). However, all three models fail completely on a small number of items. The qualitative analysis shows that these failures stem predominantly from misinterpretations of visual content rather than from deficits in physics knowledge, and that LLM errors differ structurally from those of students: when models err, they converge on a single distractor with high consistency, whereas student errors are more broadly distributed. These findings indicate that chatbot reliability on conceptual physics is item-dependent and unpredictable, with direct implications for how concept inventories are administered.

• The paper demonstrates that multimodal AI chatbots achieve high accuracy (up to 97%) on a controlled relativity concept inventory.
• It employs structured chain-of-thought output and qualitative coding across visual, reasoning, and coordination axes to delineate failure modes.
• Findings underscore that visual interpretation errors, rather than physics reasoning, are the dominant limitation affecting AI performance.

Evaluation of AI Chatbot Performance and Failure Modes on a Novel Classical Relativity Concept Inventory

Summary of Methodology

This paper addresses critical limitations in evaluating LLMs on physics concept inventories, specifically the issue of item leakage from training datasets. The authors utilize the Classical Relativity Concept Inventory (CRCI), a 21-item diagnostic instrument on Galilean relativity and non-inertial reference frames, intentionally unavailable online during model training. Three state-of-the-art multimodal LLMs (GPT-5.2, Gemini 3. Pro, Gemini 3. Flash) are tested under controlled conditions: 30 independent prompt iterations per item, temperature fixed at $T=0.7$ for model parity, and structured chain-of-thought output (visual description, physics reasoning, answer selection) parsed via API.

Qualitative coding is applied across three axes—visual interpretation, physics reasoning, and answer coordination—to distinguish nuanced failure modes. Quantitative comparison utilizes performance of 267 first-year physics students from the CRCI validation cohort, enabling robust benchmarking.

Major Results and Numerical Performance

Gemini 3. Flash emerges with mean accuracy of $97\%$ across CRCI items, slightly outperforming Gemini 3. Pro ($89\%$), and both exceeding GPT-5.2 ($73\%$, reasoning disabled) and student mean performance ($62\%$). Item-level results highlight that all LLMs produce nearly flawless outputs on most questions, but catastrophic failure occurs on select items, often with zero inter-iteration variance.

Enabling reasoning in GPT-5.2 increases its mean accuracy to $85$–$86\%$, bringing it in line with Gemini 3. Pro; however, visual interpretation remains a persistent bottleneck (e.g., GPT-5.2 achieves $0\%$ on Q4 regardless of reasoning depth, due to misreading spatial arrangements).

Student error profiles are distributed across multiple distractors, exhibiting characteristic human misconceptions and cognitive instability. In contrast, LLMs converge on single distractors deterministically when failing, suggesting error mechanisms not analogous to human cognition.

Failure Mode Analysis

The qualitative breakdown reveals four principal failure types:

• Visual interpretation errors: Predominant, especially on items involving non-trivial spatial diagrams, inclined planes, or reference frame transformations (e.g., Q15, Q16).
• Semantic misalignment: Example Q1, where translation artifacts induce systematic model failure despite simple physical content.
• False positives: LLMs sometimes produce correct answers through flawed chain-of-thought, undermining reliability of reasoning-based explanations.
• Coordination errors: The selected answer fails to logically follow from the model's stated reasoning (rare, mainly observed in GPT-5.2).

Visual processing remains the fundamental constraint: when models extract the correct diagram features, physics reasoning is robust; deficient visual parsing propagates failure, unaffected by reasoning depth.

Distractor Distribution and Comparative Error Patterns

The response distributions demonstrate that LLMs, when mistaken, select incorrect options with high consistency, often distinct from the distractors chosen by students. For example, GPT-5.2 selects the same distractor in $100\%$ of iterations on Q4, whereas student errors are broadly distributed.

Shannon entropy analysis confirms statistically higher response concentration for LLMs versus students ($p < 0.005$ for all models), reinforcing the conclusion that LLM error mechanisms are structurally different.

Implications for AI in Physics Education

These findings demand recalibration of expectations for LLM reliability in conceptual physics education:

• Item-dependent performance: High aggregate accuracy does not guarantee consistent correctness or explanatory validity across items.
• Visual interpretation limits: Persistent failures on diagrammatic or spatially complex questions restrict the pedagogical utility of current multimodal architectures.
• Assessment integrity: If chatbots can achieve near-perfect scores on concept inventories, unsupervised “take-home” administration becomes invalid as a measure of student understanding.

Model selection should be empirically guided; larger or more expensive models do not guarantee superior domain performance (inverse scaling noted). Critical use and supervised environments are recommended for instructors to preserve assessment validity and mitigate risk of AI-driven misconception reinforcement.

Limitations and Directions for Future Research

The study’s conclusions are shaped by configuration asymmetries and rapidly evolving model versions, as noted with Gemini 3. Pro’s API deprecation. Item types and visual demands influence generalizability. Prompt engineering and more elaborate multimodal prompting strategies remain unexplored but may offer marginal improvement.

The CRCI furnishes a unique testbed for future longitudinal benchmarking as AI systems progress. Systematic replication and comparison across multiple physics domains, assessment types, and LLM versions will be necessary to track scalability and robustness.

Theoretical and Practical Implications

Structurally, LLMs encode and optimize for internal consistency, manifesting deterministic error convergence not representative of human reasoning. This distinction underlines the necessity of aligning educational AI usage with awareness of non-human error typologies, and cautious integration in formative and summative assessment contexts.

Further developments in AI vision, multimodal reasoning, and prompt structuring may alleviate current bottlenecks, but item-specific failure modes will likely persist without substantial advances in deep perceptual integration and semantic nuance. The CRCI-based evaluation establishes a high-resolution baseline for these evolving challenges.

Conclusion

LLMs exhibit high, but unpredictable, item-dependent reliability on conceptual physics inventories not found in training data. Visual misinterpretation, not physics reasoning, is the dominant failure axis, and when failure occurs it is systematic and structurally distinct from student errors. Accurate answers may be accompanied by incorrect reasoning, underscoring risks in using LLMs for tutoring or assessment guidance. Empirical, task-specific benchmarking and supervised administration remain imperative. The CRCI provides a crucial baseline for monitoring AI progress in physics conceptual understanding (2605.09602).