ChatGPT: 3.5, 4, and 4o Evolution

• ChatGPT Versions 3.5, 4, and 4o are generative transformer models evolving through significant architectural advances and RLHF-based optimization.
• Benchmarking reveals statistically significant error reductions and enhanced performance across domains such as healthcare, engineering, and Bayesian reasoning.
• Practical guidance recommends using advanced versions for high-stakes tasks while enforcing human oversight for persistent, domain-specific error modes.

ChatGPT, a family of LLMs developed by OpenAI, has undergone significant evolution from GPT-3.5 through GPT-4 and GPT-4o, resulting in substantial improvements in domains ranging from software engineering and healthcare to statistics and nuanced language classification. These models have been extensively evaluated using rigorous academic benchmarks, revealing nuanced strengths, persistent limitations, and sharp inter-version performance differentials (Garousi, 26 Apr 2025, McGee et al., 17 Dec 2024, Green et al., 29 Nov 2025, Mu et al., 14 Apr 2025).

1. Architecture and Versional Evolution

GPT-3.5, GPT-4, and GPT-4o represent ascending generations of transformer-based LLMs. While all operate as generative, autoregressive models trained on corpora comprising web text, code, and domain-specific datasets, major architectural and procedural advances differentiate these versions. GPT-4 introduced a “two-tier” large multimodal architecture and improved chaining of reasoning, while GPT-4o (“GPT-4 omniverse”) employed infrastructure optimized for mixture-of-experts inference and chain-of-thought stabilization, built on a Mistral tech stack. Model capacity increased by more than 10× between GPT-3.5 and GPT-4/4o, accompanied by broader pretraining data and optimization through reinforcement learning from human feedback (RLHF) (Mu et al., 14 Apr 2025).

The result has been a substantial reduction in both factual and arithmetic errors, improved handling of symbolic and mathematical reasoning, and expanded multimodal capabilities (including image understanding in GPT-4 and 4o) (McGee et al., 17 Dec 2024). These advances manifest as improved benchmark performance in standard NLP, code synthesis, and reasoning tasks.

2. Benchmarking Methodologies and Error Metrics

Academic and industry researchers employ standardized protocols to quantify model accuracy, precision, recall, F1 score, and error rates across domains. A broadly adopted approach is the Multivocal Literature Review (MLR), integrating empirical data from IEEE Xplore, ACM Digital Library, SpringerLink, arXiv, company whitepapers, OpenAI documentation, and public benchmarks up to early 2025 (Garousi, 26 Apr 2025).

Key metrics and statistical tools include:

• Error rate: $E_r = 100\% - A$, or $$E_r = \frac{N_{\rm errors}}{N_{\rm total}}\times 100\%$$
• Accuracy: $(TP + TN) / (TP + TN + FP + FN)$
• Precision: $TP / (TP + FP)$
• Recall: $TP / (TP + FN)$
• F1 Score: $$2 \cdot \frac{\mathrm{Precision}\cdot\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$$
• Cohen’s d (effect size): $d = \frac{\bar{E}_1 - \bar{E}_2}{s_p}$
• 95% confidence intervals, McNemar’s test for paired dichotomous outcomes, and ordinal logistic regression for quality gradation (McGee et al., 17 Dec 2024, Garousi, 26 Apr 2025).

This enables direct, statistically significant pairwise comparison of model versions and robust characterization of domain/task-level error variability.

3. Performance Across Domains and Tasks

A. Healthcare

Error rates are highly variable, ranging from 28%–72% for GPT-3.5 and 23%–83% for GPT-4 depending on decision complexity; rare-disease diagnosis remains particularly error-prone (up to 83%). Median error is $\approx$30% for GPT-3.5 and $\approx$25% for GPT-4. GPT-4-turbo and 4o are presumed within $\pm$2% of GPT-4, though fine-grained data are lacking (Garousi, 26 Apr 2025). Human-in-the-loop review remains categorically necessary.

B. Business and Economics

GPT-3.5 error ranges from 15–47% (median 31%) and GPT-4 from 15–20% (median 17%), with pronounced improvements on formal exams (e.g., MBA and accounting). Open-ended economic reasoning (Caplan-type questions) incurs higher GPT-3.5 error (up to 69%), while GPT-4 reduces such errors substantially (Garousi, 26 Apr 2025).

C. Engineering and Computer Science

Engineering tasks show median error reductions from 45% (GPT-3.5) to 25% (GPT-4) (Garousi, 26 Apr 2025). For programming, GPT-3.5 achieves ~82% correctness on structured test cases (error 18%), but performance deteriorates (error 52%) in open-ended Stack Overflow scenarios. GPT-4 improves structured-case success to 87.5% (12.5% error) but open-ended debugging remains unstable (Garousi, 26 Apr 2025).

D. Social Media Classification and Statistical Reasoning

In a nuanced social-media annotation benchmark, GPT-4 and 4o reach F1 scores of 0.61–0.63, notably outperforming GPT-3.5 (0.49–0.53). Prompt structure strongly modulates performance; simple label-based prompts outperform more complex binary/reasoning chains, particularly for GPT-4o. All versions struggle with cultural euphemisms, slang, and ambiguous targets, necessitating continued human oversight (Green et al., 29 Nov 2025).

For introductory–graduate statistics exams, GPT-4 attains 81.7% accuracy versus 50.5% for GPT-3.5, with especially large margins (over 60%) for image-based questions (e.g., interpreting visual summaries). McNemar’s test yields $p = 1.2 \times 10^{-5}$, signifying a highly significant advantage for GPT-4 (McGee et al., 17 Dec 2024).

E. Bayesian Reasoning

On binary classification tasks where Bayes’ rule is normatively optimal, GPT-3.5 displays “representativeness heuristic” behavior (ignoring priors in 82% of cases, with heavy algebraic errors and low efficiency: 75%–85%). GPT-4 demonstrates conceptual grasp of Bayes (1–2% prior-ignoring, moderate arithmetic errors, 93%–96% efficiency). GPT-4o eliminates most computational noise, matching or exceeding human performance (efficiency 97%–99%). The progression is from heuristic matching (3.5), to conceptual Bayes (4), to accurate Bayesian (4o) (Mu et al., 14 Apr 2025).

4. Statistical Evidence of Model Gains

Paired-sample and cross-sectional studies confirm statistically significant error reductions from GPT-3.5 to GPT-4 across multiple domains:

• Structured financial/accounting tasks: error drops from 47% to 15% ($\Delta=32$ pp; $d\gtrsim1.0$)
• Standardized statistics/data-science exams: overall accuracy gain $\approx$31% (McNemar’s $p=1.2\times10^{-5}$)
• Social media annotation: F1 gain of +0.12 (Prompt 1), with larger gains on frequent classes (Green et al., 29 Nov 2025)
• Bayesian task efficiency: GPT-4o achieves 99.4% efficiency, surpassing average human performance (Mu et al., 14 Apr 2025)

Effect sizes are large (Cohen’s $d\gtrsim1.0$), and 95% confidence intervals confirm robustness. Where reported, model upgrades have $p<0.05$ for accuracy improvements in classroom, social annotation, and structured task settings (McGee et al., 17 Dec 2024, Garousi, 26 Apr 2025).

5. Failure Modes, Error Typologies, and Task Sensitivities

Despite global improvements, persistent and version-specific weaknesses prevail:

• Healthcare: Extreme outliers for rare or ambiguous cases, reflecting training-data scarcity (Garousi, 26 Apr 2025).
• Open-ended reasoning: GPT-3.5 and GPT-4 manifest high variability (10–50% error) in debugging, code maintenance, and open-ended Q&A.
• Social annotation: Models—especially GPT-4 and 4o—misclassify cultural euphemisms, internet slang (acronyms, emojis), and cannot reliably disambiguate “ad hominem” vs. “content threat” (Green et al., 29 Nov 2025).
• Interpretive and arithmetic errors: GPT-3.5–4 exhibit both chain-of-thought and symbolic manipulation errors, though drastically reduced in 4o (Mu et al., 14 Apr 2025).
• Equity and access: Free-tier users restricted to GPT-3.5 or throttled GPT-4o confront 30%–80% lower accuracy on some STEM tasks, especially visual/statistical interpretation, perpetuating digital divides (McGee et al., 17 Dec 2024).

Median and interquartile range analyses reveal that routine requirements/design (5%–20% error) are reliably managed, but implementation, testing, and maintenance display significantly wider variance (Garousi, 26 Apr 2025).

6. Practical Guidance and Task–Version Selection

Empirical syntheses provide the following actionable guidance:

• High-stakes, structured tasks: Deploy GPT-4 or turbo/4o variants, especially for standardized exams, financial/professional analysis, or image+text multimodality (McGee et al., 17 Dec 2024, Garousi, 26 Apr 2025).
• Low-risk prototyping/drafting: GPT-3.5 may suffice for basic code generation or informal writing when cost is critical.
• Healthcare/critical domains: Irrespective of version, enforce mandatory human-in-the-loop review and validation before acting on outputs.
• Prompt engineering: Refinements yield 1%–2% additional accuracy, especially in engineering domains (Garousi, 26 Apr 2025).
• Social annotation, ambiguity-prone tasks: Always cross-validate against a human-annotated gold standard; Krippendorff’s $\alpha$ remains the baseline for reliable annotation standards (Green et al., 29 Nov 2025).
• Equity warning: GPT-4o’s free-tier access is throttled; once quotas are reached, users revert to GPT-3.5, reinforcing a “tiered” tutor-access regime (McGee et al., 17 Dec 2024).

7. Future Directions, Reliability, and Open Problems

While technological improvements have delivered measurable, statistically robust performance advances—especially with GPT-4o—persistent risks and context dependence remain unsolved. No version eliminates domain-specific outliers (e.g., 83% error on rare-disease diagnosis) or reliably interprets cultural subtext in language annotation. Even in domains where LLMs now approach or exceed human benchmarks (e.g., Bayesian classification), task diversity requires continuous local benchmarking due to potential performance drift following model retraining or API changes (Garousi, 26 Apr 2025, Green et al., 29 Nov 2025).

The advent of GPT-4o and its broad (albeit quota-limited) free-tier availability partially address equity barriers, especially for vision tasks. However, usage caps maintain a “tiered” access regime and reinforce the need for critical, ongoing validation of outputs in applied and educational settings (McGee et al., 17 Dec 2024).

Comprehensive evaluation—domain-specific, prompt-optimized, and benchmarked against robust human standards—remains essential to ensure safe, reliable, and equitable deployment.