An Empirical Study of Security Calibration in Large Language Models for Code

Abstract: LLMs are rapidly transforming software development, yet their use in security-critical contexts raises a key question: do models know when their generated code is insecure? This property, known as calibration, measures whether a model's confidence aligns with the true correctness of its outputs. We present the first large-scale empirical study of security calibration in LLM-generated code. We evaluate GPT-4o-mini, Gemini-2.0-Flash, and Qwen3-Coder-Next across multiple temperature settings on two complementary benchmarks: self-contained security tasks and multi-language repository-level contexts. Our results suggest that overconfidence is prevalent across the evaluated LLMs. Functional calibration is consistently worse than security calibration, suggesting that models estimate security outcomes more reliably than functional correctness, potentially because functional correctness depends on complex execution behavior. We also examine whether calibration-guided automated repair can help remediate vulnerabilities in LLM-generated code, finding only limited improvements while frequently introducing functional regressions. Moreover, we study different mitigation strategies for reducing False Trust, where models assign high confidence to vulnerable code. The results show that although architectural gating improves calibration on controlled benchmarks, calibration deteriorates in realistic repository-level settings, increasing the risk of high-confidence vulnerable outputs.

• The paper quantifies miscalibration in LLM-generated code by comparing self-reported confidence with verified security outcomes.
• It reveals significant overconfidence, with error rates peaking in repository-level tasks compared to isolated function generation.
• It evaluates mitigation strategies, showing execution gating improves calibration while automated repair remains largely ineffective.

Security Calibration in LLM-Generated Code: A Technical Assessment

Introduction and Motivation

The deployment of LLMs for code generation in security-critical workflows introduces substantial risk due to their tendency to produce vulnerable programs while expressing unjustified confidence. The study "An Empirical Study of Security Calibration in LLMs for Code" (2606.31159) undertakes a rigorous multi-model, multi-benchmark empirical analysis to quantify when, and to what extent, LLM self-reported confidence aligns with the security correctness of their outputs—a property termed security calibration. The work considers three advanced LLM architectures (GPT-4o-mini, Gemini-2.0-Flash, Qwen3-Coder-Next), multiple temperature settings, and leverages both self-contained tasks (SALLM) and repository-level contexts (AICGSecEval), addressing the operational relevance of calibration for software security and the ongoing false trust problem in AI-aided code development.

Methodology

Security calibration is operationalized as the alignment between an LLM’s probabilistic self-assessment and the ground-truth security status of generated code, as verified through functional and vulnerability test harnesses. Four confidence estimation methods are compared: (i) token-probability aggregation, (ii) verbalized confidence elicitation, (iii) sampling-based consistency, and (iv) self-consistency prompting with chain-of-thought rationale. The methodology spans 36,000 generations per model, with extensive temperature sweeps and stratified measurement by vulnerability type, as well as analysis of calibration-guided repair strategies and architectural/prompt-level mitigations. This dual-benchmark design enables direct comparison between isolated code generation and real-world, multi-file repository repair.

Figure 1: Overview of the study’s empirical pipeline, from prompt and generation through functional/security test harnesses, confidence elicitation, and calibration/repair evaluation.

Main Results

1. Systematic Overconfidence and Calibration Error

LLMs demonstrate pronounced and persistent overconfidence in security-critical code generation. For verbalized confidence, GPT-4o-mini and Qwen3-Coder-Next exhibit severe miscalibration (Expected Calibration Error ECE ≈ 0.41–0.48), while Gemini-2.0-Flash is less extreme (ECE ≈ 0.25–0.26), yet still above thresholds generally regarded as “well-calibrated” ($\mathrm{ECE} \leq 0.01$). Token probability-based posteriors yield even worse calibration (ECE approaching 0.80), confirming that next-token probabilities are not trustworthy for security prediction at the program level. Sampling-based consistency, while perfectly calibrated at the aggregate prompt level, is ineffective for per-sample discrimination.

High False Trust rates are ubiquitous: for GPT-4o-mini, approximately one-third of code predicted as "high-confidence" secure ($p_{\text{sec}} \ge 0.8$) is in fact vulnerable. In repository-level repair settings, this risk escalates further, with false trust rates exceeding 70% for all models.

2. Calibration Asymmetry: Security vs. Functional Correctness

A clear asymmetry emerges: functional correctness is always worse calibrated than security correctness, across all models and temperatures. The mean $\Delta\mathrm{ECE}$ (security minus functional) ranges from -0.15 (GPT-4o-mini) to -0.53 (Qwen3-Coder-Next) depending on the architecture. This suggests that static vulnerability pattern recognition—e.g., spotting use of unsafe APIs—is a more tractable task for LLM self-assessment than reasoning about holistic functional correctness, which is contingent on runtime and environmental factors that are generally opaque to the model during generation.

3. Calibration Degradation in Repository-level Contexts

Contrasting self-contained function generation (SALLM) with repository-level patching tasks (AICGSecEval), calibration error increases by a factor of 2–4 when confronted with cross-file, multi-language dependencies. For example, GPT-4o-mini’s ECE rises from 0.41 to >0.69; Gemini and Qwen3-Coder-Next both display comparable deterioration, with repository-level False Trust routinely >70% regardless of target language. Decomposition analyses attribute much of this to build/test fragility, but even when filtering out non-building samples, calibration remains significantly degraded relative to isolated function code, highlighting persistent uncertainty amplification due to real-world context complexity.

4. Automated Repair and Mitigations

Calibration-guided repair—resubmitting low-confidence/insecure generations for iterative fixing—provides minor improvements in risk stratification but negligible practical reduction in vulnerabilities. Among all models, repair success is generally below 3%, with functional breakage often in excess of 60%. Specifically, vulnerabilities requiring sink/API replacement are almost unrepairable under standard prompting regimes, signifying a rigidity barrier that impedes effective automated mitigation.

Among evaluated mitigation strategies, execution gating (i.e., filtering out code that fails preliminary functional tests prior to security assessment) provides the most consistent calibration improvements, reducing ECE by up to 39% at rejection rates of 46–86%. In contrast, prompt-based interventions (few-shot exemplars, chain-of-thought) improve vulnerability removal rates but rarely yield harness-passing outputs without further structural integration.

Implications and Prospective Developments

The strong miscalibration and prevalence of false trust in LLMs for code generation indicate substantial operational risk when these models are used as security-critical developer assistants. Notably, the findings demonstrate that:

• Model confidence self-reports cannot be used blindly for trust decisions; substantial fractions of high-confidence generations remain insecure even after post-hoc calibration.
• Security self-assessment is more consistent than functional self-assessment, but both degrade sharply in real-world, repository-level repair scenarios, amplifying the need for context-sensitive calibration benchmarks and continued evaluation beyond “toy” datasets.
• Mitigation via execution gating is effective and straightforward, justifying its adoption in practical AI-assisted development pipelines. However, calibration-based repair is fundamentally constrained by model inability to perform robust API migration or handle structurally complex vulnerability patterns.

For future AI system design, these results underscore the necessity of rigorous, context-aware calibration evaluation as a standard metric in secure code generation pipelines. Likewise, ongoing research into improved post-hoc calibration (e.g., learned transformations of intermediate activations [liu2024litcab], adaptive temperature scaling [xieetal2024calibrating]), uncertainty modeling, and better integration of static/dynamic analysis frameworks will be required to achieve safe, reliable deployment. Automating code repair for complex security flaws remains elusive for current LLMs, especially as real-world structure increases in richness.

Conclusion

This study delivers a comprehensive empirical assessment of security calibration in LLM-based code generation, highlighting systematic overconfidence, the persistence and operational significance of false trust, calibration asymmetry between functionality and security, and the compounding difficulty of reliable code repair in complex, repository-level contexts. Theoretical and practical implications stress the centrality of calibration as a core trustworthiness criterion and its limitations for fully-automated security assurance. Architectural interventions such as execution gating currently yield the most robust improvements; further advances will require greater foundation-model awareness of context, external verification, and more effective calibration-aware training paradigms.

---

References:

(2606.31159) An Empirical Study of Security Calibration in LLMs for Code [xieetal2024calibrating] Calibrating LLMs with Adaptive Temperature Scaling [liu2024litcab] LitCab: Lightweight LLM Calibration over Short- and Long-form Responses

(Additional references from main paper omitted for brevity.)