Will It Break in Production? Metric-Driven Prediction of Residual Defects in Python Systems

Abstract: Python's dynamic nature complicates testing and increases the possibility that some defects evade detection, so an effective fault prediction becomes essential. We examine whether post-release faults can be predicted using modern ML and DL. Using a balanced dataset of over 4,000 labeled faults with 83 product, process, statistical, and Python-specific metrics plus normalized code representations, we conduct cross-project experiments. LLMs and unsupervised models fail to distinguish residual from non-residual faults, while supervised metric-based models (RandomForest, XGBoost, CatBoost) perform far better, yielding a 0.85-0.9 recall and cutting false negatives by an order of magnitude. Process metrics, especially age, churn, and developer-activity, alongside class and file size, consistently prove most predictive. Notably, the Principal Component Analysis shows that metrics and code embeddings occupy distinct regions of the representation space, suggesting that they capture complementary rather than redundant information.

• The paper demonstrates that process and size metrics can reliably predict residual defects in Python systems by explicitly distinguishing them from pre-release faults.
• It employs a comprehensive methodology with over 4,000 faults and 83 metrics, validated through manual annotation with 95% accuracy.
• Supervised metric-based models outperformed LLMs, achieving F1 scores between 0.71 and 0.73 and substantially reducing false negatives in defect prediction.

Metric-Driven Prediction of Residual Defects in Python Systems

Problem Motivation and Distinctiveness

Despite substantial advancements in defect prediction, the enduring challenge of identifying residual faults—defects that evade pre-release validation and emerge only in production—remains unresolved in industrial and open-source software systems. The operational and economic consequences of such failures are well documented, highlighting the need for reliable cross-project predictors capable of effectively ranking components by their risk of escaping testing. Notably, residual bugs are not merely a subset of typical post-release issues; their manifestation depends on subtle, context-dependent interactions and usage scenarios that elude classic coverage and static analysis.

Figure 1: The software defect iceberg: visible bugs found pre-release vs. hidden, context-dependent residual defects in real-world use.

Traditional fault prediction approaches conflate pre-release and post-release defects, thus suffering from survivorship bias in their predictive models. The present work explicitly disentangles these two categories, targeting the prediction of residual defects with specialized metrics and evaluation regimes.

Methodology and Dataset Construction

The methodology incorporates four main stages: data collection, residual/non-residual classification, metric extraction, and model development and evaluation.

Figure 2: Detailed methodology adopted in this work.

• Data Collection: Over 4,000 Python faults are constructed from PyResBugs and augmented with 500 additional instances from BugsInPy, producing a balanced, reproducible dataset. Residual defects are identified via issue analysis, commit inspection, and temporal evidence, emphasizing code-level verifiability. The labeling heuristic is validated through manual annotation, yielding inter-rater reliability (Cohen's $\kappa > 0.80$) and $\approx 95\%$ accuracy.
• Metric Extraction: 83 metrics are employed, covering code complexity, Halstead metrics, size and documentation indicators, coupling, inheritance, Python-specific syntactic features, process-oriented measures (e.g., age, churn, developer activity), and statistical properties (entropy via KenLM on Py150).
• Split Strategy and Structure Analysis: A 90/10 train/validation split is enforced, with cross-project testing via BugsInPy commits. All splits are verified to avoid data leakage, and distributions of token counts and structural characteristics are assessed.

  Figure 3: Histograms of the train and test splits showcasing the distribution of token counts across residual and non-residual faults.

The dataset design ensures that deep learning and metric-based models are evaluated in realistic, cross-project settings, mitigating project-specific confounders.

LLM and Deep Learning Baselines

The utility of code representations learned by LLMs and fine-tuned DL models is interrogated using both zero-shot and supervised protocols.

• Closed-Source LLMs: Gemini 3 Pro, Claude 4.5 Sonnet, and GPT-4.1 are prompted to differentiate residual from non-residual code changes. Their behavior is dominated by class imbalance and degenerate predictions (e.g., Gemini producing recall $=0.99$ but at the expense of a massive false positive rate), revealing an inability to internalize discriminative boundaries in the absence of explicit metric guidance. F1 scores range from 0.43 (GPT-4.1) to 0.72 (Gemini 3 Pro), but none of the models generalizes meaningfully.
• Open-Weight DL Models: Fine-tuned CodeT5+, CodeLlama, and DeepSeek-Coder all exhibit limited classification competency ($\mathrm{F1}=0.32$ to $0.46$). Confusion matrices indicate trade-offs between recall and precision, but these models fundamentally lack robust signals even with identifier/literal normalization and structured pre-processing.

These results provide negative evidence regarding the applicability of LLMs and DL code embeddings alone for defect escape prediction in Python. The output variance further underscores class confusion and effective inability to deal with subtle context-dependent boundary cases in residual faults.

Metric-Based Model Performance

A comprehensive ablation of metric-based unsupervised and supervised learners is conducted:

• Unsupervised Models: IF, One-Class SVM, and LOF offer marginal precision but catastrophically poor recall (e.g., IF recall = 0.34, LOF recall = 0.13). These models fail to capture the latent structure of fault escape, indicating that residual defects are not statistical outliers in metric space.
• Supervised Models: Random Forest, XGBoost, and CatBoost substantially outperform all other methods, achieving F1 scores $0.71$–$0.73$ and recall in the $0.85$–$0.90$ range. This corresponds to a one order of magnitude reduction in false negatives compared to LLMs and unsupervised models.

Key Predictive Factors and Feature Orthogonality

Feature attribution (via model importance and SHAP) indicates that process history metrics (method age, bug density, average commits per author), alongside class/file size indicators, dominate predictive power.

Figure 4: Top-10 feature importances for the three supervised models used in RQ2.

Specifically, higher method age correlates with increased residual-fault risk, while high bug density and collaborative author activity shift the prediction towards non-residual. Large class/file size, navigational complexity, and documentation density are also significant. These findings are stable across RandomForest, XGBoost, and CatBoost, with each model’s attribution indicating slightly different emphases but consistent directionality.

A principal component and canonical correlation analysis of metric vectors and code embeddings (extracted from CodeLlama) reveals pronounced orthogonality: maximum Spearman $\rho = 0.27$, with a mean near $\approx 95\%$0, and canonical correlations $\approx 95\%$1 across top components.

Figure 5: PCA projection of metric and embedding components. The two X’s represent the centroids. The two clusters exhibit a near-orthogonality of the two representations.

This demonstrates that code embeddings and software metrics encode complementary information. However, hybrid models that naively concatenate these representations underperform, suggesting more advanced integration mechanisms are necessary to exploit both views effectively.

Theoretical and Practical Implications

The results indicate that structural and historical process metrics remain the most reliable predictors of defect escape in Python systems, outperforming code representations even when large-scale LLMs are available. This not only validates the dominant role of process-aware metrics for reliability engineering but also challenges expectations that DL/LLMs can supplant established metric-based regimes for high-stakes cross-project fault prediction.

From a practical standpoint, supervised metric-driven models are lightweight, interpretable, and efficient, suitable as robust safety filters in CI/CD pipelines for prioritizing verification and code review. Their moderate precision but high recall aligns well with operational risk management.

The performance ceiling of code representations, and the finding of feature orthogonality, suggest that future work should focus on architectures capable of structured information integration, potentially by leveraging graph-based models, domain-adaptation, or hierarchical contrastive techniques that can align structural and semantic features across projects.

Conclusion

This work demonstrates that despite the sophistication of current LLMs and code-based models, software metrics—especially process and size indicators—are currently the most robust signals for predicting residual (production-escaping) faults in Python systems. While code embeddings offer orthogonal information, naive hybridization degrades, rather than enhances, performance in cross-project settings. The findings recommend a dual-track research agenda: (1) expanding multi-language residual fault datasets for broader generalizability, and (2) designing representation learning strategies explicitly tailored to combine semantic embeddings and process metrics for defect escape prediction.

Advances along these axes are expected to improve the cost-effectiveness and coverage of predictive quality assurance in large-scale, dynamic, and safety-critical software environments.

---

Citation: "Will It Break in Production? Metric-Driven Prediction of Residual Defects in Python Systems" (2604.26667)