An Empirical Evaluation of Locally Deployed LLMs for Bug Detection in Python Code

Abstract: LLMs have demonstrated strong performance on a wide range of software engineering tasks, including code generation and analysis. However, most prior work relies on cloud-based models or specialized hardware, limiting practical applicability in privacy-sensitive or resource-constrained environments. In this paper, we present a systematic empirical evaluation of two locally deployed LLMs, LLaMA 3.2 and Mistral, for real-world Python bug detection using the BugsInPy benchmark. We evaluate 349 bugs across 17 projects using a zero-shot prompting approach at the function level and an automated keyword-based evaluation framework. Our results show that locally executed models achieve accuracy between 43% and 45%, while producing a large proportion of partially correct responses that identify problematic code regions without pinpointing the exact fix. Performance varies significantly across projects, highlighting the importance of codebase characteristics. The results demonstrate that local models can identify a meaningful share of bugs, though precise localization remains difficult for locally executed LLMs, particularly when handling complex and context dependent bugs in realistic development scenarios.

• The paper demonstrates that locally deployed LLMs, LLaMA 3.2 and Mistral, achieve roughly 43-45% accuracy in bug detection on real-world Python code.
• It details a methodology using function-level context extraction and keyword-based evaluation on the BugsInPy benchmark to assess bug localization efficacy.
• Results highlight that structured codebases yield higher accuracy while complex projects result in more partial responses, underscoring opportunities for model enhancements.

Evaluation of Locally Deployed LLMs for Real-World Python Bug Detection

Motivation and Research Context

The proliferation of LLMs has catalyzed diverse applications within software engineering, notably in code generation, understanding, and automated debugging tasks. Notwithstanding impressive advances in cloud-hosted foundation models, practical deployment remains hindered by issues related to privacy, cost, and infrastructure constraints. Locally executable LLMs, notably LLaMA 3.2 (8B) and Mistral (7B), have emerged as plausible alternatives, promising competitive performance without external dependencies. However, rigorous empirical investigations into their efficacy for realistic bug detection scenarios, particularly on production-level open-source Python code, remain sparse. This study systematically assesses locally deployed LLMs on the BugsInPy benchmark, employing zero-shot function-level prompts and automated keyword-based evaluation to establish baseline reliability and nuances of their failure modes.

Experimental Setup and Methodology

Dataset and Function Extraction

The analysis leverages the BugsInPy benchmark, comprising 349 processed bugs from 17 mainstream Python projects. The selection ensures coverage across disparate domains (data science, web, ML, tools), emphasizing ecological validity. For each bug instance, function-level context is extracted (by identifying enclosing def or class blocks) as preliminary tests disclosed that file-level prompts diluted the task focus, whereas line-level context lacked sufficient behavioral cues.

Prompting Protocols

A standardized zero-shot prompt explicitly signals the presence of a bug (“It contains a bug. Find the bug and explain how to fix it.”), steering models away from trivial misclassifications while concentrating the evaluation on explanation and localization. This protocol closely mirrors realistic developer interactions with local code assistants outside bespoke prompt engineering.

Response Evaluation

The paper introduces an automated keyword-based assessment: fixes are parsed for new tokens, which are cross-matched in generated responses (after exclusion of trivial English stopwords and short tokens). Outcomes are trisected into “correct”, “partial” (detects problematic region but not the fix), and “wrong”. Partial responses, although not pinpointing the bug, signal practical value by narrowing diagnostic focus. Manual inspection of a random subset validated high concordance between keyword-labeling and human judgment, strengthening the reliability of reported metrics.

Models and Hardware

Both LLaMA 3.2 and Mistral are executed via Ollama on consumer-grade Apple M1 Pro hardware (16GB RAM), quantized to 4-bit precision for efficiency. Response times remain practical (7–13s per bug), affirming the feasibility of local inference for iterative debugging cycles.

Bug Detection Performance: Aggregate and Project-Level Outcomes

The core empirical finding is that local LLMs attain approximately 43–45% accuracy in function-level bug detection on real-world Python defects, with Mistral marginally outperforming LLaMA 3.2. About half of responses are “partial”, highlighting the models’ ability to flag suspicious code without precisely articulating the fix (Figure 1).

Figure 1: Global bug detection results, depicting the distribution among correct, partial, and wrong responses for LLaMA 3.2 and Mistral.

Performance exhibits substantial project-level heterogeneity. Structured codebases with regular patterns (e.g., PySnooper, black, fastapi) are amenable to bug localization, exhibiting correctness rates above 70%. In contrast, projects with complex abstractions or hidden dependencies (e.g., tqdm, sanic, luigi) witness pronounced performance degradation, with partial responses predominating (Figures 2, 3, 4).

Figure 2: Bug detection accuracy stratified by project, highlighting variances across codebases.

Figure 3: Score distribution for LLaMA 3.2 by project, revealing the fraction of correct, partial, and wrong classifications.

Figure 4: Score distribution for Mistral, accentuating partial responses in complex projects.

McNemar's test on paired predictions yields $p = 0.68$, confirming the lack of statistically significant difference between the models’ correct detection rates.

Fine-Grained Analysis by Bug Type

Manual curation enables stratification of bug types (Null/None Check, Return Value, Conditional Logic, Indexing, Error Handling, Loop Logic, Comparison Operator, Type Conversion, Other/Complex). Both models demonstrate the highest competency in Null/None Check and Return Value bugs (approx. 60% and 51%), where local reasoning suffices and target patterns are syntactically regular. Conversely, Type Conversion and Other/Complex categories elicit minimal correctness (0–22%), elucidating current LLM limitations in scenarios requiring non-local context or multi-component reasoning. Notably, Mistral achieves higher accuracy in Indexing and Error Handling, suggesting model-specific strengths for certain bug typologies (Figure 5).

Figure 5: Bug detection accuracy by bug category for both models; numbers denote bug counts per category.

Implications, Limitations, and Prospective Directions

Practical Utility

Local LLMs, even absent custom hardware or cloud APIs, reliably flag the presence of bugs in a significant fraction of real-world functions, offering privacy-preserving, cost-free assistance for preliminary code review. The prevalence of partial responses implies tangible utility in narrowing probable bug regions, even when fixes are not precisely elucidated.

Limitations

Significant contextual deficiencies—function boundaries, multi-function dependencies, implicit contracts—impede localization accuracy for complex bugs. Keyword evaluation, while scalable, may undervalue semantically correct responses using alternate terminologies. Explicit prompts informing the model of a bug further accentuate detection rates compared to blind classification scenarios.

Theoretical and Future Directions

Improving semantic labeling and evaluation fidelity (e.g., by deploying semantic similarity metrics, leveraging human annotation) is paramount for robust assessment. Integrating richer context (file-level, inter-function relationships), few-shot in-context learning with task exemplars, or retrieval-based augmentation may substantially enhance bug identification and localization. Ensemble approaches exploiting complementary strengths across models warrant further exploration.

Expansion to diverse datasets (beyond BugsInPy) and programming languages, alongside benchmarking additional open-weight LLMs, will be critical for generalizing current findings and optimizing AI-assisted debugging pipelines.

Conclusion

Locally deployed LLMs—specifically LLaMA 3.2 and Mistral—exhibit meaningful utility for real-world Python bug detection, attaining near 45% correctness in challenging scenarios and generating practically useful partial diagnoses. Model effectiveness is sensitive to codebase heterogeneity and bug typology, with substantial limitations in context-dependent and complex cases. The study affirms the accessibility of AI-driven debugging on consumer hardware and motivates future refinement in evaluation methodology, context augmentation, and cross-model ensembling to advance precision and reliability in offline LLM deployment for software engineering tasks.