Eliminating Hallucination-Induced Errors in LLM Code Generation with Functional Clustering

Abstract: Modern code-generation LLMs can already solve a large fraction of programming problems, yet they still hallucinate subtle bugs that make their outputs unsafe for autonomous deployment. We present functional clustering, a black-box wrapper that eliminates nearly all hallucination-induced errors while providing a tunable confidence score. The wrapper samples many candidate programs, executes each on a self-generated test suite, and clusters candidates whose I/O behavior is identical; the empirical mass of the largest cluster serves as an exact confidence estimate. A single scalar threshold on this estimate lets users trade coverage for reliability with exponential guarantees. On LiveCodeBench our verifier preserves baseline pass@1 on solvable tasks yet slashes the error rate of returned answers from ~65% to 2%, and drives it to 0% at a conservative threshold while still answering 15.6% of prompts. Manual audits show that the few residual mistakes stem from prompt misinterpretation, not random generation noise, narrowing future work to specification clarity. Because the method requires only sampling and sandbox execution, it applies unchanged to closed-source APIs and future models, offering a practical path toward dependable, autonomous code generation. Our code is available on Github (https://github.com/20ChaituR/functional-clustering).

• The paper proposes a model-agnostic functional clustering technique that groups equivalent code outputs to reduce hallucination-induced errors.
• It leverages behavior-based clustering with a custom test suite, cutting error rates on benchmarks from 65% to 0% at conservative thresholds.
• The method provides scalable confidence estimation, enabling autonomous, high-confidence code generation across diverse LLM architectures.

Eliminating Hallucination-Induced Errors in LLM Code Generation with Functional Clustering

Introduction

The increasing proficiency of LLMs in code generation brings closer the possibility of autonomous, machine-led software development. However, a significant barrier remains: the propensity of LLMs to generate code snippets with minute, yet critical, bugs—hallucination-induced errors such as off-by-one errors and incorrect operator usage. The paper proposes a model-agnostic technique called "functional clustering" to address these hallucination-induced mistakes. This technique effectively minimizes such errors while providing a scalable, exact confidence estimate for the generated outputs.

Methodology

Functional Clustering

Functional clustering is introduced as a black-box approach that leverages behavior-based clustering of multiple candidate code solutions. Instead of relying on syntactic analysis or semantic embeddings, this method assesses the functional equivalence of code outputs based on their performance on a custom-generated test suite. The methodology involves:

1. Candidate and Input Sampling: The LLM generates multiple candidate solutions and a set of diverse inputs.
2. Execution and Clustering: Each candidate solution is executed using the test inputs in a secure sandbox. Candidates producing identical outputs form equivalence classes.
3. Confidence Estimation: The largest equivalence class's empirical probability mass is used to estimate the model's confidence.

This approach automatically generates its test cases, obviating the need for human-labeled data. It is model-agnostic and can be applied to any LLM without requiring alterations to the model's architecture or access to its internal workings.

Empirical Results

The paper reports significant error reduction on LiveCodeBench, a standard benchmark for code generation tasks. At a conservative threshold, the error rate drops from approximately 65% to 0%, with a coverage of 15.6%. Importantly, the method's robustness is demonstrated with two leading code LLMs: GPT-4o and Claude-3-Haiku. An analysis of correctness versus confidence on the HumanEval benchmark reveals that functional clustering successfully isolates specification errors from random generation noise, facilitating reliable failure-mode identification.

Figure 1: Correctness versus estimated confidence on HumanEval.

Implications and Future Work

Functional clustering extends beyond error reduction; it provides an operational pathway to scalable, reliable, and autonomous code generation. The technique supports fine-tuned control over the trade-off between accuracy and coverage, allowing practitioners to adjust thresholds per project requirements. This balance ensures that models return only high-confidence results, thereby enhancing their viability in production settings.

Future work may focus on integrating dynamic specification refinements to handle multiple valid outputs and exploiting parallelization to manage computational overheads. Automated error diagnosis and iterative prompt clarification also pose promising avenues for advancing LLM capabilities in software development.

Conclusion

Functional clustering effectively addresses the persistent problem of hallucination-induced errors in LLM-generated code by focusing on functional equivalence through behavioral analysis, thereby setting a foundation for dependable, autonomous code generation. The methodological simplicity and broad applicability of functional clustering underscore its potential as a foundational tool in the evolution of machine-led software development. This operational framework not only enhances the model's reliability but also steers future research toward improving LLM understanding and adaptability in handling varied and complex programming tasks.