AutoCodeBench-Complete Framework

• AutoCodeBench-Complete is a modular, fully automated framework for constructing and evaluating execution-based code completion benchmarks with realistic, developer-centric scenarios.
• It employs a structured five-stage pipeline—including repository selection, LLM-based sandboxing and test generation, and iterative debugging—to ensure robust and reproducible evaluation.
• The framework leverages execution metrics such as Pass@k, execution success rate, and edit similarity to provide actionable insights into LLM performance in real-world coding contexts.

AutoCodeBench-Complete is a modular, fully automated framework for constructing and evaluating execution-based code completion and generation benchmarks at both the snippet and repository level. It integrates best practices and algorithmic strategies from recent advances in LLM benchmarking, focusing on realistic code scenarios, developer-centric coverage, continuous difficulty calibration, and reproducibility. The design synthesizes pipeline innovations from CodeBenchGen (Xie et al., 2024), semantic masking and evaluation protocols from ExecRepoBench (Yang et al., 2024), and developer workflow emulation from Codev-Bench (Pan et al., 2024).

1. System Architecture and Pipeline Stages

AutoCodeBench-Complete operates through a structured, five-stage pipeline, enabling seamless automation, extensibility, and traceability for large-scale code evaluation:

1. Repository and Snippet Selection: Repositories are sampled based on domain diversity and activity, using metadata (e.g., stars, contributors). File-level filters exclude non-executable or I/O-heavy code. Abstract syntax trees (ASTs) are parsed to extract candidate code fragments implementing appropriate size and complexity constraints. Each selected code snippet is uniquely identified by repository, file, and line range (Xie et al., 2024).
2. LLM-Based Sandboxing: Arbitrary code fragments are transformed into isolated, executable Python mini-modules via LLM prompting. The input consists of the code context plus required imports and helper functions. Sandboxing adheres to context-size and security constraints (e.g., banning certain operations), and iteratively refines the transformed module until it compiles successfully.
3. LLM-Based Test Generation: For each sandboxed module, the LLM generates a test suite, typically requiring at least three independent unit tests that cover edge, normal, and failure cases. Prompts employ few-shot learning with labeled example pairs and may leverage chain-of-thought priming to encourage coverage of boundary conditions (Xie et al., 2024).
4. Filtering and Iterative Debugging: Synthesized code and corresponding tests are executed in a containerized or versioned environment. If failures occur, LLM-based auto-debugging patches are suggested and applied iteratively, up to a set limit. Only code passing all tests and safety checks proceeds to the next stage.
5. Benchmark Assembly and Validation: Validated examples are aggregated, with all relevant metadata (dependencies, provenance, LLM prompts, environment specs). The system emits a benchmark manifest (JSON), dependency files, and CI scripts for reproducible, automated evaluation and deployment (Xie et al., 2024).

Each pipeline stage communicates via standardized JSON metadata, supporting easy interchange, inspection, and future extensions.

2. Benchmark Granularity and Scenario Construction

AutoCodeBench-Complete supports construction of both granular snippet-based and developer-centric repository-level benchmarks:

• Block- and Line-Level Samples:

The system enables sampling at the granularity of statements, expressions, full or inner blocks (e.g., functions, control structures), matching real-world developer workflows observed in IDE completion logs (Pan et al., 2024). Multi-level grammar-based masking on ASTs enables extraction of masking spans at different syntactic units for code infilling evaluation (Yang et al., 2024).

• Repository-Scale Scenarios:

Entire repositories are processed by automated agents to configure execution environments, identify unit tests, parse dynamic call chains, and construct completion holes for evaluation. Scenarios range from full block to inner-block and incomplete-suffix completion, as well as retrieval-augmented completions, anchoring each task to a specific unit test and code context (Pan et al., 2024).

Scenario frequencies and prompt lengths are derived from empirical developer usage, ensuring coverage of both common and challenging real-world editing actions.

3. Evaluation Protocols and Key Metrics

Rigorous evaluation is central to AutoCodeBench-Complete, operating directly on execution outcomes and semantic similarity metrics:

• Execution-Based Metrics:
  • Pass@k: Probability that at least one of $k$ independently sampled completions passes all unit tests for a sample of $n$ model outputs with $c$ correct, formalized as:

  $$\mathrm{Pass}@k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}$$

  Used for both greedy ($k=1$, first decode) and top-$k$ suggestion accuracy, with averaging across all benchmark samples (Xie et al., 2024, Yang et al., 2024, Pan et al., 2024). - Execution Success Rate: Fraction of examples for which at least one sampled output passes all tests.

• Edit Similarity (ES):
  • Normalized Levenshtein score quantifies how close the LLM's output $g_i$ is to the reference $r_i$:

  $$\mathrm{ES}_i = 1 - \frac{\mathrm{Levenshtein}(g_i, r_i)}{\max(|g_i|, |r_i|)}$$

  The overall ES is the mean across all samples (Yang et al., 2024, Pan et al., 2024).

• Exact Match Rate: Fraction of completions that exactly (token-for-token) match the ground truth.

Functional correctness (Pass@k) is prioritized over syntactic similarity, but edit metrics are reported to reveal over-generation, off-by-one errors, or usability issues.

4. Solvability and Difficulty Calibration

Human subject studies calibrate task solvability and difficulty. For example, CodeBenchGen measured that 81.3% of Exec-CSN benchmark examples are solvable by graduate students, with 61% requiring substantial effort. Difficulty bands and ratings (e.g., quick, effortful, expert-only) are collected via participant surveys and fed back to the pipeline, allowing pruning or re-prompting of trivial or unsolvable samples (Xie et al., 2024).

A similar calibration loop can be integrated through a web UI, logging user attempts, time spent, usage of external resources, and post-task difficulty assessments. This data supports the automatic tagging of scenarios by challenge level, ensuring that the benchmark offer granular insights into model performance as a function of real-world task difficulty.

5. Scaling, Automation, and Reproducibility

Robust scalability and reproducibility are ensured through several system practices:

• Diverse Repository Sampling:

Stratified by topics, contributors, and popularity, with periodic refresh to avoid overfitting (Xie et al., 2024).

• Immutable Execution Environments:

Containerization (e.g., Docker, Nix) with strict version pinning for all dependencies ensures every example and completion task is reproducible, independent of host system drift (Xie et al., 2024).

• Automated CI/CD Integration:

Continuous verification via automated smoke tests on random subsets detects drift or flakiness. All runs are versioned, including LLM parameters, prompts, and random seeds.

• Comprehensive Metadata and Cataloging:

Every example is indexed by domain, difficulty, lines of code, AST depth, and provenance. Queryable registries enable targeted mini-benchmark selection (e.g., by topic or complexity) (Xie et al., 2024).

• Agent-Based Repository Management:

LLM-driven agents orchestrate repository crawling, environment setup, dynamic program analysis, and automated test harness construction, minimizing the need for manual annotation or intervention (Pan et al., 2024).

6. Integration of Execution-Based, Developer-Centric, and Repository-Level Evaluation

AutoCodeBench-Complete synthesizes multiple dimensions of code evaluation in a single extensible platform:

• Execution-Driven Verification:

All completions are verified by rerunning the native or synthesized unit tests on actual code with the LLM completion inserted. No reliance on string-matching heuristics or external gold annotations.

• Developer-Intent and Scenario Modeling:

Scenario sampling, prompt lengths, and completion “hole” locations are derived from empirical developer interaction data, matching the completion patterns observed in practice (Pan et al., 2024).

• Multi-Level Grammar-Based Completion:

ASTs are employed both for identifying valid completion units and for generating challenging masked spans at the statement, expression, or function level. This multi-level setup increases robustness against model overfitting to simplistic or synthetic patterns (Yang et al., 2024).

• RAG-Style Augmentation:

Completion contexts can be enhanced with retrieval-augmented generation, leveraging relevant code fragments from within the same repository for improved realism and performance assessment (Pan et al., 2024).

7. Representative Experimental Findings

Empirical results from related frameworks illustrate the impact of execution-based, context-rich benchmarks:

• Qwen2.5-Coder-Instruct-C (7B):

Achieves 44.2% average greedy Pass@1 on ExecRepoBench, more than doubling the rate of prior 7B models, primarily attributed to grammar-based masking and cross-file context (Yang et al., 2024).

• Codev-Bench Scenarios:

Code-oriented LLMs (FIM-mode) outperform general LLMs on both full-block and inner-block completions, with Pass@1 for Codegemma-7b up to 77.4% on inner-block tasks versus 15.5% for GPT-4 in the same scenario (Pan et al., 2024). Correlations between edit similarity and pass rates highlight the importance of functional, not merely syntactic, correctness.

Conclusion

AutoCodeBench-Complete represents an overview of contemporary best practices for execution-based, developer-centric, and repository-level evaluation of code generation and completion LLMs. Through modular LLM-driven pipelines, rigorous execution-oriented metrics, and difficulty-driven calibration, it enables scalable, reproducible, and realistic benchmarking aligned with the workflows and needs of modern software development (Xie et al., 2024, Yang et al., 2024, Pan et al., 2024).