CodeAssistBench: Multi-turn Code Evaluation

• CodeAssistBench is a benchmark and dataset framework that rigorously evaluates multi-turn code assistance using real-world GitHub issues.
• It employs automated dataset creation, containerized builds, and simulated dialogues to replicate realistic, project-level interactions.
• CAB reveals significant capability gaps in current LLMs, highlighting the need for improved environment-aware and interactive code generation.

CodeAssistBench (CAB) is a benchmark and dataset framework designed to rigorously evaluate multi-turn, chat-based code assistance in realistic, project-grounded environments. Addressing vital shortcomings of single-turn and isolated-code benchmarks, CAB automatically constructs large-scale test corpora from public GitHub issues, integrates full project context through containerized builds, and evaluates LLM-powered programming agents via simulated user–assistant dialogues. The benchmark reveals substantial capability gaps in current state-of-the-art models, particularly in tasks requiring interactive clarification, deep repository reasoning, and complex environment orchestration (Kim et al., 14 Jul 2025).

1. Motivation and Distinguishing Features

CAB was created in response to limitations of prior code QA benchmarks, which focus on single-turn generation of code snippets or patches (e.g., HumanEval, MBPP, CodeContests), and the more recent Stack Overflow–derived Q&A benchmarks (InfiBench, StackEval), which are still single-turn, manual in curation, and devoid of project-scale context. Empirical evidence indicates the centrality of multi-turn interactions—debugging, codebase exploration, iterative clarifications—in real software development workflows (with ≥77% of 34,000+ surveyed developers highlighting these needs).

CAB distinguishes itself through a fully automated and scalable pipeline for extracting, filtering, and operationalizing real-world GitHub issues. Dialogues are evaluated within their original project environments, following a multi-turn, chat-based paradigm that includes up to ten user–assistant exchanges. This setup enables robust assessment of a code assistant’s ability not only to generate code but also to reason about and act upon complex, stateful software ecosystems.

Key features include:

• Automated dataset construction: Large-scale harvesting and filtering of GitHub issues by configurable repository characteristics (creation date, stars, license).
• Multi-turn dialogues: Structured user–assistant turns reconstitute the conversational nature of real programming help requests.
• Full project context: Repositories at the relevant commit (including build/test scripts) are containerized for reproducibility.
• Scalable containerization: Automated Dockerfile synthesis/repair ensures robust build/test execution.
• Automated agent evaluation: Simulated “User” agents drive clarifications; “Judge” agents grade correctness and interaction quality.

2. Dataset Construction and Testbed Composition

CAB employs a three-phase pipeline: repository collection, issue filtering, and data preparation.

1. Repository Collection: Public GitHub repositories are filtered using:
   • Star count $s(r) \ge S_{min}$ (e.g., $S_{min} = 10$)
   • Creation date $t(r) \ge t_0$
   • Approved license $\ell(r)$
   • Community score: $CS(r) = Q(r) + H(r)$, where $Q(r)$ is the number of closed “question” issues and $H(r)$ is the count of “help wanted” issues.
   • Top-$K$ by $CS(r)$ selected as $R_K$.
2. Issue Filtering: For each repository, closed issues with the appropriate tags are filtered by:
   • Presence of at least two distinct participants.
   • LLM-validated binary criteria: includes checks for resolution, specificity, clarity, safety, reproducibility.
   • Non-contributory comments are stripped.
   • Each filtered issue is parsed into a sequence of (user, maintainer) dialogue turns.
3. Data Preparation:
   • Environment Reproduction: The relevant commit ($S_{min} = 10$0) is identified and used to synthesize Docker environments. Up to 5 candidate Dockerfiles are generated and repaired iteratively; the first that builds is retained.
   • Satisfaction Condition Extraction: An LLM derives a set $S_{min} = 10$1 of verifiable conditions for success.
   • User-Response Reference Mining: A BM25 index over historical dialogues is built to sample user-style clarifications for simulation.

The resultant CAB dataset consists of 3,286 real-world programming tasks from 231 repositories (seven languages: Python, Java, C++, C#, JavaScript, TypeScript, C), with coverage of domains including web frameworks, build tools, cloud infrastructure, and data processing. Success rates for Docker builds are reported as 78.2% (all-time) and 97.6% (recent issues) (Kim et al., 14 Jul 2025).

Metric	Total	Python	Java	C++	C#	JS	TS	C
Final Retained Issues	3,286	647	537	493	536	444	441	188
Recent Subset (post-Nov 2024)	308	159	8	31	25	39	26	20

3. Evaluation Protocol and Tooling

CAB’s evaluation employs three agent roles within a controlled, containerized sandbox:

• User Agent: Initiates dialogue with the issue content and repository context. Judges replies (via satisfaction conditions) and requests follow-up, but does not execute code directly.
• Maintainer Agent: Instantiated by a code-capable LLM (e.g., ChatGPT 4.1 Mini, Sonnet 3.7), allowed up to five environment exploration operations per turn (file operations, code execution). Supplies solutions, explanations, or configuration advice.
• Judge Agent: At dialogue end (upon user satisfaction or after ten turns), an LLM-driven judge assigns:
  • Technical Correctness (does final state achieve all satisfaction conditions and build/run)
  • Satisfaction Completeness (fraction of $S_{min} = 10$2 met)
  • Interaction Quality (conciseness, clarity)
  • Outputs: Correct / Partially Correct / Incorrect

Metrics include:

• Resolution Rate $S_{min} = 10$3
• Average turn count per verdict
• Verbosity and (not reported in original paper but derivable) Response latency (Kim et al., 14 Jul 2025).

4. Benchmark Results and Capability Gap Analysis

A principal finding is the pronounced capability gap between isolated, standalone QA benchmarks and CAB’s multi-turn, environment-grounded tasks.

Comparison Summary

Model	Correct (Recent)	Correct (All)	Avg. CAB Turns (Correct)
ChatGPT 4.1 Mini	16.49%	29.14%	2.94
DeepSeek R1	11.34%	27.14%	2.82
Llama 3.3 70B	9.33%	13.58%	3.22
Haiku 3.5	7.22%	16.86%	3.86
Sonnet 3.7	11.34%	25.71%	2.36
Sonnet 3.7(Think)	13.40%	27.43%	2.50

By contrast, success rates on StackOverflow-style benchmarks remain much higher (InfiBench ≈ 70.64%, StackEval ≈ 83.0%). For recent, more challenging CAB issues, no model exceeds 16.49% correctness.

• Language effect: Statically typed languages (C#, C++, Java) are systematically harder (most models <13% on recent C# issues); dynamic languages (Python, JS) are moderately easier but also drop below 15% on recent tasks.
• Turn count: Successful CAB dialogues average 2–3 turns, similar to real GitHub threads; unsuccessful sessions routinely require 1–2 additional clarifications.

The difficulty arises from the need for codebase-wide exploration, interaction with unfamiliar build/test systems, iterative clarification, and multi-file, environment-aware edits—capabilities that currently overstrain existing LLMs.

5. Methodological Extensions: Structure-Aware Approaches and Feedback Dynamics

Recent research leverages CAB to investigate both structure-aware code agent interfaces and interactive evaluation protocols.

Structure-Aware Edits with AST Primitives: The CodeStruct framework, evaluated on CAB, reframes editing from brittle string-substitution to node-level AST manipulation via readCode (for retrieval) and editCode (for validated transformation). On CAB, CodeStruct yields accuracy improvements of +0.8 to +4.4 percentage points across six LLMs, with significant resource savings (up to 33.7% fewer input tokens and 33.3% lower cost for GPT-5-mini). AST-based navigation eliminates string-match and line-range guessing errors, particularly benefiting mid- and lower-capacity models. Qwen3-32B demonstrates the most pronounced accuracy jump under CodeStruct (+4.4 pp), albeit with increased token and cost overhead, suggesting model-size–dependent tradeoffs (Kim et al., 7 Apr 2026).

Interactive Feedback Protocols: CAB was also featured in an analysis of LLM code agents under interactive feedback (Pan et al., 25 Feb 2025). In this setup, static coding problems are obfuscated to enforce information asymmetry, requiring agents to iteratively solicit and incorporate simulated user feedback (sentence, paragraph, code correction, or query rephrasing). Results show that detailed feedback (paragraphs, code corrections) produces largest gains in functional correctness (up to +0.10 improvement relative to self-critique), while model rankings shift non-negligibly between interactive and static evaluation (Spearman’s Footrule $S_{min} = 10$4). CAB’s conversational loop elicits more realistic model behaviors, with stronger models making small, impactful edits and weaker models exhibiting superficial but less functional code modifications.

6. Limitations, Evaluation Challenges, and Open Directions

CAB exposes technical and methodological challenges:

• Satisfaction Condition Extraction: Achieves 86.3% precision but only 65.7% recall. Improving recall (e.g., hybrid LLM–heuristic or human-in-the-loop) is a key direction (Kim et al., 14 Jul 2025).
• Simulated User Modeling: Current BM25-based strategies generate plausible clarifications but lack coverage of user uncertainty, exploratory behavior, or nonstandard error-handling.
• Environment Reasoning: Success rates sharply decline for tasks demanding multi-step debugging, environment orchestration, and implicit project-structure inference.

Suggested improvements include richer user simulation techniques, integration of tool-use primitives (debuggers, linters, REPLs), support for additional programming languages (Rust, Go, Swift, ML domains), expanded metrics (including developer satisfaction, response latency, resource usage), and fine-grained scoring for partial satisfaction or agent confidence.

Potential CAB extensions involve modeling multi-agent collaboration (e.g., LLM pair-programming), supporting version control workflows (branch merges, reviews), continual “nightly” evaluation on new repository commits, and broader benchmarking of tool-coupled agents.

7. Significance and Influence

CAB establishes a new standard for evaluating code assistants as interactive, full-context agents within realistic software engineering workflows. By demonstrating that leading models fail on over 80% of contemporary, real-world multi-turn tasks, CAB highlights the limitations of current LLM architectures and benchmarks, guiding development of next-generation developer tools and more robust, collaborative LLMs (Kim et al., 14 Jul 2025, Pan et al., 25 Feb 2025, Kim et al., 7 Apr 2026).

The adoption of CAB by subsequent work underscores its value as a foundation for both structure-aware agent research and studies of agent steerability under interactive conditions. As the field transitions to more agentic, project-aware evaluation, CAB provides a rigorously defined, extensible, and automatable testbed, aligning benchmarks with the demands of actual professional programming practice.