CodeAssistBench: Multi-turn Programming Assist

• CodeAssistBench is a benchmark framework that evaluates multi-turn, chat-based programming assistance in full repository contexts using realistic Docker environments.
• It automates dataset generation from GitHub issues and recreates containerized environments to simulate iterative dialogues for debugging and configuration tasks.
• Empirical findings highlight a significant performance gap with single-turn benchmarks, emphasizing the challenges of sustaining accurate, multi-step technical interactions.

Searching arXiv for the CAB paper and closely related benchmarks to ground citations. CodeAssistBench (CAB) is a benchmark and evaluation framework for multi-turn, chat-based programming assistance in realistic, project-specific contexts. It was introduced as “the first benchmark framework for evaluating multi-turn programming assistance in realistic settings that address real-world questions about actual codebases,” with automatic dataset generation from question-related GitHub issues, automatic containerization of codebases for evaluation, and model assessment through simulated users in containerized environments with full codebase access (Kim et al., 14 Jul 2025). CAB is explicitly positioned against benchmarks centered on single-turn code generation or isolated question answering, and instead targets iterative, dialogue-driven support on real GitHub issues within full repositories and, where needed, reproducible containerized environments (Kim et al., 14 Jul 2025).

1. Motivation and problem setting

CAB is motivated by the claim that prevailing programming-assistant benchmarks are too narrow. Popular Q&A-style benchmarks derived from Stack Overflow, including InfiBench and StackEval, are described as single-turn, isolated, and heavily curated; models already perform well on them, with reported success rates of 70.64% on InfiBench and 83.0% on StackEval (Kim et al., 14 Jul 2025). CAB treats these results as evidence that such datasets “no longer represent significant challenges for contemporary LLMs” and that they fail to capture full project context, build systems, and the iterative back-and-forth of real developer workflows (Kim et al., 14 Jul 2025).

The benchmark therefore shifts emphasis from single-shot code synthesis on toy problems to conversational assistance over complete repositories. The underlying problem formulation includes clarification, environment setup, debugging, and navigation of repository-specific constraints, rather than merely generating a code snippet. The port-mapping example described in CAB is used to show that test-only oracles are insufficient: the assistant must infer project topology and configuration, explain constraints such as a hard-coded proxy port, and adapt over turns to satisfy evolving user needs (Kim et al., 14 Jul 2025). A plausible implication is that CAB evaluates not only answer correctness in the narrow sense, but also the capacity to sustain technically grounded interaction under repository-specific constraints.

CAB’s core contributions are framed in four parts: automated, scalable dataset generation from GitHub issues; multi-turn evaluation with simulated users and full codebase access; automatic containerization and environment reconstruction; and continuous expansion through configurable repository creation dates that permit “recent” cohorts after model-training cutoffs (Kim et al., 14 Jul 2025). In this design, the benchmark is not merely a static dataset, but a configurable framework for constructing and refreshing evaluation cohorts.

2. Dataset construction pipeline

CAB converts raw GitHub issues into structured evaluation instances in three stages: repository collection, issue filtering, and data preparation (Kim et al., 14 Jul 2025). Repository selection begins from all public GitHub repositories, denoted $RGH$, and uses configurable filters over stars, creation date, license, and language:

$$R = \{ r \in RGH \mid s(r) > S_{\min},\ t(r) > t_0,\ \ell(r) \in L \}$$

Here, $s(r)$ is star count, $t(r)$ is creation date, and $\ell(r)$ is an SPDX permissive license such as MIT or Apache-2.0 (Kim et al., 14 Jul 2025). CAB then defines a community score

$CS(r) = Q(r) + H(r)$

where $Q(r)$ and $H(r)$ are the counts of closed issues labeled “question” and “help wanted,” respectively; the framework selects the top-$K$ repositories by this score, breaking ties by stars (Kim et al., 14 Jul 2025).

Issue filtering starts from closed issues in the selected repositories. Each issue $i$ is represented as

$$R = \{ r \in RGH \mid s(r) > S_{\min},\ t(r) > t_0,\ \ell(r) \in L \}$$0

and is processed by regex-based rules and LLM-based filters (Kim et al., 14 Jul 2025). The regex rules require at least two distinct participants, in order to ensure genuine Q&A dynamics, and remove issues containing media such as URLs, images, or videos to focus on text-only assistance (Kim et al., 14 Jul 2025). CAB then applies a 7-question issue-level LLM filter with binary Yes/No decisions over resolution, specificity, clarity, safety, and reproducibility; only issues that are clearly resolved by someone other than the author, technically specific, reproducible, and free of sensitive content are retained (Kim et al., 14 Jul 2025). A second, message-level LLM filter removes comments with “absolutely no support value,” including examples such as “+1,” “Thanks,” and “Bump,” while preserving other comments to maintain realistic conversational flow (Kim et al., 14 Jul 2025).

Retained issues are restructured into complete user–maintainer exchanges:

$$R = \{ r \in RGH \mid s(r) > S_{\min},\ t(r) > t_0,\ \ell(r) \in L \}$$1

so that the issue becomes

$$R = \{ r \in RGH \mid s(r) > S_{\min},\ t(r) > t_0,\ \ell(r) \in L \}$$2

(Kim et al., 14 Jul 2025). CAB then prepares each issue for evaluation by reconstructing the repository state near the issue timestamp, generating environment specifications when needed, extracting satisfaction conditions, and producing references for realistic follow-up behavior. The resulting dataset entry is

$$R = \{ r \in RGH \mid s(r) > S_{\min},\ t(r) > t_0,\ \ell(r) \in L \}$$3

where $$R = \{ r \in RGH \mid s(r) > S_{\min},\ t(r) > t_0,\ \ell(r) \in L \}$$4 is the commit closest to issue creation time, $$R = \{ r \in RGH \mid s(r) > S_{\min},\ t(r) > t_0,\ \ell(r) \in L \}$$5 is the successful environment configuration when applicable, $$R = \{ r \in RGH \mid s(r) > S_{\min},\ t(r) > t_0,\ \ell(r) \in L \}$$6 is the set of extracted satisfaction conditions, and $$R = \{ r \in RGH \mid s(r) > S_{\min},\ t(r) > t_0,\ \ell(r) \in L \}$$7 is a set of user response references retrieved via BM25 over historical maintainer–user pairs (Kim et al., 14 Jul 2025).

The satisfaction-condition component is central to CAB’s notion of success. Sonnet 3.7 is prompted to distill explicit, verifiable, solution-agnostic criteria that the user expects fulfilled, producing $$R = \{ r \in RGH \mid s(r) > S_{\min},\ t(r) > t_0,\ \ell(r) \in L \}$$8 (Kim et al., 14 Jul 2025). These conditions are verified for JSON format, completeness as condition-plus-explanation, and conversational grounding; conditions that fail verification are discarded, and issues without surviving conditions are excluded (Kim et al., 14 Jul 2025). This makes the benchmark’s target variable a structured set of user-grounded requirements rather than a single reference answer.

3. Environment reconstruction and multi-agent evaluation

A distinguishing feature of CAB is automatic environment preparation. For each issue, CAB identifies the commit closest to issue creation time, clones the repository at that snapshot, extracts artifacts such as the README, Dockerfiles, workflows, and file tree, and synthesizes build scripts or Dockerfiles (Kim et al., 14 Jul 2025). Candidate generation and iterative repair are automated via Sonnet 3.7 prompts. The prompt context includes the repository URL, issue title, commit SHA, truncated body, README, repository structure, up to two workflows, and a reference Dockerfile if one exists; the model is instructed as an “expert Docker engineer” and returns raw Dockerfile content without Markdown fences so it can be passed directly to docker build (Kim et al., 14 Jul 2025).

If all initial candidates fail, CAB captures the build logs and re-prompts Sonnet 3.7 to fix the observed errors, permitting up to three repair attempts and stopping at the first success (Kim et al., 14 Jul 2025). Build candidates must install dependencies, clone and check out the specified commit, and build the project. The initial build phase intentionally avoids running tests or the user’s code; those actions are deferred to the maintainer agent during evaluation if needed (Kim et al., 14 Jul 2025). Across 294 environment-dependent issues, CAB reports 81% overall success, with per-language examples including Java at 90%, JavaScript at 97%, Python at 86%, and C++ at 53% (Kim et al., 14 Jul 2025).

Evaluation proceeds through three agents. The user agent presents the initial question and context, monitors progress relative to the extracted satisfaction conditions $$R = \{ r \in RGH \mid s(r) > S_{\min},\ t(r) > t_0,\ \ell(r) \in L \}$$9, provides follow-ups, and signals when satisfied; it does not directly execute commands, but it can incorporate observed execution outcomes (Kim et al., 14 Jul 2025). The maintainer agent, which is the model under test, can read repository files, run commands in the container, and edit code or configuration files as needed. Per response, it may take up to five exploration steps, where each step can be a file operation or command execution, in order to balance efficiency and thoroughness (Kim et al., 14 Jul 2025). The judge agent evaluates the outcome once the dialogue ends, either because the user is satisfied or because the 10-turn cap has been reached (Kim et al., 14 Jul 2025).

The judge’s criteria integrate technical correctness, satisfaction completeness, and interaction quality. For Docker-tagged issues, execution success is mandatory; technically sound but non-executable answers count as incorrect (Kim et al., 14 Jul 2025). CAB logs the full dialogue history, environment outputs, and file changes, maintaining state across turns for reproducibility (Kim et al., 14 Jul 2025). The user follow-up process is grounded by BM25 retrieval over similar historical issues, which is intended to preserve realistic conversational patterns while remaining fully automated (Kim et al., 14 Jul 2025).

4. Metrics, resolution criteria, and corpus characteristics

CAB reports correctness and verbosity, while also defining formal metrics for success, turn efficiency, condition coverage, and human alignment of extracted satisfaction conditions (Kim et al., 14 Jul 2025). Overall correctness is measured by success rate:

$s(r)$0

where ResolvedIssues counts judge verdicts of “Correct” (Kim et al., 14 Jul 2025). Mean dialogue length is

$s(r)$1

where $s(r)$2 is the number of turns for issue $s(r)$3 (Kim et al., 14 Jul 2025). CAB also defines condition-level alignment by

$s(r)$4

where $s(r)$5 is the number of satisfaction conditions marked TRUE by the judge and $s(r)$6 is the total number of satisfaction conditions for issue $s(r)$7; aggregate alignment may be reported either as mean $s(r)$8 or as

$s(r)$9

(Kim et al., 14 Jul 2025).

A separate human study evaluates the extracted satisfaction conditions through precision and completeness. CAB defines

$t(r)$0

and

$t(r)$1

and reports 86.27% accuracy and 65.71% completeness across 663 conditions, with language-specific breakdowns (Kim et al., 14 Jul 2025). Resolution itself is tri-valued: “Correct” means fully accurate and satisfying all user conditions, with build or test success required for Docker issues; “Partially correct” indicates minor flaws or partial condition satisfaction; and “Incorrect” covers major errors, unmet conditions, or failed Docker validation (Kim et al., 14 Jul 2025).

The corpus statistics show both scale and curation. CAB begins from 25,656 closed GitHub issues with “question/help-wanted” tags and retains 3,286 after regex and LLM filtering plus condition verification (Kim et al., 14 Jul 2025). The paper reports the final retained set in two ways: the abstract states 3,286 issues across 231 repositories, whereas Section 4.1 reports 3,286 issues across 214 repositories, with 294 requiring Docker and 238 successfully built (Kim et al., 14 Jul 2025). CAB attributes these differences primarily to environment retention and license or activity constraints, and directs readers to repository metadata for exact counts per release (Kim et al., 14 Jul 2025). This discrepancy is not presented as a contradiction in task count, but as a release-accounting issue tied to retention criteria.

The benchmark spans seven programming languages, with final retained totals of Python 647, Java 537, C++ 493, C# 536, JavaScript 444, TypeScript 441, and C 188, summing to 3,286 (Kim et al., 14 Jul 2025). Over half of issues across all languages involve multiple turns, and Python, C++, and TypeScript have the highest rates of three or more turns (Kim et al., 14 Jul 2025). The framework also defines two temporal cohorts, “All-Time” and “Recent (after Nov 2024),” so that evaluation can distinguish historically accumulated issues from post-cutoff material that is intended to remain challenging as models evolve (Kim et al., 14 Jul 2025).

5. Empirical findings and comparative position

CAB evaluates ChatGPT 4.1 Mini, DeepSeek R1, Llama 3.3 70B, Haiku 3.5, Sonnet 3.7, and Sonnet 3.7 (Think mode) (Kim et al., 14 Jul 2025). Dialogue generation temperature is fixed at 0 for reproducibility, while Dockerfile synthesis uses temperature 0.7 to encourage candidate diversity (Kim et al., 14 Jul 2025). The evaluation subset is obtained by balanced stratified sampling over language and turn length, with up to five Dockerfile-containing issues per language, yielding 350 datasets for All-Time and 194 for Recent (Kim et al., 14 Jul 2025).

The headline result is a substantial gap between Stack Overflow-like Q&A benchmarks and CAB. Models achieve 70–83% success on StackOverflow-derived benchmarks, specifically 70.64% on InfiBench and 83.0% on StackEval, but on CAB Recent repositories the best model, ChatGPT 4.1 Mini, reaches only 16.49% correctness; on All-Time, the same model reaches 29.14% (Kim et al., 14 Jul 2025). The Recent correctness range across evaluated models is 7.22% for Haiku 3.5 to 16.49% for ChatGPT 4.1 Mini, while the All-Time range is 13.58% for Llama 3.3 70B to 29.14% for ChatGPT 4.1 Mini (Kim et al., 14 Jul 2025). DeepSeek R1 is noted as having the lowest error rate on All-Time (Kim et al., 14 Jul 2025).

CAB also reports turn-level effects. Correct answers typically require two to three turns, whereas incorrect cases tend to prolong CAB dialogues by one to two turns relative to the original GitHub threads, which is described as consistent with users seeking clarifications when assistance is vague or incomplete (Kim et al., 14 Jul 2025). Language-specific patterns indicate that statically typed languages—C#, C++, and Java—are harder, especially in Recent cohorts, where correctness often falls below approximately 15% (Kim et al., 14 Jul 2025). Dynamically typed languages—JavaScript and Python—show stronger All-Time performance, with examples of up to approximately 44% correctness on JavaScript for some models, but they also drop markedly on Recent cohorts (Kim et al., 14 Jul 2025). No model dominates across both typed and dynamic languages in Recent repositories (Kim et al., 14 Jul 2025).

In relation to prior work, CAB is contrasted with InfiBench and StackEval by its multi-turn interactions, full repository and environment access, and automated scalable generation requiring minimal manual curation (Kim et al., 14 Jul 2025). It is also distinguished from code-generation benchmarks such as HumanEval, MBPP, and CodeContests, and from maintenance evaluations such as SWE-Bench and SWE-PolyBench, because CAB focuses on conversational assistance and user satisfaction over multiple turns rather than pass@1 or single-shot test success (Kim et al., 14 Jul 2025). Relative to multi-turn code-conversation benchmarks such as ConvCodeWorld, MINT, and TICODER, CAB is characterized by grounding dialogues in real repositories and GitHub issues with environment validation rather than in controlled synthetic tasks centered on code synthesis (Kim et al., 14 Jul 2025). A plausible implication is that CAB occupies a distinct evaluation niche at the intersection of repository-grounded software engineering, conversational agents, and reproducible execution.

6. Reproducibility, limitations, and practical use

CAB is publicly released with dataset, scripts, prompts, and evaluation framework, and it provides judge and maintainer prompts, Dockerfile generation prompts, and full interaction logs for exact replication (Kim et al., 14 Jul 2025). Repository selection is restricted to permissive licenses such as MIT and Apache-2.0, and issue-level filtering removes PII in order to reduce legal and privacy risks (Kim et al., 14 Jul 2025). The reported hardware includes AWS g5.16xlarge with 1 A10G GPU, 64 vCPUs, and 256 GB RAM; the GPU was unused for inference because models were API-based, while CPU and RAM were significant for Docker builds (Kim et al., 14 Jul 2025). Recommended local resources for large-scale builds are approximately 1 TB of storage and 32 GB of RAM (Kim et al., 14 Jul 2025).

The benchmark’s limitations are explicit. Satisfaction-condition extraction emphasizes precision over recall: the human-alignment study shows approximately 86.3% accuracy but only approximately 65.7% completeness, so models may be penalized for criteria missing in $t(r)$2 (Kim et al., 14 Jul 2025). Evaluation is performed on a stratified sample of 544 issues because of compute cost, which means rare edge cases may be underrepresented (Kim et al., 14 Jul 2025). Simulated user behavior is BM25-based and may underrepresent exploratory or ambiguous behaviors (Kim et al., 14 Jul 2025). CAB includes only issues with successful environment synthesis, skewing the benchmark toward modern repositories with standard tooling, while legacy or atypical builds are underrepresented (Kim et al., 14 Jul 2025). The scope is further limited to seven languages and permissively licensed repositories, leaving enterprise or proprietary codebases outside the benchmark (Kim et al., 14 Jul 2025). Temperature settings are fixed for reproducibility, but confidence intervals and significance tests are not reported (Kim et al., 14 Jul 2025).

CAB also provides practical guidance for deployment as an evaluation framework. The recommended workflow is to configure repository selection through $t(r)$3, $t(r)$4, languages, and licenses; run the repository and issue collection scripts; execute the dataset-generation pipeline to synthesize Docker environments, extract satisfaction conditions, and create user-response references; and then use the evaluation orchestrator to run multi-agent simulations against a chosen LLM via API and collect judge outputs (Kim et al., 14 Jul 2025). For interpretation, the framework recommends using SuccessRate for overall solvability, Alignment_i for condition-level coverage, and MeanTurns for turn-efficiency and friction; environment-dependent failures are treated as hard negatives (Kim et al., 14 Jul 2025). Suggested baselines include ChatGPT 4.1 Mini, Sonnet 3.7 (Think), DeepSeek R1, and an open-weight model such as Llama 3.3 70B (Kim et al., 14 Jul 2025). For organizational adaptation, CAB recommends mirroring the pipeline on sanitized internal issues, defining satisfaction conditions under internal policies, integrating CI for environment validation, and using BM25 retrieval over internal issue archives to seed realistic user follow-ups (Kim et al., 14 Jul 2025).

The port-remapping example illustrates the framework’s intended granularity. CAB reconstructs the repository at the issue timestamp, extracts satisfaction conditions such as requiring the main application to run on a user-specified port without conflicts and requiring that the Cloudflare proxy port not be exposed unless explicitly necessary, and then evaluates a multi-turn exchange in which the maintainer agent edits docker-compose.yml, runs docker-compose up, and verifies the absence of port conflicts (Kim et al., 14 Jul 2025). The judge then marks all satisfaction conditions met and returns the verdict “Correct” (Kim et al., 14 Jul 2025). This example encapsulates CAB’s central premise: the relevant unit of evaluation is not a detached answer string, but a repository-grounded conversational trajectory that can include explanation, configuration changes, execution, and satisfaction of explicit user conditions.