---
title: 'CodeAssistBench: Multi-turn Programming Assist'
url: https://www.emergentmind.com/topics/codeassistbench-cab
type: topic
---

# CodeAssistBench: Multi-turn Programming Assist

Searching arXiv for the CAB paper and closely related benchmarks to ground citations.
CodeAssistBench (CAB) is a benchmark and evaluation framework for multi-turn, chat-based programming assistance in realistic, project-specific contexts. It was introduced as “the first benchmark framework for evaluating multi-turn programming assistance in realistic settings that address real-world questions about actual codebases,” with automatic dataset generation from question-related GitHub issues, automatic containerization of codebases for evaluation, and model assessment through simulated users in containerized environments with full codebase access [2507.10646]. CAB is explicitly positioned against benchmarks centered on single-turn code generation or isolated question answering, and instead targets iterative, dialogue-driven support on real GitHub issues within full repositories and, where needed, reproducible containerized environments [2507.10646].

## 1. Motivation and problem setting

CAB is motivated by the claim that prevailing programming-assistant benchmarks are too narrow. Popular Q&A-style benchmarks derived from Stack Overflow, including InfiBench and StackEval, are described as single-turn, isolated, and heavily curated; models already perform well on them, with reported success rates of 70.64% on InfiBench and 83.0% on StackEval [2507.10646]. CAB treats these results as evidence that such datasets “no longer represent significant challenges for contemporary LLMs” and that they fail to capture full project context, build systems, and the iterative back-and-forth of real developer workflows [2507.10646].

The benchmark therefore shifts emphasis from single-shot code synthesis on toy problems to conversational assistance over complete repositories. The underlying problem formulation includes clarification, environment setup, debugging, and navigation of repository-specific constraints, rather than merely generating a code snippet. The port-mapping example described in CAB is used to show that test-only oracles are insufficient: the assistant must infer project topology and configuration, explain constraints such as a hard-coded proxy port, and adapt over turns to satisfy evolving user needs [2507.10646]. A plausible implication is that CAB evaluates not only answer correctness in the narrow sense, but also the capacity to sustain technically grounded interaction under repository-specific constraints.

CAB’s core contributions are framed in four parts: automated, scalable dataset generation from GitHub issues; multi-turn evaluation with simulated users and full codebase access; automatic containerization and environment reconstruction; and continuous expansion through configurable repository creation dates that permit “recent” cohorts after model-training cutoffs [2507.10646]. In this design, the benchmark is not merely a static dataset, but a configurable framework for constructing and refreshing evaluation cohorts.

## 2. Dataset construction pipeline

CAB converts raw GitHub issues into structured evaluation instances in three stages: repository collection, issue filtering, and data preparation [2507.10646]. Repository selection begins from all public GitHub repositories, denoted $RGH$, and uses configurable filters over stars, creation date, license, and language:

$$
R = \{ r \in RGH \mid s(r) > S_{\min},\ t(r) > t_0,\ \ell(r) \in L \}
$$

Here, $s(r)$ is star count, $t(r)$ is creation date, and $\ell(r)$ is an SPDX permissive license such as MIT or Apache-2.0 [2507.10646]. CAB then defines a community score

$$
CS(r) = Q(r) + H(r)
$$

where $Q(r)$ and $H(r)$ are the counts of closed issues labeled “question” and “help wanted,” respectively; the framework selects the top-$K$ repositories by this score, breaking ties by stars [2507.10646].

Issue filtering starts from closed issues in the selected repositories. Each issue $i$ is represented as

$$
i = (title(i),\ body(i),\ messages(i)), \quad messages(i)=\{m_{i,1}, m_{i,2}, \ldots\}
$$

and is processed by regex-based rules and LLM-based filters [2507.10646]. The regex rules require at least two distinct participants, in order to ensure genuine Q&A dynamics, and remove issues containing media such as URLs, images, or videos to focus on text-only assistance [2507.10646]. CAB then applies a 7-question issue-level LLM filter with binary Yes/No decisions over resolution, specificity, clarity, safety, and reproducibility; only issues that are clearly resolved by someone other than the author, technically specific, reproducible, and free of sensitive content are retained [2507.10646]. A second, message-level LLM filter removes comments with “absolutely no support value,” including examples such as “+1,” “Thanks,” and “Bump,” while preserving other comments to maintain realistic conversational flow [2507.10646].

Retained issues are restructured into complete user–maintainer exchanges:

$$
turn_{i,k} = (m_{i,k}^{author}, m_{i,k}^{maintainer})
$$

so that the issue becomes

$$
i = (title(i), body(i), \{turn_{i,1}, turn_{i,2}, \ldots\})
$$

[2507.10646]. CAB then prepares each issue for evaluation by reconstructing the repository state near the issue timestamp, generating environment specifications when needed, extracting satisfaction conditions, and producing references for realistic follow-up behavior. The resulting dataset entry is

$$
d_i = (r_i, i, sha_i, e_i, s_i, u_i)
$$

where $sha_i$ is the commit closest to issue creation time, $e_i$ is the successful environment configuration when applicable, $s_i$ is the set of extracted satisfaction conditions, and $u_i$ is a set of user response references retrieved via BM25 over historical maintainer–user pairs [2507.10646].

The satisfaction-condition component is central to CAB’s notion of success. Sonnet 3.7 is prompted to distill explicit, verifiable, solution-agnostic criteria that the user expects fulfilled, producing $s_i = \{s_{i,1}, \ldots, s_{i,K}\}$ [2507.10646]. These conditions are verified for JSON format, completeness as condition-plus-explanation, and conversational grounding; conditions that fail verification are discarded, and issues without surviving conditions are excluded [2507.10646]. This makes the benchmark’s target variable a structured set of user-grounded requirements rather than a single reference answer.

## 3. Environment reconstruction and multi-agent evaluation

A distinguishing feature of CAB is automatic environment preparation. For each issue, CAB identifies the commit closest to issue creation time, clones the repository at that snapshot, extracts artifacts such as the README, Dockerfiles, workflows, and file tree, and synthesizes build scripts or Dockerfiles [2507.10646]. Candidate generation and iterative repair are automated via Sonnet 3.7 prompts. The prompt context includes the repository URL, issue title, commit SHA, truncated body, README, repository structure, up to two workflows, and a reference Dockerfile if one exists; the model is instructed as an “expert Docker engineer” and returns raw Dockerfile content without Markdown fences so it can be passed directly to `docker build` [2507.10646].

If all initial candidates fail, CAB captures the build logs and re-prompts Sonnet 3.7 to fix the observed errors, permitting up to three repair attempts and stopping at the first success [2507.10646]. Build candidates must install dependencies, clone and check out the specified commit, and build the project. The initial build phase intentionally avoids running tests or the user’s code; those actions are deferred to the maintainer agent during evaluation if needed [2507.10646]. Across 294 environment-dependent issues, CAB reports 81% overall success, with per-language examples including Java at 90%, JavaScript at 97%, Python at 86%, and C++ at 53% [2507.10646].

Evaluation proceeds through three agents. The user agent presents the initial question and context, monitors progress relative to the extracted satisfaction conditions $s_i$, provides follow-ups, and signals when satisfied; it does not directly execute commands, but it can incorporate observed execution outcomes [2507.10646]. The maintainer agent, which is the model under test, can read repository files, run commands in the container, and edit code or configuration files as needed. Per response, it may take up to five exploration steps, where each step can be a file operation or command execution, in order to balance efficiency and thoroughness [2507.10646]. The judge agent evaluates the outcome once the dialogue ends, either because the user is satisfied or because the 10-turn cap has been reached [2507.10646].

The judge’s criteria integrate technical correctness, satisfaction completeness, and interaction quality. For Docker-tagged issues, execution success is mandatory; technically sound but non-executable answers count as incorrect [2507.10646]. CAB logs the full dialogue history, environment outputs, and file changes, maintaining state across turns for reproducibility [2507.10646]. The user follow-up process is grounded by BM25 retrieval over similar historical issues, which is intended to preserve realistic conversational patterns while remaining fully automated [2507.10646].

## 4. Metrics, resolution criteria, and corpus characteristics

CAB reports correctness and verbosity, while also defining formal metrics for success, turn efficiency, condition coverage, and human alignment of extracted satisfaction conditions [2507.10646]. Overall correctness is measured by success rate:

$$
SuccessRate = \frac{ResolvedIssues}{TotalIssues} \times 100\%
$$

where `ResolvedIssues` counts judge verdicts of “Correct” [2507.10646]. Mean dialogue length is

$$
MeanTurns = \frac{1}{N}\sum_{i=1}^{N} T_i
$$

where $T_i$ is the number of turns for issue $i$ [2507.10646]. CAB also defines condition-level alignment by

$$
Alignment_i = \frac{Met_i}{C_i}
$$

where $Met_i$ is the number of satisfaction conditions marked TRUE by the judge and $C_i$ is the total number of satisfaction conditions for issue $i$; aggregate alignment may be reported either as mean $Alignment_i$ or as

$$
ConditionCoverage = \frac{\sum_i Met_i}{\sum_i C_i}
$$

[2507.10646].

A separate human study evaluates the extracted satisfaction conditions through precision and completeness. CAB defines

$$
Accuracy\ (Precision) = \frac{CorrectConditions}{TotalConditions}
$$

and

$$
Completeness\ (Recall) = \frac{ConditionsExtracted}{ConditionsNeeded}
$$

and reports 86.27% accuracy and 65.71% completeness across 663 conditions, with language-specific breakdowns [2507.10646]. Resolution itself is tri-valued: “Correct” means fully accurate and satisfying all user conditions, with build or test success required for Docker issues; “Partially correct” indicates minor flaws or partial condition satisfaction; and “Incorrect” covers major errors, unmet conditions, or failed Docker validation [2507.10646].

The corpus statistics show both scale and curation. CAB begins from 25,656 closed GitHub issues with “question/help-wanted” tags and retains 3,286 after regex and LLM filtering plus condition verification [2507.10646]. The paper reports the final retained set in two ways: the abstract states 3,286 issues across 231 repositories, whereas Section 4.1 reports 3,286 issues across 214 repositories, with 294 requiring Docker and 238 successfully built [2507.10646]. CAB attributes these differences primarily to environment retention and license or activity constraints, and directs readers to repository metadata for exact counts per release [2507.10646]. This discrepancy is not presented as a contradiction in task count, but as a release-accounting issue tied to retention criteria.

The benchmark spans seven programming languages, with final retained totals of Python 647, Java 537, C++ 493, C# 536, JavaScript 444, TypeScript 441, and C 188, summing to 3,286 [2507.10646]. Over half of issues across all languages involve multiple turns, and Python, C++, and TypeScript have the highest rates of three or more turns [2507.10646]. The framework also defines two temporal cohorts, “All-Time” and “Recent (after Nov 2024),” so that evaluation can distinguish historically accumulated issues from post-cutoff material that is intended to remain challenging as models evolve [2507.10646].

## 5. Empirical findings and comparative position

CAB evaluates ChatGPT 4.1 Mini, DeepSeek R1, Llama 3.3 70B, Haiku 3.5, Sonnet 3.7, and Sonnet 3.7 (Think mode) [2507.10646]. Dialogue generation temperature is fixed at 0 for reproducibility, while Dockerfile synthesis uses temperature 0.7 to encourage candidate diversity [2507.10646]. The evaluation subset is obtained by balanced stratified sampling over language and turn length, with up to five Dockerfile-containing issues per language, yielding 350 datasets for All-Time and 194 for Recent [2507.10646].

The headline result is a substantial gap between Stack Overflow-like Q&A benchmarks and CAB. Models achieve 70–83% success on StackOverflow-derived benchmarks, specifically 70.64% on InfiBench and 83.0% on StackEval, but on CAB Recent repositories the best model, ChatGPT 4.1 Mini, reaches only 16.49% correctness; on All-Time, the same model reaches 29.14% [2507.10646]. The Recent correctness range across evaluated models is 7.22% for Haiku 3.5 to 16.49% for ChatGPT 4.1 Mini, while the All-Time range is 13.58% for Llama 3.3 70B to 29.14% for ChatGPT 4.1 Mini [2507.10646]. DeepSeek R1 is noted as having the lowest error rate on All-Time [2507.10646].

CAB also reports turn-level effects. Correct answers typically require two to three turns, whereas incorrect cases tend to prolong CAB dialogues by one to two turns relative to the original GitHub threads, which is described as consistent with users seeking clarifications when assistance is vague or incomplete [2507.10646]. Language-specific patterns indicate that statically typed languages—C#, C++, and Java—are harder, especially in Recent cohorts, where correctness often falls below approximately 15% [2507.10646]. Dynamically typed languages—JavaScript and Python—show stronger All-Time performance, with examples of up to approximately 44% correctness on JavaScript for some models, but they also drop markedly on Recent cohorts [2507.10646]. No model dominates across both typed and dynamic languages in Recent repositories [2507.10646].

In relation to prior work, CAB is contrasted with InfiBench and StackEval by its multi-turn interactions, full repository and environment access, and automated scalable generation requiring minimal manual curation [2507.10646]. It is also distinguished from code-generation benchmarks such as HumanEval, MBPP, and CodeContests, and from maintenance evaluations such as SWE-Bench and SWE-PolyBench, because CAB focuses on conversational assistance and user satisfaction over multiple turns rather than pass@1 or single-shot test success [2507.10646]. Relative to multi-turn code-conversation benchmarks such as ConvCodeWorld, MINT, and TICODER, CAB is characterized by grounding dialogues in real repositories and GitHub issues with environment validation rather than in controlled synthetic tasks centered on code synthesis [2507.10646]. A plausible implication is that CAB occupies a distinct evaluation niche at the intersection of repository-grounded software engineering, conversational agents, and reproducible execution.

## 6. Reproducibility, limitations, and practical use

CAB is publicly released with dataset, scripts, prompts, and evaluation framework, and it provides judge and maintainer prompts, Dockerfile generation prompts, and full interaction logs for exact replication [2507.10646]. Repository selection is restricted to permissive licenses such as MIT and Apache-2.0, and issue-level filtering removes PII in order to reduce legal and privacy risks [2507.10646]. The reported hardware includes AWS `g5.16xlarge` with 1 A10G GPU, 64 vCPUs, and 256 GB RAM; the GPU was unused for inference because models were API-based, while CPU and RAM were significant for Docker builds [2507.10646]. Recommended local resources for large-scale builds are approximately 1 TB of storage and 32 GB of RAM [2507.10646].

The benchmark’s limitations are explicit. Satisfaction-condition extraction emphasizes precision over recall: the human-alignment study shows approximately 86.3% accuracy but only approximately 65.7% completeness, so models may be penalized for criteria missing in $s_i$ [2507.10646]. Evaluation is performed on a stratified sample of 544 issues because of compute cost, which means rare edge cases may be underrepresented [2507.10646]. Simulated user behavior is BM25-based and may underrepresent exploratory or ambiguous behaviors [2507.10646]. CAB includes only issues with successful environment synthesis, skewing the benchmark toward modern repositories with standard tooling, while legacy or atypical builds are underrepresented [2507.10646]. The scope is further limited to seven languages and permissively licensed repositories, leaving enterprise or proprietary codebases outside the benchmark [2507.10646]. Temperature settings are fixed for reproducibility, but confidence intervals and significance tests are not reported [2507.10646].

CAB also provides practical guidance for deployment as an evaluation framework. The recommended workflow is to configure repository selection through $S_{\min}$, $t_0$, languages, and licenses; run the repository and issue collection scripts; execute the dataset-generation pipeline to synthesize Docker environments, extract satisfaction conditions, and create user-response references; and then use the evaluation orchestrator to run multi-agent simulations against a chosen LLM via API and collect judge outputs [2507.10646]. For interpretation, the framework recommends using `SuccessRate` for overall solvability, `Alignment_i` for condition-level coverage, and `MeanTurns` for turn-efficiency and friction; environment-dependent failures are treated as hard negatives [2507.10646]. Suggested baselines include ChatGPT 4.1 Mini, Sonnet 3.7 (Think), DeepSeek R1, and an open-weight model such as Llama 3.3 70B [2507.10646]. For organizational adaptation, CAB recommends mirroring the pipeline on sanitized internal issues, defining satisfaction conditions under internal policies, integrating CI for environment validation, and using BM25 retrieval over internal issue archives to seed realistic user follow-ups [2507.10646].

The port-remapping example illustrates the framework’s intended granularity. CAB reconstructs the repository at the issue timestamp, extracts satisfaction conditions such as requiring the main application to run on a user-specified port without conflicts and requiring that the Cloudflare proxy port not be exposed unless explicitly necessary, and then evaluates a multi-turn exchange in which the maintainer agent edits `docker-compose.yml`, runs `docker-compose up`, and verifies the absence of port conflicts [2507.10646]. The judge then marks all satisfaction conditions met and returns the verdict “Correct” [2507.10646]. This example encapsulates CAB’s central premise: the relevant unit of evaluation is not a detached answer string, but a repository-grounded conversational trajectory that can include explanation, configuration changes, execution, and satisfaction of explicit user conditions.

Source: https://www.emergentmind.com/topics/codeassistbench-cab