CodeFlowBench: Multi-Turn Code Generation Benchmark

• CodeFlowBench is a benchmark suite that evaluates language models and static analysis tools by decomposing real-world coding tasks into iterative, multi-turn steps.
• It features a structured six-stage pipeline that extracts, decomposes, and analyzes code tasks from sources like Codeforces, using metrics such as overall-turns and dependency structure complexity.
• It employs dual evaluation protocols—multi-turn and single-turn—that reveal the impact of dependency complexity on model performance and highlight specific error types in code generation.

CodeFlowBench is a benchmark suite developed to rigorously evaluate the ability of LLMs and static analysis tools to perform complex code-generation and data-flow reasoning in realistic, multi-turn software development scenarios. Distinct from conventional single-turn code-generation or static analysis benchmarks, CodeFlowBench formalizes and measures the multi-step, dependency-driven process of code composition—termed "codeflow"—which is essential for scalable and maintainable software engineering. It comprises an extensive set of code-generation problems derived from real-world sources, a structurally labeled decomposition pipeline, a robust evaluation framework for both codeflow and data-flow analyses, and supports comprehensive error and metric reporting across algorithmic, architectural, and tool boundaries (Wang et al., 30 Apr 2025, Weideman et al., 30 May 2025).

1. CodeFlow: Paradigm and Formalization

The codeflow paradigm models how real-world software is built: incrementally, by composing new functionality via explicit reuse of previously written functions or modules. Formally, a problem is decomposed into $T$ subproblems (turns), each associated with a function $F_t$ and natural-language statement $S_t$. Each step conditions on prior solutions, inherited dependencies, and a background description $B$:

$$C_t \;\sim\; p\Bigl(\,\cdot\mid F_t,\,S_t,\,\{F_{k_1},\ldots,F_{k_d}\},\,\{C_1,\ldots,C_{t-1}\},\,B\Bigr),\quad t=1,\ldots,T$$

A comparison is drawn to the single-turn oracle where all subproblems are solved jointly. CodeFlowBench is the first benchmark designed to comprehensively target this iterative, function-reuse paradigm in LLM evaluation (Wang et al., 30 Apr 2025).

2. Dataset Construction, Structure, and Scale

CodeFlowBench aggregates 5,258 tasks scraped from Codeforces, each split into modular components via automated dependency analysis:

Six-Stage Pipeline:

1. Extract problem statements, constraints, and metadata via Codeforces API and HTML parsing.
2. Scrape and normalize official editorials and canonical solutions.
3. Synthesize and validate single-file, compilable solutions using LLMs and the Codeforces judge.
4. Parse solution ASTs to extract functions/subproblems, constructing the dependency graph and assigning turn structure.
5. Instrument verified solutions to gather per-function unit tests, deduplicated and capped per function.
6. Assign two structural complexity metrics: Overall-Turns (number of subproblems) and Overall-Depth (maximal dependency tree depth).

Approximately 60% of problems decompose into 2–4 functions, with an average overall-depth of 1.94. The depth and branching structure of the dependency graph (formally, Dependency Structure Complexity, $$\mathrm{DSC} = \frac{\mathrm{OverallTurns}}{\mathrm{OverallDepth}}$$) are critical for evaluating model robustness. High DSC reflects combinatorially branched or iteratively reused utilities (Wang et al., 30 Apr 2025).

3. Evaluation Protocol and Metrics

Dual Assessment Protocol:

• Multi-turn: The model is sequentially prompted with each function/subproblem, along with all previously generated code and explicit dependencies. Each generated function is compiled/tested before proceeding.
• Single-turn: The model receives all function signatures and specifications, producing implementations in a single response, which is then evaluated end-to-end.

Primary metric is execution-based Pass@k (typically Pass@1), computed as:

$$\mathrm{Pass@}k = 1 - \frac{\binom{n - c}{k}}{\binom{n}{k}}$$

for $n$ sampled outputs and $c$ correct ones. To fine-grain multi-turn progress, Pass Depth (PD) is used: if the first failing subproblem is at depth $d$ in a problem of root depth $F_t$0, then $F_t$1, or $F_t$2 if entirely correct; average PD (APD) is reported over the dataset. DSC is found to inversely correlate with multi-turn Pass@1, with Pearson $F_t$3 (Wang et al., 30 Apr 2025).

4. Error Typology and Analysis

Extensive error annotation on CodeFlowBench highlights codeflow-unique failure types:

• Incomplete Reasoning (IR): Model-generated logic lacks coverage for boundary or adversarial cases.
• Insufficient Globalization (IG): Failures arise from missing imports, global constants, or shared state not inferable from local context.
• Instruction Misinterpretation (IM): Incorrect invocation or sequencing of dependencies, violating the intended codeflow.

IR accounts for 40–50% of multi-turn failures, while IG and IM comprise an additional 45%, underlining that multi-turn program synthesis exposes modeling challenges not apparent in conventional benchmarks (Wang et al., 30 Apr 2025).

5. Quantitative Model Evaluation

Results on sixteen LLMs reveal severe challenges for multi-turn codeflow:

Model	Pass@1 (Multi-turn)	Pass@1 (Single-turn)	APD (Overall)
o1-mini	20.8%	37.8%	0.541
GPT-4o-mini	13.8%	22.0%	0.423
DeepSeek-R1	20.5%	46.1%	0.569

For problems with $F_t$4, multi-turn Pass@1 converges to zero across all tested models. Smaller, code-specialized models perform relatively well on shallow or unbranched subproblems but fail rapidly as dependency complexity increases. These findings signal fundamental barriers in current LLM architectures for realistic incremental development scenarios (Wang et al., 30 Apr 2025).

6. Extensibility and Recommendations

The CodeFlowBench paradigm—inclusive of its codeflow formalization, hierarchical dataset, and iterative evaluation—has been recognized as essential for future code-generation research. Recommendations based on systematic insights include:

1. Modular pipeline design for large-scale, automatically decomposed benchmarks.
2. Comprehensive dependency tracking to enable fine-grained multi-turn evaluation.
3. Diverse language and code structure coverage to capture real-world programming practices.
4. Standardized execution-based success metrics to enable apples-to-apples model comparison.
5. Transition to full-repository and cross-file codeflow modeling to extend beyond single-file or function scope.

A plausible implication is that future model and framework development should explicitly address globalization, dependency resolution, and sequential reasoning to achieve robust multi-turn code synthesis.

7. Impact and Future Directions

CodeFlowBench defines a critical axis for evaluation, uncovering model limitations that are obscured in end-to-end or single-file settings. The structure of the benchmark allows principled study of how architectural, finetuning, or prompting changes influence multi-step reasoning and codebase assembly capabilities. Future expansion is proposed to apply codeflow evaluation to full software repositories and to catalyze the development of LLM architectures and agent frameworks that can exploit and maintain global state, orchestrate dependency-aware planning, and self-verify iterative outputs (Wang et al., 30 Apr 2025).

CodeFlowBench thus serves as a reference point for assessing real-world code-generation and analysis workflows, providing a reproducible and scalable bedrock for both static and generative model evaluation in complex, multi-turn development scenarios.