HAI-Eval: Human-AI Coding Collaboration

• HAI-Eval is a benchmark designed to measure human-AI coding partnerships by using tasks that are unsolvable by standalone humans or AI, emphasizing the need for effective collaboration.
• It employs 45 collaboration-necessary templates across SDE, MLE, and DS tracks at varying difficulty levels, ensuring tasks compel joint reasoning and contextual interpretation.
• Empirical results demonstrate that human-AI collaboration markedly improves pass rates and efficiency, underscoring the value of co-reasoning in complex coding scenarios.

HAI-Eval is a unified benchmark designed to measure the synergy of human-AI partnership in coding, rather than the performance of unaided humans or standalone LLMs on fully specified programming tasks. Its core innovation is the use of “Collaboration-Necessary” problem templates: tasks that are intractable for both standalone LLMs and unaided humans, but solvable through effective collaboration. The benchmark combines 45 templates, a standardized IDE for human participants, and a reproducible toolkit with 450 task instances for LLMs. In a within-subject study with 45 participants and benchmarks against 5 state-of-the-art LLMs under 4 different levels of human intervention, standalone LLMs and unaided participants achieved poor pass rates of 0.67% and 18.89%, whereas human-AI collaboration reached 31.11% (Luo et al., 30 Nov 2025).

1. Conceptual basis and definition

HAI-Eval is motivated by the claim that both human-oriented and LLM-oriented coding benchmarks are misaligned with AI-assisted development. Human benchmarks such as LeetCode and Codeforces assume a perfectly defined problem with clear input/output, emphasize algorithmic problem solving and manual implementation, and evaluate the developer as an isolated “code producer.” LLM benchmarks such as HumanEval, MBPP, SWE-Bench, DS-1000, and LiveCodeBench move toward more realistic repositories or environments, but still frame tasks as fully specified coding problems. HAI-Eval is designed for the part of software work that depends on interpreting ambiguous requirements, requirement engineering, strategic planning and decomposition, and orchestrating and validating AI assistance (Luo et al., 30 Nov 2025).

The benchmark formalizes “necessary collaboration” through two constraints. First, tasks are AI-Incomplete: current standalone LLMs almost never solve them. Second, tasks are constructed so that humans obtain a significant performance boost with AI assistance because doing everything manually is suboptimal under realistic constraints. The paper states this with the conditions

$$\Pr(\text{Solve}(T, \mathcal{A})) \le \theta_{\text{low}} \quad\text{and}\quad \mathbb{E}[\text{Score}(s_{\mathcal{H}+\mathcal{A}})] - \mathbb{E}[\text{Score}(s_\mathcal{H})] \ge \delta,$$

where $\theta_{\text{low}}$ is a very low success probability for any standalone LLM and $\delta > 0$ is a collaboration incentive margin (Luo et al., 30 Nov 2025).

This construction is grounded in distributed cognition, human-AI collaboration literature, and relational complexity theory. The underlying design choice is not to maximize algorithmic novelty, but to wrap a relatively standard algorithmic core in contextual complexity that obstructs standalone LLM performance while making delegation attractive for humans. This suggests that HAI-Eval is less a conventional coding benchmark than a benchmark of joint problem solving.

2. Collaboration-Necessary task design

HAI-Eval’s task universe is organized as a Problem Template Bank of 45 templates arranged in a 3×3 matrix. The three tracks are SDE, MLE, and DS; the three difficulty levels are Easy, Medium, and Hard. Each template encodes a core algorithmic or data-processing task, domain context, and a collaboration-necessary structure. The intended failure mode for LLMs is not merely weak coding, but difficulty with relational or contextual reasoning, including underspecified or ambiguous requirements, multimodal or symbolic specifications such as UML or ER diagrams, legacy code with minimal documentation, and domain-specific business logic. The intended difficulty for humans is scale, repetition, configuration overhead, or boilerplate, so that AI delegation improves performance under time limits (Luo et al., 30 Nov 2025).

Task instances are generated by an Agentic Task System powered by GPT-4.1. The system uses four named tools: TechnicalParameterTool, ImplementationConstraintTool, ContextualVariableTool, and InterfaceSpecificationTool. The generator first fixes logic-critical numerical parameters, then environment constraints such as framework and Python versions, then contextual narratives and interface elements. It performs internal consistency checks and retries if parameters conflict. For human experiments, this generation is dynamic. For LLM benchmarking, the benchmark freezes 10 instances per template into a static set of 450 tasks to ensure reproducibility (Luo et al., 30 Nov 2025).

The benchmark also imposes a formal validation procedure for AI-incompleteness. For a template $T$, using 20 task instances for a fidelity test and 10 for a generalization test, the paper requires

$$\frac{1}{|S(T)|} \sum_{t\in S(T)} \text{Pass}(\mathcal{A}_\text{base}, t) \le 0.05$$

and

$$\forall \mathcal{A}\in \mathbb{A}_\text{SOTA}:\quad \frac{1}{|S'(T)|} \sum_{t\in S'(T)} \text{Pass}(\mathcal{A}, t) \le 0.10.$$

Here $\mathcal{A}_\text{base}$ is Claude Sonnet-4, and $\mathbb{A}_\text{SOTA}$ is {Claude Sonnet-4, GPT-4.1, Gemini-2.5-Pro}. All 45 templates satisfy these constraints, and 42 have 0% pass across all models (Luo et al., 30 Nov 2025).

3. Benchmark infrastructure and experimental protocol

HAI-Eval was built to preserve what the paper calls ecological validity. Human participants work in GitHub Codespaces with full VS Code in the browser, with GitHub Copilot in Agent mode. Each task ships with a devcontainer specifying the operating system, Python version, dependencies, and extensions. Participants submit via ./scripts/submit.sh, which packages code and sends it to a FastAPI backend for evaluation. The benchmark defines ecological validity along three axes: interaction method, assistance strength, and task requirement (Luo et al., 30 Nov 2025).

For autonomous evaluation of LLMs, the benchmark provides HAI-Eval Controller (HAI-EC), a VS Code extension that spins up Docker containers mirroring the task devcontainers, interacts programmatically with Copilot Agent, feeds it the README and project files, runs tests, returns logs, and stops when all visible tests pass or a time limit is reached. This keeps the automated setting close to the human one, but controlled and reproducible (Luo et al., 30 Nov 2025).

The study defines four conditions:

Condition	Description	Overall Pass@1
$C_H$	Human-only	18.89%
$C_0$	Fully autonomous AI	0.67%
$\theta_{\text{low}}$0	Minimally-intervened AI	2.89%
$\theta_{\text{low}}$1	Human-AI collaboration	31.11%

The participant pool comprises 45 experts from CS and related fields in East Asia: 15 SDE, 15 MLE, and 15 DS. All were at least 18, proficient in Python and VS Code, and regular users of coding assistants at least three times per week; about 84% were daily users. The study used a within-subject design: each participant completed four tasks, two under $\theta_{\text{low}}$2 and two under $\theta_{\text{low}}$3, with task and condition order counterbalanced via Latin Square and split into two sessions with a 24-hour gap (Luo et al., 30 Nov 2025).

4. Scoring, metrics, and evaluated models

HAI-Eval records functional test pass/fail per test case, solution execution time, peak memory, total completion time, and token usage. To reduce platform dependence in efficiency scoring, the benchmark runs a canonical efficient solution and an inefficient but correct solution before evaluation and uses their runtimes and memory as dynamic thresholds. Efficiency is then converted into binary pass/fail per task. The benchmark reports Overall Pass, which requires passing all functional and efficiency tests; Partial Pass, the fraction of test cases passed; Completion Time; and Token Usage. For LLM-only settings it also reports pass@1 and pass@10 (Luo et al., 30 Nov 2025).

The benchmark evaluates 5 state-of-the-art models through Copilot Agent mode with default settings: Claude-Sonnet-4, Claude-Sonnet-3.7, GPT-4.1, GPT-4o, and Gemini-2.5-Pro. The paper treats this as realistic usage rather than bespoke prompt or sampling optimization, because Copilot controls temperature, top-$\theta_{\text{low}}$4, and agent orchestration (Luo et al., 30 Nov 2025).

This measurement scheme places HAI-Eval in a different category from classic code-generation benchmarks. It does not score only unit-test correctness under a static specification. It evaluates the joint ability to interpret verbose READMEs, adapt to starter code and environment constraints, meet efficiency requirements, and use an IDE-centered workflow. A plausible implication is that HAI-Eval measures a compound of requirement interpretation, planning, implementation, debugging, and delegation.

5. Empirical findings and the co-reasoning partnership

The central empirical finding is that current SOTA LLMs perform very poorly on HAI-Eval, even with multiple attempts and limited procedural assistance. On the 450 static tasks, Claude-Sonnet-4 reaches 0.67% Overall Pass@1 in $\theta_{\text{low}}$5 and 2.89% in $\theta_{\text{low}}$6; Claude-Sonnet-3.7 reaches 0.00% and 1.56%; GPT-4.1 reaches 0.00% and 1.78%; GPT-4o reaches 0.00% in both; and Gemini-2.5-Pro reaches 0.22% and 2.22%. The paper characterizes this as hitting a higher-order reasoning wall: even “Easy” tasks remain close to zero pass for standalone agents because the dominant barrier is extracting the real specification and planning under contextual complexity, not merely writing code (Luo et al., 30 Nov 2025).

Against this background, the human and collaborative conditions are decisive. Averaged over all difficulties and tracks, humans alone in $\theta_{\text{low}}$7 achieve 18.89% Overall Pass@1, while human-AI collaboration in $\theta_{\text{low}}$8 achieves 31.11%. By difficulty, the gains are strongest on harder tasks: on Hard tasks, $\theta_{\text{low}}$9 reaches 6.67% and $\delta > 0$0 reaches 23.33%; on Medium, 13.33% versus 26.67%; on Easy, 36.67% versus 43.33%. Completion times also improve under collaboration: for example, Easy tasks fall from 2165 seconds in $\delta > 0$1 to 1379 seconds in $\delta > 0$2 (Luo et al., 30 Nov 2025).

The paper interprets this as an emerging co-reasoning partnership. Participants reported high trust in AI code suggestions (4.09/5) and AI explanations (4.22/5). All 45 participants used AI to generate boilerplate, repetitive, or utility code; 41 used it to debug errors; 36 used it to brainstorm or explore solution strategies; and 23 used it to suggest a fundamentally different approach or algorithm. Among the top 15 performers, 12 used AI for fundamentally different approaches or algorithms. This pattern supports the paper’s claim that the human-AI relationship is not simply “human strategizes, AI implements.” Strategic breakthroughs can originate from either side (Luo et al., 30 Nov 2025).

The qualitative Clickomania-style case study exemplifies this interpretation. The agent initially failed due to TLE and misinterpreted rules. The human then recognized that global search was unnecessary and proposed a greedy strategy. The AI subsequently refined that strategy by maintaining a candidate queue of potentially matching cells and updating only affected rows and columns after each elimination, reducing runtime substantially. The final passing solution emerged from shared reasoning, not one-way delegation (Luo et al., 30 Nov 2025).

6. Position within human-AI evaluation and stated limitations

HAI-Eval belongs to a broader shift from evaluating isolated models to evaluating human-AI systems. A closely related human-centred TEVV perspective argues that the system under test is the human plus the AI plus the context, not the AI in isolation, and that evaluation must address realized operational performance with humans in the loop (Helmer et al., 2024). SPHERE provides a complementary documentation framework for such systems through five questions—what is being evaluated, how evaluation is conducted, who participates, when evaluation is conducted, and how evaluation is validated—highlighting that human-AI evaluation design itself requires structured reporting (Ma et al., 24 Mar 2025).

In adjacent work, Oak National Academy describes a “clean HAI-Eval loop” in which humans define the rubric, label a sample, critique model rationales, and then re-evaluate a revised automatic evaluator (Clark et al., 23 Jan 2025). HypoEval, although it does not mention “HAI-Eval” explicitly, is described as conceptually targeting the same goals as a human-aligned evaluation framework like “HAI-Eval”: alignment, robustness, and interpretability, with only 30 human evaluations per aspect or dataset (Li et al., 9 Apr 2025). In healthcare, RubricsTree is presented as an HAI-Eval infrastructure for open-ended, agentic personal health AI, using an expert-curated hierarchical rubric taxonomy of over 100 atomic Boolean rubrics and a context-aware adaptive router (Zhang et al., 16 Jun 2026). These comparisons suggest that the term “HAI-Eval” now spans both a specific collaborative coding benchmark and a wider methodological orientation toward human-aligned, auditable, and context-sensitive evaluation.

The benchmark’s own stated limitations are narrower and concrete. It is currently Python-only; model access is constrained by what Copilot Agent mode supports, excluding many API-only or open-source models; the participant sample consists of East-Asian students and recent graduates who already use AI tools heavily; and the binary conversion of correctness and efficiency increases comparability but can hide gradations in performance (Luo et al., 30 Nov 2025). Future work proposed in the paper includes expansion to other languages such as Java and C++, additional tracks such as DevOps and security, broader model evaluations through direct API integration, more diverse human studies, and richer evaluation metrics for code readability, maintainability, and extensibility (Luo et al., 30 Nov 2025).