Can Vibe Coding Beat Graduate CS Students? An LLM vs. Human Coding Tournament on Market-driven Strategic Planning (2511.20613v1)

Abstract: The rapid proliferation of LLMs has revolutionized AI-assisted code generation. This rapid development of LLMs has outpaced our ability to properly benchmark them. Prevailing benchmarks emphasize unit-test pass rates and syntactic correctness. Such metrics understate the difficulty of many real-world problems that require planning, optimization, and strategic interaction. We introduce a multi-agent reasoning-driven benchmark based on a real-world logistics optimization problem (Auction, Pickup, and Delivery Problem) that couples competitive auctions with capacity-constrained routing. The benchmark requires building agents that can (i) bid strategically under uncertainty and (ii) optimize planners that deliver tasks while maximizing profit. We evaluate 40 LLM-coded agents (by a wide range of state-of-the-art LLMs under multiple prompting methodologies, including vibe coding) against 17 human-coded agents developed before the advent of LLMs. Our results over 12 double all-play-all tournaments and $\sim 40$k matches demonstrate (i) a clear superiority of human(graduate students)-coded agents: the top 5 spots are consistently won by human-coded agents, (ii) the majority of LLM-coded agents (33 out of 40) are beaten by very simple baselines, and (iii) given the best human solution as an input and prompted to improve upon, the best performing LLM makes the solution significantly worse instead of improving it. Our results highlight a gap in LLMs' ability to produce code that works competitively in the real-world, and motivate new evaluations that emphasize reasoning-driven code synthesis in real-world scenarios.

• The paper demonstrates that human-coded agents achieve win rates of around 97%, outperforming LLM-generated strategies in complex, market-driven planning tasks.
• The paper evaluates a novel Auction, Pickup, and Delivery Problem benchmark to test multi-agent optimization and strategic reasoning in realistic road networks.
• The paper reveals that while LLMs produce syntactically correct code, they struggle with integrated strategic planning, resulting in suboptimal competitive performance.

LLM vs. Human Coding in Real-World Strategic Planning: An Evaluation with the Auction, Pickup, and Delivery Problem

Introduction

This paper presents a large-scale evaluation of LLM-based code generation in a domain that transcends conventional unit-test-based benchmarks, specifically targeting reasoning-intensive, multi-agent optimization tasks. The central question is whether current LLMs, as coding assistants or via “vibe coding,” can compete at the graduate student level—designing agents for a complex market-driven logistics optimization challenge. The findings indicate substantial limitations in LLM-generated code when facing real-world competitive, multi-agent strategic environments, despite their impressive syntactic reliability.

Figure 1: Conventional code benchmarks focus on functional correctness via unit tests, while the proposed benchmark emphasizes open-ended, strategy-driven tasks such as planning, optimization, and competitive algorithm development.

Benchmark Overview: The Auction, Pickup, and Delivery Problem

The paper proposes a novel benchmarking suite, the Auction, Pickup, and Delivery Problem (APDP). APDP integrates core aspects from combinatorial optimization (pickup and delivery planning, PDP), online sequential decision-making (bidding in auctions), and competitive multi-agent strategic reasoning. Unlike traditional static vehicle routing formulations, APDP introduces the following complexities:

• Staged Decision Problem: Agents first participate in a series of reverse first-price auctions for tasks, often under information constraints and stochastic task arrivals. Each agent's bidding policy must jointly consider its own cost structure, task bundling synergies, and estimation of opponents' behavior.
• Profit Maximization under Constraints: Following the auction, agents optimize parcel delivery plans over real road networks, respecting vehicle capacity, route feasibility, and operational cost.
• Opponent and Future Planning: Rational bidding in APDP must explicitly anticipate future auctions, synergies among tasks, and the dynamics induced by competitive adversaries.
• Real-World Topologies: Evaluation spans multiple realistic road networks (Switzerland, France, UK, Netherlands) and agent/company initializations.

  Figure 2: The APDP setting requires agents to strategically bid for tasks under capacity, spatial, and market constraints, then optimize vehicle routing to maximize profit.

  

  Figure 3: Example network topologies used for APDP tournaments; vehicles (colored triangles) are embedded in authentic road networks, increasing environmental complexity.

Experimental Design

The evaluation covered 57 agents: 40 LLM-coded (4 SOTA LLMs × 5 prompting methodologies × 2 completions each) and 17 human-coded baselines (12 graduate student submissions, 5 carefully designed “simple” baselines). Agents were evaluated over 12 double round-robin tournaments (each agent versus every other, company assignments swapped for fairness), amassing approximately 40,000 matches.

Prompting Strategies:

The experiment systematically explores prompting effects, including “vibe coding” (simply pasting the full assignment to the LLM), iterative self-improvement, peer-LMM code review/citation, and LLM-optimized prompts.

Bug Handling:

A rigorous debugging procedure ensured all agents executed without syntax errors, though LLMs commonly violated semantic constraints (timeouts, capacity, etc.), requiring multiple repair cycles. Importantly, student code was not manually debugged—any runtime crash resulted in an automatic win for the opponent.

Results

Preliminary Test Cases

Syntactic competence is robust: LLMs reliably output code that compiles and runs for simple variants (e.g., single-agent PDP, basic search). However, failures in basic algorithmic understanding and planning are common:

• LLMs often fail to implement admissible heuristics in A*, despite explicit instructions, resulting in suboptimal or incorrect planning.
• Generated plans frequently underutilize available vehicles or ignore cost minimization, even when these principles are stated in prompts.
• Substantial variance is seen between LLM instances and across prompt strategies.

Full Tournament Evaluation

The core result is that human-coded agents, particularly graduate student submissions, consistently and by a significant margin outperform LLM-coded agents:

• The top 5 leaderboard slots are all occupied by human-coded agents, with win rates exceeding 92%.
• 33 out of 40 LLM-coded agents are outperformed even by trivial baseline strategies (e.g., cost-fixed bid, naive sequential delivery).
• The most competitive LLM solutions reach human baselines only in a minority of configurations and never surpass top-performing student agents.
• When explicitly asked to “improve” the winning student agent, the best LLM (GPT-5 Pro) consistently degrades the solution, causing a drop from first to 10th place.

Key Numerical Findings:

• Top student agent winrate: ~97%
• Best LLM agent winrate: ~85%
• Majority of LLM agents' winrates: < 60%; often close to naïve random policies

Analysis of Failure Modes

Despite producing syntactically correct and debuggable code, LLMs routinely fail at:

• Reasoning about Multi-agent Dynamics: Generated bidding strategies are myopic, poorly anticipating opponents and future state changes.
• Combining Optimization Subproblems: LLMs struggle to integrate auction/strategic reasoning and vehicle routing.
• Leveraging Prior Context: Even with perfect reference solutions (via in-context learning), LLMs do not successfully adapt or enhance competitive strategies.

Limitations and Implications

The results suggest that LLMs’ code synthesis capabilities—when unaided by extensive external scaffolding, domain-specific fine-tuning, or human intervention—are fragile in settings requiring integrated, long-horizon planning and multi-agent strategic adaptation. While LLM output is functionally executable, passing unit tests is insufficient for complex real-world tasks.

Practical implications are clear:

• For Industry: Reliance on LLMs for automation of complex, market-facing planning logic remains unwarranted; human intervention or end-to-end design remains necessary.
• For Benchmarking AI: The present unit-test/pass@k–centric paradigm dramatically overestimates true “real-world” code generation capacity. Benchmarks must shift toward competitive, open-ended, and strategic problem domains.
• For AI Development: These observations reinforce ongoing questions regarding LLMs' limits on compositional reasoning, implicit planning, and agent modeling [kambhampati2024can].

Prospects for Future Work

Several future directions are motivated:

• Enhanced Evaluation Protocols: Develop and adopt benchmarks requiring agents to compose, plan, and compete in open-ended, strategic, and adversarial settings.
• Advanced LLM Architectures: Research is needed into architectures or training protocols emphasizing structured planning, recursive self-improvement, and game-theoretic reasoning.
• Hybrid Human-AI Autonomy: Investigate workflows that robustly blend LLM syntactic fluency with human strategic insight; automated code generation may need domain-specific scaffolding and program synthesis constraints.
• Meta-Learning for Prompting and Repair: Systematic paper of prompt/feedback designs or LLM self-improvement techniques that can reliably induce strategic, competitive behaviors in code generation.

Conclusion

This comprehensive evaluation demonstrates that current LLMs fall short of graduate student-level code synthesis capability in strategic, planning-centric, multi-agent environments such as APDP. The primary challenge is not syntactic fluency but deep integration of optimization, strategic reasoning, and competitive adaptation—skills essential for real-world software engineering beyond boilerplate synthesis. The research community should prioritize benchmarks and architectures aligned with these challenges to more faithfully drive progress in automated code generation and agent design.

---

Reference:

"Can Vibe Coding Beat Graduate CS Students? An LLM vs. Human Coding Tournament on Market-driven Strategic Planning" (2511.20613)