---
title: Greenfield-5 Benchmark
url: https://www.emergentmind.com/topics/greenfield-5-benchmark
type: topic
---

# Greenfield-5 Benchmark

Greenfield-5 is a custom benchmark suite for **greenfield, repository-level code generation from vague, high-level intent** in what its authors call the “Vibe Coding” setting. It was introduced to evaluate whether an agent can turn an ambiguous project request into a **runnable, interactive, multi-file codebase** without architectural collapse, and it is presented as a bridge “between algorithmic scripts and system-level architectures.” In contrast to suites centered on repair of existing repositories, Greenfield-5 targets **greenfield synthesis from high-level intents**, with explicit stress on coherent architecture, valid cross-file symbol linkage, and task-level behavior under ambiguity [2604.13100].

## 1. Definition and evaluative scope

Greenfield-5 is organized around a **“Complexity Spectrum of five greenfield repositories.”** Its defining object is not a function, patch, or issue resolution trace, but a newly generated repository that must satisfy both **structural** and **behavioral** constraints. The benchmark therefore evaluates more than whether code executes: it measures whether a system can produce a repository with a coherent file layout, stable interfaces, valid imports, and interactive functionality aligned with a high-level prompt [2604.13100].

The benchmark is explicitly framed against a **Context-Fidelity Trade-off**. In the authors’ formulation, vague user intents and growing implementation context interact poorly with linear generation pipelines, producing “Symbolic Hallucination,” invented APIs, inconsistent interfaces, and eventual architectural collapse. This motivates the benchmark’s emphasis on multi-file synthesis under under-specification rather than on single-file completion or brownfield bug fixing. Formally, the repository-generation objective is written as
$$
\mathcal{R}^* = \operatorname*{arg\,max}_{\mathcal{R}} P(\mathcal{R} \mid \mathcal{I}),
$$
where $\mathcal{R} = \{f_1, f_2, ..., f_N\}$ is the repository and $\mathcal{I}$ is under-specified user intent. The paper also highlights the sequential factorization
$$
P(\mathcal{R} \mid \mathcal{I}) = \prod_{i=1}^{N} P(f_i \mid f_{<i}, \mathcal{I}),
$$
to explain how early defects can propagate across later files in baseline systems [2604.13100].

A common misconception is to treat Greenfield-5 as a generic coding benchmark. The benchmark is narrower and more specific: it targets **repo-level autonomous generation from vague intent**, with strict multi-file structure and interactive task requirements. It is therefore best understood as a benchmark for **architectural orchestration and cross-module consistency** in greenfield synthesis rather than as a general-purpose software engineering scorecard [2604.13100].

## 2. Composition of the five-task suite

The suite contains five tasks arranged in increasing architectural complexity. The tasks are all repository-generation problems, and several impose explicit file-count constraints. The later tasks stress event-driven structure, extensibility, polymorphism, and deeper cross-file dependency chains.

| Task | Characterization | Defining constraints |
|---|---|---|
| **Gomoku** | Logic-Intensive / Logic | Gomoku program with AI so players can play against AI |
| **Plane Battle** | Action-Oriented | Player movement, bullets, enemies, enemy death on hit, enemy bullets, win/lose conditions |
| **City Sim** | Resource Management | “Micro City Sim,” resource balancing, menu placement, extensibility, **above 10 files** |
| **Snake++** | Event-Driven | Modular Snake variant, power-up factory, multiple layouts, leaderboard/save, **13–15 files** |
| **Roguelike** | System Architecture / Scalability stress test | Modular Pygame roguelike, event bus, polymorphic entities, cross-module inventory effects, **at least 15 files** |

The benchmark descriptions are unusually concrete. **City Sim** is specified as a grid-based city builder in Python/Pygame with real-time Money/Energy simulation, Houses that generate money but consume energy, Power Plants that generate energy but cost money, taxes that stop when energy is insufficient, menu-based building placement, at least Roads, Residential Zones, and Industrial Zones, HUD/resource alerts, and easy extensibility for new building types, “implicitly requiring polymorphism.” **Snake++** requires growth logic, Speed Boost, Teleportation Walls, a modular power-up factory, multiple arena layouts, a leaderboard/save system, and an explicit architecture constraint of **13–15 files**. **Roguelike** requires isolated procedural map generation, rendering separated from data, an event-driven UI with Message Log and HUD via Event Bus rather than direct coupling, polymorphic entities including Player, Orc, and Mage, cross-module inventory and item interactions such as a Scroll revealing Fog of War, and **at least 15 files**, elsewhere described as **15–25 interdependent files** [2604.13100].

This progression is central to the benchmark’s design. **Gomoku** and **Plane Battle** primarily expose logic and game-loop organization; **City Sim** adds resource simulation and extensible design; **Snake++** stresses event-driven extensibility; **Roguelike** acts as the benchmark’s scalability stress test, combining procedural generation, polymorphism, UI/event separation, and deep inter-file coordination [2604.13100].

## 3. Construction principles and why the benchmark is hard

Each Greenfield-5 task begins from a **high-level prompt—the “Vibe”**—rather than from a fully formal specification. This is deliberate. The benchmark assumes that real users often do not provide complete blueprints, and it therefore tests whether a system can infer a viable architecture from ambiguous intent. The generated artifact must satisfy **repo-level generation requirements**: multi-file decomposition, coherent internal imports and symbol references, and project structure constraints; monolithic scripts do not satisfy the intended benchmark setting [2604.13100].

Greenfield-5 is difficult because it combines several burdens that are usually separated in other evaluations. The paper identifies these explicitly: **architectural decomposition**, **symbol grounding across files**, **interactive behavior**, **game-specific logic fidelity**, **strict multi-file structure**, and **high-level under-specification**. The benchmark also includes a **reference architecture** against which generated file sets are compared. This makes it possible to score both missing modules and unnecessary file proliferation, rather than treating any runnable output as equivalent [2604.13100].

The benchmark’s static difficulty is reinforced by a dynamic coupling effect. Later tasks force generated files to remain globally consistent over longer horizons, which the paper characterizes as **context exhaustion**. In baseline sequential systems, code context expands linearly while the original intent becomes diluted. A result is that a repository may execute partially yet still exhibit “zombie code,” interface drift, or unreachable logic. Greenfield-5 is designed to expose exactly these failure modes [2604.13100].

A plausible implication is that Greenfield-5 evaluates a distinct regime of software-generation ability: not just local code correctness, but the capacity to preserve **symbolic consistency** and **architectural topology** under ambiguity. That emphasis differentiates it from both unit-test-centric code-generation benchmarks and brownfield repair suites.

## 4. Evaluation protocol and metrics

The evaluation protocol combines **dynamic functional scoring** with **static structural analysis**. The paper reports **10 independent runs per method/task**, with means as the reported numbers. For the academic methods, the reported configuration uses **gpt-4o-2024-11-20**, temperature **0.0**, and a **16k token** context window. The hardware reported is a **MacBook Pro (M2 Pro), 32GB unified memory, 512GB SSD, macOS 15**. Commercial IDE baselines are evaluated in their own listed versions and modes, including Lingma, CodeBuddy, Trae, and Gemini Studio [2604.13100].

The dynamic metric is
$$
S_{overall} = \frac{1}{3} (S_{exec} + S_{inter} + S_{rule}) \times 100\%.
$$
Here **Executability** $S_{exec} \in \{0,1\}$ checks whether dependencies install and the entry point launches without runtime crash for a **60-second keep-alive window**; **Interactivity** $S_{inter} \in [0,1]$ measures whether a standardized UI action sequence elicits valid responses; and **Rule Adherence** $S_{rule} \in [0,1]$ measures task-specific logic correctness. The appendix describes this as a **triple-blind human scoring protocol** with three independent reviewers [2604.13100].

The interaction and rule checks are benchmark-specific. For example, **Gomoku** requires that placing a stone changes state and triggers AI response, and that 5-in-a-row is detected correctly. **Plane Battle** requires directional control and bullet firing, with collisions reducing health or triggering game-over. **City Sim** checks menu selection, grid placement, HUD updates, and enforcement of the energy-tax dependency loop. **Snake++** checks direction changes, visible speed boost behavior, and rule-level handling of shield and teleportation walls. **Roguelike** checks room movement, interaction with enemies/items, and that the procedural map remains traversable [2604.13100].

Static structure is evaluated with two explicit metrics. **Architectural Fidelity** is
$$
S_{arch} = \frac{2 \cdot |F_{gen} \cap F_{ref}|}{|F_{gen}| + |F_{ref}|},
$$
where $F_{gen}$ and $F_{ref}$ are the generated and reference file-path sets. This penalizes both **“Hollow Skeleton”** repositories with missing files and **bloated** repositories with redundant files. **Linkage Consistency** is
$$
S_{link} = \frac{\sum_{i \in \mathcal{I}} \mathbb{I}(i \text{ is valid})}{|\mathcal{I}|},
$$
where $\mathcal{I}$ is the set of internal import statements extracted from ASTs, and an import is valid iff the target file exists and defines the imported symbol [2604.13100].

## 5. Reported empirical results

Greenfield-5’s headline empirical pattern is a sharp increase in difficulty from the early tasks to the later ones. In the main results table, **Contract-Coding** reports **100** on **Gomoku**, **100** on **Plane Battle**, **87** on **City Sim**, **80** on **Snake++**, and **47** on **Roguelike**. The abstract summarizes the hardest-task result as **“47\% functional success while maintaining near-perfect structural integrity.”** The paper’s central interpretation is that the benchmark distinguishes between systems that fail through architectural collapse and systems that preserve architecture but still make local logic errors [2604.13100].

For the easier tasks, several baselines reach perfect or near-perfect scores. On **Gomoku**, OpenHands, Lingma, Gemini Studio, Ours w/o HEG, and Ours (Full) each report **100**, while ChatDev reports **93** and FLOW **96**. On **Plane Battle**, Gemini Studio, CodeBuddy, Ours w/o HEG, and Ours (Full) each report **100**, while ChatDev and OpenHands report **90** and Trae **80**. These results indicate that Greenfield-5 is not uniformly hard; rather, its difficulty is concentrated in tasks with deeper structural coupling [2604.13100].

The mid-complexity tasks show clearer divergence. On **City Sim**, the main table reports **87** for Ours (Full), **87** for Gemini Studio, **85** for Trae, **85** for Ours w/o HEG, and lower numbers for MetaGPT, ChatDev, FLOW, OpenHands, Lingma, and CodeBuddy. On **Snake++**, the main table reports **83** for Gemini Studio, **80** for Ours (Full), **78** for Ours w/o HEG, **65** for Trae, and lower values for the other baselines. The benchmark thus begins to separate systems once file-count constraints, event-driven coordination, and extensibility become binding [2604.13100].

**Roguelike** is the suite’s clearest stress test. In the main table, **Gemini Studio** reports **63**, **Trae** **47**, **Ours w/o HEG** **47**, **Ours (Full)** **47**, **CodeBuddy** **40**, **Lingma** **33**, **OpenHands** **30**, **MetaGPT** **10**, **ChatDev** **10**, and **FLOW** **0**. The appendix provides a structural and functional breakdown for Roguelike in which **Ours (Full)** achieves **100** structural integrity, **92** dependency consistency, **90** executability, **20** interactivity, **30** rules, and **47%** overall. By contrast, **Trae** is also at **47%** overall but with **23.9 files** in the main table versus **14.2 files** for Ours (Full), while **Gemini Studio** leads overall at **63%** but reports **80** structure and **80** dependency rather than perfect structural scores [2604.13100].

The ablation results reinforce the benchmark’s architectural emphasis. Comparing **Ours w/o HEG** with **Ours (Full)**, the score remains **100 → 100** on Gomoku and **100 → 100** on Plane Battle, but time drops from **205s → 136s** and **201s → 117s** respectively. On **City Sim**, the score changes **85 → 87** with time **480s → 257s**; on **Snake++**, **78 → 80** with time **465s → 198s**; and on **Roguelike**, the overall score remains **47 → 47** while file count changes **25.0 → 14.2** and time **510s → 232s**. The paper interprets this as evidence that HEG improves efficiency and repository cleanliness more than raw functional score on the hardest task [2604.13100].

## 6. Failure modes, interpretation, and relation to adjacent work

Greenfield-5 is designed not merely to rank systems, but to expose distinct **pathologies of repo-level generation**. The appendix names these failure modes explicitly. Among academic baselines, **MetaGPT** exhibits **“Hollow Skeleton”** and **“Implementation Sparsity”**; **ChatDev** shows a **“Reflection-Action Gap”** and **“Flat Structure Fallacy”**; **FLOW** exhibits **“Aggregative Context Collapse”**; and **OpenHands** is characterized as **“Lost-in-the-Middle.”** Among commercial systems, **Gemini Studio** is associated with **“Probabilistic Drift,”** **Lingma** with **“Logical Detachment,”** **Trae** with **“Orphaned Logic”** and **“Unreachable Logic,”** and **CodeBuddy** with **“Calling Hallucination.”** The paper argues that Contract-Coding changes the dominant error type from architectural collapse to **“Local Logic Error”** on the hardest tasks [2604.13100].

This interpretation matters for how Greenfield-5 should be read. The benchmark’s principal claim is not that one method dominates every absolute score, but that some systems preserve **repository skeleton and inter-file topology** under scale while others do not. On Roguelike in particular, the paper emphasizes that Contract-Coding’s failures are mostly local logic limitations rather than missing modules, invalid imports, or monolithic collapse. In that sense, Greenfield-5 functions as an **architecture-sensitive diagnostic benchmark** [2604.13100].

The benchmark also sits within a small cluster of related but non-identical greenfield evaluations. The paper itself contrasts Greenfield-5 with **SWE-bench-like** settings that focus on **“Brownfield Repair.”** A distinct 2026 study, “Vibe Coding Ate My Homework,” introduces **an evaluation suite for analysing an LLM’s proficiency in carrying out simple, isolated greenfield programming tasks in Python**, with **5 unique tasks** at **3 prompt levels** and a total of **60 data samples**; however, that study does **not** define or name a benchmark called Greenfield-5, and its scope is explicitly limited to simple isolated tasks rather than repository-scale synthesis [2606.18293].

A further point of caution concerns internal reporting consistency. The Greenfield-5 paper itself contains discrepancies between the main table and appendix for some mid-complexity tasks. For example, **City Sim** and **Snake++** have appendix values for Trae and Ours w/o HEG that do not perfectly align with the main table, and the paper explicitly notes such inconsistency in the supplied synthesis. The safest reading is therefore the one given in the source summary: Greenfield-5 consistently presents Contract-Coding as structurally stronger on hard repo-level tasks, while some task-level appendix/main numbers do not perfectly align [2604.13100].

Source: https://www.emergentmind.com/topics/greenfield-5-benchmark