Greenfield-5 Benchmark

• Greenfield-5 is a benchmark designed to evaluate repository-level code synthesis from under-specified intents, emphasizing coherent architecture and valid inter-file linkages.
• It employs a five-task suite that progressively tests logical organization, event-driven interactions, and scalability using dynamic execution and static structural metrics.
• The results reveal that while some systems maintain strong structural integrity, many struggle with local logic consistency, exposing issues like context exhaustion and symbolic hallucination.

Greenfield-5 is a custom benchmark suite for greenfield, repository-level code generation from vague, high-level intent in what its authors call the “Vibe Coding” setting. It was introduced to evaluate whether an agent can turn an ambiguous project request into a runnable, interactive, multi-file codebase without architectural collapse, and it is presented as a bridge “between algorithmic scripts and system-level architectures.” In contrast to suites centered on repair of existing repositories, Greenfield-5 targets greenfield synthesis from high-level intents, with explicit stress on coherent architecture, valid cross-file symbol linkage, and task-level behavior under ambiguity (Lin et al., 10 Apr 2026).

1. Definition and evaluative scope

Greenfield-5 is organized around a “Complexity Spectrum of five greenfield repositories.” Its defining object is not a function, patch, or issue resolution trace, but a newly generated repository that must satisfy both structural and behavioral constraints. The benchmark therefore evaluates more than whether code executes: it measures whether a system can produce a repository with a coherent file layout, stable interfaces, valid imports, and interactive functionality aligned with a high-level prompt (Lin et al., 10 Apr 2026).

The benchmark is explicitly framed against a Context-Fidelity Trade-off. In the authors’ formulation, vague user intents and growing implementation context interact poorly with linear generation pipelines, producing “Symbolic Hallucination,” invented APIs, inconsistent interfaces, and eventual architectural collapse. This motivates the benchmark’s emphasis on multi-file synthesis under under-specification rather than on single-file completion or brownfield bug fixing. Formally, the repository-generation objective is written as

$$\mathcal{R}^* = \operatorname*{arg\,max}_{\mathcal{R}} P(\mathcal{R} \mid \mathcal{I}),$$

where $\mathcal{R} = \{f_1, f_2, ..., f_N\}$ is the repository and $\mathcal{I}$ is under-specified user intent. The paper also highlights the sequential factorization

$$P(\mathcal{R} \mid \mathcal{I}) = \prod_{i=1}^{N} P(f_i \mid f_{<i}, \mathcal{I}),$$

to explain how early defects can propagate across later files in baseline systems (Lin et al., 10 Apr 2026).

A common misconception is to treat Greenfield-5 as a generic coding benchmark. The benchmark is narrower and more specific: it targets repo-level autonomous generation from vague intent, with strict multi-file structure and interactive task requirements. It is therefore best understood as a benchmark for architectural orchestration and cross-module consistency in greenfield synthesis rather than as a general-purpose software engineering scorecard (Lin et al., 10 Apr 2026).

2. Composition of the five-task suite

The suite contains five tasks arranged in increasing architectural complexity. The tasks are all repository-generation problems, and several impose explicit file-count constraints. The later tasks stress event-driven structure, extensibility, polymorphism, and deeper cross-file dependency chains.

Task	Characterization	Defining constraints
Gomoku	Logic-Intensive / Logic	Gomoku program with AI so players can play against AI
Plane Battle	Action-Oriented	Player movement, bullets, enemies, enemy death on hit, enemy bullets, win/lose conditions
City Sim	Resource Management	“Micro City Sim,” resource balancing, menu placement, extensibility, above 10 files
Snake++	Event-Driven	Modular Snake variant, power-up factory, multiple layouts, leaderboard/save, 13–15 files
Roguelike	System Architecture / Scalability stress test	Modular Pygame roguelike, event bus, polymorphic entities, cross-module inventory effects, at least 15 files

The benchmark descriptions are unusually concrete. City Sim is specified as a grid-based city builder in Python/Pygame with real-time Money/Energy simulation, Houses that generate money but consume energy, Power Plants that generate energy but cost money, taxes that stop when energy is insufficient, menu-based building placement, at least Roads, Residential Zones, and Industrial Zones, HUD/resource alerts, and easy extensibility for new building types, “implicitly requiring polymorphism.” Snake++ requires growth logic, Speed Boost, Teleportation Walls, a modular power-up factory, multiple arena layouts, a leaderboard/save system, and an explicit architecture constraint of 13–15 files. Roguelike requires isolated procedural map generation, rendering separated from data, an event-driven UI with Message Log and HUD via Event Bus rather than direct coupling, polymorphic entities including Player, Orc, and Mage, cross-module inventory and item interactions such as a Scroll revealing Fog of War, and at least 15 files, elsewhere described as 15–25 interdependent files (Lin et al., 10 Apr 2026).

This progression is central to the benchmark’s design. Gomoku and Plane Battle primarily expose logic and game-loop organization; City Sim adds resource simulation and extensible design; Snake++ stresses event-driven extensibility; Roguelike acts as the benchmark’s scalability stress test, combining procedural generation, polymorphism, UI/event separation, and deep inter-file coordination (Lin et al., 10 Apr 2026).

3. Construction principles and why the benchmark is hard

Each Greenfield-5 task begins from a high-level prompt—the “Vibe”—rather than from a fully formal specification. This is deliberate. The benchmark assumes that real users often do not provide complete blueprints, and it therefore tests whether a system can infer a viable architecture from ambiguous intent. The generated artifact must satisfy repo-level generation requirements: multi-file decomposition, coherent internal imports and symbol references, and project structure constraints; monolithic scripts do not satisfy the intended benchmark setting (Lin et al., 10 Apr 2026).

Greenfield-5 is difficult because it combines several burdens that are usually separated in other evaluations. The paper identifies these explicitly: architectural decomposition, symbol grounding across files, interactive behavior, game-specific logic fidelity, strict multi-file structure, and high-level under-specification. The benchmark also includes a reference architecture against which generated file sets are compared. This makes it possible to score both missing modules and unnecessary file proliferation, rather than treating any runnable output as equivalent (Lin et al., 10 Apr 2026).

The benchmark’s static difficulty is reinforced by a dynamic coupling effect. Later tasks force generated files to remain globally consistent over longer horizons, which the paper characterizes as context exhaustion. In baseline sequential systems, code context expands linearly while the original intent becomes diluted. A result is that a repository may execute partially yet still exhibit “zombie code,” interface drift, or unreachable logic. Greenfield-5 is designed to expose exactly these failure modes (Lin et al., 10 Apr 2026).

A plausible implication is that Greenfield-5 evaluates a distinct regime of software-generation ability: not just local code correctness, but the capacity to preserve symbolic consistency and architectural topology under ambiguity. That emphasis differentiates it from both unit-test-centric code-generation benchmarks and brownfield repair suites.

4. Evaluation protocol and metrics

The evaluation protocol combines dynamic functional scoring with static structural analysis. The paper reports 10 independent runs per method/task, with means as the reported numbers. For the academic methods, the reported configuration uses gpt-4o-2024-11-20, temperature 0.0, and a 16k token context window. The hardware reported is a MacBook Pro (M2 Pro), 32GB unified memory, 512GB SSD, macOS 15. Commercial IDE baselines are evaluated in their own listed versions and modes, including Lingma, CodeBuddy, Trae, and Gemini Studio (Lin et al., 10 Apr 2026).

The dynamic metric is

$$S_{overall} = \frac{1}{3} (S_{exec} + S_{inter} + S_{rule}) \times 100\%.$$

Here Executability $S_{exec} \in \{0,1\}$ checks whether dependencies install and the entry point launches without runtime crash for a 60-second keep-alive window; Interactivity $S_{inter} \in [0,1]$ measures whether a standardized UI action sequence elicits valid responses; and Rule Adherence $S_{rule} \in [0,1]$ measures task-specific logic correctness. The appendix describes this as a triple-blind human scoring protocol with three independent reviewers (Lin et al., 10 Apr 2026).

The interaction and rule checks are benchmark-specific. For example, Gomoku requires that placing a stone changes state and triggers AI response, and that 5-in-a-row is detected correctly. Plane Battle requires directional control and bullet firing, with collisions reducing health or triggering game-over. City Sim checks menu selection, grid placement, HUD updates, and enforcement of the energy-tax dependency loop. Snake++ checks direction changes, visible speed boost behavior, and rule-level handling of shield and teleportation walls. Roguelike checks room movement, interaction with enemies/items, and that the procedural map remains traversable (Lin et al., 10 Apr 2026).

Static structure is evaluated with two explicit metrics. Architectural Fidelity is

$$S_{arch} = \frac{2 \cdot |F_{gen} \cap F_{ref}|}{|F_{gen}| + |F_{ref}|},$$

where $F_{gen}$ and $\mathcal{R} = \{f_1, f_2, ..., f_N\}$0 are the generated and reference file-path sets. This penalizes both “Hollow Skeleton” repositories with missing files and bloated repositories with redundant files. Linkage Consistency is

$\mathcal{R} = \{f_1, f_2, ..., f_N\}$1

where $\mathcal{R} = \{f_1, f_2, ..., f_N\}$2 is the set of internal import statements extracted from ASTs, and an import is valid iff the target file exists and defines the imported symbol (Lin et al., 10 Apr 2026).

5. Reported empirical results

Greenfield-5’s headline empirical pattern is a sharp increase in difficulty from the early tasks to the later ones. In the main results table, Contract-Coding reports 100 on Gomoku, 100 on Plane Battle, 87 on City Sim, 80 on Snake++, and 47 on Roguelike. The abstract summarizes the hardest-task result as “47\% functional success while maintaining near-perfect structural integrity.” The paper’s central interpretation is that the benchmark distinguishes between systems that fail through architectural collapse and systems that preserve architecture but still make local logic errors (Lin et al., 10 Apr 2026).

For the easier tasks, several baselines reach perfect or near-perfect scores. On Gomoku, OpenHands, Lingma, Gemini Studio, Ours w/o HEG, and Ours (Full) each report 100, while ChatDev reports 93 and FLOW 96. On Plane Battle, Gemini Studio, CodeBuddy, Ours w/o HEG, and Ours (Full) each report 100, while ChatDev and OpenHands report 90 and Trae 80. These results indicate that Greenfield-5 is not uniformly hard; rather, its difficulty is concentrated in tasks with deeper structural coupling (Lin et al., 10 Apr 2026).

The mid-complexity tasks show clearer divergence. On City Sim, the main table reports 87 for Ours (Full), 87 for Gemini Studio, 85 for Trae, 85 for Ours w/o HEG, and lower numbers for MetaGPT, ChatDev, FLOW, OpenHands, Lingma, and CodeBuddy. On Snake++, the main table reports 83 for Gemini Studio, 80 for Ours (Full), 78 for Ours w/o HEG, 65 for Trae, and lower values for the other baselines. The benchmark thus begins to separate systems once file-count constraints, event-driven coordination, and extensibility become binding (Lin et al., 10 Apr 2026).

Roguelike is the suite’s clearest stress test. In the main table, Gemini Studio reports 63, Trae 47, Ours w/o HEG 47, Ours (Full) 47, CodeBuddy 40, Lingma 33, OpenHands 30, MetaGPT 10, ChatDev 10, and FLOW 0. The appendix provides a structural and functional breakdown for Roguelike in which Ours (Full) achieves 100 structural integrity, 92 dependency consistency, 90 executability, 20 interactivity, 30 rules, and 47% overall. By contrast, Trae is also at 47% overall but with 23.9 files in the main table versus 14.2 files for Ours (Full), while Gemini Studio leads overall at 63% but reports 80 structure and 80 dependency rather than perfect structural scores (Lin et al., 10 Apr 2026).

The ablation results reinforce the benchmark’s architectural emphasis. Comparing Ours w/o HEG with Ours (Full), the score remains 100 → 100 on Gomoku and 100 → 100 on Plane Battle, but time drops from 205s → 136s and 201s → 117s respectively. On City Sim, the score changes 85 → 87 with time 480s → 257s; on Snake++, 78 → 80 with time 465s → 198s; and on Roguelike, the overall score remains 47 → 47 while file count changes 25.0 → 14.2 and time 510s → 232s. The paper interprets this as evidence that HEG improves efficiency and repository cleanliness more than raw functional score on the hardest task (Lin et al., 10 Apr 2026).

6. Failure modes, interpretation, and relation to adjacent work

Greenfield-5 is designed not merely to rank systems, but to expose distinct pathologies of repo-level generation. The appendix names these failure modes explicitly. Among academic baselines, MetaGPT exhibits “Hollow Skeleton” and “Implementation Sparsity”; ChatDev shows a “Reflection-Action Gap” and “Flat Structure Fallacy”; FLOW exhibits “Aggregative Context Collapse”; and OpenHands is characterized as “Lost-in-the-Middle.” Among commercial systems, Gemini Studio is associated with “Probabilistic Drift,” Lingma with “Logical Detachment,” Trae with “Orphaned Logic” and “Unreachable Logic,” and CodeBuddy with “Calling Hallucination.” The paper argues that Contract-Coding changes the dominant error type from architectural collapse to “Local Logic Error” on the hardest tasks (Lin et al., 10 Apr 2026).

This interpretation matters for how Greenfield-5 should be read. The benchmark’s principal claim is not that one method dominates every absolute score, but that some systems preserve repository skeleton and inter-file topology under scale while others do not. On Roguelike in particular, the paper emphasizes that Contract-Coding’s failures are mostly local logic limitations rather than missing modules, invalid imports, or monolithic collapse. In that sense, Greenfield-5 functions as an architecture-sensitive diagnostic benchmark (Lin et al., 10 Apr 2026).

The benchmark also sits within a small cluster of related but non-identical greenfield evaluations. The paper itself contrasts Greenfield-5 with SWE-bench-like settings that focus on “Brownfield Repair.” A distinct 2026 study, “Vibe Coding Ate My Homework,” introduces an evaluation suite for analysing an LLM’s proficiency in carrying out simple, isolated greenfield programming tasks in Python, with 5 unique tasks at 3 prompt levels and a total of 60 data samples; however, that study does not define or name a benchmark called Greenfield-5, and its scope is explicitly limited to simple isolated tasks rather than repository-scale synthesis (Barbour, 15 Jun 2026).

A further point of caution concerns internal reporting consistency. The Greenfield-5 paper itself contains discrepancies between the main table and appendix for some mid-complexity tasks. For example, City Sim and Snake++ have appendix values for Trae and Ours w/o HEG that do not perfectly align with the main table, and the paper explicitly notes such inconsistency in the supplied synthesis. The safest reading is therefore the one given in the source summary: Greenfield-5 consistently presents Contract-Coding as structurally stronger on hard repo-level tasks, while some task-level appendix/main numbers do not perfectly align (Lin et al., 10 Apr 2026).