MiniAppBench Benchmark Suite

• MiniAppBench is a comprehensive benchmark that evaluates LLM-generated MiniApps by testing both static fidelity and dynamic interactive logic.
• It assesses applications on three dimensions—intention, static presentation, and dynamic behavior—ensuring adherence to real-world principles.
• The suite employs agentic evaluation with browser automation and LLM evaluator agents to generate robust performance metrics.

MiniAppBench is a comprehensive benchmark and evaluation suite designed to rigorously assess the generation of principle-driven, interactive HTML/CSS/JavaScript applications ("MiniApps") by LLMs. Developed to fill the gap left by previous code and web layout benchmarks, it addresses the new paradigm in LLM-powered assistants: synthesizing not just static answers, but executable, interactive artifacts for end-users across a diverse set of real-world domains (Zhang et al., 10 Mar 2026).

1. Motivation: From Static Text to Dynamic MiniApps

The evolution of LLM assistants has shifted user expectations from static text, code snippets, or layout reconstructions to fully interactive applications directly executable in the browser. Traditional benchmarks such as HumanEval and MBPP emphasize algorithmic correctness in isolation, whereas web layout benchmarks (Pix2Code, FullFront, Web2Code) focus primarily on pixel or DOM fidelity to a mockup, typically omitting complex interaction logic.

Critical challenges in this emergent evaluation regime include:

• Fidelity to real-world principles: Generated artifacts must adhere to discipline-specific rules (e.g., physics laws, correct calendar arithmetic).
• Synthesis of custom interaction logic: Requirements often go beyond CRUD or simple workflows, demanding state management, event-driven logic, and simulation.
• Flexible evaluation of open-ended design: Benchmarking must accommodate multiple valid realizations of a user’s task, instead of enforcing a single implementation as ground truth.

MiniAppBench was created to offer a systematic, scalable, and principle-grounded framework for these needs (Zhang et al., 10 Mar 2026).

2. MiniApp: Definition, Structure, and Distinction

A MiniApp is a self-contained web application composed of:

• UI Layout: An explicit HTML DOM structure augmented with CSS, exposing user controls and displays.
• Interaction Logic: Custom JavaScript logic implementing event-driven behaviors, finite state machines, numerical simulation, or game mechanics.
• Real-World Principle Adherence: Explicit or implicit encoding of physics, biology, scheduling, scoring, or similar rules.

Unlike prior benchmarks:

• Code generation datasets test local algorithms but offer little abstraction to UI or domain rules.
• Web mockup datasets test how faithfully a static visual can be replicated—but typically lack required interactivity or principle grounding.
• MiniAppBench requires both novel, goal-driven interaction and explicit real-world logic, demanding integration rather than compartmentalized solutions (Zhang et al., 10 Mar 2026).

3. Dataset Construction and Task Taxonomy

The MiniAppBench dataset is derived from tens of millions of user queries to a production LLM-powered assistant, filtered and stratified to surface tasks requiring both complex interaction and principle adherence.

Multi-stage pipeline:

1. Identify candidate queries demanding both interaction and implicit real-world logic.
2. Data augmentation via LLM to introduce variants; in-depth manual standardization.
3. LLM- and human-driven curation of “eval-references” ($r_i$), which enumerate checklists for three evaluation axes (Intention, Static, Dynamic).
4. Stratified sampling yields 500 finalized tasks, balanced by domain and difficulty.

Task Representation:

$\tau_i = \langle q_i, (c_i, s_i), r_i, d_i \rangle$

• $q_i$: natural-language user query
• $c_i$: coarse domain $\in$ {Science, Games, Tools, Humanities, Lifestyle, Visualization}
• $s_i$: subclass (fine-grained type)
• $r_i$: JSON-format “eval reference” with dimension-specific checklists
• $d_i$: difficulty (Easy, Medium, Hard)—determined by baseline pass rates

Domain and Difficulty Breakdown:

Domain	Count	% of 500
Science	187	37.4
Games	121	24.2
Tools	57	11.4
Humanities	47	9.4
Lifestyle	32	6.4
Visualization	64	11.2

Difficulty: 30% Easy, 40% Medium, 30% Hard (Zhang et al., 10 Mar 2026).

4. MiniAppEval: Agentic, Three-Dimensional Assessment

MiniAppEval introduces an automated, human-like evaluation protocol leveraging browser automation and LLM evaluator agents to overcome the absence of a fixed “correct” implementation.

Evaluation Dimensions:

1. Intention: Alignment with user’s high-level goals (e.g., correct periodic simulation in physics, intended game rules).
2. Static: Structural and presentational accuracy (UI components, code structure, accessibility).
3. Dynamic: Robustness and correctness under interactive, multi-step operations, including edge case handling.

Scoring Criterion:

• Each dimension produces a scalar score $S^\mathrm{I}$, $S^\mathrm{S}$, $\tau_i = \langle q_i, (c_i, s_i), r_i, d_i \rangle$0.
• A binary “pass” if $\tau_i = \langle q_i, (c_i, s_i), r_i, d_i \rangle$1, where $\tau_i = \langle q_i, (c_i, s_i), r_i, d_i \rangle$2.

Agentic Evaluation Framework:

• Uses Playwright automation to enact exploratory interaction traces on generated MiniApps.
• An LLM “evaluator agent” plans steps, accumulates interaction evidence, and aggregates it into scores with natural language justifications.
• Eval-references $\tau_i = \langle q_i, (c_i, s_i), r_i, d_i \rangle$3 serve as flexible guides, enabling pass-through for legitimate novel designs (Zhang et al., 10 Mar 2026).

5. Experimental Results, Patterns, and Failure Modes

Empirical evaluation across leading LLMs demonstrates several concrete trends:

Group	Pass Rate (Overall)	Pass Rate (Easy)	Pass Rate (Medium)	Pass Rate (Hard)
All Models (mean)	17.05%	34.05%	13.89%	4.34%
GPT-5.2 (best)	45.46%	-	-	-

Closed-source models (GPT-5.1/5.2, Claude Opus, Gemini Pro) outperform open models by a significant margin. Highest pass rates appear in Lifestyle (33.3%) and Visualization (31.6%), with Science (11.6%) and Games (14.7%) considerably lower, corresponding to their higher requirements for domain-specific reasoning.

Frequent Failure Modes:

• Static failures: Missing or incorrectly styled UI components.
• Principle failures: Implementation deviates from implicit rules (e.g., non-conservation in physics).
• Dynamic failures: Unstable or incomplete interaction flows, especially under multi-action sequences or invalid input.

Strong positive correlation observed between token usage and pass rate (Pearson 0.84), with moderate correlation to inference time (0.74). This suggests that larger, more resource-intensive models are better equipped for generative tasks requiring integrated logic and robust interactivity (Zhang et al., 10 Mar 2026).

6. Usage, Extensibility, and Reproducibility

MiniAppBench is released for reproducibility and flexible research deployment via github.com/MiniAppBench. Quick install and execution pipelines are provided:

• pip install miniappbench
• miniappbench run --query_file data/miniappbench.json --model gpt-5.2

System Features:

• Supports both monolithic HTML and modular React+TypeScript+Vite scaffolding.
• Includes build scripts and prompt templates for research reproducibility.
• Evaluation modes include ablations: code-only (static), no-eval-reference (unguided), and no-agent (template scripts).

Extending MiniAppBench:

• Add new tasks by appending JSON entries with schema $\tau_i = \langle q_i, (c_i, s_i), r_i, d_i \rangle$4.
• Customize $\tau_i = \langle q_i, (c_i, s_i), r_i, d_i \rangle$5 to target new domains or principles.
• Integrate new LLMs via model selection flags or decoding-settings table.

The benchmark design emphasizes reproducibility, scalability, and openness to future LLM and task integration (Zhang et al., 10 Mar 2026).

---

For complete implementation, task definitions, prompts, evaluation code, and configuration guidelines, refer to the official repository. MiniAppBench establishes a rigorous foundation for research into LLMs’ abilities to generate nontrivial, interactive, principle-driven web applications.