WebJudge: Automated Web Dev Evaluation

• WebJudge is a framework for automated web development evaluation that combines LLMs, MLLMs, and agentic workflows with rubric-based scoring.
• It employs both static (code and screenshot analysis) and dynamic (interactive UI testing) modes to comprehensively assess code quality and user experience.
• The system integrates deduplication and ranking of reference solutions alongside pairwise comparisons to benchmark LLM performance against human judgments.

WebJudge is a specialized framework and benchmark infrastructure for automated, scalable evaluation of web development quality using LLMs, multimodal LLMs (MLLMs), and agentic workflows. It is designed to measure the extent to which such automated systems can replicate, surpass, or augment human expert judgments on open-ended web development tasks, with a strong emphasis on both code and user interface quality, interactive behavior, and reducing redundancy in reference solutions (Li et al., 21 Oct 2025, Shirafuji et al., 2023).

1. Motivation and Design Objectives

WebJudge addresses two central challenges in the modern evaluation of web development:

• The open-ended and interactive nature of real-world web development complicates automated scoring, as it involves code, rendered interface, and user experience, all changing in response to user actions.
• Manual human evaluation is resource-intensive and not scalable for ongoing large-scale automated coding platforms.

WebJudge aims to:

• Assess the viability and limits of LLMs, MLLMs, and agentic agents as judges of web development quality.
• Provide systematic, high-quality, rubric-anchored ground truth to facilitate rigorous head-to-head comparisons against diverse automated evaluators.
• Offer a mechanism to deduplicate and rank reference solutions, minimizing redundancy and presenting learners or evaluators with representative, non-redundant solution sets (Li et al., 21 Oct 2025, Shirafuji et al., 2023).

2. Benchmark Construction and Evaluation Modes

Data Pipeline

The WebJudge corpus is derived from a base set of 10,501 user queries and paired code outputs, filtered using a two-stage process:

• Query filtering: eliminating duplicates, unsafe or infeasible requests, and poorly specified prompts.
• Environment filtering: deploying code in a standardized Next.js environment and discarding any output that fails to launch or presents a blank page.

This yields 1,713 high-quality, query–implementation pairs, each annotated by dual expert raters under a rubric-driven protocol, with inter-annotator agreement at 89.7% (allowing for ties).

Dual Evaluation Modes

WebJudge supports two orthogonal modalities for evaluation:

• Non-interactive (static): Evaluation is conducted on source code and/or rendered screenshots, with no interaction allowed.
• Continuous interactive (dynamic): The evaluator or LLM-based agent directly interacts with a live web page via standard UI operations (clicking, typing, scrolling), facilitating end-to-end dynamic assessment.

Rubric Structure

Each query–implementation pair is assessed via a three-tiered rubric:

Rubric Dimension	Example Criteria
Intention (I)	Were core user-requested features met?
Static Quality (II)	UI layout, code structure
Dynamic Behavior(III)	Interactive functionalities

At the leaf level, each subcriterion is an atomic “implemented/not implemented” binary. Let $L^d$ be the leaves for dimension $d$, each with weight $w_i$ and binary implementation $pass_i$, then the pass-rate for dimension $d$ is:

$$r_d = \frac{\sum_{i \in L^d} w_i \cdot pass_i}{\sum_{i \in L^d} w_i}$$

The global rubric score $S$ across dimensions is:

$$S = \sum_{d \in \{I, II, III\}} \alpha_d \cdot r_d \qquad (\alpha_d \ge 0, \sum \alpha_d = 1)$$

In all reported experiments, $\alpha_d = 1/3$ (Li et al., 21 Oct 2025).

3. Automated Solution Deduplication and Ranking

To optimize the presentation of reference solutions and reduce cognitive burden, WebJudge deploys a normalization and deduplication process (Shirafuji et al., 2023):

• Normalization (Norm): Strips comments, doc-strings, blank lines, whitespace; re-tokenizes; anonymizes user-defined identifiers; pretty-prints canonical code.
• Near-duplicate detection: Two solutions $P_i$, $d$0 are deemed duplicates if $d$1, i.e., normalized forms exactly match ($d$2). For relaxed deduplication, a token-based similarity threshold $d$3 can be applied.
• Deduplication Algorithm: Submissions are grouped by normalized form. Each group is represented by a canonical program; multiplicities ($d$4) are tracked.
• Ranking: Unique solutions are ranked by duplicate count $d$5 (frequency among all accepted submissions):

$d$6

Displaying the top $d$7 ranked solutions yields substantial coverage: empirically, the top 10 unique solutions cover, on average, $d$8 of submitted solutions in programming contexts (Shirafuji et al., 2023).

4. Evaluation Metrics

WebJudge emphasizes agreement between automated and human expert judges using multiple metrics:

• Accuracy (Agreement Rate):

$d$9

• Cohen’s kappa ($w_i$0): Quantifies agreement beyond chance:

$w_i$1

where $w_i$2 is observed agreement and $w_i$3 is the chance agreement.

• Pearson correlation ($w_i$4) and Spearman correlation ($w_i$5): Measure linear and rank correlations respectively between model and human scores.
• Dynamic feasibility (WebDevJudge-Unit): Uses standard classification metrics (precision $w_i$6, recall $w_i$7, $w_i$8, and accuracy $w_i$9).

For deduplication coverage, the “top-n coverage” metric is:

$pass_i$0

5. Experimental Findings and Performance Limits

LLM/MLLM vs. Human Benchmarks

• Single-answer grading: Likert-style (1–5) per subcriterion produces mean accuracy rates of 56–59%.
• Pairwise comparison: Increases accuracy to 63–66%, with Claude-3.7 Sonnet and GPT-4.1 attaining 65.14% and 66.06%, respectively; human expert agreement stands at 84.82%. A gap of approximately 15–20 percentage points persists between the best LLM/MLLM judges and human experts.

Agentic Workflows

Agentic workflows (planner → executor → summarizer pipeline) on dynamic aspects yield accuracy $pass_i$1, not surpassing plain LLMs. Major limitations include brittle and over/under-specified planning, navigation errors, and compounding execution noise.

Deduplication and Reference Presentation

Using normalized exact-match deduplication, an average deduplication rate of 60.20% is achieved across benchmark datasets, compared to 29.59% for raw exact-match. The average number of unique solutions per problem is reduced from 2,408 to 1,361. Displaying the top 10 reference solutions provides a mean coverage of 29.95% of all accepted solutions (see table below):

Method	Avg. #Unique	DedupRate
Baseline (raw exact)	2,408	29.59%
Ours (normalized exact)	1,361	60.20%

n (top)	Avg. Cov(n)
1	13.16%
5	24.37%
10	29.95%
20	36.84%

6. Failure Modes and Systematic Challenges

WebJudge reveals three principal failure domains:

• Functional equivalence: LLMs often fail to recognize alternative surface implementations as functionally matching the specification, whereas humans accept paraphrases and structural synonyms.
• Feasibility verification: Static code-based LLMs exhibit high recall (90%) but low precision (≈72%), often failing to verify execution; interactive approaches do the converse (precision ≈82%, recall ≈70%). Neither alone suffices; hybrid protocols are indicated.
• Model biases: Positional bias in pairwise mode (skew towards first/second presented option, <90% post-swap consistency) and verbosity bias (favoring longer outputs) are persistent and cause measurable degradations in scoring robustness (Li et al., 21 Oct 2025).

A plausible implication is that robust automated evaluation in web development requires explicit hybridization of static reasoning with live dynamic validation and advanced mechanisms for equivalence detection.

7. Research Insights and Recommendations

WebJudge supports several operational and methodological guidance points:

• Pairwise comparison is empirically more robust for open-ended tasks than single-answer grading; future automated judges should default to relative scoring.
• Rubric-anchored binary assessment outperforms multi-point Likert grading; evaluators benefit from anchor-free checklists over subjective scales.
• Functional equivalence detection (e.g., paraphrase or canonical test queries) must be integrated to bridge model–human gaps.
• Hybrid static–interactive assessment is necessary to simultaneously realize high recall and precision in behavioral validation.
• Debiasing strategies (swap-and-filter, calibration prompts, learning-based debiasing layers) are required to mitigate systematic LLM biases.
• Future research directions include multi-round interactive judging, collaborative agent protocols, and fine-tuning on rubric-annotated data to improve calibration and granularity of automated judgments (Li et al., 21 Oct 2025).

WebJudge establishes that while LLMs and related models have acquired a substantial repertoire of evaluative competencies, they currently fall short of reliably replacing human judgment for interactive, multifactorial domains such as web development. Ongoing advances in hybrid evaluation, dynamic equivalence checking, and bias mitigation remain critical for progress.