RealBench: Realistic AI Benchmarks

• RealBench is a framework of benchmarks that replicates real-world constraints across domains like hardware design, photorealism, multimodal reasoning, code generation, and weather forecasting.
• It standardizes evaluation with structured specifications, rigorous verification methods—including formal equivalence for Verilog—and objective metrics such as pass rates and error rates.
• The benchmarks expose critical challenges in current AI models, revealing low success rates in formal tasks and significant gaps in handling complex, realistic scenarios.

RealBench refers to a diverse set of rigorously designed benchmarks, each targeting a specific scientific or engineering domain, unified by a common philosophy: to faithfully reproduce the complexity, structure, and constraints of real-world tasks while exposing the limits of current AI and machine learning models. Benchmarks named “RealBench” have been proposed in hardware design automation (Verilog generation), photorealistic image synthesis, Chinese multi-image reasoning, hybrid human–AI text detection, repository-level code generation, and operational AI weather forecasting. Each addresses critical shortcomings in prevailing evaluation protocols—specifically the lack of realistic task design, representative data, comprehensive assessment criteria, and operationally relevant verification procedures.

1. Verilog Generation with Real-World IP Designs

The original “RealBench” in digital hardware design automation provides the first benchmark for evaluating LLMs on IP-level Verilog code generation using authentic, structured, multi-modal specifications and stringent verification (Jin et al., 22 Jul 2025). Previous benchmarks were limited to trivial, flat modules (≈50 LOC); RealBench’s modules average ≈180 LOC and encompass deep hierarchies, with sources drawn from OpenCores and RISC-V Hummingbirdv2.

Key Features:

• Data and Task Structure: Four open-source IP cores (SD card controller, AES top, AES helpers, RISC-V E203), totaling 60 module-level tasks and 4 system-level (whole-IP) generation challenges. Module-level requires generating a module from multi-modal specs and golden submodules; system-level requires synthesizing an entire IP from scratch.
• Specification Format: Structured input includes block diagrams, port tables, submodule relations, textual descriptions with state-transition/data-flow diagrams, register maps, and timing constraints—matching industrial practice.
• Verification Methodology:
  • 100% line-coverage testbenches: Every reference code line is exercised via random/directed inputs; passing code must achieve full coverage.
  • Formal equivalence checking: Both reference and generated Verilog are synthesized into netlists for verification by JasperGold; only designs that are formally equivalent are considered correct.
• Evaluation Metrics: Distinguishes syntax@k (no parser errors), func@k (100% testbench pass), formal@k (testbench + formal equivalence). Metrics computed as pass@k for “top-k” generations.
• Results: Even the best LLM (o1-preview) only achieves 13.3% formal@1 at module level; all models give 0% on full-system generation.

This paradigm exposes LLMs’ failure to transition from toy examples to true IP designs—rooted in errors such as faulty submodule instantiation, FSM omissions, and semantic hallucinations, and points to limitations in current vision–LLM backbones for structured diagram parsing.

2. Automated Photorealism Evaluation for Text-to-Image Generation

RealBench in photorealistic image synthesis offers the first scalable, human-free benchmark for measuring “indistinguishable-from-real” quality in T2I models (Ye et al., 29 Nov 2025). This benchmark responds to the inadequacy of preference-based and human-in-the-loop evaluations, which confound aesthetics and realism and are expensive to scale.

Core Components:

• Metrics:
  • Detector-Scoring: Aggregates the outputs of state-of-the-art synthetic-image detectors—semantic (e.g., Forensic-Chat, GPT-5) and feature-level (e.g., OmniAID, Effort)—to quantify “realness”; see $S_{\text{det}}(I)$ formula in data.
  • Arena-Scoring: GPT-5 LLM is used as a forced-choice judge in pairwise comparison between model outputs (or real photos), yielding a win-rate for realism ($S_{\text{arena}}$).
• Dataset: 1,000 text prompts with ground-truth real images from free photography sources, spanning seven photographic domains at 512×512 resolution.
• Procedures: Each model generates one image per prompt; detectors assess realism batchwise; Arena-Scoring aggregates >3,000 judge pairings per model.
• Results: RealGen (the model motivating this benchmark) achieves up to 50.15% “confusion” with real images (GPT-5 mistaken half the time), and its Detector-Score correlates (Pearson $r \approx 0.72$) with human preference scores.

RealBench here directly links measurable detector artifacts to perceptual realism, providing an objective yardstick for “realness” distinct from style or prompt compliance. Lower detector-identified artifacts predict higher user confusion.

3. Chinese Multi-image Reasoning and Content Understanding

In multimodal research, RealBench is the pioneering large-scale Chinese benchmark for multi-image understanding, grounded in real user-generated content (Zhao et al., 22 Sep 2025). Unlike prior English-centric or synthetically merged datasets, this dataset is sourced from Xiaohongshu social posts, capturing the entropy, layout complexity, and contextual variability of actual Chinese digital media.

Salient Properties:

• Scale & Composition: 9,393 posts, 69,910 images (avg. 7.4 per post); coverage of 36 categories with high-resolution and complex layouts.
• Task Suite: Four tasks—multi-image retrieval, ranking, extraction, and reasoning—each with “easy” and “hard” tracks, requiring integration across images and robust Chinese-language understanding.
• Annotation: GPT-4o–assisted generation plus rigorous, two-stage expert human curation.
• Metrics: Accuracy for retrieval/ranking; ROUGE-L F1 for extraction/reasoning.

Empirical results show a pronounced gap between closed-source (Qwen-VL-Max: 46.88% overall) and open-source models (average: 16.24%), with the most sophisticated MLLMs still bottlenecked on cross-image information fusion and Chinese-specific text/graphic alignment.

4. Hybrid Human–AI Text Attribution via Hierarchical Structured Detection

For the attribution of “hybrid” texts co-produced by humans and AI (editing, translation, paraphrasing), RealBench introduces a comprehensive, automatically labeled corpus designed to reflect the real multiprocess landscape of collaborative writing (He et al., 20 Oct 2025).

Features:

• Dataset Construction:
  • Aggregates 5.63M “basic” texts (human and LLM-only) and 10.77M hybrid samples covering paraphrase, translation, extension, and polish transformations.
  • Hybrid texts labeled at fine granularity (e.g., human_polish_Gemini1.5, Llama3_paraphrase_Qwen2.5-7B); supports adversarial attack variants.
  • 1,204 categories capturing both source and transformation pipeline.
• Detection Method: Employs a Hierarchical Affinity Tree (HAT) to reflect similarities among transformation processes and structures a Tree-Structured Contrastive Loss for representation learning, stabilizing classification across over a thousand hybrid classes.
• Benchmarks and Metrics: Supports supervised (IID), OOD, zero-shot, few-shot, and robustness testing using F1, AUROC, and TPR@FPR metrics.
• Results: DETree trained on RealBench achieves up to 97.75% supervised F1 and near-perfect OOD performance with limited support samples, showing strong generalization and resilience to adversarial attacks.

The dataset illuminates the distinct “signatures” of different collaborative workflows, pushing detection well beyond binary paradigms.

5. Repository-Level Code Generation under Realistic Software Development Practices

RealBench in software engineering provides the first code generation benchmark aligning with enterprise software practice: full repositories, natural-language requirements, and UML system designs (Li et al., 24 Apr 2026). This bridges the representational gap between prior single-file benchmarks and the true workflow of team-based, spec-driven software development.

Structure:

• Data: 61 real-world, GitHub-vetted Python repositories (avg. 1,201 LOC), accompanied by concise requirement docs, two-level UML extracted and human-corrected, and multi-level, human-verified test suites (avg. 50 tests, 79.8% coverage).
• Specification Format: JSON-formatted package and class diagrams capture architectural structure, exports, and relations.
• Generation Strategies:

1. Holistic: Single-shot full-repo generation.
2. Incremental: File-by-file, in UML order.
3. Retrieval-Augmented: Context-augmented by already-generated modules.

• Metrics:
  • Class-level: Completion@1, Execution@1, Pass@1 (all tests passed).
  • Repo-level scores: Human-graded Requirement@1 and Architecture@1, Grammar and Logic Error Rates (GER, LER), and AST-based F1.
• Empirical Findings: Top LLMs (GPT-4o: 20.1% Pass@1) generate correct module skeletons but falter in logic (LER up to ~60%) and syntax (GER up to ~40%), especially for large codebases (>1000 LOC). Pass@1 rates fall to ≈10–15% for projects over 2000 LOC.

This suggests that current LLMs are far from automating real-world repo synthesis and highlights best-practices: holistic generation for small repos, incremental for large, always including complete UML, and aggressive error mitigation via postprocessing and test-first prompting.

6. Operational-Realistic Numerical Weather Forecasting Evaluation

RealBench for AI weather forecasting addresses the operational disconnect of earlier benchmarks by introducing a testbed based on true real-time constraints, held-out 2025 atmospheric states, and direct verification with global in-situ station networks (Li et al., 24 May 2026).

Benchmark Structure:

• Data:
  • Training/Validation: Historical ERA5 and operational ECMWF analysis up to 2024.
  • Test: Strictly OOD, covering 2025 (no leakage).
  • Verification: ECMWF operational analyses (±3h 4D-Var, true real-time) and 11,539 WEATHER-10K station records.
• Metrics:
  • Global: Weighted RMSE, ACC, Bias, spatial Activity.
  • Event-centric: Object-based CSI/POD/FAR for heatwaves/cold surges, and DPE, MAE, Bias for tropical cyclone position/intensity.
• Findings:
  • Performance is significantly worse against station observations compared to grid-based reanalyses; for 2m temperature, RMSE increases by 0.3–0.5 K.
  • For extreme events, AI models (e.g., AIFS, GraphCast) suffer sharp drops in object CSI (e.g., 0.513→0.215 for heatwaves from day 1 to 10) and systematically underpredict intensity.
  • Direct evaluation with near-real-time analysis reveals unaccounted-for positive biases in cyclone intensity and degraded anomaly correlation for surface winds.

The benchmark encourages architectural innovations such as event-specific losses, more aggressive downscaling, and verification against multi-source observation networks.

7. Synthesis and Significance across Domains

Collectively, RealBench benchmarks operationalize a philosophy of “evaluation under real-world constraints”: authentic task data, multi-modal specification, verification mirroring actual deployment, and metrics aligned with target use-cases. The core model limitations repeatedly surfaced—be it context-length bottleneck (Verilog, codegen), modality fusion challenges (image/diagram reasoning), semantic hallucinations (hardware, translation), or failure to capture extreme outcomes (weather forecasting)—support the thesis that realistic benchmarking is essential for genuine progress toward deployable AI. Future iterations of RealBench are expected to broaden scope (more domains, modalities, languages), enhance specification realism, and refine verification protocols.