---
title: 'AECBench: Benchmark for AEC LLM Evaluation'
url: https://www.emergentmind.com/topics/aecbench
type: topic
---

# AECBench: Benchmark for AEC LLM Evaluation

Searching arXiv for the benchmark and closely related work to ground the article.
arxiv_search.query({"search_query":"all:AECBench OR ti:\"AECBench\"","start":0,"max_results":10})
arxiv_search.query({"search_query":"ti:\"A Hierarchical Benchmark for Knowledge Evaluation of Large Language Models in the AEC Field\"","start":0,"max_results":5})
arxiv_search.query({"search_query":"ti:\"A Multimodal Benchmark for Agentic Systems in Architecture, Engineering, and Construction\" OR ti:\"AECV-Bench\" OR ti:\"Evidence-Centered Benchmark Design for NLP\"","start":0,"max_results":10})
AECBench is a domain-specific benchmark for evaluating large language models in the Architecture, Engineering, and Construction field, designed to measure performance on real-world, safety-critical engineering tasks rather than generic exam-style question answering. It was introduced as a Chinese benchmark with a five-level, cognition-oriented framework spanning Knowledge Memorization, Understanding, Reasoning, Calculation, and Application, and it uses a 4,800-question dataset derived primarily from authentic AEC practice and validated through a two-round expert review process [2509.18776].

## 1. Purpose and problem setting

AECBench was created in response to two perceived deficiencies in earlier AEC-oriented evaluation sets. First, prior benchmarks were described as covering only limited slices of AEC knowledge, often concentrating on a single subtopic or discipline. Second, they were described as insufficiently realistic because they relied heavily on multiple-choice questions, whereas actual AEC practice requires code interpretation, constraint reasoning, quantitative calculation, compliance checking, and long-form document generation [2509.18776].

The benchmark is explicitly grounded in the safety-critical character of AEC work. Within this framing, benchmark quality is not treated as a purely academic matter: incorrect outputs may correspond to code violations, structural or fire-safety risks, incorrect calculations, flawed design documents, or poor compliance judgments. Accordingly, AECBench is intended to support practitioners seeking model-selection guidance, researchers and developers seeking standardized weakness analysis, and regulators seeking evidence relevant to safety and governance frameworks [2509.18776].

Its central design premise is that AEC competence is hierarchical rather than monolithic. The benchmark therefore evaluates not only whether a model can recall domain facts, but also whether it can interpret provisions, reason over constraints, execute discipline-specific calculations, and produce professional outputs appropriate to document-centric workflows. This hierarchy is adapted from Bloom’s taxonomy but specialized for AEC use [2509.18776].

## 2. Hierarchical cognitive framework and task taxonomy

AECBench is organized as a five-level hierarchical cognitive framework. The five levels are Knowledge Memorization, Knowledge Understanding, Knowledge Reasoning, Knowledge Calculation, and Knowledge Application. The final level is further divided into Analysis, Evaluation, and Creation, reflecting practical professional workflows in AEC [2509.18776].

| Level | Focus | Representative tasks |
|---|---|---|
| 1. Knowledge Memorization | Recall of facts | Code Memorization Retrieval; Terminology Memorization; Abbreviation Memorization |
| 2. Knowledge Understanding | Interpretation and contextualization | Code Provision Interpretation; Design Literacy QA; Code Provision Interpretation (Tabular Data) |
| 3. Knowledge Reasoning | Inference and decision-making | Design Decision Formulation; Building Type Inference; Design Decision Formulation (Tabular Data) |
| 4. Knowledge Calculation | Quantitative problem-solving | Calculation in Architectural Design; Calculation in MEP Design; Calculation in Construction; Calculation in Engineering Economics |
| 5. Knowledge Application | Analysis, evaluation, creation | Document Classification; Document Proofreading; Compliance Checking; Brand Compliance Verification; Information Extraction; Evaluation of Documents in Architectural Design; Evaluation of Documents in Structural Design; Conceptual Design Proposal Generation; Specialized Report Generation; Tender Review Report Generation |

The benchmark contains 23 representative tasks distributed across these levels. At the Memorization level, tasks focus on basic recall, including code provisions, terminology, and abbreviations. At the Understanding level, tasks shift from recall to interpretation, including code provision interpretation and design literacy, with one task explicitly centered on tabular code data. The Reasoning level introduces logical inference and decision-making, including building type inference and a tabular-data variant of design decision formulation. The Calculation level spans architectural design, MEP design, construction, and engineering economics. The Application level then moves toward professional document workflows through classification, proofreading, compliance checking, evaluation, and open-ended creation [2509.18776].

A key feature of this taxonomy is that it treats AEC work as composite. The benchmark states that a typical engineering problem may require recalling a code clause, understanding its meaning, reasoning about constraints, carrying out calculations, and finally generating a compliant document. This suggests that the hierarchy is intended less as a strict partition of abilities than as a structured progression of increasingly integrated professional cognition [2509.18776].

## 3. Dataset construction, sources, and question formats

The benchmark comprises 4,800 questions and was constructed in three stages: data collection, data cleaning, and data review. The primary source material was questions crafted by domain engineers rather than merely scraped from tests. These engineers drew on building codes, proprietary internal AEC documents, domain-specific materials, and official licensure examinations, which anchors the dataset in authentic engineering practice rather than purely pedagogical assessment [2509.18776].

During data cleaning, the authors standardized the raw material and applied question deduplication using n-gram overlap and vector similarity, outlier detection to correct factual and spelling issues, and content completion to ensure answers and options were complete. Review proceeded in two rounds. The first round was conducted by mid-level engineers to assess reasonableness, identify obvious errors, and confirm topic coverage. The second round was conducted by senior experts with more than 10 years of experience to resolve ambiguities, identify subtle errors, and validate representativeness and technical accuracy. The reported example in which “university” was revised to “primary school” to remove code-related ambiguity illustrates the benchmark’s emphasis on precise interpretation in regulatory contexts [2509.18776].

AECBench also uses multiple question formats in order to better approximate professional AEC work. These formats are multiple-choice, classification, extraction, generation, and open-ended questions. Multiple-choice is used for most tasks in Memorization, Understanding, Reasoning, Calculation, and one Analysis task. Classification is used for document classification, extraction for information extraction, generation for proofreading and brand compliance verification, and open-ended formats for all tasks in the Creation sub-level. The stated rationale is to reduce multiple-choice biases such as lucky guessing and to better capture real professional complexity [2509.18776].

The dataset design therefore combines breadth of task type with procedural curation. In methodological terms, this places AECBench close to benchmark-design arguments that treat benchmarks as measurement instruments whose validity depends on explicit content choices, adaptation choices, and evidence extraction rules rather than on dataset size alone. That broader measurement perspective is articulated in “Evidence-Centered Benchmark Design for NLP” [2406.08723].

## 4. Evaluation protocol, scoring rules, and LLM-as-a-Judge

AECBench uses task-dependent evaluation metrics rather than a single universal scoring rule. Accuracy is used for multiple-choice tasks, F1 for classification, F0.5 for proofreading, Soft-F1 for some extraction and verification tasks, Kendall’s tau for evaluation tasks, and normalized scores for comparison across tasks. For generation tasks, reported scores are normalized to a 0–100 scale; when Kendall’s tau is positive, the score is computed as
$$
\text{Score} = \tau \times 100
$$
[2509.18776].

A major methodological contribution is its rubric-based LLM-as-a-Judge pipeline for long-form and open-ended responses. The process is two-stage: experts first define detailed rubrics specifying what counts as correct, partially correct, or incorrect, often decomposing a task into sequential steps with partial credit; an LLM then scores model outputs against those rubrics. This is used especially for generation tasks, where exact-match scoring would not capture factual correctness, code compliance, technical coherence, or professional expression [2509.18776].

The judge model was not selected arbitrarily. Multiple models were first evaluated on the Evaluation tasks—architectural document evaluation and structural document evaluation—and their outputs were compared with human expert scores using Kendall’s tau. DeepSeek-R1 was reported as the strongest judge and was subsequently used to score the generation tasks. The benchmark also studies judge bias, reporting that LLM judges tend to overestimate low-quality outputs and underestimate high-quality outputs, thereby compressing scores into a narrower range. To mitigate this, the authors applied isotonic regression and piecewise linear regression, improving mean absolute error from 2.947 to 1.926 and 2.015, respectively [2509.18776].

This scoring design makes AECBench notable within AEC evaluation because many target tasks are not naturally reducible to exact matching. It also aligns with broader benchmark methodology emphasizing that evidence extraction and evidence accumulation must be justified as appropriate to the capability being measured, rather than adopted solely by precedent [2406.08723].

## 5. Empirical results and observed failure modes

AECBench evaluates nine LLMs: Moonshot-v1-128k, GLM-4-Plus, Qwen-Turbo, QwQ-32B, DeepSeek-V3, DeepSeek-R1, GPT-4o, GPT o3-mini, and Hunyuan-TurboS. The central empirical pattern is a clear performance decline as cognitive complexity increases. Models perform well on memorization and basic understanding, but performance drops on reasoning, calculation, and application, especially on creation tasks. This is the benchmark’s main headline result: current LLMs can recall AEC knowledge more reliably than they can deploy it in realistic workflows [2509.18776].

The strongest area is memorization. The paper reports that all models score very highly on terminology memorization, with many exceeding 95%, which is interpreted as evidence that a substantial amount of AEC factual knowledge is already embedded in model parameters. By contrast, a major weakness appears in tasks involving tables in building codes, specifically code table interpretation and design decision formulation with tabular data. These perform markedly worse than expected, including relative to some application tasks, indicating that the bottleneck is not only logical reasoning but also the encoding and retrieval of tabular regulatory knowledge [2509.18776].

Additional weaknesses appear in complex reasoning, quantitative calculation, and practical compliance tasks. Architectural design calculation and engineering economics calculation are identified as the hardest calculation tasks because they require multi-step reasoning, formula selection, parameter handling, and numerical consistency. Compliance checking and brand compliance verification are difficult because they require the comparison of multiple pieces of structured information, frequently under table or code constraints. At the Application level’s Creation sub-level, conceptual design proposals, specialized reports, and tender review reports all remained below a 60-point threshold across models, indicating that long-form professional generation is still challenging [2509.18776].

Model-level differences are also informative. DeepSeek-R1 is the strongest overall performer and outperforms DeepSeek-V3 on 19 of 23 tasks, which the paper interprets as evidence that reasoning-oriented training materially improves AEC performance. Even so, DeepSeek-R1 still exhibits substantial weaknesses in table-heavy code interpretation, long-form generation, and advanced reasoning and calculation tasks. The paper also reports that performance on code-table tasks improves substantially when the table content is reformatted into more model-friendly representations, either through manual expert textual description or automated HTML conversion from table images, with the manual method performing better because automated conversion may lose layout details such as spanning cells. This suggests that some observed failures arise from access and representation problems rather than from complete absence of the underlying knowledge [2509.18776].

## 6. Significance, limitations, and relation to other AEC benchmarks

AECBench’s significance lies in its attempt to measure whether LLMs can assist in the building lifecycle under conditions that resemble professional practice. Its structured design separates failures of factual recall, misunderstanding, reasoning, calculation, and practical application, offering a more fine-grained view than aggregate exam-style scores. The benchmark’s conclusions are cautious: LLMs are promising for knowledge recall and some assistant functions, but they are not yet robustly reliable for high-stakes AEC deployment, particularly where table interpretation, numerical reasoning, compliance assessment, or professional document generation are involved [2509.18776].

The benchmark also identifies concrete methodological needs for future work: better reasoning and calculation, better handling of code tables, better long-form generation, calibrated automated evaluation, and multimodal benchmarks that include architectural drawings and other non-textual artifacts. These are limitations in both model capability and benchmark scope. The present benchmark is centered on textual and document-centric knowledge evaluation; it does not itself provide a multimodal test of drawing literacy or interactive coordination [2509.18776].

That limitation is clarified by comparison with adjacent AEC benchmarks. “AECV-Bench: Benchmarking Multimodal Models on Architectural and Engineering Drawings Understanding” evaluates multimodal and vision-language models on floor-plan object counting and drawing-grounded document QA, showing that OCR and text-centric QA are strongest while symbol-centric counting of doors and windows remains unsolved [2601.04819]. “AEC-Bench: A Multimodal Benchmark for Agentic Systems in Architecture, Engineering, and Construction” instead evaluates agentic systems on intra-sheet, intra-drawing, and intra-project coordination tasks over real construction documents, emphasizing retrieval, spatial grounding, and cross-sheet reasoning [2603.29199]. Taken together, these benchmarks suggest a division of labor within AEC evaluation: AECBench concentrates on hierarchical knowledge evaluation for LLMs, AECV-Bench on drawing understanding, and AEC-Bench on multimodal agentic coordination.

Within that broader landscape, AECBench functions as a foundational benchmark for textual and knowledge-centric assessment in AEC. Its practical contribution is not merely the 4,800-question dataset, but the argument that meaningful evaluation in a safety-critical engineering domain must be aligned with authentic tasks, expert review, and scoring procedures capable of handling complex professional outputs [2509.18776].

Source: https://www.emergentmind.com/topics/aecbench