ArchEval: Diverse Evaluation Frameworks

Updated 7 July 2026

ArchEval is a term that denotes evaluation approaches across multiple domains, including LLM-based computer architecture benchmarks, production CMS assessments, and web archivability analysis.
It emphasizes dynamic, workflow-level evaluations using rigorous protocols, canonical verifiers, and baseline-normalized performance metrics.
Its methodologies support detailed post-session analysis, production-derived feedback, and archival fidelity assessment to drive practical improvements.

ArchEval is a name used for several evaluation-oriented constructs in recent research. One recent usage denotes a benchmark and platform for measuring LLM agents as computer architects, where architectural work is treated not merely as code generation or parameter tuning but as a process involving workload interpretation, mechanism selection, simulator use, performance prediction, hard-constraint satisfaction, and artifact production (Wang et al., 3 Jul 2026). In other work, ArchEval denotes the production-trace-based evaluation layer used to assess the Archi support agent for CMS Computing Operations (Lugato et al., 3 Jun 2026), and, in a web-archiving context, it denotes reasoning about website archivability as a property that changes over time as sites move from static HTML toward client-side JavaScript, Ajax, and interaction-triggered loading (Kelly et al., 2013).

1. Disambiguation and scope

The term has distinct meanings across the literature and should be interpreted from context.

Usage	Context	Source
ArchEval	Benchmark and platform for evaluating LLM agents as computer architects	(Wang et al., 3 Jul 2026)
ArchEval	Evaluation layer and question corpus for the Archi CMS CompOps deployment	(Lugato et al., 3 Jun 2026)
ArchEval	Reasoning about website archivability as a property that changes over time	(Kelly et al., 2013)

These usages share an emphasis on evaluation under realistic operational constraints, but they do not describe the same artifact. In computer architecture, ArchEval is a benchmark suite with simulators and verifiers. In scientific operations, it is a production-derived benchmark and judging framework. In web preservation, it is an analytic lens for understanding why the archivability of pages changes as publishing practices evolve.

2. Benchmarking AI agents as computer architects

In the computer-architecture sense, ArchEval was introduced to fill a gap left by prior benchmarks that evaluate finished artifacts, isolated knowledge, or agents operating inside a prebuilt design-space exploration loop. The benchmark asks whether an LLM agent can do the work of a computer architect: interpret the workload, choose mechanisms, build or use simulator workflows, predict performance before feedback, obey hard constraints, and decide which feasible design is worth evaluating.

The suite contains 20 challenges spanning five domains: CPU core mechanisms, system architecture, memory systems, accelerators, and compute-in-memory. The challenge set includes branch prediction, BTB design, LLC replacement, prefetching, co-design of multiple core mechanisms, cache hierarchy configuration, distributed training and collective communication choices, DRAM controller tuning, RowHammer mitigation, systolic-array partitioning, DNN layer mapping, ReRAM CIM design-space search, and NN + CIM co-design. The eight integrated simulators are ChampSim, gem5, Ramulator, DRAMSys, ASTRA-sim, SCALE-Sim, Timeloop, and MNSim (Wang et al., 3 Jul 2026).

A defining feature of the benchmark is that it treats architecture work as a process rather than a single terminal submission. Each challenge is posed under three support levels. In L1 full harness, the agent receives a prepared optimization loop, starter files, a narrow interface, a complete simulator and evaluation toolchain, and repeated verifier feedback. In L2 simulator-code container, the agent receives simulator source and a build environment but must inspect code, run probes, and assemble its own workflow. In L3 agent-only, the agent receives only a static task package and no runnable simulator feedback before submission. This progressively removes scaffolding while holding the task constant, thereby separating assisted optimization from autonomous experimental workflow construction and pre-feedback judgment.

3. Experimental protocol and measurement logic

Each challenge is specified by a challenge card containing the prompt, deliverable format, workload evidence, simulator or verifier, metric, baseline, constraints, and visibility and feedback policy. Evaluation always uses a canonical verifier. Submitted artifacts are executed in isolation, and the verifier returns typed outcomes such as SIM_OK, BUILD_FAIL, VALIDATION_REJECT, and SIM_TIMEOUT.

ArchEval reports baseline-normalized performance rather than raw simulator metrics. Values above $1.0\times$ mean that the submission beats the baseline, whereas values below $1.0\times$ indicate underperformance. The score is defined as

$\text{score} = \frac{m_{\text{agent}}}{m_{\text{base}}}$

when higher is better, and

$\text{score} = \frac{m_{\text{base}}}{m_{\text{agent}}}$

when lower is better. The benchmark aggregates results using geometric mean, median, win rate over all 20 challenges, and hard-failure counts. Invalid submissions are not hidden; failures to build, timeouts, constraint violations, or unevaluable artifacts are reported explicitly.

The platform also records the full trajectory: prompt, rationale when visible, tool calls, simulator observations, file diffs, build logs, verifier outcomes, and timestamps. This supports post-session analysis through six rubric dimensions: task compliance, simulator and tool use, workload-grounded design, performance judgment, constraint awareness, and integrity and originality. The trajectory machinery is central to the benchmark’s design because it distinguishes an agent that genuinely constructs an architecture workflow from one that only produces a superficially valid final artifact (Wang et al., 3 Jul 2026).

4. Empirical findings on autonomy, prediction, and novelty

The initial evaluation compares four agent configurations: GPT-5.5 + Codex, Gemini 3.5 Flash, Gemma 4 31B, and Gemini 3.1 Flash-Lite. In L1, all four reach or exceed baseline overall. The reported suite geomeans are $1.74\times$ , $1.68\times$ , $1.75\times$ , and $1.00\times$ , with win rates of 85%, 85%, 85%, and 60%, respectively. This indicates that, when given a prepared optimization loop and repeated feedback, current agents can perform assisted optimization across diverse simulators.

L2 exposes a different capability boundary. Here the agent must turn simulator source into a working experiment loop. The suite geomeans are $1.88\times$ for GPT-5.5 + Codex, $1.90\times$ for Gemini 3.5 Flash, $1.0\times$ 0 for Gemma 4 31B, and $1.0\times$ 1 for Gemini 3.1 Flash-Lite. Process logs show limited simulator execution before first submission: 60% for Gemini 3.5 Flash, 30% for Gemma 4 31B, and 15% for Gemini 3.1 Flash-Lite. This suggests that reading simulator code is easier than converting it into actionable experimentation.

L3 is the sharpest stress test because the agent must predict performance without simulator feedback. Only GPT-5.5 + Codex remains above baseline, reaching a $1.0\times$ 2 suite geomean and a 65% win rate. Gemini 3.5 Flash, Gemma 4 31B, and Gemini 3.1 Flash-Lite fall below baseline at $1.0\times$ 3, $1.0\times$ 4, and $1.0\times$ 5, with win rates of 25%, 35%, and 30%. The benchmark therefore frames current agents as useful optimization assistants rather than autonomous architects.

A central empirical result concerns performance prediction. Across agents, the median relative error for self-evaluation of submitted designs is 81–99%, and only 0–22% of predictions contain the true verifier result inside the agent’s own uncertainty interval. Agents usually overestimate their own designs. For GPT-5.5 + Codex, the median error is 93%, the range hit is 15%, and the overestimate ratio is 65%. Agent-written surrogate models are also weak. GPT-5.5 + Codex produces an executable model on 18 of 20 L3 tasks but achieves within-session Kendall’s $1.0\times$ 6, cross-session agreement of $1.0\times$ 7, and a 15% performance-modeling pass rate. Gemma 4 31B reaches 5%, and both Gemini agents reach 0%.

Constraint handling is stronger when requirements are explicit and program-checkable. The paper gives a cache-replacement example in which an agent satisfies a 256 B SRAM metadata budget exactly. Even so, satisfying constraints does not imply high performance. The recorded trajectories also reveal integrity issues: some submissions copy starter code as final output, some artifacts do not match the written rationale, and some designs achieve validity mainly by exploiting interface details or by making trivial modifications. Novelty is particularly weak. Across audited code-authoring tasks, almost all gains come from recombinations of known mechanisms with parameter tuning. Only one submission in the suite is judged genuinely novel, and even that design performs at only $1.0\times$ 8 of LRU (Wang et al., 3 Jul 2026).

5. Production-grounded evaluation in agentic operations

In the Archi framework for CMS Computing Operations, ArchEval does not denote a standalone assistant. It denotes the evaluation layer and question corpus used to assess how well Archi answers authentic operator questions derived from production traces. The motivation is the operational complexity of CompOps, where documentation, tickets, logs, dashboards, APIs, and service-specific tools are fragmented, partially outdated, and distributed across time zones and teams; even a single task typically spans five or more sources.

Archi itself consists of a data manager, an agent runtime, and a user interface. The data manager ingests heterogeneous sources such as generic web scraping, JIRA tickets, git repositories, and local files, and stores text and vectors in PostgreSQL via pgvector. Retrieval can use BM25 keyword search, vector search, and metadata filtering. The agent runtime supports multiple LLM providers and tools, including MCP servers and custom tools, and deployments define agent classes in LangGraph. In the CMS CompOps deployment, the agent is a ReAct-style assistant that alternates reasoning steps with tool calls over an ingested corpus of roughly ten thousand documents and live tools for current operational state. Every tool call is recorded in a visible trace.

ArchEval’s evaluation methodology is explicitly production-derived. The authors report two question sets collected from production traces and curated by an experienced CMS operator: a 63-question set used for both human and automated grading, and a larger 270-question set, a superset of the 63Q, used only for automated grading. These questions cover factual lookup, live data queries, ticket investigations, procedure retrieval, exploratory investigation, and debugging. Of the 270 questions, 199 require no live tools and 71 require current live operational state. The paper compares Bare, Single-shot RAG, Agent-no-live, and Agent-with-live, the last of which matches the production deployment.

The judging setup combines blinded human review and automated LLM judges. For the 63Q human panel, CMS domain experts and operators score correctness and usefulness on 1–5 scales and rank responses per question, allowing ties. For automated evaluation, the rubric uses four dimensions—relevance, completeness, specificity, and helpfulness—each scored 1–5, with the paper reporting their unweighted mean; a fifth source-faithfulness dimension is reported separately when retrieved sources are returned. The reported results show a clear advantage for the production-like agentic configuration. On the 63Q human panel, GPT-5.5 Agent-with-live scores 4.25 on correctness and 4.07 on usefulness, with an average of $1.0\times$ 9 and the best rank among configurations. On the automated four-judge panel for 63Q, GPT-5.5 Agent-with-live leads at $\text{score} = \frac{m_{\text{agent}}}{m_{\text{base}}}$ 0. Production feedback complements these benchmark results: between 13 February and 29 April 2026, the assistant served 20 distinct users out of 22 with access, across 393 conversations and 598 user messages, with a mean response time of 107 seconds and 99 voluntary feedback events, 70% positive and 30% negative. The paper also concludes that locally hosted open-weight models are competitive and are “keeping pace with frontier models with a modest delay,” which is significant because they can run on CMS-internal infrastructure with no external API calls (Lugato et al., 3 Jun 2026).

A distinct but adjacent evaluation method is AgentArcEval, a scenario-based software architecture evaluation method for FM-based agents inspired by ATAM. It is not ArchEval, but it clarifies the wider evaluation landscape in which agent systems are assessed. AgentArcEval extends scenario-based architecture analysis to compound architectures, goal-directed autonomy, non-deterministic behavior, continuous evolution, guardrails, and runtime assurance. It provides a catalogue of 11 agent-specific general scenarios—accuracy, adaptability, efficiency, privacy, security, fairness, availability, observability, transparency, safety, and contestability—and is demonstrated through the Luna tax-copilot case study (Lu et al., 23 Oct 2025).

6. Archival and web-preservation usages

In a web-archiving context, ArchEval denotes a way of reasoning about website archivability as a property that changes over time. The core claim is that archivability declines when content is no longer directly accessible in the initial markup. Static pages composed of HTML, CSS, images, semantically structured content, and resources whose URIs are present in the initial source are generally easy to crawl and preserve. Pages become harder to archive when important content depends on Ajax-driven loading, DOM manipulation, interaction-triggered loading, runtime-generated URIs, hidden or conditional content, client-side gating, infinite scroll, lazy loading, or other runtime behaviors. The argument is tied explicitly to accessibility: if content remains broadly accessible in a standards-compliant sense, it is also generally easier for archiving systems to capture. The paper relates this to WCAG notions of perceivability, operability, and robustness. Its methodology uses the Memento framework to request archived versions via TimeMaps and TimeGates, samples roughly one memento per year when possible, and replays pages with and without JavaScript using PhantomJS while recording screenshots, HTTP response codes, and the resources needed to assemble the page. The case studies of Google Maps, YouTube, NASA, and the White House show several failure modes, including missing interactive behavior, runtime loading from the live web rather than the archive, resource-chain failures, and abrupt drops in preserved CSS resources (Kelly et al., 2013).

Several archival studies are adjacent to this usage because they evaluate preservation not only as storage of source files but also as fidelity, interoperability, and reusability. The ArchOnto work on Portuguese archives proposes a modular CIDOC CRM-based linked open data model, an intermediate ISAD Ontology for legacy ISAD(G) text fields, and explicit migration rules for automatic population. Its evaluation uses manual representation of a sample of 25 records to validate completeness and conformity, and its reported class and property counts show CIDOC CRM as the backbone, DataObject as essential for literal validation, and the ISAD Ontology as a transitional layer (Koch et al., 2023).

A different archival line treats annotations as first-class carriers of digital scholarship. That work argues that archives should preserve not only authoritative source versions but also query results, feature-value assignments, topic labels, keywords, and associated metadata and provenance. It develops a local, portable annotation model, connects portability to FRBR’s work-level distinction, and demonstrates annotation-driven applications for the CKCC and DTHB projects, including QFA and TKA (Roorda et al., 2014).

Another adjacent framework addresses aggregation across private, personal, and public web archives. It introduces the Memento Meta-Aggregator, StarGate, enriched TimeMaps using CDXJ-like syntax, archival query precedence and short-circuiting, and OAuth-based access attributes for private captures. The framework extends archival negotiation beyond time and makes privacy-sensitive aggregation a first-class concern (Kelly et al., 2018).

Taken together, these usages suggest that ArchEval functions less as a single canonical object than as a recurring label for evaluation regimes in which dynamic behavior, provenance, workflow fidelity, and constraint handling matter as much as final outputs. In computer architecture, this means simulator-grounded assessment of agentic design loops. In CMS operations, it means production-derived benchmarking of authentic troubleshooting work. In web preservation, it means analyzing whether archives can faithfully reconstruct what users originally saw as web technologies evolve.