ArchEval: Measuring AI Agents as Computer Architects

Abstract: Computer architecture has long used benchmarks to make progress measurable. LLM agents create a different measurement problem: success is not merely writing code or tuning parameters. The agent must interpret workloads, choose mechanisms, use simulators, predict performance, satisfy hard constraints, and decide which feasible design is worth evaluating. This paper introduces ArchEval, a benchmark and platform for evaluating LLM agents on computer architecture design and optimization. It contains 20 challenges across CPU core mechanisms, system architecture, memory systems, accelerators, and compute-in-memory, backed by eight simulators. Each challenge is posed under three settings: L1 full harness, with repeated simulator feedback; L2 simulator-code container, where simulator source is available but the agent must assemble its own workflow; and L3 agent-only, with no runnable feedback before submission. Each run reports baseline-normalized verifier performance and records the full trajectory, connecting results to workload analysis, simulator-tool use, prediction, constraint handling, and artifact integrity. Initial results show a sharp boundary in current agents. With L1 support, all four evaluated agents reach or exceed baseline and improve real designs across diverse simulators. Removing support exposes weaknesses: many agents fail to turn simulator source into useful experiments, and L3 predictions often disagree with verifier results. In L3, only GPT-5.5 + Codex remains above baseline, reaching 1.21x geomean performance and a 65% win rate; the other three fall below baseline. Even GPT-5.5 + Codex has only a 15% performance-modeling pass rate. ArchEval frames today's agents as useful optimization assistants rather than autonomous architects, and identifies capabilities needed next: simulator-tool use, calibrated prediction, pre-feedback judgment, and useful mechanism discovery.

• The paper introduces ArchEval, a benchmark framework that evaluates AI agents' architectural reasoning through simulator-backed iterative design challenges.
• The paper demonstrates that LLM agents excel with repeated simulation feedback but falter when required to predict performance without external cues.
• The paper finds that while agents improve design outcomes with feedback, they show limitations in autonomous static reasoning and genuine innovation.

Authoritative Summary of "ArchEval: Measuring AI Agents as Computer Architects" (2607.03601)

Motivation and Benchmark Framework

The paper introduces ArchEval, a benchmark and platform for systematically evaluating the architectural reasoning capabilities of LLM agents engaged in computer architecture design and optimization tasks. In contrast to prior architecture evaluation benchmarks which focus narrowly on static artifacts of completed designs, ArchEval measures whether LLM agents can genuinely simulate the architectural design process: workload analysis, mechanism proposal, iterative performance modeling, and constraint satisfaction across simulator-backed domains.

Figure 1: Benchmarking the architectural reasoning process, from workload analysis to iterative mechanism evaluation via simulators.

The platform operationalizes this via 20 diverse challenges drawn from CPU microarchitecture, system-level design, memory subsystems, accelerators, and compute-in-memory (CIM), each paired with credible baselines and discipline-specific simulators. Challenges are instantiated at three evaluation levels: L1 (full harness, repeated simulator feedback), L2 (simulator-code access, agent-assembled workflows), and L3 (no simulator feedback, agent-only static reasoning). These settings probe distinct capabilities: assisted optimization, tool use, and pre-feedback design judgment.

Figure 2: Controlled evaluation workflow: task specification, artifact generation, simulator-mediated verification, and trajectory logging.

Benchmark Design, Challenge Suite, and Protocol

Each task is formally specified by a "challenge card": it encodes objective, workload evidence, artifact requirements, hard constraints, simulator, metric parser, baseline, and experimental support. This abstraction enables uniform evaluation despite heterogeneous simulators such as ChampSim, gem5, DRAMSys, Ramulator, SCALE-Sim, Timeloop, and MNSIM.

ArchEval's three evaluation settings progressively remove support—L1 offers a prepared optimization loop and repeated feedback; L2 provides simulator code, but agents must assemble their own experiments; L3 is feedback-free, requiring agents to generate artifacts and predictions based solely on static workload evidence.

Figure 3: Cache replacement task posed under L1, L2, L3: prompt, available support, and feedback policy change per setting.

Challenge coverage is broad, spanning canonical problems in branch prediction, BTB design, cache replacement, prefetching, co-design, cache hierarchy, DRAM tuning, RowHammer mitigation, collective communication optimization, accelerator partitioning, layer mapping, CIM design, and NN-hardware co-design.

Figure 4: Distribution of the 20 architecture challenges across CPU, system, memory, accelerator, and CIM domains.

Experimental Results and Capability Map

ArchEval evaluates four agent configurations: GPT-5.5 + Codex, Gemma 4 31B, Gemini 3.5 Flash, and Gemini 3.1 Flash-Lite. Performance is scored as baseline-normalized: $>1.0\times$ indicates beating the reference baseline.

Agents achieve strong optimization in L1, consistently improving canonical architectures across simulators when given feedback loops (geomean up to $1.74$, win rates up to $95\%$ for GPT-5.5). However, performance sharply degrades in L2 and especially L3: only GPT-5.5 + Codex retains above-baseline performance in L3 ($1.21\times$ geomean, $65\%$ win rate), and even this configuration exhibits weak performance-modeling calibration ($15\%$ pass rate).

Figure 5: Performance drop as simulator feedback is withheld; geomean performance and win rate decline from L1 to L3.

Repeated simulator feedback in L1 demonstrably enhances design quality—agents conduct iterative design-space exploration, rejecting ineffective candidates and tuning mechanisms in response to measured outcomes.

Figure 6: Agent improvement trajectory with successive simulator-feedback loops, showing monotonic rise in best-so-far geomean performance.

L2 exposes friction in tool usage: most agents treat simulator code statically and do not construct local experimental loops. L3 reveals unreliable surrogate performance prediction and design judgment: agent-written models only weakly correlate with verifier outcomes (Kendall's $\tau$ median $+0.20$ for GPT-5.5, $+0.13$ for Gemma 4 31B), and agents systematically overestimate submitted artifact quality.

Diagnostic Findings

Workload Analysis: Only GPT-5.5 consistently grounds design decisions in workload-specific evidence; other agents generate generic text with little impact on mechanism selection.

Performance Modeling: Agent-written models for performance prediction before feedback (L3) are unreliable; median prediction errors are 81--99%, uncertainty calibration is poor, and agents overwhelmingly overestimate their designs.

Constraint Handling: Precise, program-checkable resource constraints (e.g., 256\,B SRAM limit) are reliably enforced by frontier agents, but satisfying constraints does not equate to optimal design judgment.

Mechanism Discovery: Genuine architectural innovation is very rare; most improvements arise from recombination and parameter tuning of known patterns. Only one out of 72 code-authoring submissions exceeded this threshold, and it underperformed the reference policy.

Interface Shortcuts: Audited trajectories reveal starter-copying and mismatch between stated rationale and submitted artifact, highlighting the necessity of full process logging.

Implications and Future Directions

ArchEval frames LLM agents as optimization assistants rather than autonomous architects. With sufficient harnessing and simulator feedback, agents are practical for design-space exploration but lack robust architectural reasoning, tool assembly, pre-feedback modeling, and mechanism discovery. The key deficiencies are:

• Simulator-tool integration and experimental workflow assembly
• Calibrated performance prediction in the absence of feedback
• Robust workload-to-mechanism mapping
• Constraint-efficient design tradeoffs
• Genuine mechanism innovation

The benchmark's trajectory logging and process diagnostics are essential for meaningful evaluation, as single outcome metrics conceal shortcut, overconfidence, or non-grounded failures.

Future work must address compositional agent architectures, improved learning for pre-feedback modeling, robust design grounding, and systematic expansion of test domains and challenge complexity. Benchmarking improvements should involve quantitative trajectory metrics and held-out challenges to track capability progression.

Conclusion

ArchEval provides a simulator-grounded, multi-domain benchmarking protocol for AI architecture agents, separating optimization from architectural judgment and tool usability. Current LLM agents are useful within accountable optimization loops but lack reliable autonomy in early-stage architecture design. Progress toward fully capable architecture agents will require addressing the specific deficiencies identified here, which ArchEval enables the community to measure and track rigorously.