Power Systems Agent Benchmark
- Power Systems Agent Benchmark is an executable evaluation framework that verifies AI agents' structured outputs using deterministic physics-based criteria.
- The framework covers 41 task families across eight power engineering domains, ensuring realistic assessments of network operations, protection, and stability.
- It employs private-seed generated held-out cases and strict feasibility checks to resist training-data contamination and ensure reproducible performance.
The Power Systems Agent Benchmark is an executable evaluation framework for AI agents in electric power engineering. Rather than grading prose or requiring a single reference answer, it gives an agent a structured task, requires a structured solution, and uses a deterministic evaluator to recompute engineering quantities, check operational constraints, and return a feasibility flag, a normalized score, and explicit violations. The benchmark contains 41 task families across eight areas of power engineering, from power flow and protection to stability, microgrids, reliability, power quality, and forecasting, with each family grounded in a citable source, standard, or documented engineering formulation (Trashchenkov, 18 Jun 2026).
1. Evaluation philosophy and departure from text-based assessment
The benchmark is built around the claim that standard question answering is a poor fit for power engineering because many tasks admit multiple valid solutions, while correctness is physical rather than textual. In this view, a relay setting, dispatch, or restoration plan is correct only if it satisfies the constraints of the actual network model. The benchmark therefore adopts the executable-evaluation logic used in software-oriented settings and applies it to power engineering: the effect of an action is judged, not the fluency of the explanation (Trashchenkov, 18 Jun 2026).
This design also distinguishes the benchmark from earlier power-domain evaluation styles described as mostly textual or narrow. Within that comparison, PowerAgentBench is identified as the closest prior evaluator-based effort, but the Power Systems Agent Benchmark is characterized as having broader coverage, seed-generated held-out cases, an agent-agnostic CLI protocol, and explicit quality-control procedures. ElecBench is described as relying more on text or judging for dispatch-like scenarios, while PFBench is described as programmatic but restricted to power flow. A common misconception is therefore that power-system agent evaluation is equivalent to answer grading or solver benchmarking; the benchmark is explicitly intended to test whether an agent can return a structured engineering artifact whose feasibility can be recomputed.
The resulting emphasis is operational rather than rhetorical. The benchmark’s central contribution is to move evaluation away from whether a model can talk about engineering and toward whether it can produce a solution that respects the physics and constraints of the grid.
2. Task contract, scoring semantics, and evaluator model
The benchmark uses a single linear contract. An agent receives a task.json, writes a structured submission.json, and the deterministic evaluator returns an evaluation-result.json. The task object contains identity and family, public inputs, submission schema, hard constraints, metrics, provenance, and a bilingual presentation block. The solution object contains the structured answer and an optional rationale. The result object contains a feasibility flag, a score in , explicit violations, and supporting evidence. The explanation is recorded but excluded from scoring (Trashchenkov, 18 Jun 2026).
Feasibility is strict. A submission is feasible only if it violates none of the hard constraints. If infeasible, its score is zero, even when it is close to the optimum. The score is normalized relative to the reference optimum: optimal solutions receive 1, feasible but suboptimal solutions receive a smaller fraction, and a submission that exceeds the reference optimum is capped at 1 while treating the reference as non-optimal for that case.
The scoring mode depends on task type. Exact numeric tasks are checked against tolerances using a 0.1% relative band plus a small absolute floor. Optimization and control tasks are scored by objective gap to the optimum. Classification and localization tasks use exact set matching with misses enumerated. Standards-compliance tasks are pass/fail with diagnostics. At run level, the benchmark reports feasible rate, perfect rate, and mean score; these are intentionally separate because a run may contain feasible but non-optimal solutions.
The current evaluators are described as compact deterministic surrogates rather than full simulator-backed checks. Some evaluators are closed-form textbook calculations, some are simplified optimization or linearized checks, and none yet invokes a full power-system simulator. The design, however, is explicitly extensible through a maturity ladder from closed-form or graph-based surrogate, to deterministic engineering model, to full simulator-backed check, to stochastic or large-scale scenario suite. Natural future backends named in the paper include pandapower, MATPOWER, OpenDSS, PowerModels / PGLib, ANDES, and PyPSA, but these are not used in the current version.
3. Task coverage, engineering grounding, and provenance
The benchmark’s 41 task families are organized into eight engineering areas, which makes its scope substantially broader than a single-study benchmark. Every family was checked for physical correctness against its source before inclusion, and every family is attached to a provenance entry consisting of source title, locator, a mapping note explaining how the source becomes an executable case, and a confidence level (Trashchenkov, 18 Jun 2026).
| Area | Count | Families |
|---|---|---|
| Network analysis and operating constraints | 5 | ybus_construction, pf_voltage_repair, contingency_screening, corrective_transmission_switching, dynamic_thermal_rating |
| Short circuit and protection | 6 | three_phase_short_circuit, earth_fault_calculation, breaker_relay_short_circuit, distance_protection_settings, overcurrent_protection, cable_ampacity_sizing |
| Stability, grid code, and inverter-based resources | 5 | critical_clearing_time, transient_stability_prediction, frt_compliance, ibr_short_circuit_frt, min_synchronous_share |
| Distributed resources, PV, EV, and storage | 5 | pv_volt_var, ev_v2g_outage_schedule, ev_v2g_voltage_support, bess_ancillary_response, commercial_pv_lcoe_uncertainty |
| Microgrids and dispatch | 6 | microgrid_economic_dispatch, rolling_microgrid_dispatch, islanded_microgrid_pq_dispatch, dispatch_uncertainty, hydro_thermal_storage_uc, n1_generation_commitment |
| Reliability and restoration | 5 | flisr_restoration, fault_section_localization, fci_placement, fci_saidi_caidi, operator_breaker_load_actions |
| Power quality, standards, assets, and cybersecurity | 7 | en50160_voltage_compliance, power_quality_event_classification, harmonic_ieee519_compliance, transformer_thermal_loading, fdi_state_estimation, protected_meter_placement, mas_frequency_handover |
| Forecasting under uncertainty | 2 | wind_power_forecast, wind_prediction_interval |
The provenance rubric has three levels. “High” denotes direct reproduction of a standard or canonical textbook method, “medium” denotes a primary source or standard engineering formulation, and “low” denotes a faithful adaptation of standard practice when no single document fully pins down the exact formulation. The reported distribution is 11 high, 28 medium, and 2 low. The two low-confidence families are dispatch_uncertainty and min_synchronous_share.
The cited source base spans standards and textbooks central to power engineering. Examples include IEC 60909 for short-circuit calculation, IEC 60076-7 for transformer loading, IEEE 738 for dynamic thermal rating, IEEE 519 for harmonic compliance, EN 50160 for voltage quality, IEC 60364-5-52 / IEC 60949 for cable sizing, IEEE 1366 for reliability indices, IEEE 2800 and ENTSO-E RfG for fault ride-through, and textbooks such as Kundur, Horowitz and Phadke, Abur and Expósito, and Anderson. This grounding is significant because it makes the benchmark a standards-linked engineering evaluation rather than a synthetic puzzle set.
4. Held-out synthesis, contamination resistance, and benchmark quality control
A major design feature is that the held-out split is not stored as fixed data. Each task family has a generator, and held-out instances are synthesized on demand from private seeds. The generator is inspectable, but the actual instances remain unknown to users. Generation proceeds through physics-aware perturbations of public cases, including network parameters, weather series, load profiles, fault locations, and indicator readings and customer complaints implied by those faults (Trashchenkov, 18 Jun 2026).
A candidate held-out draw is accepted only if three conditions are met: the reference solver can solve it, the evaluator declares it feasible, and no small perturbation of the solution scores higher. The stated goal is to ensure that held-out cases are solvable, diverse, not near-duplicates of the public split, and resistant to training-data contamination. Because the seed remains private while the mechanism remains inspectable, the design is placed in the same family as dynamic benchmarks such as Dynabench and LiveBench.
The benchmark also treats evaluation as a quality-control instrument. A methodological lesson stated in the paper is that unanimous failure by all agents on the same task should trigger suspicion about the task or evaluator rather than immediate attribution of the failure to the models. This diagnostic process uncovered specification defects such as mismatched fields, unstated conventions, and over-strict comparisons; after correction, agents recovered.
The most prominent example is a held-out faulted-section localization case in which all three reference agents identified the same faulted section, but the evaluator demanded a different one. The defect was traced to an off-by-one error in the indicator model. Because the self-checking reference solver used the same underlying function as the evaluator, solver and evaluator agreed with each other and the defect was invisible to self-consistency checks. It was exposed only because independent agents, using correct physics, disagreed with the evaluator. This episode clarifies a broader point: executable evaluation does not eliminate benchmark errors, but it creates new opportunities to detect them through diverse agent cross-checks.
5. Reference evaluation, ceiling effects, and harness dependence
The reference evaluation used three command-line agents on the public case of each family and on one freshly generated held-out case per family: Codex (codex-cli 0.139.0) on gpt-5.5, Cursor (cursor-agent 2026.06.12) on composer-2.5, and OpenCode (opencode 1.15.13) on deepseek-v4-flash-free. The reported metrics are feasible rate, perfect rate, and mean score out of 41 tasks (Trashchenkov, 18 Jun 2026).
| Agent | Public: feasible / perfect / mean | Held-out: feasible / perfect / mean |
|---|---|---|
| Codex | 40/41, 38/41, 0.973 | 41/41, 40/41, 0.995 |
| Cursor | 40/41, 39/41, 0.973 | 41/41, 41/41, 1.000 |
| OpenCode | 39/41, 39/41, 0.951 | 40/41, 39/41, 0.973 |
The main empirical takeaway is that the strongest agents are near ceiling, public and held-out performance are broadly consistent, and OpenCode trails modestly but remains strong. The paper also states that the held-out split was not harder on average and was marginally easier in this snapshot. For that reason, the benchmark is explicitly not presented as a frontier-stumping leaderboard in its current compact tier. Its stated value lies instead in executable evaluation feasibility, contamination resistance, and practical filtering among agents that may all appear competent in textual settings.
A second empirical result concerns harness effects. On a separate public-split grid, OpenCode outperformed Aider across four models: Kimi K2.5 scored 0.93 versus 0.85, GLM-4.6 scored 0.78 versus 0.44, Claude Haiku 4.5 scored 0.71 versus 0.51, and Qwen3-Coder-Next scored 0.66 versus 0.53. The benchmark therefore supports a distinction between model capability and agent implementation. The same base model can behave like a different agent depending on the CLI or scaffolding, and the benchmark records that dependence rather than abstracting it away.
6. Place within the power-systems benchmark ecosystem
The Power Systems Agent Benchmark belongs to a broader but heterogeneous lineage of power-systems benchmark design. A literature-based feasibility study on Multi-Agent Systems in Bangladesh is not itself a benchmark paper, but it is explicitly described as a benchmark-design source because it identifies power-system problems that agent systems are expected to solve, the architectures and algorithms already used in the literature, and operational constraints that a benchmark should reproduce. It organizes MAS applications into protection, control, monitoring, network operation, network management, electricity market, and planning, and it highlights benchmark-relevant properties such as decentralization, distributed renewable integration, resilience, self-healing, and communication-constrained coordination. This suggests a conceptual precursor to the executable benchmark’s multi-area scope and its interest in realistic grid constraints (Hassan et al., 2022).
Several neighboring 2026 benchmarks target more specific agentic workflows. PowerAgentBench-SS defines a steady-state framework with a public tool API, hidden evaluator, evidence log, validation budget, and risk-sensitive metrics such as submitted recall, evidence-backed recall, found recall, false-safe penalties, severity regret, and mitigation reduction; its pilot is a DC thermal contingency-search task on deterministic IEEE 39-bus variants (Mylonas et al., 17 Jun 2026). PowerAgentBench-Dyn addresses dynamic studies through interactive environments with constrained action spaces, simulation budgets, and repeated-run success rates; its initial tasks are Dynamic Model Quality Review and Dynamic Security Risk Screening (Zhang et al., 18 Jun 2026). PFAgent contributes a deterministic multi-turn scenario suite on ANDES in which an agent must generate executable code, preserve cross-turn continuity, and return machine-checkable JSON results (She et al., 12 Apr 2026). EnergyAgentBench shifts the focus toward live energy-infrastructure analysis, with 70 task variants across datacenter siting, long-horizon portfolio reasoning, lifetime LCOE ranking, and causal grid diagnosis (Curcio, 13 May 2026).
Other benchmark substrates predate these agent-focused systems and define adjacent evaluation traditions. PowerGridworld is a lightweight, modular, customizable framework for constructing power-systems multi-agent reinforcement-learning environments rather than a fixed benchmark suite, making it a benchmark-enabling substrate for heterogeneous, power-flow-coupled tasks (Biagioni et al., 2021). The EEA-ENB is a 26-area, real-data-based benchmark for distributed Load Frequency Control with renewable energy sources and energy storage systems, using centralized MPC as a performance and computation-time reference for distributed control methods (Riccardi et al., 2024). The Hycon2 Power Network System benchmark targets the AGC layer in interconnected power systems and provides centralized MPC milestone performance across three scenarios (Riverso et al., 2012). EMSx is a numerical benchmark for microgrid energy management over 70 industrial sites, with explicit dynamics, uncertainty, and controller scoring for battery-charging decisions under PV and load forecasts (Franc et al., 2020).
Taken together, these works indicate an evolution from control benchmarks and environment-building frameworks toward executable, tool-grounded, auditable agent benchmarks. Within that trajectory, the Power Systems Agent Benchmark is characterized by breadth across subfields, deterministic recomputation of feasibility, private-seed held-out generation, and explicit use of benchmark execution as a mechanism for both evaluation and quality control (Trashchenkov, 18 Jun 2026).