PowerAgentBench-Dyn: A Benchmark for Agentic AI in Power System Dynamic Studies

Abstract: LLM-based agents are increasingly being used to automate multi-step engineering work flows by interacting with software tools, interpreting intermediate results, and autonomously planning subsequent actions. Power system dynamic studies represent a particularly promising yet largely unexplored application domain for these agents. Unlike static computational tasks, dynamic studies often require more time on model parameter calibration, engineering judgment, and decision making under constrained action spaces. This paper introduces PowerAgentBench-Dyn, a benchmark designed to evaluate Agentic AI systems on power system dynamic-analysis tasks. The benchmark targets problems that cannot be reduced to a single optimization or coding task, but instead require a type of reasoning, tool usage, and iterative experimentation routinely performed by experienced power system engineers. The proposed framework includes two initial benchmark tasks. The first, the Dynamic Model Quality Review Benchmark, evaluates agents' ability to validate and diagnose dynamic models based on model-quality compliance criteria specified by system operators. The second, the Dynamic Security Risk Screening Benchmark, assesses agents' capability to leverage semantic memory and a limited simulation budget to identify, rank, and analyze the most critical short-circuit contingencies from an unseen fault dataset, as well as propose and evaluate possible mitigation measures. For each task, we define the simulation environment, observation and action spaces, and evaluation metrics. The benchmark is reproducible in a metric-based sense: released cases and simulator settings define a deterministic evaluator, while stochastic agent behavior is assessed over repeated runs using success rates and other metrics. The benchmark supports the development of future Agentic AI for power system operation and planning.

• The paper presents a benchmark that formalizes interactive, engineering-centric tasks for power system dynamics using constrained actions and iterative simulation.
• The methodology leverages agentic AI with iterative simulator interactions to diagnose and repair dynamic models while complying with strict operational guardrails.
• Empirical results demonstrate that modern LLM agents achieve high workflow success and precise KPI estimation, paving the way for reproducible power system studies.

PowerAgentBench-Dyn: Agentic Benchmarking for Power System Dynamic Studies

Background and Motivation

Power system dynamic studies underpin grid reliability, especially as inverter-based resources (IBRs), storage, and flexible loads increasingly affect transient stability and dynamic security. Conventional AI approaches targeting power system dynamics typically restrict themselves to supervised learning paradigms for contingency classification or transient stability assessment relying on fixed training distributions and well-defined labels [10858603, mehrzad2023review]. However, real engineering workflows require multi-step, tool-driven decisions that encompass case preparation, disturbance selection, iterative simulation, model diagnosis, and constrained parameter edits—tasks fundamentally incompatible with deterministic scripts or one-shot optimizations.

Agentic AI systems, notably LLM-driven agents, have recently demonstrated the capability to handle such complex workflows by synergizing reasoning, tool usage, feedback-driven action revision, and iterative planning [react, agentbench, toolllm, swebench]. PowerAgentBench-Dyn (2606.20401) introduces the first benchmark specifically for evaluating agentic AI on power system dynamic-analysis tasks. Its aims are to (i) formalize interactive engineering-centric tasks with constrained action spaces and (ii) produce reproducible metrics that distinguish agent proficiency in multitool orchestration, outcome-oriented reasoning, and adherence to operational guardrails.

Figure 1: PowerAgentBench framework overview, illustrating environment structure and agent-tool interaction cycles.

Framework and Task Definitions

Interactive Environment Formalism

Each benchmark task is posed as an interactive environment $$\mathcal{T} = (\mathcal{O}, \mathcal{A}, \mathcal{E}, \mathcal{C}, \mathcal{G}, \mathcal{M})$$, where $\mathcal{O}$ is the observation space (plots, logs, status), $\mathcal{A}$ is a constrained action space (allowed tool invocations and parameter edits), $\mathcal{E}$ is the simulation environment (e.g., PSS/E, PowerFactory), $\mathcal{C}$ comprises user/engineering constraints (parameter lock files, iteration budgets), $\mathcal{G}$ is the hidden ground truth/evaluator, and $\mathcal{M}$ is optional semantic memory.

Benchmarks are strictly tool-grounded: the agent must interact with user-specified simulators and analysis platforms. Engineering guardrails are enforced, limiting actions to whitelisted parameters; all outcomes are scored using reliability- and constraint-oriented metrics such as pass/fail rates, waveform fits, ranking accuracy, mitigation success, and wall-clock completion times.

Dynamic Model Quality Review Benchmark

The Dynamic Model Quality Review Benchmark (DMQ Benchmark) assesses the agent's ability to diagnose and repair dynamic models using only allowed actions. For example, in the WECC solar PV case, agents are permitted to modify only four REECAU1 gains within a five-iteration budget. The DMView platform exposes model-quality tests (flat-start, voltage/frequency steps, ride-through) to the agent, which must then iteratively repair controller gains based on simulation feedback and DMView reports.

Agents are scored both on full-suite pass rate (all DMView tests passed within constraints), process metrics (iterations used, waveform fit improvement, LVRT recovery time), and strict compliance (only whitelisted parameters modified, evidence-based edits). Numerical results show that Anthropic's Opus 4.8 and Sonnet 4.6 achieve optimal repair in all runs with single-shot updates (median final gains reduced ~10x to pass criteria), whereas Haiku 4.5 occasionally fails within the action budget and demonstrates more cautious parameter reduction.

Dynamic Security Risk Screening Benchmark

The Dynamic Security Risk Screening (DSR) Benchmark task evaluates whether agents can use semantic memory and limited simulation budgets to identify, rank, and analyze critical short-circuit contingencies. The agent must query prior results, retrieve contextually similar fault cases, estimate KPIs (voltage extrema, stability flag, severity score), rank candidates, simulate critical scenarios, and evaluate mitigation measures—under stringent constraint files and simulator interfaces.

Performance metrics include top-$k$ contingency selection overlap, absolute KPI error relative to deterministic ground truth, mitigation success, and wall-clock time. Case studies with Opus 4.7, Sonnet 4.6, Gemini 3.1 Pro, GPT-5.5, Haiku 4.5, and Qwen2.5-Coder-7B show that proprietary LLMs achieve 90–100% workflow success rate, consistently ranking the most severe contingencies and correctly flagging instability. Open-weight models without sufficient tool orchestration fail to complete the full pipeline. KPI estimation error is minimal: voltage and severity scores closely match ground truth even in stochastic agent runs.

Implications and Future Directions

PowerAgentBench-Dyn concretely demonstrates that agentic AI can handle open-ended, constraint-driven workflows relevant to real-world power engineering. Unlike token-level accuracy or supervised classifiers, these benchmarks demand: (i) iterative simulator interaction, (ii) diagnostic reasoning, (iii) constraint compliance, and (iv) auditable process tracking. Full reproducibility is bifurcated between deterministic task configurations (model files, simulator versions, scoring code) and probabilistic agent outputs (LLM sampling, tool-call retry/recovery logic, simulation stochasticity).

The benchmark provides a reproducible platform for quantitative comparison of agentic systems, decoupling tool orchestration from outcome metrics. Practical implications span faster engineering workflows, improved dynamic model validation, and more robust dynamic security screening. Future directions include expansion to broader dynamic model families (EMT studies, oscillatory analysis, PMU replay), public/synthetic grids for benchmarking, and integration with advanced agentic frameworks and human-in-the-loop checkpoints.

Agentic evaluation, as exemplified here, is increasingly critical for deploying AI in operational power systems. It enables assessment of cognitive and practical engineering skills beyond deterministic optimization, fostering advancements in reliable, interpretable, and constraint-enforcing AI for complex cyberphysical domains.

Conclusion

PowerAgentBench-Dyn (2606.20401) formalizes agentic benchmarking for power system dynamic studies, shifting evaluation from static prediction or optimization to interactive multi-tool workflows with stringent constraints. Through two comprehensive tasks—dynamic model quality review and dynamic security risk screening—the benchmark quantitatively distinguishes agent proficiency in diagnosis, repair, ranking, and mitigation under operational guardrails. Empirical results validate the viability of contemporary LLM agents for practical power engineering workflows, laying a foundation for further advances in reproducible, auditable, and outcome-oriented AI in power system dynamics.