A Benchmark on LLM-Based Power Flow Computation: Do More Structured Prompts Help?

Abstract: We present a controlled benchmark evaluating three LLMs -- Claude Sonnet 4.5, Gemini 2.5 Pro, and GPT-3.5 Turbo -- across four prompt formats (from concise narrative to structured JSON with explicit iteration trace) on Gauss--Seidel AC power flow computation for a three-bus system. Against 50 test cases with reference solutions computed numerically, Gemini 2.5 Pro with the simplest narrative prompt achieves the lowest mean absolute error (MAE = 0.257 MW/MVar, 54\% of cases within 5\% relative error), while the same model with a JSON-structured prompt raises MAE to 0.789 -- a 3.1$\times$ increase. Adding a worked example degrades accuracy for Gemini but provides a marginal gain for Claude. GPT-3.5 Turbo fails on at least 90\% of cases under all prompt formats. An independent 100-case replication with related prompt-format families confirms the qualitative ordering (Gemini $>$ Claude $>$ GPT-3.5): the best 100-case configuration (Gemini with explicit iteration trace) achieves MAE = 0.402 and 53\% within 5\%, while Claude Sonnet 4.5's near-flat accuracy profile ($\approx$38\% within 5\% across formats) and GPT-3.5's near total ineffectiveness (92--97\% above 20\% error) both replicate. In neither evaluation does any configuration achieve sufficient reliability for use as a direct numerical solver. These findings offer a diagnostic baseline for practitioners and researchers evaluating LLMs for smart-grid decision-support assistance.

• The paper establishes a benchmark for evaluating LLM performance on Gauss–Seidel power flow computations using various prompt structures.
• It compares models like Gemini 2.5 Pro, Claude Sonnet 4.5, and GPT-3.5 Turbo, revealing model-dependent sensitivities and error distributions.
• Results show that increased prompt structure can degrade performance, emphasizing the need for empirical prompt-model tuning in smart-grid computations.

Benchmarking LLM-Based Gauss–Seidel Power Flow Computation: Prompt Structure and Model Reliability

Introduction

This study establishes a rigorous empirical benchmark assessing LLM performance on the direct execution of Gauss–Seidel AC power flow calculations for a three-bus system. Motivated by practical decision-support requirements in smart-grid contexts, the authors systematically evaluate three LLMs—Gemini 2.5 Pro, Claude Sonnet 4.5, and GPT-3.5 Turbo—across four prompt paradigms, ranging from concise narrative to highly structured JSON input with explicit iterative trace output requirements. Unlike prior work, which predominantly examines LLMs as natural-language interfaces or retrieval/classification agents, this work isolates LLMs' ability to serve as unassisted iterative numerical solvers, thereby constituting a controlled testbed for prompt and model interaction in a nontrivial engineering computation regime.

Experimental Setup and Methodology

The benchmark is instantiated on a three-bus system (one slack bus, two PQ buses) with randomized, physically plausible line and load parameters. The test set comprises 50 primary and 100 replication cases where the target is the post-convergence slack-bus active ($P_g$) or reactive power ($Q_g$). For each case, ground-truth is calculated by a high-precision NumPy implementation, and LLM results are compared against this baseline. Evaluation metrics include mean absolute error (MAE), relative error bins, and parseability (robustness to formatting/output errors).

Prompt formats are explicitly delineated along two axes: input representation (natural language vs. machine-readable) and output specification (final value vs. convergence trace). The four configurations are:

• Narrative–Final: Textbook-style question, final answer only.
• Narrative+Example: As above, but prepended with a worked example.
• Procedural–Text: Numbered procedure, bullet-point data, per-iteration trace plus final answer.
• Structured–JSON: JSON-encoded input/output, with procedural description and trace request.

Model Performance across Prompt Structures

The results demonstrate marked differences both in overall accuracy and in prompt sensitivity across model tiers. Gemini 2.5 Pro achieves the best single-configuration performance (narrative–final, MAE = 0.257 MW/MVar, 54% of cases within 5% relative error; see Figure 1), but exhibits high prompt-sensitivity: the transition from narrative–final to structured–JSON increases MAE by a factor of 3.1, accompanied by a sharp rise in large errors and unparseable outputs. This quantitatively undermines the common assumption that maximal explicitness or step-wise scaffolding improves accuracy for arithmetic tasks.

Figure 1: MAE heatmap for 12 LLM–prompt configurations on 50 three-bus Gauss–Seidel power flow cases.

Claude Sonnet 4.5 demonstrates near-flat accuracy profiles irrespective of prompt type, consistently placing ≈38% of cases within 5% relative error and yielding a narrow MAE spread (0.545–0.772). Notably, procedural-style prompts, which degrade Gemini's performance, marginally benefit Claude. In contrast, the baseline GPT-3.5 Turbo model performs poorly in all settings (MAE > 1.8 in all configurations); 90–96% of its outputs exceed 20% relative error regardless of prompt design.

Prompt–model interaction is nontrivial: for instance, including a worked example improves Claude's accuracy but significantly worsens Gemini's. These divergent effects likely reflect different internal attention and calibration mechanisms, or, alternately, the impact of pretraining distribution and in-context anchoring.

Error Distributions and Output Characteristics

Error distributions underscore the model–format interaction. For Gemini 2.5 Pro under the optimal prompt, half of outputs cluster below 5% relative error, but a nontrivial tail remains (see Figure 2). With increasing prompt complexity, the error distribution shifts, and catastrophic errors or failures become dominant. For Claude, distributional shape remains consistent across formats, confirming format insensitivity is not a sampling artifact.

Figure 2: Distribution of relative errors for all model–prompt configurations on 50 cases; high error and failure rates dominate for GPT-3.5 Turbo.

Distinct output variable types also matter: recovering reactive power ($Q_g$) is systematically harder for all models than real power ($P_g$), with up to 5× higher MAE for $Q_g$. This is plausibly attributable to bias in training corpora or inherent difficulty in complex-number arithmetic, particularly imaginary component extraction.

Generalization and Replication

A 100-case replication robustly reproduces all primary findings. The model capability ordering (Gemini ≫ Claude > GPT-3.5) persists, with no model+prompt configuration achieving solver-grade accuracy (<1% relative error in most cases). Claude again remains invariant to prompt details, while Gemini's optimal prompt shifts from narrative–final to procedural–text (MAE = 0.402 on 100 cases, 53% of cases within 5%, see Figure 3), suggesting that distributional properties of the sampled instances or subtle prompt length effects influence performance.

Figure 3: Error-bin distribution for 100-case replication; all models exhibit significant large-error and failure rates, with overall patterns recapitulating main experiment.

Implications and Discussion

Strong empirical claims: Increasing prompt complexity, especially structured or verbose procedural prompts, do not necessarily improve and may degrade LLM performance on iterative numerical tasks for models of Gemini’s and Claude’s class. LLMs fine-tuned for reasoning on closed-form math benchmarks cannot be assumed to generalize to iterative engineering algorithms, even with worked examples or detailed procedures.

Practical implication: No tested model–prompt combination attains reliability suitable for direct solver replacement in industrial or operational settings. LLMs retain diagnostic or didactic value, potentially assisting with approximate reasoning or result explanation, but must not be trusted for critical smart-grid computation.

Theoretical implication: The interaction between representation format and model architecture/capability is highly non-monotonic and model-dependent. Prompt engineering for numerical solvers should be evaluated empirically per-model and per-task, with no expectation of monotonic improvement with increasing structure or explicitness.

Conclusion

This work establishes a detailed controlled benchmark for LLM-based Gauss–Seidel power flow computation, systematically evaluating prompt format effects and their interaction with model capability. Results reveal that prompt–model interactions are complex and often counterintuitive, with substantial performance drops observed for more structured or explicit task configurations. LLMs examined herein fall short of providing solver-grade numerical reliability, though differences in their prompt-sensitivity patterns suggest opportunities for future research on context calibration or hybrid program-aided pipelines. These findings provide a diagnostic baseline for future LLM development and deployment in smart-grid and engineering domains.