Attribution of reported ML gains to model design versus data-generation choices

Determine whether the performance improvements reported for machine-learning methods in steady-state transmission grid analysis are attributable to the models’ architectural/design choices or to the dataset generation assumptions (including loads, generator dispatch, and topology).

Background

The paper notes that many prior studies rely on custom, unpublished data pipelines with differing assumptions about loads, dispatch, and topology, which undermines reproducibility and fair comparisons. In this context, the authors explicitly state that it is unclear whether reported gains are due to model design choices or data-generation choices, identifying a concrete unresolved question about attribution of performance improvements.

gridfm-datakit is proposed as a unified, scalable platform to standardize and diversify data generation, potentially enabling more reliable benchmarking and the disentanglement of model effects from data effects. However, the causal attribution of performance gains remains explicitly identified as unclear in the current literature.

References

Limited incentives to generate diverse or complex scenarios also make it unclear whether reported gains arise from model design or data choices.

— gridfm-datakit-v1: A Python Library for Scalable and Realistic Power Flow and Optimal Power Flow Data Generation  (2512.14658 - Puech et al., 16 Dec 2025) in Motivations, bullet “Lack of reproducibility and benchmarking” (Section 2)

The repair is composite and the attribution ablation covers two cells and one equal-budget comparator that equalizes only the perturbation budget, so we cannot fully separate correct localization from perturbing the value block at all; and the two held-out statistics, though never used in fitting, diagnosis, or repair, were selected after the repair rather than pre-registered, making them an independence audit rather than a confirmatory test.

— Testing, not presuming, adequacy: calibrating generative social simulators against emergent network structure  (2609.24012 - Shao et al., 21 Sep 2026) in Limitations, “The repair attribution and held-out statistics are suggestive, not confirmatory”

Message-passing surrogates such as PowerFlowNet report 4--48× speed-ups over Newton--Raphson, iterative self-ensembling schemes improve physical consistency, and large GNN OPF solvers scale to thousands of buses and $N-1$ contingencies, though generalisation across topologies remains the open problem.

— Physics-Aware Distillation of a Grid Foundation Model for Real-Time Edge Inference  (2610.21833 - Sănduleac et al., 15 Oct 2026) in Section 2, State of the Art, paragraph “Learning-based power flow and grid foundation models”