gridfm-datakit-v1: A Python Library for Scalable and Realistic Power Flow and Optimal Power Flow Data Generation (2512.14658v1)

Abstract: We introduce gridfm-datakit-v1, a Python library for generating realistic and diverse Power Flow (PF) and Optimal Power Flow (OPF) datasets for training Machine Learning (ML) solvers. Existing datasets and libraries face three main challenges: (1) lack of realistic stochastic load and topology perturbations, limiting scenario diversity; (2) PF datasets are restricted to OPF-feasible points, hindering generalization of ML solvers to cases that violate operating limits (e.g., branch overloads or voltage violations); and (3) OPF datasets use fixed generator cost functions, limiting generalization across varying costs. gridfm-datakit addresses these challenges by: (1) combining global load scaling from real-world profiles with localized noise and supporting arbitrary N-k topology perturbations to create diverse yet realistic datasets; (2) generating PF samples beyond operating limits; and (3) producing OPF data with varying generator costs. It also scales efficiently to large grids (up to 10,000 buses). Comparisons with OPFData, OPF-Learn, PGLearn, and PF$Δ$ are provided. Available on GitHub at https://github.com/gridfm/gridfm-datakit under Apache 2.0 and via pip install gridfm-datakit.

• The paper introduces gridfm-datakit-v1, a unified Python library that generates scalable and diverse power flow and optimal power flow datasets for transmission grids.
• It employs hybrid load, topology, and admittance perturbations to simulate realistic scenarios including constraint violations and varying economic conditions.
• The framework supports large-scale grids, integrates with ML pipelines, and enhances reproducibility for grid-aware machine learning model evaluations.

gridfm-datakit-v1: Scalable and Realistic Data Generation for ML-based Power Flow and OPF

Introduction and Motivations

The deployment and benchmarking of ML approaches for steady-state transmission grid analysis—such as Power Flow (PF) and AC Optimal Power Flow (OPF)—require access to large, realistic, and diverse datasets. Existing data generation libraries fragment state-of-the-art perturbation methods across disparate ecosystems, are computationally restrictive in grid size, or fail to adequately capture the diversity and out-of-distribution scenarios critical for robust ML model evaluation. gridfm-datakit-v1 addresses these deficiencies by presenting a unified Python-based framework for scalable and realistic data generation, supporting both PF and OPF use cases for the transmission grid.

Figure 1: gridfm-datakit unifies and advances state-of-the-art methods for data generation that were previously scattered across existing libraries, providing the most scalable Python-based framework for generating both Power Flow and Optimal Power Flow datasets for the transmission grid.

Limitations of Existing Approaches

Contemporary libraries for synthetic grid data generation exhibit fundamental limitations:

• Fragmented perturbation modeling: Perturbation strategies for loads, topologies, generators, and admittances are implemented heterogeneously across more than ten libraries, limiting comprehensive experimentation.
• Restrictive scenario coverage: Most PF datasets only include operating points that are OPF-feasible, excluding violation cases (e.g., overloads, voltage excursions) and thus biasing ML training data. PF$\Delta$ [pfdelta] circumvents this but introduces unrealistic clusters of constraint violations.
• Under-diversified OPF datasets: Fixed generator cost functions across scenarios in OPF libraries (OPFData [lovett2024opfdatalargescaledatasetsac], PGLearn [pglearn], OPF-Learn [opflearn]) result in low generalization to varying economic conditions.
• Scale constraints: Data generation for large-scale grids (>2,000 buses) remains challenging in most open-source toolchains due to computational bottlenecks and design.

gridfm-datakit advances the state of the art by unifying complete perturbation strategies, enabling scenario diversity including violations, and enabling scalable data generation on grids up to 30,000 buses for PF, and 10,000 buses for OPF.

Design and Methodology

Grid Selection and Scalability

gridfm-datakit supports MATPOWER format inputs and all PGLib benchmarks. It scales up to 30,000 buses for PF and 10,000 for OPF, using PowerModels.jl [powermodels] as a backend, and achieves high sample throughput with fast convergence rates.

Hybrid Load Perturbation

Previous libraries have introduced either global or local load perturbation, but not both with realism and scalability. gridfm-datakit introduces a hybrid mechanism where each bus's nominal load is scaled by a global temporal reference derived from real-world system-level aggregated profiles (e.g., from ERCOT), and further perturbed by multiplicative local noise, jointly preserving both spatiotemporal correlation and scenario diversity.

Topology and Admittance Perturbations

While most libraries only generate $N-1$ (single-outage) scenarios, gridfm-datakit enables arbitrary $N-k$ contingencies by exhaustively or randomly sampling subsets of grid elements (branches, transformers, generators) for outage, with feasibility checks (no islanding). It also incorporates branch admittance perturbations via uniform scaling of resistance and reactance, which are absent in most other libraries.

Generator Setpoints and Constraint Violations

gridfm-datakit offers two modes:

• OPF mode: Generator dispatches are solved via ACOPF, with random cost coefficient permutations to drive economic diversity across datapoints.
• PF mode: Generator setpoints are determined under base (pre-outage) topology and then evaluated under contingent topologies without recomputing optimal dispatch. This creates natural, scenario-driven constraint violations (overloads, voltage excursions, slack/generation bounds, etc.) reflecting true cascades and realism, in contrast to constraint-agnostic OPF solutions (as in PF$\Delta$).

Data Diversity and Distributional Characteristics

gridfm-datakit's diversity is quantitatively assessed using normalized mean per-feature Shannon entropy, evaluating not just load attributes ($P_d$, $Q_d$) but also generator output, branch flows, voltages, and topology states. The distributional breadth, as measured by entropy, is compared directly to PF$\Delta$, OPFData, OPF-Learn, and PGLearn.

Figure 2: Power Flow (PF) datasets generated by gridfm-datakit exhibit diverse feature distributions, supporting varied ML training regimes.

On PF data, gridfm-datakit achieves slightly lower entropy for load variables than PF$\Delta$, since the latter samples the feasible set directly (producing high power factor variability, but lower physical realism and no temporal correlation). However, gridfm-datakit maintains a balanced mix of in-limit and out-of-limit scenarios; PF$\Delta$ yields >75% violation rates on certain buses (Figure 3).

Figure 3: Distribution of reactive power generation ($Q_g$) for PF datasets (gridfm-datakit vs. PF$\Delta$); note severe reactive violation rates in PF$\Delta$.

For branch constraints, gridfm-datakit produces realistic overload rates ($1.2\%$ branch overloading, $79\%$ of scenarios with any overload), contrasting with universal overloads in PF$\Delta$.

On OPF data, gridfm-datakit achieves drastically higher entropy for generator outputs (normalized mean entropy for $P_g$ is 2.3$\times$ that of PGLearn and 2.8$\times$ that of OPFData) due to cost permutation and admittance perturbation. This supports robust model generalization to unseen dispatch regimes.

Outputs, Validation, and ML Foundation Model Integration

gridfm-datakit delivers outputs in ML-ready structured formats, with per-sample inclusion of both AC and fast DC baseline solutions and accompanying metadata and runtime. Automated validation routines verify constraint consistency and balance. Built-in statistical reporting assists rapid dataset characterization.

Figure 4: Example output of the gridfm_datakit stats command on PF data generated for IEEE 118.

Direct integration with gridfm-graphkit enables seamless transformation of grid data to PyTorch Geometric objects for graph-centric ML paradigms. gridfm-graphkit facilitates GNN/Graph Transformer pretraining, fine-tuning, and deployment; supports self-supervised objectives (e.g., masked feature reconstruction and physics-informed losses); and allows for scalable multi-GPU experimentation and transfer to unseen grid topologies.

Implications, Limitations, and Future Prospects

gridfm-datakit sets a new standard for reproducibility, diversity, and scalability in power systems ML datasets generation. It closes a longstanding gap in the research ecosystem by bringing together the full spectrum of perturbation models—spanning loads, topologies, generator economics, and network physics—in a high-throughput, validated, and open-source toolchain.

Practically, this will support the emergence of robust foundation models for grid operation and planning, as outlined in [hamann2024perspectivefoundationmodelselectric], and catalyze fair benchmarking for neural solvers [Khaloie]. From a theoretical perspective, this enables more stringent tests of model generalization under stress, supports zero-shot transfer, and unlocks research into domain adaptation and out-of-distribution detection for power systems.

Future development goals include per-bus load profile support, subgraph/topology sampling, large-scale dataset hosting (e.g., via HuggingFace), and dual variable (LMP) inclusion for advanced primal-dual learning [primaldual].

Conclusion

gridfm-datakit-v1 constitutes a significant advance in synthetic data generation methodology for PF and OPF, providing a unified, extensible, and highly scalable framework. By jointly addressing diversity, physical realism, and computational efficiency, it enables reproducible ML benchmarking and accelerates research toward grid-aware foundation models. The broader adoption of unified frameworks of this type will be essential for both method development and deployment within the emerging grid-AI landscape (2512.14658).