GENIE: Simulation-Aware Database Integration

• GENIE is a database extension that integrates simulation tools with PostgreSQL via virtual columns and dynamic simulator invocation to streamline data analysis.
• It employs simulator registries, gray-box integration, and adaptive algorithms to balance runtime efficiency and analytical accuracy through progressive refinement.
• The platform has demonstrated transformative improvements in environmental modeling and disaster risk assessment by reducing redundant simulations and computational overhead.

GENIE refers to a database extension paradigm designed to tightly integrate physics-based simulators with relational databases in order to enable dynamic, interactive, and iterative data exploration and scientific discovery. Traditional workflows treat simulators as detached, external pre-processing, with analysts manually orchestrating runs, exporting results, and loading data for analysis—resulting in inefficiency, high latency, and poor support for explorative or what-if analyses. GENIE re-architects this approach by making the database engine itself "simulation-aware": simulation tasks are invoked dynamically and in response to SQL queries, with results cached, reused, and refined as needed for rapid, query-driven, incremental scientific analysis. GENIE is implemented as an extension to PostgreSQL 14 with PostGIS and supports multiple simulators such as WRF-SFIRE, HYSPLIT, SLOSH, and HEC-RAS, demonstrating transformative reductions in time-to-insight for risk assessment and scientific studies (Colaco et al., 15 Nov 2025).

1. Technical Foundations and Database–Simulator Integration

GENIE fundamentally reconceives the database as a platform not just for storing simulation outputs, but for orchestrating, parameterizing, and invoking simulators as first-class components based on analytical need. The core principles are:

• Virtual Attributes: Simulation outputs are registered as virtual columns in database tables (e.g., smoke dispersion "concentration"), computed by external simulators only for required spatiotemporal regions and resolutions. These columns are defined via SQL DDL extensions—e.g., [ALTER](https://www.emergentmind.com/topics/alter) TABLE smoke_dispersion [ADD](https://www.emergentmind.com/topics/adversarial-diffusion-distillation-add) COLUMN concentration REAL GENERATED BY SIMULATOR hysplit ….
• Simulator “Gray-Box” Integration: Simulators are treated as opaque external programs but their interfaces (parameters, outputs, dependencies) are registered in the database. GENIE leverages these exposed controls to optimize execution but does not attempt to introspect or directly manipulate internal solvers.

Integration is achieved through PostgreSQL catalog extensions, including system tables (pg_simulators, pg_virtual_columns) populated by the new DDL commands. Simulator adapters translate high-level database requests into simulator-specific job specifications (e.g., WRF namelists, HYSPLIT CONTROL files) and manage the full execution and re-import cycle, handling formats such as NetCDF and HDF5.

2. GENIE Architecture

The GENIE system is realized as three interacting core components:

• Simulator Registry & Catalog: Track registered simulators, parameters (e.g., spatial/temporal resolution, particle count), output formats, and dependencies (e.g., smoke dispersion depending on fire emissions from a different simulator).
• Generator Driver (in PostgreSQL executor):
  • Planner: Intercepts SQL, recognizes simulator-generated columns, extracts predicates, and constructs a dependency graph across simulators.
  • State Manager: Tracks a spatiotemporal coverage map (e.g., via PostGIS) detailing at which extents and resolutions simulation data exists; identifies gaps needing computation.
  • Query Log: Records past queries to inform prefetching and proactive simulation.
• Simulator Adapters: For each simulator, platform-specific logic translates high-level database requests into job execution, monitors status, parses results, and ingests outputs into spatially indexed database structures.

GENIE's planner solves an internal optimization problem for each query: minimize estimated runtime $T(\mathcal{P})$ subject to achieving user-specified or default analytical accuracy $A(\mathcal{P}) \ge Q_\text{req}$, with $\mathcal{P}$ denoting the parameter vector across simulators. Heuristic and future learned models are used to select parameter settings and manage trade-offs between runtime and precision.

3. SQL Abstraction, Dependency Management, and Orchestration

GENIE introduces SQL extensions for registering simulators, linking virtual columns, and expressing ensemble computations (e.g., combining outputs from multiple simulators with weighted averaging).

• Example DDL:

REGISTER SIMULATOR hysplit EXECUTABLE '/path/hysplit' PARAMETERS (spatial_res REAL DEFAULT 0.1, temporal_res REAL DEFAULT 1.0, particle_count INT DEFAULT 1000) OUTPUT_FORMAT netcdf; ALTER TABLE smoke_dispersion ADD COLUMN concentration REAL GENERATED BY SIMULATOR hysplit DEPENDS ON (fire_emissions.emission_rate);

• Query Example:

SELECT s.station_id, AVG(d.concentration) FROM monitoring_stations s JOIN smoke_dispersion d ON ST_DWithin(s.location, d.grid_cell, 1000) WHERE d.timestamp BETWEEN '2024-08-15' AND '2024-08-17' GROUP BY s.station_id HAVING AVG(d.concentration)>35;

Upon query interception, the planner determines what subset of temporal and spatial simulation outputs are required, invokes only necessary simulation jobs at appropriate resolutions, and reuses existing data where available. The "WITH HINT" clause can be used to force desired resolutions or parameter settings.

4. Algorithms for (Re)Simulation, Reuse, and Progressive Refinement

GENIE executes an adaptive, incremental algorithm designed for efficiency and rapid refinement:

• Step 1 (Region Identification): Analyze spatial and temporal predicates from the query to determine required extents and tolerable resolutions.
• Step 2 (Coverage Query): Intersect query window with existing simulated data (coverage map); identify gaps or under-resolved regions.
• Step 3 (Multi-Epoch Simulation Scheduling):
  • Epoch 0: Proactive coarse simulation coverage during idle periods.
  • Epoch 1: Immediate coarse simulations for missing regions upon query.
  • Epoch 2: Selective refinement on high-interest subregions (e.g., where concentrations exceed thresholds).
  • Epoch 3: User-requested or further on-demand refinement.
• Step 4 (State Update): Refined results update the coverage map, enabling subsequent queries to benefit from improved data without rerunning the full pipeline.

This design supports computational reuse and avoids redundant simulation, yielding significant reductions in both runtime and resource consumption.

5. Use Cases: Wildfire Analysis and Hurricane Risk Assessment

GENIE enables interactive, practical use cases in environmental modeling:

• Wildfire Smoke Dispersion: Chained simulation with WRF-SFIRE (fire spread) generating fire_emissions, and HYSPLIT (atmospheric transport) generating smoke_dispersion. Queries for air-quality at monitoring stations trigger only the minimal necessary simulation tasks and can be interactively refined with varying spatial or temporal granularity.
• Hurricane Hazard Assessment: Integrated workflow using wind/surge/flood simulators (e.g., SLOSH, HEC-RAS) feeding into damage_assessment via chained dependencies. GENIE enables rapid approximation and targeted refinement for risk mapping, outperforming static workflows.

Performance evaluation on a 48-core, 192 GB cluster demonstrated an order-of-magnitude speedup (8–12×), rapid coarse-to-fine refinement, and approximately 40% reduction in redundant computations over 10 sequential queries, compared to static pipelines.

6. Experimental Results

Quantitative benchmarks in the wildfire use case show:

Query Type	Static (Fine)	Static (Coarse)	GENIE Adaptive
Regional AVG	145 min, 100%	12 min, 70%	18 min, 92%
Station Time Series	—	—	35 min, 96%
Plume Evolution	—	—	42 min, 94%

Progressive overview accuracy: 80% in 2 min, 95% in 10 min, full high-res in 15 min (vs. 145 min cold start). Computation reuse reduced runtime by 40%, number of simulator invocations, and total data generated.

7. Challenges, Limitations, and Future Research

GENIE's strengths in performance and interactive analysis introduce several technical and research challenges:

• Multi-Simulator Integration: Standardizing interfaces, handling I/O formats, coordinate systems, resource management, and error recovery across heterogeneous simulators.
• Dynamic Parameter Tuning: Scaling to large parameter spaces across chained simulators, with highly nonlinear runtime/accuracy trade-offs; planned advances include cost-accuracy prediction via Bayesian optimization or reinforcement learning.
• Consistency and Uncertainty: Managing incremental refinement, propagation of errors and resolutions through dependent simulation workflows, and ensuring consistency of results.
• Scalability and Parallelism: Current prototype executes simulators sequentially; scaling requires distributed state managers, pipeline parallelism, and asynchronous refinement.
• Usability and Democratization: Reducing the technical barrier for new simulator integration, supporting GUI-based parameter tuning, and tracking provenance of simulation results.

GENIE is positioned as a canonical approach for integrating complex simulations with scientific data systems, with implications for domains ranging from environmental science and engineering to disaster response and beyond (Colaco et al., 15 Nov 2025).