SmartSim: HPC & ML Coupling Framework
- SmartSim is a Python-based, open-source framework designed to couple HPC simulations with machine learning via an in-memory Redis orchestration layer.
- It enables legacy MPI/OpenMP applications to exchange tensors with ML models in real time without rewriting codes or relying on file-based workflows.
- Its architecture, comprised of SmartRedis, an Infrastructure Library, and an Orchestrator, supports scalable online analytics and distributed model serving.
Searching arXiv for recent SmartSim papers and related uses in HPC/CFD. SmartSim is a Python-based, open-source software framework for coupling traditional high-performance computing simulations with machine learning and online analytics at runtime. It was developed to let MPI/OpenMP applications written in Fortran, C, or C++ exchange tensors with ML models online, during the simulation timestep loop, without requiring simulation developers to rewrite legacy codes in Python or to rely on file-based workflows. Its defining architectural choice is a distributed in-memory coupling layer, centered on an Orchestrator built on Redis plus Redis modules, especially RedisAI, and exposed through lightweight language-specific clients (Partee et al., 2021, Laszewski et al., 30 Jul 2025).
1. Design problem and historical setting
SmartSim addresses a practical convergence problem in scientific computing. Large production simulations are commonly implemented in compiled languages and parallelized with MPI and OpenMP, whereas contemporary ML tooling evolves rapidly in Python and is increasingly optimized for heterogeneous CPU/GPU systems. The central difficulty is therefore not only language interoperability, but efficient data exchange at scale between simulation state and ML services. SmartSim was positioned as a systems layer that decouples simulation codes from ML frameworks while still permitting live, in-memory interaction on HPC platforms (Partee et al., 2021).
The framework’s original intent was threefold: to describe and execute large ensembles of simulations, to incorporate in-the-loop inference capabilities for simulations, and to store and consume data for online visualization, analysis, and surrogate model training. In later work on experiment execution, SmartSim was characterized as more opinionated than Cloudmesh, with a design philosophy oriented toward application and domain scientists rather than toward a broad plugin architecture (Laszewski et al., 30 Jul 2025).
This positioning places SmartSim between conventional workflow tools and direct library embedding. It is neither a purely file-mediated workflow engine nor merely a solver-side inference library. Its role is that of a runtime integration substrate for loosely coupled, heterogeneous scientific applications.
2. Core software architecture
SmartSim has three central architectural elements: SmartRedis, the Infrastructure Library, and the Orchestrator. The division of responsibility is explicit.
| Component | Function | Key properties |
|---|---|---|
| SmartRedis | Client library compiled into applications | APIs in Fortran, C, C++, and Python; puts/gets tensors, runs scripts, executes ML models |
| Infrastructure Library (IL) | Python orchestration layer | Launches and manages simulations and supporting infrastructure; integrates with schedulers and local execution |
| Orchestrator | Distributed in-memory data and model service | Built on Redis plus Redis modules, especially RedisAI; shared-nothing clustering via CRC16 hashing |
SmartRedis is the application-facing layer. Users work with native arrays in the host language; the library exposes tensor operations, script execution, and model invocation. The Infrastructure Library is the control plane. It launches and monitors simulations, the in-memory service infrastructure, and ensembles; in the 2021 ocean-climate workflow it was used from JupyterLab. The Orchestrator is the runtime data plane: a distributed in-memory datastore and model-serving substrate that applications address logically as a single storage service, even when deployed across multiple nodes (Partee et al., 2021).
The Orchestrator is more than a cache. It serves as a parameter-server-like rendezvous point for coupled workflows. Simulations can push tensors into the database, analysis or visualization tools can consume them online, and ML models stored in the Orchestrator can execute directly against data already resident there. This avoids filesystem mediation and the associated latency and scalability penalties of file I/O. Because the Orchestrator is database-centric rather than peer-to-peer, the communication topology scales more favorably than fully connected application coupling; an OpenFOAM example reduced a fully connected topology from 524,288 connections to 16,896 in a hub-and-spoke design centered on the database (Maric et al., 2024).
3. Data model, execution semantics, and model serving
SmartSim moves data through opaque tensor abstractions. In addition to individual tensors, SmartRedis provides a DataSet abstraction that bundles multiple tensors plus metadata under one logical key. The framework also supports in-database preprocessing via stored TorchScript programs: JIT-traced Python scripts execute on CPU or GPU inside the Orchestrator and operate directly on tensors already present there. This enables online computational pipelines in which preprocessing and inference occur inside the same in-memory substrate (Partee et al., 2021).
Model serving is handled through RedisAI. The Orchestrator can host TensorFlow, Keras, PyTorch, TensorFlow-Lite, and ONNX models. Although these models are authored in Python, the paper notes that they are JIT-compiled and executed in a C runtime. A key design feature is model replication: when client.set_model() is called, a copy of the model is distributed to every database node. During inference, SmartRedis selects the model instance co-located with some or all input data, and if necessary transparently moves temporary copies of missing input or output tensors between database nodes. The user therefore sees a simple remote inference call rather than distributed database mechanics (Partee et al., 2021).
The execution model is loosely coupled. Simulations are launched as ordinary HPC jobs with SmartRedis embedded, while the Orchestrator can run in the same allocation, on adjacent nodes, or in batch mode. SmartSim’s communication pattern is database-centric and key-driven: clients put, get, poll, and delete data rather than exchange direct messages. In OpenFOAM workflows this semantics supports both synchronous and asynchronous patterns, with synchronization implemented by storing, deleting, and polling flags in the database. Because SmartRedis internally manages I/O asynchronously, polling is used to ensure that consumers observe completed writes (Maric et al., 2024).
Two recurrent misconceptions are explicitly contradicted by the published system descriptions. First, SmartSim does not require embedding a Python runtime into legacy simulation codes; coupling is mediated through SmartRedis clients and the Orchestrator. Second, a logically shared ML service does not imply a single physical model instance; operationally, models are replicated across database nodes, which is how many concurrent simulations can invoke the “same” model while exploiting all Orchestrator hardware (Partee et al., 2021).
4. Climate-scale demonstration with MOM6
The first large-scale demonstration of SmartSim was a global ocean climate modeling campaign using MOM6, a Fortran ocean general circulation model. The scientific task was to replace or augment a conventional turbulence parameterization for eddy kinetic energy with a neural network that predicts EKE from resolved-flow features. The reference high-resolution simulation was a global ocean configuration called ER with about 7.5 million ocean grid points; two lower-resolution configurations with about 2.7 million ocean grid points were then used, one with the conventional MEKE parameterization and one with the SmartSim-based ML replacement, SmartSim-EKE (Partee et al., 2021).
The DNN training data were derived from the ER run integrated for 20 years to equilibrate the eddy field. Four 2D predictors were extracted from coarsened ER output: surface mean kinetic energy, Rossby radius of deformation normalized by horizontal grid area, surface relative vorticity, and column-averaged isopycnal slope. The network, EKEResNet, was described as a small residual CNN-like model. Because each sample is effectively a point with four features, the network begins with transposed convolutions to expand spatial extent to 7, followed by three bottleneck residual blocks with group convolutions, and ends with two fully connected layers producing predicted surface EKE.
One preprocessing transform was specified explicitly. Relative vorticity was mapped by
with , and in this work . The target variable was , because EKE appeared approximately log-normal. Training used stochastic gradient descent with learning rate , a step-wise schedule peaking at epoch 5 out of 100, mean squared error loss with penalty , parallel training on 8 NVIDIA P100 or V100 GPUs, and local batch size 512. A weighted sampling strategy based on inverse probability density over 0 was adopted to better represent tail values (Partee et al., 2021).
In the online MOM6 workflow, SmartSim was embedded directly into the simulation timestep loop. At each inference event, MOM6 computed the four predictor fields on local subdomains, the Fortran SmartRedis client sent those arrays to the Orchestrator, the JIT-traced PyTorch EKEResNet model stored in the Orchestrator was queried remotely, and predicted EKE values were returned to the simulation. Those estimates were converted into the coefficient controlling the Gent–McWilliams parameterization of eddy effects on resolved circulation. The online inference portion of MOM6 was called 8 times per simulated day, on about 2.7 million grid points distributed across 910 cores per ensemble member, which in wall-clock terms occurred about every 4 seconds.
The ensemble itself was large by climate-model standards. SmartSim generated and launched a 12-member SmartSim-EKE ensemble; each member used 19 compute nodes and 910 physical CPU cores, and all members communicated with the same logical ML architecture backed by a shared Orchestrator using 16 NVIDIA P100 GPUs. Each ensemble member ran for 10 additional simulated years after branching from a 20-year spin-up. Across the full campaign, the ensemble covered 120 simulated years and served approximately 970 billion inferences, with an average sustained online inference rate of 1.86 million inferences per second.
The reported runtime overhead was modest. For a representative member, total elapsed time spent in put_tensor was 24 seconds over the full run, whereas total run_model time was about 2 hours in a total wall time of 136 hours. The authors estimated SmartSim-EKE overhead at about 6% relative to the existing MEKE parameterization, and noted that essentially the same 1 overhead was observed when an ensemble member was run individually, suggesting that the shared Orchestrator was not saturated by the 12-member workload. All 12 ensemble members completed their 10-year integrations with no evidence of numerical instability. The paper described this as the first climate-scale numerical ocean simulations improved through distributed online inference of deep neural networks using SmartSim (Partee et al., 2021).
5. Expansion into CFD, reinforcement learning, and experiment execution
Subsequent work generalized the SmartSim pattern beyond the MOM6 use case. In CFD, SmartSim was used to couple PHASTA to distributed ML training and inference on Polaris through an architecture with four parts: a data producer, a data consumer, an in-memory database, and SmartRedis as the communication library. The major systems contribution was a co-located deployment in which simulation, database, and ML resources share each node: the CFD code uses CPU cores, the database is pinned to a subset of CPU cores, and GPUs are used for ML or RedisAI inference. On Polaris, this design delivered perfect scaling efficiency for both send and retrieve in weak-scaling studies and perfect scaling of total inference cost under co-location, while framework overhead was much less than 1% of PDE integration time and data retrieval overhead was just over 1% of training time in the in situ autoencoder application (Balin et al., 2023).
OpenFOAM integration emphasized reusable coupling points inside a large CFD ecosystem. SmartSim was used to couple pre-processing and post-processing applications, solvers, function objects, and mesh motion solvers. The authors introduced an OpenFOAM-side smartRedisClient API and the fieldsToSmartRedis function object so that existing solvers could stream internal-field and boundary-patch data to the Orchestrator without direct solver-source modification. Three example workflows were documented: Bayesian optimization over a turbulence-model parameter space, partitioned singular value decomposition as an online analysis pipeline, and ANN-based mesh motion in the loop (Maric et al., 2024).
In reinforcement learning, SmartSim became the coupling layer for online control problems in which a simulation acts as the RL environment. Relexi used SmartSim for two major tasks: launching and managing MPI-parallel HPC workloads and implementing communication between the CFD solver FLEXI and a TensorFlow/TF-Agents PPO driver through an in-memory database. The deployment split ML work onto a GPU-equipped head node and MPI simulations onto worker nodes; SmartSim orchestrated repeated launches within a single batch allocation and mediated online state/action exchange through the Orchestrator. This framework scaled to up to 1024 parallel environments on up to 2048 cores (Kurz et al., 2022).
SmartFlow extended this line further by combining SmartSim with a newly introduced SmartRedis-MPI layer and Stable-Baselines3. The resulting framework was presented as nearly CFD-solver-agnostic: SmartSim Infrastructure Library launches the Orchestrator and CFD jobs, SmartRedis-MPI gathers distributed state and reward data across MPI ranks and scatters actions back, and the Python RL side polls the datastore and updates policies. In the CaLES integration, the authors reported that only five additional lines of code were required to add the SmartRedis-MPI calls needed for initialization, state transfer, reward transfer, action retrieval, and finalization (Xiao et al., 1 Aug 2025).
A broader systems interpretation appeared in work on experiment execution for community benchmark workflows. There SmartSim was presented as an experiment-execution framework with four principal abstractions—Experiment, Model, Ensemble, and Orchestrator—and as a tool especially well suited to hybrid AI/ModSim workflows involving simulation ensembles, in-the-loop inference, online visualization and analysis, and surrogate-model training. That paper also situated SmartSim within “benchmark carpentry,” an educational and methodological effort to make complex benchmark workflows reusable and teachable (Laszewski et al., 30 Jul 2025).
6. Limitations, comparative context, and recurring issues
SmartSim’s published evaluations emphasize numerical stability, runtime overhead, and scalability, but they do not make fault tolerance a central topic. No detailed fault-recovery mechanism for the runtime itself was described in the original ocean-climate paper; “stability” there referred primarily to the fact that all coupled MOM6 integrations remained numerically stable over full 10-year runs. Later CFD studies likewise focused on scaling, deployment strategy, and coupling overhead rather than on formal resilience protocols (Partee et al., 2021).
Several practical constraints recur across the literature. Co-located deployments scale extremely well when data are naturally local to each node, but a paper on in situ CFD coupling noted that if a workflow requires arbitrary cross-node data access, extra MPI coordination may be required. OpenFOAM integrations showed that SmartSim does not remove the need to design synchronization protocols carefully; users still need deterministic key naming, polling intervals, aggregation keys, and cleanup logic. RL-oriented frameworks identified startup overhead as a bottleneck when launching large numbers of fast simulations, especially on GPU systems, and discussed MPMD launch and node-local storage as mitigations (Balin et al., 2023, Maric et al., 2024, Xiao et al., 1 Aug 2025).
SmartSim also differs sharply from file- and schema-centric workflow systems. SIMEX, for example, supports modular start-to-end simulations in photon science through sequential “Calculators” connected by HDF5 and increasingly openPMD interfaces. It does not present a distributed runtime, a database-backed coordination layer, or in-memory ML model serving. The contrast is instructive: SmartSim’s integration backbone is database-centric and online, whereas SIMEX’s is predominantly staged and file-mediated (Fortmann-Grote et al., 2016).
Finally, the experiment-execution literature identifies one external risk specific to SmartSim’s software stack: dependence on Redis and RedisAI, which that paper states must be compiled from source because of license restrictions. The Redis license change was described there as an external risk affecting a key SmartSim component. This does not negate the framework’s role, but it qualifies the portability and distribution story that often accompanies runtime middleware (Laszewski et al., 30 Jul 2025).
SmartSim’s significance in the arXiv literature lies in this combination of properties: a lightweight multi-language client layer for legacy HPC applications, a distributed in-memory datastore that doubles as a model-serving substrate, and a Python orchestration layer that represents simulations, ensembles, and experiments explicitly. Across ocean climate modeling, CFD, OpenFOAM integration, reinforcement learning, and experiment execution, it has functioned as middleware for online, loosely coupled HPC+ML workflows rather than as a domain-specific solver or a conventional file-based workflow engine.