ATLAS Library for Geophysical Modeling

• ATLAS Library is a flexible middleware providing advanced abstractions for grids, meshes, and fields in weather and climate modeling.
• It utilizes C++ and Fortran interoperability to enable scalable parallel computing on both CPUs and accelerators.
• Engineered for high-performance portability, ATLAS minimizes overhead in halo exchanges and supports modern NWP applications.

ATLAS Library

The term "ATLAS library" refers primarily to the open-source software library developed at the European Centre for Medium-Range Weather Forecasts (ECMWF) for scalable, high-performance geophysical modeling. Within the atmospheric and climate science communities, ATLAS serves as a C++/Fortran middleware that provides advanced abstractions for grid generation, mesh handling, parallel data distribution, and field operations on spherical geometries, targeting exascale Numerical Weather Prediction (NWP) and climate models. It is explicitly engineered for performance portability across CPU and accelerator architectures, with native support for both traditional structured grids and emerging unstructured and regional configurations (Deconinck, 2019, Deconinck, 2019).

1. Design Objectives and Architectural Principles

The core mission of ATLAS is to encapsulate the fundamental data structures and distributed-memory communication patterns required by modern weather and climate modeling dwarfs. The library is written in C++11 with zero-overhead Fortran 2008 interfaces, optimized for integration into legacy and production NWP codes. Key objectives include:

• Flexibility: Native support for structured, reduced, regional, and unstructured grids; arbitrary projections through polymorphic [Projection](https://www.emergentmind.com/topics/predictive-head-projection) and Domain interfaces.
• Performance Portability: Host/device data management using a storage abstraction compatible with GridTools for GPUs; MPI and OpenMP threading for scalable parallelism.
• Interoperability: A mixed C++/Fortran design enabling seamless coupling across language boundaries; direct pointer passing and type-bound procedures ensure limited overhead.
• Scalability: Minimal overhead for halo-exchange, gather/scatter primitives, and in-place spectral transforms, with demonstrated scaling to O(10⁴) MPI ranks (Deconinck, 2019, Deconinck, 2019).

2. Core Data Structures and Abstractions

ATLAS exposes four foundational abstractions:

• Grid: An ordered list of 2D coordinates, typically with no connectivity. Grids can be regular (lat-lon, Gaussian), reduced, or fully unstructured. Each grid derives from the base Grid class and includes explicit support for projections and domains (e.g., global, rectangular, zonal bands).
• Mesh: A collection of elements (cells, nodes, edges) with connectivity information, supporting both cell-centered and node-centered numerical schemes. Block, irregular, and multi-block connectivity structures enable efficient storage and iteration.
• Field: Multi-dimensional arrays (scalars or vectors) associated with a Grid or Mesh. Fields can be realized as host-native arrays or GPU-resident buffers with explicit synchronization semantics.
• FunctionSpace: An interpretation layer specifying how a Field is laid out (node-based, cell-based, spectral) and providing distribution-aware operations (halo exchange, gather/scatter, reductions) (Deconinck, 2019, Deconinck, 2019).

The software stack also includes interfaces for mesh I/O (read/write via MeshReader), dynamic partitioning (Partitioner, MatchingMeshPartitioner), and mapping of fields across different meshes.

3. Parallelism, Accelerator Support, and API Integration

Parallel distribution in ATLAS is achieved via domain decomposition at both Grid and Mesh levels using a configurable partitioner (checkerboard, equal-regions). Each MPI rank operates on a local MeshPartition with ghost layers (halos) to support stencil operations and communication.

Accelerator support: ATLAS leverages a unified data descriptor that registers host and device pointers, enabling lazy synchronization and minimizing data transfer. The GridTools backend supports device-resident storage, CUDA or OpenACC kernels (Fortran bindings), and direct GPU-to-GPU MPI communication that avoids host staging.

API exposure: All major C++ classes are accessible from Fortran through derived types holding C_PTR handles, with interface routines mirroring the C++ signatures. FunctionSpace and Field operations (e.g., haloExchange, createField, view access) are available from both languages without runtime penalties outside the actual function call (Deconinck, 2019, Deconinck, 2019).

4. Remapping, Operators, and Domain-Specific Language (DSL) Integration

ATLAS provides comprehensive support for remapping and interpolation, critical for multi-grid models and coupling disparate components. The remapping operator generically computes

$$f_{\rm to}(\mathbf{x}_j) = \sum_{i=1}^N R_{ji}\;f_{\rm from}(\mathbf{x}_i)$$

where the remapping matrix $R_{ji}$ is constructed via linear element projection for 3D fields. Weight matrices $W_{ij}$ can also be generated for nearest-neighbor, bilinear, or conservative volume projections.

The library integrates with grid-aware DSLs (notably via GridTools), allowing fields to reside on device-resident storages and kernels to be dynamically generated by DSL tools for CPUs and GPUs. This enables in-model operators—such as gradient, divergence, and spectral transforms—to execute at device memory bandwidths, with field storage and code-generation targets abstracted from the NWP user's perspective (Deconinck, 2019).

5. Usage Patterns, Extensibility, and Ecosystem Integration

Canonical usage flow in high-performance NWP or climate applications involves:

1. Grid/mesh generation: Choose or define one or more (possibly distinct) global or regional grids; construct associated meshes.
2. Partitioning and function spaces: Apply a Distribution/Partitioner to decompose fields for parallel execution. MatchingMesh enables aligning decompositions between coupled components.
3. Field allocation and initialization: Allocate multi-level fields (e.g., temperature, velocity) and initialize values pointwise using analytic or data-driven methods.
4. Distributed computation: Perform local and global operations with built-in primitives for halo exchange, gather/scatter, remapping/interpolation, and in-model numerical operators (e.g., finite-volume differencing, spectral transforms).
5. Communication and finalization: Complete distributed operations with minimal host-device synchronization.

Example C++ usage:

atlas::Grid grid("O32", atlas::option::distribution("equal_regions")); atlas::Mesh mesh = atlas::MeshGenerator("structured").generate(grid); atlas::FunctionSpace fs(mesh, atlas::option::name("CellCentre")); atlas::Field T = fs.createField<double>(atlas::option::levels(60), atlas::option::name("T")); fs.haloExchange(T);

(Deconinck, 2019)

Extensibility: New projections, mesh types, and communication backends can be introduced by subclassing core abstractions and adhering to the storage and indexing conventions. The library is released under Apache-2.0, with comprehensive documentation, user guides, and a clear pathway for external contributions (Deconinck, 2019).

Integration: The API is designed to slot into larger end-to-end workflows, including domain-specific DSLs and model coupling layers. Explicit Fortran support enables legacy and next-generation applications to utilize ATLAS capabilities with no need for extensive rewriting.

6. Limitations, Performance, and Future Directions

Current limitations include partial support for asynchronous halo exchange (overlap of computation and communication), ongoing work on limited-area mathematical operators for regional models, and the need for more sophisticated backend support to fully exploit heterogeneous architectures.

Internal benchmarks demonstrate near-ideal weak scaling for local stencil kernels and distributed halo exchanges up to O(10⁶) points per GPU in a 32-GPU cluster environment, with under 10% overhead due to host-device data movement for typical field sizes.

Development is ongoing to extend the mathematical operator repertoire (gradient, divergence, curl) for regional projections, complete asynchronous communication functionality, and expand automatic code-generation support for future exascale architectures (Deconinck, 2019).

7. Summary Table: Structural Overview (ATLAS Library)

Component	Primary Abstraction	Performance/Notes
Grid	Ordered list of points	Structured/unstructured, projections
Mesh	Elem., connectivity	Cells, nodes, edges (MPI dist.)
Field	Multi-dim. array	Host/device sync, metadata
FunctionSpace	Field layout & comm.	Halo exchange, partitioning
DSL Interface	Device code-generation	GridTools/CUDA/OpenACC
Fortran Binding	C_PTR to C++ wrappers	Type-bound procs, no loop penalty

• "Development of Atlas, a flexible data structure framework" (Deconinck, 2019)
• "Public release of Atlas under an open source license, which is accelerator enabled and has improved interoperability features" (Deconinck, 2019)