Transforming the Use of Earth Observation Data: Exascale Training of a Generative Compression Model with Historical Priors for up to 10,000x Data Reduction

Abstract: Earth observation is becoming one of the largest data-producing activities in science, yet current pipelines still treat compression as a storage and transmission tool rather than a new way to use data. We present a generative compression framework that learns from historical Earth observation archives and enables on-demand 100x to 10,000x data reduction across downstream tasks. Unlike general visual data, Earth observation repeatedly measures the same evolving planet, making historical-prior learning feasible for extreme compression. To realize this paradigm, we train large generative compression models at exascale on the LineShine Armv9 CPU supercomputer, with co-optimization across model design, kernels, memory hierarchy, runtime, and parallelism. Our implementation sustains 1.54 EFLOP/s and peaks at 2.16 EFLOP/s in end-to-end training. This work shows that historical-prior generative compression can turn Earth observation data into an active, task-adaptive foundation for acquisition, delivery, storage, and scientific use.

• The paper introduces an exascale-trained generative compression model that leverages historical EO priors for up to 10,000x data reduction.
• It employs the novel D2AR framework with modular sensor decoupling and topology-aligned parallelism on Arm-based supercomputers to ensure high data fidelity.
• Empirical results demonstrate that extreme compression preserves key scientific indicators, supporting efficient downstream tasks such as scene classification.

Exascale Generative Compression for Earth Observation Data with Historical Priors

Introduction

This paper presents a new paradigm for Earth observation (EO) data utilization via exascale training of a generative compression model endowed with historical priors, achieving 100×–10,000× data reduction. The approach leverages the inherent spatiotemporal redundancy in global EO archives and introduces a learning-based compression framework that enables on-demand, task-adaptive data access. The implementation combines algorithmic, model, and system-level co-design, targeting the Armv9-based LineShine supercomputer, achieving over 1.54 EFLOP/s sustained training throughput. The following analysis examines the key contributions, technical innovations, and implications of this work.

Motivation and Context

EO archives have grown to the exabyte scale, yet current data pipelines rely on storage- and transmission-centric compression, resulting in significant bottlenecks for downstream scientific usage. Typical compression techniques (e.g., CCSDS, JPEG2000, VVC) and even learned codecs optimize rate-distortion with local bitstreams, rendering them suboptimal at ultra-low bitrates due to degradation of semantically or physically meaningful features. The observation that EO data repeatedly samples the same evolving planet motivates the core hypothesis: global historical priors can be distilled into generative models that reconstruct high-fidelity data from minuscule compressed signals, thus transforming raw archives into an active knowledge resource.

Framework and Model Innovations

The proposed Dual-Decoupled Asymmetric Compression and Reconstruction (D2AR) framework reorganizes EO data handling at both the algorithmic and user-access levels. The core pipeline performs lightweight, on-orbit compression to produce feature tokens, which act as control signals for a ground-based, large-scale generative reconstruction model. This allows compression and reconstruction to be decoupled, with downstream users accessing data at suitable compression levels on-demand. The architecture supports flexible decoding across arbitrary ratios and sensor platforms.

Figure 1: Overview of the proposed historical-prior generative compression framework, emphasizing generative prior learning, reconstruction workflow, and exascale training system design.

The D2AR model introduces modular sensor-specific decoupling, using EQ-VAE-based physical adapters so that a standardized Flow Matching backbone learns to generate the EQ-VAE latent space. This separation ensures the core optimization is platform-agnostic and efficient, maintaining fidelity in multispectral and SAR imagery. Critically, D2AR injects learned global spatiotemporal priors via continuous embeddings into the generative process, substantially improving the reconstruction, particularly when compressing by several orders of magnitude.

System-Level Innovations

Training such priors at the exascale entails highly nontrivial system challenges. The LineShine supercomputer, equipped with Armv9 LX2 CPUs, introduces complex NUMA and hierarchical HBM/DDR memory. The team presents optimizations across kernels, memory management, and parallelization:

• SME-Oriented GEMM Tiling: Two kernel shapes are selected at runtime based on problem geometry, maximizing reuse and cache residency. Operand-specific memory policies (prefetching streamed operands to HBM, direct thread-private accumulation) minimize bandwidth contention.

  Figure 2: SME-GEMM workflow with thread partitioning, data packing, and accumulation optimized for HBM/DDR locality and cache efficiency.

• Operator and Lifetime-Aware HBM Placement: HBM is reserved for attention output activations and all backward intermediates, with offloading of long-lived gradients to DDR. Analytical latency profiles drive these allocation decisions.

  Figure 3: Operator forward/backward latency across memory placements, demonstrating the criticality of HBM for attention operations.

• Topology-Aligned Parallelism: A hybrid combination of Sequence Parallelism (SP) and Hybrid Shared Data Parallelism (HSDP) is deployed, ensuring communication is localized within fast shared-memory domains and optimizer states are efficiently sharded across clusters.

  Figure 4: Performance under different parallelism configurations, balancing memory, compute, and communication efficiency.

• Asynchronous Runtime: Introduces overlapped operator dispatch, allocation, and communication, reducing overhead by an order of magnitude compared to off-the-shelf PyTorch.

Empirical Results

Single-Node Performance

Each sequential optimization—HBM-aware memory management, SME kernel optimization, communication improvements, and asynchronous runtime—contributes significant speedup. The 6B parameter D2AR-rec model achieves a 10.3× overall speedup post-optimization on a single node.

Figure 5: Successive optimization stages yield drastic reductions in per-iteration runtime for all model sizes.

Exascale Weak Scaling

The framework demonstrates sustained weak-scaling efficiency to 20,480 nodes (1.54 EFLOP/s at 76% scaling efficiency for the 6B model), supporting training on globally distributed historical archives. Increased node counts directly translate into increased data coverage rather than mere speedup for fixed datasets.

Figure 6: Near-linear weak scaling as data and node counts scale, enabling efficient ingestion of massive EO archives.

Reconstruction Quality and Downstream Task Utility

Empirically, D2AR enables 1,000×–10,000× compression with negligible to minimal accuracy reductions for physically and semantically important indicators (e.g., NDVI, LPIPS, MS-SSIM). Ablation demonstrates strong dependence on the richness of the global prior—reduced spatial coverage measurably impairs generalization to unseen regions.

Downstream scene classification (e.g., land cover on DynamicEarthNet) reveals only minor drops in F1 and mAP when models are trained on reconstructions from extreme compression, supporting the claim that D2AR preserves utility for scientific tasks.

Visual and Spectral Validation

Qualitative results under 100×–10,000× compression show that the model sustains macroscopic structure and spectral integrity, with only micro-texture loss at the highest compression, thereby preserving analysis-critical information for most scientific workloads.

Figure 7: Reconstruction maintains visual and spectral fidelity even under extreme lossy compression, particularly when global historical priors are exploited.

Implications and Future Directions

This work redefines compression from a passive archival utility to an active, adaptive data service that leverages generative priors from vast historical archives. Methodologically, it demonstrates the viability of task-decoupled, prior-driven, large-model reconstruction at the exascale, with significant implications:

• Scientific Impact: Enables democratized access to global-scale EO data, on-demand, with minimal bandwidth and storage, supporting equitable scientific analysis and timely applications (e.g., climate study, disaster response).
• Systemic Transformation: Proposes a data ecosystem where central archives become continuously updated generative prior engines, satellites transmit features rather than pixels, and data centers serve model-mediated access.
• Architectural Relevance: Illustrates that CPU-centric, control/data-customizable supercomputers (esp. Arm variants) can support large-scale scientific foundation model workloads efficiently, free from accelerator-dominated frameworks.
• Generality: The paradigm may extend to other scientific domains characterized by repeated measurements and strong cross-temporal/instrument redundancy, including cosmology and environmental monitoring.

Conclusion

This paper establishes a comprehensive route to exascale generative compression for EO data by combining global-prior learning, model-physics modularization, and system-level co-optimization. The demonstrated 1,000–10,000× reduction ratios are achieved without sacrificing utility for downstream scientific analysis, supporting the shift toward more efficient, scalable, and equitable scientific data systems. This architecture forms the basis for a broader historical-prior-driven paradigm in data-intensive scientific discovery.

---

Reference: "Transforming the Use of Earth Observation Data: Exascale Training of a Generative Compression Model with Historical Priors for up to 10,000x Data Reduction" (2605.08633)