Can Deep Neural Networks Improve Compression of Very Large Scientific Data?

Abstract: Error-bounded lossy compression is a fundamental technique for managing the rapidly growing volumes of scientific data produced by modern simulations and observational instruments. Most state-of-the-art-compressors follow a prediction-residual paradigm, where compression effectiveness depends on the quality of the predictor: more accurate predictions generate smaller residuals that are easier to compress. This observation raises a question: can modern machine learning models serve as superior predictors for scientific data compression? Answering this question directly is challenging because developing compression-specific ML predictors requires substantial resources. Instead, we leverage the climate domain where highly accurate pretrained weather forecasting foundation models already exist, making them an ideal testbed. We present a framework that integrates spatial and temporal deep learning models into a conventional error-bounded compression pipeline. The framework supports auto-regressive forecasting models and avoids error accumulation. Using ERA5 climate data as a representative large-scale scientific dataset, we evaluate three distinct ML predictors: a VAEformer-based codec (CRA5), a graph neural network forecaster (GraphCast), and a vision-transformer forecaster (Aurora), against the state-of-the-art compressor SZ3.1 under identical quantization and entropy-coding backends. Our evaluation over approximately 1.7 TB of data reveals a surprising result: although ML predictors generate more accurate predictions and can improve reconstruction quality by up to 91% while achieving up to 9.6x higher compression ratios for highly predictable variables, they do not improve overall dataset-level compression ratio. We show that prediction accuracy alone is insufficient: the spatial structure of the resulting residuals plays a decisive role in entropy coding efficiency.

• The paper demonstrates that substituting classical predictors with DNNs improves reconstruction fidelity but does not enhance overall compression ratio due to the structured nature of residuals.
• The study benchmarks various models on the ERA5 dataset, revealing that ML predictors excel for smooth, predictable variables while underperforming for turbulent fields.
• The analysis highlights the need for spatially-aware entropy coders and compression-aware training objectives to fully leverage DNN advantages in scientific data compression.

Deep Neural Networks as Predictors for Bounded Lossy Compression of Large-Scale Scientific Data

Introduction

Error-bounded lossy compression is a linchpin technique for managing exabyte-scale scientific datasets, providing guarantees on the maximum deviation between original and reconstructed values. The central architectural paradigm—prediction-residual encoding—realizes its efficacy through predictively minimizing the entropy of the stored residual stream. Contemporary compressors such as SZ3.1 employ spatial polynomial predictors to exploit local coherence, but they are limited in capturing physical priors and long-range dependencies.

This study interrogates the viability of leveraging pretrained, domain-specific deep neural networks (DNNs) as predictors within an error-bounded pipeline, benchmarking them against classical methods using a unified backend in quantization and entropy coding. Uniquely, the experiment is not limited to compression-specific neural architectures, but repurposes highly-accurate weather forecasting foundation models (GraphCast, Aurora, CRA5) as generic predictors, conducted on the canonical ERA5 climate dataset ($\sim$1.7 TB).

Compression Framework and Models

The core compression pipeline involves four stages: (1) prediction, (2) error-bounded quantization, (3) entropy encoding, and (4) outlier handling. Typical methods’ performance is tightly coupled to the predictor module that feeds quantization and entropy coding with small-magnitude, spatially coherent residuals, requisite for maximal compression.

Figure 1: Prediction-residual error-bounded compression pipeline, highlighting the central role of the predictor for overall efficiency.

Baselines

• SZ3.1: High-performance spatial predictor based on multi-dimensional polynomials, executes each timestep independently—serves as the SOTA baseline.
• CRA5: Transformer-based VAE codec optimized for ERA5, producing spatial predictions per variable.
• GraphCast: GNN auto-regressive temporal forecaster, mapping current states to subsequent predictions on an icosahedral grid.
• Aurora: Large-scale Swin Transformer U-Net, operating on patches, trained on multi-source reanalysis data.

  Figure 2: Illustration of the spatial prediction/interpolation principle, where black symbols are processed, and arrows denote interpolation.

These ML models are integrated as drop-in predictors, with the error-bounded back-end enforcing uniform quantization ($\epsilon r_X$ bin width) and fixed, lossless entropy coding.

Figure 3: A single ERA5 field (e.g., 2-meter temperature) at one timestep, showcasing the highly structured, spatially coherent nature of the data relevant to predictor effectiveness.

Experimental Protocol

The evaluation used 1,997 six-hourly timesteps spanning nine common ERA5 variables, quantized at three error bounds ($\epsilon=10^{-2},\,10^{-3},\,10^{-4}$), processed independently per variable/level.

• Objective: Evaluate aggregate and per-variable compression ratio, fidelity (MAE/RMSE), quantized-residual statistics, and throughput.
• Uniform Backend: All predictors share an identical quantizer and entropy code, isolating the impact of the prediction module.

Aggregate Compression and Fidelity Results

Dataset-Level Compression Ratio

Table results demonstrate that SZ3.1 consistently outperforms all ML-based predictors in aggregate compression ratio at all error bounds, with ML-based models only reaching comparable ratios when model-weight overhead is ignored or for highly predictable variables. For instance, at $\epsilon=10^{-2}$, SZ3.1 achieves $\rho\sim 200$, compressing 1.7 TB to 8.6 GB.

Speed

As anticipated, traditional methods exhibit much higher throughput due to simple predictors, while Aurora and GraphCast—especially given their auto-regressive architecture and, for GraphCast, CPU-bound execution—are orders of magnitude slower.

Reconstruction Fidelity

Despite not winning in compression, ML-based predictors (notably CRA5 and Aurora) demonstrate significant improvements in per-variable mean absolute error (up to 91% lower for geopotential), with error distributions that are smoother and less structured than SZ3.1, which is prone to quantization artifacts.

Figure 4: CDFs of pointwise absolute error at $\epsilon=10^{-2}$, illustrating variable-dependent differences in error concentration around zero.

Spatial and Per-Variable Analysis

ML predictors yield markedly better compression ratios and fidelity for smooth, highly predictable fields like geopotential and mean sea level pressure, often exceeding SZ3.1 by up to $9.6\times$ in per-variable CR. For less predictable variables (winds, humidity), their advantage vanishes or becomes negative compared to SZ3.1.

Figure 5: Time-averaged per-pixel MAE at $\epsilon=10^{-2}$ demonstrates that ML-based errors are spatially organized, following synoptic patterns.

Figure 6: Per-variable compression ratio at $\epsilon=10^{-2}$, log-scale, highlights dramatic gains by ML-based predictors for select variables.

Figure 7: Per-level analysis shows Aurora’s compression ratio for geopotential surging above $45,000\times$ at upper levels, with diminishing returns at lower/turbulent fields.

Analysis of Prediction, Residual Structure, and Coding

Entropy Bottleneck

Although DNN predictors lower pointwise RMSE/MAE, they do not always reduce the entropy of the quantized residual stream. The key result is that improved accuracy does not guarantee lower compressibility: ML residuals are spatially structured, with corrections concentrated along physical gradients or weather features, rather than as spatially uncorrelated zero-centered noise that 1D entropy coders exploit efficiently.

Figure 8: Binary support mask for quantized residuals (T2m, $\epsilon r_X$0): ML correction regions are spatially contiguous, classical (SZ3.1) are randomly distributed pixels.

This 2D structure is not utilized by conventional row-major, 1D entropy coders (e.g., Huffman+Zstd), thus the actual benefits from lower RMSE are not realized in the final compression ratio. A significant implication is that further improvement will require spatially/temporally-aware residual coding, or predictors co-trained with an information-theoretic objective.

Stability and Overhead

Regret from initialization overhead (e.g., seed timesteps for auto-regressive models) is amortized over long sequences; the error-bounded feedback loop ensures stability with no error accumulation over thousands of steps.

Figure 9: Compression ratio versus sequence length, showing rapid amortization of seed overhead for long sequences except at the tightest ratios.

Figure 10: Normalized MAE over 1,997-step autoregressive rollouts confirms absence of error drift, enabled by step-wise error correction.

Implications and Future Directions

Practically, this analysis reveals that the main bottleneck in realizing compression gains from ML predictors in error-bounded frameworks now lies in entropy coding, not in prediction. Theoretical implications include:

• Residual entropy, not only prediction error, governs compression ratio.
• Spatial coherence in residuals, beneficial for DNN error correction, is detrimental under backends blind to such structure.
• Hybrid frameworks must consider joint optimization of predictors and coders.
• Dataset-level results are dominated by turbulent, high-entropy fields; ML gains reside in smooth-structured fields.

Key directions for future work include developing 2D/3D spatial-entropy coders, compression-aware training objectives (minimizing residual entropy under error constraint), or adaptive, variable-dependent hybrid pipelines.

Conclusion

The systematic experimental investigation on error-bounded lossy compression using deep neural network predictors finds that, for large-scale scientific data, substituting classical spatial predictors with DNN-based temporal or learned-spatial predictors does not increase dataset-level compression ratio when exchanging the predictor alone. ML-based predictors yield improved reconstruction fidelity and per-variable gains for smooth variables, but overall compression is limited by the entropy characteristics of the residuals and the inability of 1D coders to exploit structured corrections. The study delineates the boundaries separating ML-centric from classical approaches, and identifies necessary innovations in residual coding and objective functions as the next frontier for ML-enabled scientific data compression.

Reference: "Can Deep Neural Networks Improve Compression of Very Large Scientific Data?" (2606.14353)