SZ3.1: State-of-the-Art Scientific Compressor

• SZ3.1 is an error-bounded lossy compressor employing spatial prediction with dynamic spline interpolation to guarantee pointwise error bounds in large scientific datasets.
• Its pipeline integrates error-controlled quantization, Huffman coding, and lossless compression to optimize residual entropy for superior compressibility.
• Benchmark studies on ERA5 and other datasets show SZ3.1 achieves competitive compression ratios and high throughput, even when compared with advanced ML predictors.

SZ3.1 is an error-bounded lossy compressor for scientific data in the SZ family, used in recent comparative studies as a state-of-the-art baseline and as a representative implementation of the prediction–residual paradigm (Alhumaidi et al., 12 Jun 2026, Li et al., 8 May 2025). In those studies it is described as using a spatial predictor based on dynamic multidimensional interpolation or dynamic spline interpolation, followed by error-controlled quantization, Huffman coding, and final lossless compression. Its contemporary importance derives from a recurring empirical result: even when deep spatial or temporal predictors yield more accurate forecasts, SZ3.1 often remains highly competitive because overall compressibility depends not only on prediction fidelity but also on the entropy and spatial organization of the quantized residuals.

1. Definition and role within scientific data compression

SZ3.1 belongs to the class of error-bounded lossy compressors designed for large numerical arrays produced by simulations and observational systems. In both recent studies that use it extensively, it is positioned as a strong baseline for scientific workloads, particularly climate and geoscience data. One study treats it as a high-performance spatial polynomial or spline interpolator baseline within a shared compression backend, while another describes it as the SZ family’s prediction-based, error-bounded lossy compressor configured with dynamic spline interpolation (Alhumaidi et al., 12 Jun 2026, Li et al., 8 May 2025).

Its defining architectural choice is spatial rather than temporal prediction. For the ERA5 study, SZ3.1 predicts from spatial neighbors within the same timestep and compresses each timestep independently. This makes it structurally different from auto-regressive temporal models such as GraphCast and Aurora, which must consume previous corrected states and roll forward sequentially. The same studies therefore use SZ3.1 to isolate a central research question: whether replacing the predictor alone, while holding quantization and coding fixed, is sufficient to improve scientific compression.

This positioning has two consequences. First, SZ3.1 serves as a canonical baseline for dataset-level storage efficiency under strict pointwise error guarantees. Second, it functions as a probe of a broader methodological issue: whether superior prediction accuracy necessarily translates into superior compressibility. The evidence summarized below indicates that, under the backends used in these studies, the answer is generally negative.

2. Compression pipeline and formal error guarantee

In the ERA5 analysis, SZ3.1 is embedded in the canonical prediction–residual pipeline. Let $x$ be the true value and $\hat{x}$ the prediction; the residual is $r = x - \hat{x}$. For a variable $X$ at timestep $k$, the residual field is written as $d_{X,k} = t_{X,k} - p_{X,k}$. The predictor uses spatial neighbors within timestep $k$ and has no access to temporal context. After prediction, error-bounded quantization is applied with a relative pointwise error bound $\epsilon$ and a fixed bin width per variable, determined from the global value range

$$r_X = \max_{k,i,j} t_{X,k}^{(i,j)} - \min_{k,i,j} t_{X,k}^{(i,j)},$$

with

$\Delta = \epsilon \cdot r_X.$

Quantization is mid-tread rounding to the nearest bin,

$\hat{x}$0

or, in the paper’s notation,

$\hat{x}$1

Decompression reconstructs

$\hat{x}$2

so that the corrected state satisfies

$\hat{x}$3

The integer residuals $\hat{x}$4 are entropy coded by Huffman using a Zlib implementation, unpredictable outliers are stored as raw true values, and the final bitstream is further compressed losslessly with Zstandard. In the shared-backend framework, these quantization and coding stages are identical across SZ3.1 and the ML predictors; only the source of $\hat{x}$5 differs (Alhumaidi et al., 12 Jun 2026).

This pipeline explains why SZ3.1 remains a useful reference point. Its prediction stage is relatively simple and purely spatial, but its error guarantee is explicit and its coding path is aligned with standard scientific compression practice. In the GraphComp study, the same broad structure appears in the description of the baseline family: error-controlled quantization, Huffman coding, and lossless compression, with the absolute bound derived from the relative bound as $\hat{x}$6 and the pointwise guarantee written as

$\hat{x}$7

The compression ratio is defined as

$\hat{x}$8

(Li et al., 8 May 2025)

3. Performance on ERA5 under identical backends

A detailed comparison against learned predictors was carried out on ERA5 climate data, using a $\hat{x}$9 global latitude–longitude grid of size $r = x - \hat{x}$0, a 6-hourly sequence of $r = x - \hat{x}$1 consecutive steps, and approximately $r = x - \hat{x}$2 TB of single-precision floats. Nine variables were evaluated across the common coverage of all predictors: single-level T2m, U10, V10, MSL, and multi-level T, Z, U, V, Q, with representative level results shown for 500 and 850 hPa. No cross-variable or cross-level sharing was used; each variable was compressed independently (Alhumaidi et al., 12 Jun 2026).

At the dataset level, SZ3.1 achieved the highest compression ratio at all tested relative error bounds.

$r = x - \hat{x}$3	SZ3.1 $r = x - \hat{x}$4	Best competing ML predictor $r = x - \hat{x}$5
$r = x - \hat{x}$6	198.80	CRA5 130.54
$r = x - \hat{x}$7	36.07	CRA5 19.83
$r = x - \hat{x}$8	11.72	CRA5 7.05

At $r = x - \hat{x}$9, the study states that SZ3.1 compresses approximately $X$0 TB to approximately $X$1 GB. The corresponding dataset-level values for the learned predictors are CRA5 130.54, GraphCast 59.87, and Aurora 55.87. Excluding model weights improves the apparent compression ratio of temporal models, especially Aurora at $X$2, but SZ3.1 still leads overall. The same pattern persists at $X$3 and $X$4 (Alhumaidi et al., 12 Jun 2026).

The per-variable picture is more nuanced. At $X$5, ML predictors can greatly exceed SZ3.1 on highly predictable smooth fields. For geopotential $X$6, Aurora reaches approximately $X$7 versus approximately $X$8 for SZ3.1, or approximately $X$9 higher. Similar gains are reported for MSL and T2m. By contrast, for turbulent fields such as mid-tropospheric $k$0, $k$1, $k$2, and specific humidity $k$3, SZ3.1 retains the advantage, and at $k$4 it leads on every variable.

Reconstruction quality likewise depends on the bound. At $k$5, CRA5 yields the lowest MAE on 8 of 9 variables and achieves up to 91% MAE reduction on geopotential $k$6 versus SZ3.1. Aurora reduces MAE by 27–34% for the smoothest fields, notably MSL and $k$7, but not for winds or humidity. Spatial error maps show that SZ3.1’s errors are relatively uniform and can introduce localized artifacts, whereas learned predictors produce smoother and meteorologically coherent errors. At $k$8, however, the crossover occurs: SZ3.1 has the lowest MAE and RMSE on most variables (Alhumaidi et al., 12 Jun 2026).

4. Residual entropy as the decisive mechanism

The most important interpretive result of the ERA5 study is that prediction accuracy alone is insufficient. For a quantized residual field $k$9, the achievable rate per value $d_{X,k} = t_{X,k} - p_{X,k}$0 is bounded below by the symbol entropy $d_{X,k} = t_{X,k} - p_{X,k}$1, and at fixed $d_{X,k} = t_{X,k} - p_{X,k}$2 the study gives the rough relation

$d_{X,k} = t_{X,k} - p_{X,k}$3

with $d_{X,k} = t_{X,k} - p_{X,k}$4 the raw bit-width per value. Compression ratio is therefore governed by residual symbol entropy rather than by forecast accuracy alone (Alhumaidi et al., 12 Jun 2026).

The paper illustrates this with explicit examples. For T2m at $d_{X,k} = t_{X,k} - p_{X,k}$5, CRA5 has better reconstruction error than SZ3.1, with $d_{X,k} = t_{X,k} - p_{X,k}$6 versus $d_{X,k} = t_{X,k} - p_{X,k}$7, but worse residual statistics: $d_{X,k} = t_{X,k} - p_{X,k}$8 versus $d_{X,k} = t_{X,k} - p_{X,k}$9, zero-bin fraction 92.4% versus 96.2%, entropy $k$0 of 0.5 bits versus 0.3 bits, and compression ratio 103.5 versus 117.4. For $k$1 at 850 hPa, Aurora has $k$2 versus SZ3.1’s $k$3, but $k$4 versus 0.8 bits, and compression ratio approximately 34.0 versus 47.9.

The spatial organization of residual support explains this behavior. Learned predictors generate corrections that cluster in meteorologically coherent regions such as coastlines, fronts, and cyclones, while SZ3.1 tends to produce scattered isolated pixels. The backend in that study consists of a global-frequency Huffman stage and a 1-D Zstd stage; it does not exploit 2-D spatial coherence. Under such a backend, structured residuals can be more expensive than sparse unstructured residuals even if their total magnitude is lower. A common misconception is therefore that a more accurate predictor must improve compression. The cited evidence shows instead that, within the current backend, isolated small corrections can compress better than fewer, larger, spatially coherent ones (Alhumaidi et al., 12 Jun 2026).

5. Standing in broader benchmark suites

A separate benchmark study compared SZ3.1 against GRAPHCOMP, HPEZ, SPERR, and ZFP on large-scale real and synthetic datasets, again under relative pointwise error bounds $k$5. In that work, SZ3.1 is explicitly described as using a dynamic spline interpolation approach enhanced by optimization strategies, and it is repeatedly characterized as a strong baseline or runner-up, especially on high-temporal geoscience datasets (Li et al., 8 May 2025).

The reported numbers show both SZ3.1’s strength and its limits. On RedSea96K temperature data, SZ3.1 reaches 105.21 at $k$6, 14.94 at $k$7, and 6.13 at $k$8. On ERA5, its compression ratios are particularly high for smoother variables: Era5-18-T gives 370.19 at $k$9, while Era5-18-G gives 858.46; Era5-23-G reaches 860.51 at $\epsilon$0. On Miranda, SZ3.1 records 574.60 at $\epsilon$1, but in that dataset it is only third or fourth among the competitors. Scale is the principal case in which SZ3.1 surpasses GRAPHCOMP, with 167.30 versus 147.36 at $\epsilon$2 and 40.40 versus 39.75 at $\epsilon$3, though HPEZ remains best across all tested $\epsilon$4 on that dataset (Li et al., 8 May 2025).

The broader pattern is that GRAPHCOMP consistently achieves the highest compression ratio across most datasets, outperforming the second-best method by margins ranging from 22% to 50%. Where SZ3.1 is the second-best method, the margins frequently fall in that interval. For example, on RedSea96K at $\epsilon$5, SZ3.1 records 105.21 while GRAPHCOMP reaches 141.75; on Era5-23-G at $\epsilon$6, SZ3.1 reaches 860.51 and GRAPHCOMP 1,130.25. The study also observes that HPEZ and SZ3.1 show only minimal gains in compression ratio as the number of timestamps increases, whereas GRAPHCOMP benefits strongly from temporal correlations. This suggests that SZ3.1’s spatial predictor remains powerful, but its lack of explicit temporal modeling becomes increasingly consequential on long sequences. Qualitative analyses in the same work show that, for RedSea96K at $\epsilon$7, SZ3.1’s decompression errors are concentrated over land regions such as Eastern Africa, the Arabian Peninsula, and India (Li et al., 8 May 2025).

6. Throughput, operational trade-offs, and research directions

SZ3.1’s continuing relevance is not reducible to compression ratio alone. In the ERA5 study, at $\epsilon$8 it compresses at approximately 349 MB/s and decompresses at approximately 1,385 MB/s. Those figures exceed the corresponding reported speeds for CRA5, GraphCast, and Aurora. The same study emphasizes three practical advantages: no model weights, no seed frames, and random access to timesteps without rolling an auto-regressive model. By contrast, temporal methods require two lossless seed timesteps and sequential decoding, and model weights can be large enough to halve compression ratio at moderate bounds if they are counted in $\epsilon$9. GraphCast was run on CPU because its JAX GPU inference is non-deterministic, while compression requires bit-reproducibility (Alhumaidi et al., 12 Jun 2026).

The limitations identified for SZ3.1 are equally clear. The GraphComp benchmark states that SZ3.1 may fail to fully capture irregular spatial relationships and does not explicitly model temporal dynamics. The ERA5 study adds a more specific diagnosis: under the shared Zlib-Huffman plus Zstd backend, the decisive bottleneck is not merely predictor accuracy but the mismatch between residual spatial structure and coder assumptions. This points directly to the research directions proposed in that work: learned entropy models for $$r_X = \max_{k,i,j} t_{X,k}^{(i,j)} - \min_{k,i,j} t_{X,k}^{(i,j)},$$0, such as context-adaptive arithmetic coding and hyperpriors; 2-D or 3-D context modeling or transforms on residual fields; run-length or block coding aligned with residual support maps rather than only row-major repeats; compression-aware predictors trained to minimize residual entropy $$r_X = \max_{k,i,j} t_{X,k}^{(i,j)} - \min_{k,i,j} t_{X,k}^{(i,j)},$$1 or a differentiable proxy; and hybrid schemes that combine temporal prediction with spatial transforms or multi-scale residual modeling (Alhumaidi et al., 12 Jun 2026).

Taken together, the recent literature portrays SZ3.1 as more than a legacy baseline. It remains a strong dataset-level compressor under shared error-bounded backends for ERA5, and a frequently competitive method across diverse scientific datasets. At the same time, the comparative results show that its strengths are tied to a specific predictor–coder compatibility: residuals that are small in magnitude and, crucially, well matched to order-0 Huffman coding and 1-D lossless backends. This suggests that future progress will depend less on replacing SZ3.1 with a more accurate predictor in isolation than on redesigning the full predictor–residual–entropy stack so that residual structure, not only reconstruction error, is optimized.