Progressive-X: Precision Data Compression & Fitting

• Progressive-X is a framework that provides progressive, precision-tunable data compression and retrieval for large-scale scientific data and robust geometric multi-model fitting.
• It incrementally builds results via a sequence of components, each reducing error and enabling adaptive trade-offs between accuracy and computational resources.
• The framework integrates with arbitrary compressors and employs modified RANSAC strategies, ensuring robust performance for both data reconstruction and model inference.

Progressive-X denotes a class of frameworks and algorithms designed to address two domains: (1) progressive-precision lossy-to-lossless data compression and retrieval for large-scale scientific data, and (2) robust geometric multi-model fitting with anytime capabilities and guaranteed solution quality if interrupted. The term encompasses both a general compressor-agnostic progressive compression technique for floating-point fields (Magri et al., 2023) and the Prog-X anytime multi-model fitting algorithm (Barath et al., 2019). Both approaches share the thematic principle of progressive, incremental construction of results, permitting tunable accuracy and adaptive resource consumption.

1. Multiple-Component Progressive Compression for Floating-Point Fields

The Progressive-X compression framework supports progressive-precision queries for floating-point fields independently of the underlying compressor or data representation (Magri et al., 2023). Let $x \in \mathbb{R}^n$ be the original vector-valued field. Progressive-X constructs a sequence of $K$ components $c_1, \dots, c_K$ such that the $K^\mathrm{th}$ partial sum $x_K = \sum_{i=1}^K c_i$ refines the reconstruction of $x$. Each $c_i$ encodes the residual $r_i = x - x_{i-1}$ (with $x_0 = 0$) using a user-defined error tolerance $\epsilon_i$, resulting in

$c_i = D(C(r_i, \epsilon_i))$

where $C$ and $D$ are the compressor and decompressor, and $\|\ r_i - c_i \|_\infty \le \epsilon_i$ with a strictly decreasing sequence $\epsilon_{1} > \ldots > \epsilon_K \ge 0$. After $i$ components, the reconstruction error $e_i = x - x_i$ satisfies $\| e_i \|_\infty \le \epsilon_i$, and by allowing $\epsilon_K$ to approach machine precision, the process yields fully lossless recovery.

2. Architectural Overview and Algorithms

The framework is implemented as a component-based data pipeline. On the producer side, the original field is iteratively decomposed into $K$ components, each stored separately (for example, as component-indexed datasets in HDF5 or ADIOS). On the consumer side, any $M \le K$ subset of the components may be fetched and decoded, reconstructing $x_M$ as a partial sum:

• Compression pseudocode: For $i=1$ to $K$, compute $r = x - x_\text{approx}$, compress $C(r, \epsilon_i)$, decompress the result to $c$, and refine $x_\text{approx} \leftarrow x_\text{approx} + c$.
• Decompression pseudocode: For requested $M$ components, successively decompress and accumulate $c_i$.

Complexity is $O(K n)$ for full compression and $O(M n)$ for decompression given $M$ components. Only $x_\text{approx}$ and a buffer for each component are required at runtime.

3. Plug-In Integration with Arbitrary Compressors

Progressive-X is agnostic to the underlying compressor, provided the compressor supports a signature $$C(\text{data}:\mathbb{R}^n, \text{tol}:\mathbb{R}) \to \text{bytes}$$ and $D(\text{bytes}) \to \mathbb{R}^n$ guaranteeing $$\|\text{data} - D(\text{bytes})\|_\infty \leq \text{tol}$$. No modifications are required to the compressor or decompressor source. The framework automatically computes the residuals and dispatches tolerances at each stage. If block-level or bit-rate-based schemes are used, the wrapper relays these parameters directly. All low-level quantization and bit-plane coding routines remain unmodified.

A summary of integration features is provided below:

Feature	Description
Compressor Interface	$C$ and $D$ with user-supplied tolerances
Data Layout	Components stored as independent fields or array dimensions
File/IO Compatibility	Direct use of HDF5/ADIOS “component” dimension
Compressor Modification	None required; wrapped externally

4. Empirical Evaluation on Scientific Datasets

Progressive-X has been evaluated with four base compressors (zfp, SZ3, SPERR, MGARD) and their multi-component variants (mzfp, msz, msperr, mmgard) on the SDRBench suite (e.g., Miranda $384^3$, S3D $500^3$, Nyx $512^3$). With $K=8$ and geometrically decreasing tolerances $\epsilon_i$, each added component approximately halves the maximum error up to the double-precision accuracy limit.

The “accuracy gain” metric $G(R) = \log_2 (\sigma / E(R)) - R$, where $\sigma$ is the standard deviation and $E(R)$ is the RMSE at rate $R$, demonstrates that Progressive-X variants match or outperform dedicated progressive (e.g., idx2, pmgard) and single-component compressors over $R \in [0,10]$ bits/value. Throughput at $K=8$ is typically 2–3× the single-compression time, and decompression overhead is proportional to the number of components. At sufficient $K$, lossless compression ratios are within 10–20% of specialized lossless codecs.

5. Task-Driven Precision and Use Cases

Progressive-X addresses the varying precision demands of downstream analysis without requiring a priori error budgeting:

• Volume rendering may require only a few (e.g., $M=3$) components ($\sim$0.6 bits/value).
• Gradient computations amplify error by $O(h^{-1})$, necessitating $M=5$ for $\epsilon_2 \approx 2^{-4}$ (3 bits/value).
• Higher-order derivatives require more components (e.g., $M=8$, 6 bits/value).

Interactive clients and workflows can request additional components on demand, retrieving progressively refined data as necessary for their computations or visualizations. The transparent, compressor-agnostic design permits seamless integration with existing client/server and storage stacks, enabling data shipping at coarse preview quality or full accuracy as required.

6. Progressive-X for Geometric Multi-Model Fitting

The “Prog-X” algorithm (Barath et al., 2019) addresses the challenge of geometric multi-model fitting in the presence of noise and outliers. Classic RANSAC and Hough-style approaches are limited to dominant single models and do not support robust estimation of multiple, potentially heterogeneous model instances. Multi-model variants based on large candidate pools, preference clustering, or global energy minimization suffer from inefficiency, lack of interruptibility, and overgeneration of hypotheses.

Prog-X interleaves hypothesis sampling (via a modified RANSAC loop), near-duplicate rejection (using MinHash-accelerated Jaccard overlap), and instance consolidation by multi-label energy minimization:

• The compound model $C$ at each iteration collects the active set of models. Candidate models are proposed with support not already explained by $C$ according to a “compound-aware” MSAC score.
• The consolidation step re-assigns data points and prunes unsupported models by minimizing

$$E(L) = \sum_{p \in P} \phi(L(p), p) + w_s \sum_{(p, q) \in \mathcal{N}} [L(p) \ne L(q)] + w_l \sum_{h \in A} \delta[\exists p : L(p) = h]$$

• Termination occurs when the expected support for any further undetected model falls below the user-specified confidence threshold, according to a RANSAC-derived bound.

At every iteration, the current model set $A$ and labeling $L$ constitute a valid solution, yielding true “any-time” capability. Empirical benchmarks demonstrate that Prog-X achieves lower or comparable misclassification error to prior methods (e.g., Multi-X, RPA) on standard tasks (homography, two-view motion, segmentation) with typically linear runtime scaling.

7. Comparative Advantages and Limitations

Progressive-X in both contexts (compression, model fitting) provides:

• Fine-grained, transparent user control over the trade-off between accuracy and resource usage.
• True anytime or progressive operation, in that partial results are meaningful and can be returned upon early termination.
• Minimal required changes to existing infrastructure, as progressive operation is achieved via external wrapping or interleaving.
• Robust quantitative guarantees: for compression, explicit $L_\infty$ bounds at each component; for model fitting, RANSAC-style statistical guarantees on completeness.

Documented limitations include the need for some parameter tuning ($\epsilon$, label cost, confidence), potential dependence on the quality of the proposal engine (e.g., NAPSAC sampling in model fitting), and increased (linear) computational cost with the number of components or true models.

In summary, Progressive-X provides a mathematically grounded, empirically validated, and compressor- or instance-agnostic solution to progressive-precision data handling, with applications spanning scientific simulation data compression, distributed and interactive analysis workflows, and geometric model inference in the presence of ambiguity and noise (Magri et al., 2023, Barath et al., 2019).