OpenZL: Graph-Based Lossless Compression

• OpenZL is a flexible, modular framework for lossless compression that uses a directed acyclic graph to orchestrate composable codecs.
• It integrates a multi-layered software stack—from a C11 core to Python bindings—facilitating rapid development and secure deployment.
• Empirical benchmarks demonstrate superior compression ratios and speeds across varied datasets, with successful enterprise-scale deployments at Meta.

OpenZL is a flexible, modular framework for lossless data compression based on a graph-theoretic formalism in which the compression process is represented by a directed acyclic graph (DAG) of specialized, composable codecs. Unlike generic compressors—whose performance improvements often come at the expense of throughput and resource use—OpenZL enables rapid development and deployment of tailored, application-specific compressors with minimal engineering overhead. By using a self-describing wire format and a universal decoder, it eliminates deployment lag, facilitates security auditing, and has demonstrated superior compression ratios and processing speeds on diverse, large-scale real-world datasets. OpenZL has been deployed internally at Meta, reducing development timelines and supporting scalable, maintainable compression in data-intensive applications.

1. Graph Model for Lossless Compression

OpenZL is founded on the "graph model" of compression, where the principal abstraction is the computational graph. Each node in the DAG embodies a codec—a pair of functions $(C, D)$ for encoding and decoding—which operate not on arbitrary byte streams, but on strictly typed messages. The formal definition is:

• Encoder: $C: I \rightarrow O$
• Decoder: $D: O \rightarrow I'$ with the property $D(C(\mu)) = \mu$ for all $\mu$ in the input space $I$.

Edges encode data dependencies: outputs from one codec are routed as inputs to others, enabling arbitrary compositions. This design generalizes compression beyond monolithic pipelines, facilitating complex workflows—such as tokenizing a sequence $\mu$ into an alphabet stream $\alpha$ and index stream $\nu$, which are then processed by different downstream codecs (e.g., Huffman, LZ77).

Because the graph is acyclic, compression proceeds via topological feed-forward execution, while universal decompression is performed in reverse (backpropagation order). The model supports dynamic "function graphs," allowing runtime expansion of the graph contingent on input data characteristics, which is resolved into a static graph before decompression.

2. Modular and Layered Implementation

OpenZL’s implementation is structured as a multi-layered software stack:

• Core Library: A C11 implementation (libopenzl) providing a deterministic, allocation-free engine for executing compression graphs. Kernels are optimized for focused responsibility and minimal state.
• C++ Façade: Supplies RAII resource management, error handling, and integration with other systems, wrapping the C API.
• Python Bindings: Enable rapid prototyping, data science integration, and direct access to buffers as NumPy arrays or PyTorch tensors.
• Codec Design: Each codec comprises an inner kernel (hot loop) for core computation and an outer binding layer for handling types, bounds, and buffer semantics.
• Type System: OpenZL enforces message type distinctions between opaque byte streams, string streams, fixed-size struct streams, and numeric streams.
• SDDL Tool: The Simple Data Description Language (SDDL) permits textual specification of data formats (e.g., CSV, Thrift), generating parsers that emit dispatch instructions, which are embedded in the compressed frame, ensuring the format remains self-describing throughout the workflow.
• Dynamicity Support: Via function graphs, OpenZL can select subgraphs dynamically at runtime. The resolved graph (static, with all dynamicism eliminated) is encoded in the compressed frame, guaranteeing that universal decoding remains deterministic and well-defined.

3. Empirical Performance and Benchmarking

Comprehensive experimental results demonstrate OpenZL’s competitive advantage over widely deployed generic compressors (xz, gzip, zstd):

• On the “ppmf_person” dataset (2020 US Census, CSV format), OpenZL achieves a compression ratio of nearly 117:1—55% higher than xz-9—and compresses at speeds an order of magnitude faster than xz.
• Benchmarks span CSV, Parquet, GRIB (climate reanalysis), and proprietary formats, showing that the OpenZL trade-off curve (compression ratio vs. speed) dominates comparable tools across most tested datasets.
• For climate data (e.g., ERA5 Pressure/Wind GRIB files), OpenZL achieves compression ratios approximately twice those of zstd, while preserving highly competitive decompression speeds, even when parsing overhead is present.
• Tabulated results detail ratio, compression throughput, and decompression rates (MiB/s), substantiating performance claims even when preprocessing is necessary for highly structured formats.

4. Deployment and Scalability in Enterprise Contexts

OpenZL is engineered for production-scale deployments:

• Meta Deployments: Integrated into infrastructure systems including Nimble (columnar data warehousing), Scribe (Thrift data streaming), and Feature Storage. For Thrift data in Scribe, OpenZL improved compression ratios by ~15% over Zstd on massive streaming workloads.
• Universal Decoding: Self-describing frame architecture and universal decoder eliminate traditional "rollout lag" for new compression graphs, enabling near-instantaneous deployment across large infrastructure footprints.
• Managed Compression: Meta’s managed system automates offline search over data samples to yield optimal compressor configurations, deployable widely via configuration files. This automation reduces engineering timelines from months to days.
• Specialized Use Cases: Successfully applied to compressing PyTorch model checkpoints (yielding ~17% storage savings), bfloat16 embedding tables, and aggregator log streams.
• Scalability: Modular graph design permits parallelization and adaptation for heterogeneous data and evolving workload requirements, both at the data center level and potentially in customer-facing applications.

5. Security and Maintainability

Security and steady maintainability are prioritized in OpenZL’s design:

• Codec Modularity: Small, self-contained codecs with explicit interface boundaries minimize the attack surface and facilitate rigorous vetting and testing.
• Universal Decoder: Frames encode their computational graph, obviating the need for external decoder logic and legacy codebases with potential vulnerabilities. The decoder operates generically on any conforming graph description.
• Standard Component Library: Publicly registered codecs, including entropy coders and conversion modules, foster reuse of audited, trusted code rather than bespoke implementations.
• SDDL Sandboxing: The parser generated from textual data format descriptions operates in a sandboxed environment, reducing exposure when processing untrusted data streams.
• These properties collectively streamline security reviews, debugging, and ongoing maintenance, while accommodating rapid iteration on new codecs and data formats.

6. Applications Across Data-Intensive Domains

OpenZL’s graph-based, typed-channel paradigm supports adaptation to a broad spectrum of real-world workloads:

• Structured Data Compression: By distinguishing and separating types (strings, numerics, structs), OpenZL efficiently compresses heterogeneous records such as CSV files and Parquet datasets, including complex sources like census and taxi trip data.
• Scientific Data: In genomics and climate modeling (GRIB formats), where data channels show intricate internal correlations, OpenZL enables type-specific clustering and transformation for superior ratio and throughput.
• Enterprise and ML Workflows: Supports backend warehousing, high-throughput logging, Thrift-serialized data, and neural network pipeline checkpointing (e.g., PyTorch), frequently bypassing serialization overhead by operating directly on multiple input streams.
• Extensibility: Future work will target improved support for textual domains (currently areas where compressors like xwrt outperform OpenZL), as well as high-dimensional data such as float images or 3D meshes.
• OpenZL’s engineering rationale: scalable integration, high-throughput operation, granular compression control, and universality of decoding in contemporary data infrastructures.

7. Significance and Prospects

OpenZL establishes a rigorous paradigm for practical lossless compression, unifying compositional codec graph methodology with robust engineering practice. The separation of concerns at the codec level, universal decoding capabilities, and embedded format self-description reconcile the need for both customization and maintainability in production settings. Performance evaluations substantiate leadership in both ratio and speed domains against established compressors, with internal deployments corroborating scalable and secure operation. The framework’s design facilitates further research and enhancement—potentially with machine learning guidance—opening paths toward future advances in data compression for heterogeneous, resource-constrained environments.