---
title: 'EDPC: Efficient Dual-path Parallel Compression'
url: https://www.emergentmind.com/topics/efficient-dual-path-parallel-compression-edpc
type: topic
---

# EDPC: Efficient Dual-path Parallel Compression

Searching arXiv for the EDPC paper and closely related compression work.
Efficient Dual-path Parallel Compression (EDPC) is a learning-based, autoregressive lossless compression framework for general-purpose compression of large, heterogeneous byte streams, designed to improve compression ratio and runtime efficiency simultaneously through two coordinated design dimensions: a modeling path for richer probability estimation and a system path for higher-throughput execution [2507.18969]. In the reported evaluation on text, audio, image, scientific floating-point, and heterogeneous archive datasets, EDPC improves compression ratio by 3.20% on average over PAC, reaches 10 213.8 KB/min $\approx 170.23$ KB/s compression throughput, and reduces resource usage relative to advanced autoregressive compression models (ACMs), including up to 4× parameter reduction and 1.91× GPU memory reduction [2507.18969].

## 1. Problem setting and formal definition

EDPC targets lossless compression of byte streams arising from multi-source multimedia and data analytics workloads, including Enwik9, Book, Sound, Image, Float, Silesia, and Backup [2507.18969]. The motivating setting is characterized by bandwidth constraints, storage pressure, and real-time or near real-time requirements. Within this setting, the paper positions EDPC against both traditional compressors such as Gzip, Zstd, and 7z, and neural ACMs such as TRACE, OREO, PAC, and NNCP, arguing that existing ACMs are limited by insufficient fine-grained feature extraction in probability modeling and by serial execution patterns that leave heterogeneous hardware underutilized [2507.18969].

Like other ACMs, EDPC models a byte sequence $X=\{x_1,x_2,\ldots,x_{\text{len}}\}$ autoregressively. After encoding the first $t$ bytes with uniform probabilities, it predicts
$$
\Pr(i)=P(x_i \mid x_{i-t},\ldots,x_{i-1}),
$$
and performs arithmetic coding with the predicted conditional distribution. The factorization follows the standard autoregressive product
$$
P(x_1,\ldots,x_{\text{len}})=\prod_{i=1}^{\text{len}} P(x_i \mid x_{<i}),
$$
with context truncated to the last $t=16$ bytes in the reported experiments [2507.18969].

A defining property of EDPC is dynamic compression. There is no separate pretraining stage; model parameters are randomly initialized and updated online during compression with cross-entropy loss,
$$
\text{loss}=\text{CrossEntropy}(\Pr(i),x_i).
$$
Because encoder and decoder begin from identical random parameters and execute the same update trajectory, no pretrained model weights need to be transmitted [2507.18969]. This places EDPC firmly within the ACM tradition while making runtime efficiency a first-class design objective rather than a secondary implementation concern.

## 2. Dual-path organization of the framework

The term “dual-path” in EDPC refers to two coordinated optimization paths rather than two entropy coders or two independent compression modes. The first path is the modeling path, which strengthens conditional probability estimation. The second is the system path, which restructures feature representation and execution flow to raise throughput and lower memory cost [2507.18969].

| Path | Core components | Immediate function |
|---|---|---|
| Modeling path | Information Flow Refinement, Multi-path Byte Refinement Block | Improve feature diversity and cross-byte dependency modeling |
| System path | Latent Transformation Engine, Decoupled Pipeline Compression Architecture | Reduce memory/FLOPs and overlap prediction with encoding |

At the modeling level, EDPC introduces Information Flow Refinement (IFR), grounded in mutual information theory, and the Multi-path Byte Refinement Block (MBRB), a multi-branch MLP-style module intended to reduce redundancy and strengthen heterogeneous feature propagation. At the system level, EDPC introduces the Latent Transformation Engine (LTE), which compresses high-dimensional features into a compact latent space with a Feature Distribution Matrix (FDM), and the Decoupled Pipeline Compression Architecture (DPCA), which decouples GPU-side probability prediction from CPU-side arithmetic coding and adds multi-process parallel encoding [2507.18969].

This division is not merely descriptive. The reported results depend on the interaction between the two paths. MBRB improves the quality of predicted distributions; LTE and DPCA make those improved predictions computationally viable at higher batch sizes and higher throughput. A plausible implication is that EDPC should be read as a co-design framework: the probability model and the runtime pipeline are optimized jointly rather than sequentially.

## 3. Modeling path: IFR and the Multi-path Byte Refinement Block

The theoretical motivation for EDPC’s modeling path is expressed through mutual information. For a skip connection $S \in \mathbb{R}^{b\times d}$ and a single-branch residual output $X \in \mathbb{R}^{b\times d}$, the fused representation is $S+X$, with mutual information
$$
I_{SB}(S;S+X)=H(S)+H(S+X)-H(S,S+X).
$$
For $k$ branches with outputs $X_1,\ldots,X_k$, EDPC defines $X^{(k)}=\sum_{i=1}^{k} X_i$ and
$$
I_{MB}^{(k)}(S;S+X^{(k)})=H(S)+H(S+X^{(k)})-H(S,S+X^{(k)}).
$$
Since exact mutual information is intractable, the paper uses a Kraskov k-NN non-parametric estimator, and then defines Information Flow Refinement as the relative change
$$
\text{IFR}_{k_1\rightarrow k_2}=
\frac{I^{(k_1)}(S;S+X^{(k_1)})}{I^{(k_2)}(S;S+X^{(k_2)})}.
$$
Empirically, $I^{(k)}$ decreases as branch count increases, IFR$_{1\rightarrow 2}$ and IFR$_{1\rightarrow 3}$ are greater than 1, and IFR$_{2\rightarrow 3}$ is modest, which the paper interprets as diminishing returns beyond two branches [2507.18969].

This analysis motivates the Multi-path Byte Refinement Block. For input $X \in \mathbb{R}^{b\times F}$, the general $k$-branch form is
$$
\begin{aligned}
X_0 &= \text{LN}(X)\\
X_i &= \text{Linear}_i(X_0), \quad i=1,2,\dots,k\\
X_{\text{fused}} &= X_1 \odot X_2 \odot \cdots \odot X_k\\
X_{\text{ff}} &= \text{GeLU}(X_{\text{fused}})\\
X_{\text{out}} &= \text{Linear}_{\text{out}}(X_{\text{ff}})+X.
\end{aligned}
$$
The element-wise multiplicative fusion is intended to emphasize feature dimensions that are consistently strong across branches, thereby encouraging heterogeneous yet complementary feature propagation [2507.18969].

EDPC instantiates two MBRBs: a Local MBRB with MLP hidden dimension 2048 and a Global MBRB with hidden dimension 4096 [2507.18969]. The local-global distinction is architectural rather than probabilistic; both blocks operate on representations derived from byte history, but the ablations indicate different importance. On Silesia, removing Local MBRB changes compression ratio from 5.29 to 5.16, whereas removing Global MBRB reduces it to 4.98, indicating that global modeling is more critical for compression ratio in the reported configuration [2507.18969].

The 2-branch design is an explicit operating point rather than a default inherited from prior work. On Backup, moving from 2-branch to 3-branch changes ratio from 1.94 to 1.96, but speed falls from 10 214.8 KB/min to 7 826.4 KB/min and memory rises from 2592 MB to 2820 MB; on Silesia, the shift is from 5.31 to 5.34 with similar penalties [2507.18969]. EDPC therefore adopts a 2-branch MBRB as its default trade-off between information-flow refinement and execution cost.

## 4. System path: LTE and decoupled pipeline execution

The Latent Transformation Engine is EDPC’s primary mechanism for lowering the cost of high-dimensional probability modeling. Given $X \in \mathbb{R}^{b\times F}$ and compression ratio $r$, LTE defines latent dimension $F'=F/r$ and down-projects features as
$$
X' = W_1 X + b_1, \qquad X' \in \mathbb{R}^{b\times F'}.
$$
To mitigate the expressive loss of naive projection, LTE introduces a learnable three-dimensional tensor
$$
\mathcal{U} \in \mathbb{R}^{b\times F' \times F'},
$$
the Feature Distribution Matrix, and applies batch matrix multiplication
$$
Y = \text{bmm}(X',\mathcal{U}), \qquad Y_j = X'_j \cdot \mathcal{U}_j.
$$
A final up-projection reconstructs the original feature dimension,
$$
X'' = W_2 Y + b_2, \qquad X'' \in \mathbb{R}^{b\times F}.
$$
In the reported configuration, LTE uses compression ratio $r=4$ and sits between the MBRB stack and the output layers [2507.18969].

The LTE ablation illustrates its role as a resource-efficiency module with limited compression-ratio cost. On Silesia at batch size 4096, full EDPC reports ratio 5.29, memory 1546 MB, parameters $4.15\times 10^7$, and speed 157 KB/s; removing LTE yields ratio 5.33, memory 6374 MB, parameters $2.93\times 10^8$, and speed 73 KB/s [2507.18969]. At the FDM-module level, LTE reduces parameters by 93.02% and FLOPs by 43.76%; at whole-model level, the paper reports memory reduction of 76.01% and parameter reduction of 85.84% with LTE [2507.18969].

DPCA addresses a different bottleneck: serial coupling between probability prediction and arithmetic coding. In a conventional ACM loop, probability prediction and arithmetic encoding are chained step by step, leaving GPU and CPU alternately idle. EDPC instead assigns probability prediction to the GPU and arithmetic encoding to the CPU in a pipeline. At time step $i$, the GPU computes $P(x_i\mid\cdot)$ while the CPU encodes $x_{i-1}$ using the previously generated distribution, with predicted distributions passed through a shared buffer or queue [2507.18969].

DPCA also adds multi-process parallel encoding by partitioning each batch’s byte stream into independent segments. In the experiments, segment size is 32 and the process pool has 32 subprocesses [2507.18969]. This turns the implementation bottleneck from a single serial arithmetic coder into a pipelined heterogeneous runtime. The execution-only ablation on Silesia shows that removing the pipeline drops speed from 157 KB/s to 70 KB/s, corresponding to a 2.24× speedup attributable to DPCA alone; the encoding-time analysis reports up to 21.73× reduction relative to a serial ACM baseline [2507.18969].

## 5. Training protocol, benchmarks, and reported performance

EDPC is evaluated on seven datasets spanning multiple modalities: Enwik9 and Book for text, Sound for audio, Image for images, Float for scientific floating-point data, and Silesia and Backup for heterogeneous archives [2507.18969]. The training setup follows dynamic compression with Adam, learning rate 0.001, and batch sizes including 4096 and 8192. The reported hardware is a GPU with 34.10 TFLOPS FP32 and 12 GB HBM and an Intel Xeon Platinum 8269CY CPU at 2.50 GHz [2507.18969].

| Metric | EDPC result | Reported comparison |
|---|---|---|
| Compression ratio | +3.20% overall vs PAC | +2.99% on homogeneous data, +3.73% on heterogeneous data |
| Compression throughput | 10 213.8 KB/min $\approx 170.23$ KB/s | 2.7× PAC, $\sim$7× TRACE, >23× NNCP |
| Resource efficiency | Up to 48.1% less GPU memory and up to 75% fewer parameters vs PAC | Up to 4× parameter reduction and 1.91× GPU memory reduction relative to advanced ACMs |

The compression-ratio results are reported as larger-is-better. Relative to PAC, EDPC improves Enwik9 from 5.97 to 6.43 (+7.71%), Book from 5.05 to 5.23 (+3.56%), Sound from 2.25 to 2.28 (+1.33%), Float from 1.29 to 1.32 (+2.33%), Silesia from 4.99 to 5.31 (+6.41%), and Backup from 1.92 to 1.94 (+1.04%), while tying on Image at 1.96 [2507.18969]. Averaged across datasets, the paper reports gains of +68.95% over Gzip, +33.71% over 7z, +45.73% over Zstd-19, +18.12% over Dzip, +10.91% over TRACE, +5.99% over OREO, and +3.20% over PAC [2507.18969].

Speed is one of the framework’s central claims. The neural-compressor comparison reports NNCP at 261–432.6 KB/min, Dzip at 645.6 KB/min, TRACE at 1422.6 KB/min, PAC at 3789.6 KB/min, and EDPC at 10 213.8 KB/min [2507.18969]. The effect is especially notable because EDPC is not presented as a low-capacity approximation: the same paper also reports improved compression ratios and lower resource consumption relative to strong ACM baselines.

The ablation study clarifies component roles. Starting from full EDPC on Silesia at batch size 4096, replacing LTE with low-rank factorization reduces ratio to 5.09 and speed to 80 KB/s, removing Local MBRB increases speed to 201 KB/s but lowers ratio to 5.16, and removing Global MBRB increases speed to 258 KB/s but lowers ratio to 4.98 [2507.18969]. This suggests that the system path primarily controls feasibility and throughput, whereas the modeling path primarily controls distribution quality.

## 6. Relation to prior work, terminology, and limitations

Within learned lossless compression, EDPC is positioned against two families of baselines. Traditional compressors such as Gzip, Zstd, and 7z rely on heuristic or dictionary-based mechanisms that the paper describes as weak at modeling complex cross-byte dependencies on multimedia and floating-point data. Neural ACMs such as Dzip, TRACE, OREO, PAC, and NNCP improve compression by per-byte conditional modeling, but are described as either architecturally heavy or constrained by single-path FFN/MLP context modules and serial pipelines [2507.18969]. EDPC’s differentiator is thus not merely “neural compression,” but the explicit combination of multi-branch probability modeling with decoupled heterogeneous execution.

A common misconception is to read “dual-path” as a generic synonym for any two-branch network. In EDPC, the phrase has a specific meaning: one path improves probability modeling through IFR-guided MBRB design, and the other path improves execution through LTE and DPCA [2507.18969]. It does not denote two parallel entropy coders, and it is unrelated to uses of the same acronym in other fields. Notably, “EDPC” also denotes “Edge-Disjoint Paths Compilation” in a surface-code compilation paper [2110.11493], which is terminologically distinct despite the acronym overlap. Later learned-compression work such as “Efficient Learned Data Compression via Dual-Stream Feature Decoupling” [2604.07239] addresses similar modeling-versus-efficiency tensions through dual-stream feature decoupling and a concurrent stream-parallel pipeline, but it is a separate framework rather than an alternate name for EDPC.

The reported limitations are correspondingly specific. Dynamic compression avoids storing pretrained weights, but it requires encoder and decoder to execute identical online learning trajectories, which increases decoding compute relative to static codecs [2507.18969]. The benefits of DPCA also depend on heterogeneous hardware with both GPU and multi-core CPU resources; lower-end devices may not realize the same gains. The compression-ratio advantage over PAC, while consistent, is moderate in absolute magnitude at 3.20% overall, so the framework’s practical distinctiveness lies as much in its throughput and memory profile as in raw ratio [2507.18969]. The paper further notes that performance may vary on specialized data distributions outside the evaluated benchmarks and suggests future work in more advanced multi-branch architectures, further pipeline optimization, and extensions to specialized data types or database-engine integration [2507.18969].

EDPC therefore occupies a specific place in the development of autoregressive lossless compression: it preserves the ACM regime of online, byte-level conditional coding, but reworks both the probability model and the runtime substrate so that compression ratio, memory footprint, and throughput are treated as coupled optimization targets rather than isolated metrics.

Source: https://www.emergentmind.com/topics/efficient-dual-path-parallel-compression-edpc