---
title: Data Processing Efficiency (DPE)
url: https://www.emergentmind.com/topics/data-processing-efficiency-dpe
type: topic
---

# Data Processing Efficiency (DPE)

Data processing efficiency denotes the effectiveness with which a system converts resource expenditure into useful data transformation. Across recent literature, the term is instantiated in several technically distinct but structurally related ways: in blockchain-enabled Metaverse systems it is the ratio of processed data bits to weighted delay-and-energy cost; in analog in-memory inference it is the ability to keep more of a model’s computation inside the memory array while minimizing data movement and digital post-processing; in Lakehouse ETL it is evaluated through loading time and resulting table size; and in GPU and near-data systems it is tied to fusing or relocating execution so that less data traverses the storage–memory–host hierarchy [2411.16083][2511.13950][2604.21449][2605.10511][1905.04767]. Taken together, these works suggest that DPE is not a single universal scalar but a workload-dependent efficiency notion whose denominator may be dominated by delay, energy, storage overhead, host orchestration, or data movement.

## 1. Definitions and domain-specific scope

Recent work uses the term “data processing efficiency” to describe different optimization targets under a common systems premise: useful work should increase faster than resource cost. In RRAM-based in-memory computing, NL-DPE frames DPE as the ability to keep more of the model’s computation inside the memory array, thereby minimizing data movement and avoiding expensive digital post-processing [2511.13950]. In blockchain-enabled wireless Metaverse systems, DPE is explicitly introduced as “processed bits per unit of resource consumption,” with cost modeled through delay and energy [2411.16083]. In Data Lakehouse benchmarking, DPE is treated operationally through ETL loading time and the file-system volume of the resulting tables, rather than through a single closed-form metric [2604.21449]. In heterogeneous acceleration and GPU query engines, DPE is bound to schedule quality, inter-device communication cost, kernel fusion, and the elimination of fragmented execution [2502.06304][2605.10511].

| Domain | Operationalization | Representative source |
|---|---|---|
| Analog IMC for DNN inference | Keep more computation inside memory arrays; avoid ADCs and digital post-processing | [2511.13950] |
| Blockchain-enabled Metaverse | Processed data bits divided by weighted delay-plus-energy cost | [2411.16083] |
| Data Lakehouse ETL | Data loading time and resulting table/storage size | [2604.21449] |
| Heterogeneous CPU/GPU/FPGA execution | Data-aware scheduling under compute and communication constraints | [2502.06304] |
| GPU analytical query processing | Fused IO, decompression, and operators in a GPU-resident data path | [2605.10511] |

A recurring misconception is to equate DPE with raw throughput alone. The surveyed literature does not support that simplification. Some formulations optimize throughput and energy jointly, some optimize storage footprint against ingest latency, and some emphasize how much intermediate data is avoided rather than how many arithmetic operations are issued. The common thread is not a specific unit, but a systems-level concern with reducing non-productive overhead.

## 2. Metric formulations and optimization objectives

The most explicit mathematical definition appears in the blockchain-enabled Metaverse formulation, where DPE is defined as
$$
\text{DPE} := \frac{\text{processed data bits}}{\text{delay}+\text{energy}}.
$$
For user-side local processing and server-side offloaded processing, the weighted costs are
$$
cost^{(u)}_n = \omega_t T^{(up)}_n + \omega_e E^{(up)}_n,
$$
and
$$
cost^{(s)}_{n,m} = \omega_t \big(T^{(ut)}_{n,m}+T^{(sp)}_{n,m}+T^{(sg)}_{n,m}+T^{(bp)}_{n,m}+T^{(sv)}_{n,m}\big) + \omega_e \big(E^{(ut)}_{n,m}+E^{(sp)}_{n,m}+E^{(sg)}_{n,m}\big).
$$
The corresponding optimization problem maximizes the sum of local and offloaded DPE ratios over user association, offloading ratio, bandwidth allocation, transmit power usage ratio, server computing allocation ratio, local computing usage ratio, and server-side data-processing versus blockchain-processing split [2411.16083].

That formulation is significant because it makes DPE a cross-layer objective rather than a purely communication-side or compute-side measure. The numerator is processed workload, but the denominator includes local computation delay, wireless uplink delay, server computation delay, block generation delay, block propagation delay, validation delay, and the associated energy terms. This explicitly separates DPE from spectral efficiency, energy efficiency, and throughput-only metrics, each of which omits at least one part of the end-to-end cost model [2411.16083].

Other papers adopt operational proxies rather than a single ratio. The Lakehouse benchmark evaluates DPE using two criteria only: data loading time and the volume of the tables formed in the file system. It does not introduce a separate composite DPE score or normalized formula; its logic is strictly comparative under identical Spark ETL conditions [2604.21449]. This matters because it shows that, in data engineering practice, DPE can be a multicriteria assessment rather than a scalar objective. A plausible implication is that DPE definitions remain workload- and system-specific even when the label is shared.

## 3. Architectural principle: minimizing data movement and in-path overhead

The broadest architectural thesis comes from near-data processing and processing-in-memory. “Moving processing to data” is presented as a response to the fact that modern data-intensive workloads are often dominated by data movement across the storage–memory–CPU hierarchy rather than by arithmetic itself. Near-Data Processing executes operations fully or partially in-situ close to the physical data location; Processing-in-Memory is a subclass of this model that targets computation directly within or very near memory chips. The stated goal is to transfer back only results rather than raw data, exploiting high internal device bandwidth and reducing latency, bandwidth consumption, and energy spent on movement [1905.04767].

NL-DPE is a specialized realization of this principle for analog inference. Conventional RRAM crossbars compute the analog dot product
$$
\mathbf{y}=\mathbf{x}\mathbf{W},
$$
which is efficient for VMMs but inadequate for arbitrary non-linear functions and data-dependent matrix multiplications. NL-DPE augments crosspoint arrays with RRAM-based Analog Content Addressable Memory (ACAM), maps non-linear functions and DMMul-related subfunctions into decision trees, and places ACAM immediately after the crossbar so that analog outputs are consumed directly without standalone ADCs. The paper reports that ADCs can consume over 30% of area and more than 50% of power in conventional paths; in one comparison, the ACAM replacement incurs about 98% lower power and 86× smaller area than an 8-bit ADC plus activation logic. End-to-end, NL-DPE achieves 28× higher energy efficiency and 249× speedup over a GPU baseline, and 22× higher energy efficiency and 245× speedup over existing IMC accelerators, while maintaining high accuracy [2511.13950].

The same movement-minimization logic appears in GPU database execution. Data Path Fusion fuses IO, decompression, and query operators into a single GPU kernel so that execution no longer fragments across CPU-orchestrated kernels with intermediate materialization in GPU global memory. In the reported prototype, the host is reduced to plan generation and kernel launch, while the GPU remains in control throughout the data path. The resulting system reports speedups of 2.66 to 6.22 on TPC-H and 3.84 to 16.81 on SSB over the state-of-the-art approach in the representative configuration [2605.10511].

SkimROOT applies a near-storage variant of the same idea to LHC skimming. Instead of transferring large ROOT baskets to remote compute nodes for decompression, deserialization, and filtering, SkimROOT runs filtering on an NVIDIA BlueField-3 DPU attached to the storage server and returns only the reduced output. In the reported Higgs-analysis evaluation at 1 Gbps, client-side LZ4 filtering takes 382.1 s, whereas SkimROOT takes 8.62 s, corresponding to a 44.3× speedup [2506.04507]. Across these systems, DPE improvements arise less from isolated kernel optimization than from reducing unnecessary traversal of the data path.

## 4. Workload-aware scheduling, precision adaptation, and resource allocation

A second major theme is that DPE depends on matching execution strategy to the structure of the data. DyPe formalizes this for irregular workloads on heterogeneous GPU–FPGA systems. Rather than assuming a manually fixed partition, it analyzes input characteristics such as sparsity, input dimensions, arithmetic intensity, sequence length, and window width, then explores a multi-objective, multi-constraint design space over mappings of kernels to devices and groupings of kernels into pipeline stages. Its dynamic-programming states take the form \(dp[i][f][g]\), representing the best schedule for the first \(i\) kernels using \(f\) FPGAs and \(g\) GPUs. On real hardware, conventional static scheduling is optimal in only 13 out of 86 cases, whereas DyPe finds the optimal schedule in 77 out of 86 cases, with an average of only 3.95% performance or energy-efficiency loss in the sub-optimal cases. It yields on average 1.53× throughput and 1.09× energy efficiency over the static schedule baseline, and 1.44× throughput and 1.66× energy efficiency over the GPU-only baseline [2502.06304].

At the arithmetic-datapath level, DHFP-PE improves DPE by matching the processing element to the precision actually required. Its 6-stage pipelined MAC engine supports FP8 (`E4M3`, `E5M2`) and FP4 (`2 × E2M1`, `2 × E1M2`) by reusing a single 4-bit unit multiplier. In FP8 mode the multiplier operates as a standard \(4\times4\) block; in FP4 mode the same physical array is partitioned into two simultaneous \(2\times2\) multipliers. The reported 28 nm implementation operates at 1.94 GHz with 0.00396 mm\(^2\) area and 2.13 mW power, with up to 60.4% area reduction and 86.6% power savings compared to state-of-the-art designs. Reported throughput and efficiency are 3.88 GFLOPS and 1818 GFLOPS/W in FP8 mode, and 7.75 GFLOPS and 3632 GFLOPS/W in FP4 mode [2604.04507].

In wireless blockchain-enabled Metaverse systems, workload adaptation is expressed through joint resource allocation rather than device scheduling. The DAUR algorithm alternates optimization of user association, work offloading ratios, task-specific computing resource distribution, bandwidth allocation, user power usage ratios, and server computing resource allocation ratios. The paper reports that DAUR achieves the best DPE among the tested methods, with 86.48 M bits/(s·J) in the conference version and 87.87 M bits/(s·J) in the journal version, and that the algorithm converges quickly in practice [2411.16083][2507.04657]. The shared systems insight is that DPE is not fixed by hardware capability alone; it is shaped by runtime matching between data properties, precision, topology, and allocation policy.

## 5. Data engineering and scientific-operations instantiations

In Data Lakehouse architectures, DPE is measured at the ETL and storage layer rather than at the arithmetic or kernel level. The benchmark compares Apache Hudi, Apache Iceberg, and Delta Lake under the same Apache Spark workflow: four sequential ETL processes, identical transformations, and identical input datasets spanning structured CSV and semi-structured JSON up to 7 GB. Delta Lake is the fastest overall for loading tasks. On the largest integrated table, `weather_hourly_events`, Delta loads in 285.18 seconds, compared with 388.55 seconds for Iceberg and 737.03 seconds for Hudi; for `air_quality_level`, the corresponding times are 9.57 s, 10.7 s, and 69.33 s [2604.21449].

Storage efficiency tells a different story. Apache Iceberg consistently produces the smallest tables: for `weather_hourly_events`, 270 MB versus 306 MB for Delta Lake and 4.41 GB for Hudi; for `weather_events`, 248 MB versus 286 MB and 2.26 GB; for `air_quality_level`, 17 MB versus 19.7 MB and 474 MB [2604.21449]. The immediate implication is that DPE in analytical data systems can be multi-objective and format-dependent: Delta Lake is recommended when fast loading of data of any volume is the priority, while Apache Iceberg is recommended when disk space savings and stability are more important. The benchmark also states that Hudi is ineffective for the loading and storage tasks studied here, but may still be effective in incremental update and streaming processing scenarios that were not experimentally evaluated [2604.21449].

Scientific monitoring systems show yet another DPE regime: sustained operational responsiveness under continuous ingest. The Daya Bay onsite Performance Quality Monitoring system runs on an 11-server offline environment, processes about 320 raw files per day of about 1 GB each from three experimental halls, and uses database polling, file-transfer status checks, PBS job submission, NuWa reconstruction, histogram generation, and web publication. The timing budget is explicit: file transfer takes less than 5 minutes, job submission about 10 seconds, NuWa processing about 30 minutes, and histogram printing plus web publishing about 2 minutes, producing a typical total latency of about 40 minutes after a raw data file is closed [1406.2104]. Efficiency here derives from incremental per-file processing, dedicated and elastic compute resources, and dynamic histogram creation that reduces output ROOT file volume and avoids unnecessary pre-allocation.

## 6. Trade-offs, limitations, and terminological ambiguity

The surveyed literature consistently treats DPE as workload-dependent and often explicitly bounded by trade-offs. The Lakehouse benchmark is limited to loading and storage; it does not evaluate read performance, query latency, update/merge/delete operations, or streaming behavior, and its results are tied to a single-node local Spark environment [2604.21449]. DPF achieves substantial speedups, but the authors note that joins are supported only when the build-side hash table fits in GPU memory, IO is restricted to raw block devices through BaM, only GPU-FOR and FSST are integrated as representative compression schemes, the loader is still host-side, and multi-GPU support is left for future work [2605.10511]. SkimROOT is presented as a prototype-stage system evaluated single-threadedly on one core, with future work aimed at better prefetching, more parallelization, and scaling across multiple DPUs [2506.04507].

Algorithmic DPE optimization also comes with non-global guarantees. The DAUR framework converts a mixed-integer sum-of-ratios problem into alternating convex subproblems using fractional programming, SDR, DC refinement, and discrete recovery, but the reported convergence target is a stationary point rather than a global optimum [2411.16083]. More generally, the near-data processing review emphasizes that effective DPE gains require new architectures, ISAs, virtual-memory support, coherence handling, programming models, and toolchains; without these, movement reduction may remain limited to specialized cases [1905.04767].

The acronym itself is not semantically unique. In efficient code-generation evaluation, DPE denotes “Differential Performance Evaluation,” a framework that builds performance-exercising tasks and assigns a Differential Performance Score (DPS) relative to clustered reference solutions; its benchmark, EvalPerf, contains 121 tasks [2408.06450]. In interactive segmentation, DPE denotes “Dynamic Prompt Embedding,” a mechanism that processes only a region of interest and can reduce latency to below 25% of full prompt embedding for predominant small objects while having marginal impact on accuracy [2507.09612]. These usages are terminologically separate from data processing efficiency proper. Their coexistence underscores a practical point: in current arXiv literature, “DPE” is an overloaded acronym, whereas “data processing efficiency” remains a systems-level concept whose concrete meaning is specified by the workload, cost model, and execution substrate under study.

Source: https://www.emergentmind.com/topics/data-processing-efficiency-dpe