Direct Compute Access (DCA) in Cache-Centric Systems

• Direct Compute Access (DCA) is a cache-centric I/O mechanism that redirects DMA traffic into the last-level cache to bypass DRAM for lower latency.
• It employs features like DDIO and RDCA to enable direct cache injection for I/O devices while introducing challenges such as cache contention and DMA leaks.
• System techniques like A4 and Lamda demonstrate that careful runtime management and cache reservation can improve throughput, reduce latency, and control DRAM bandwidth usage.

Searching arXiv for the cited work and closely related DCA/DDIO/RDCA papers. Direct Compute Access (DCA) denotes the broad idea of bringing data as close as possible to compute units—ideally into CPU or accelerator caches—while bypassing main memory and heavyweight software paths. In the materials considered here, DCA appears in two closely related senses. First, on modern Intel Xeon servers, Direct Cache Access and Intel “Data Direct I/O” redirect DMA traffic from DRAM into the last-level cache (LLC), allowing I/O devices to source data out of the LLC rather than always going to memory (Park et al., 12 Jun 2025). Second, in RDMA-based datacenter networks, “Remote Direct Cache Access” (RDCA) extends the same principle by making cache—not memory—the primary endpoint for NIC writes in the receiver host datapath (Li et al., 2022). Across both settings, the central technical issue is that cache-centric I/O can reduce latency and DRAM bandwidth consumption, but it also introduces new forms of cache contention, residency constraints, and microarchitectural coupling that become pronounced once network and storage devices operate at hundreds of Gbps.

1. DCA as a cache-centric DMA mechanism

In the Xeon setting, DCA is fundamentally a mechanism to redirect DMA traffic from DRAM into the LLC, and to allow I/O devices to source data out of the LLC rather than always going to memory (Park et al., 12 Jun 2025). Without DCA, the classical DMA path on a non-inclusive Xeon is device DMA-write into DRAM, followed by CPU loads from DRAM into private mid-level caches, with LLC fills occurring only as a side effect of evictions. With DCA, for DMA writes, if the target cache line is already in LLC, DMA performs an in-place write update; otherwise, the LLC performs a write allocate in a special subset of ways, referred to as DCA ways, and writes directly there, bypassing DRAM in the steady case (Park et al., 12 Jun 2025). For DMA reads, if the data is present in MLC or LLC, the device DMA-reads from the cache hierarchy; if it is not cached, the device DMA-reads from DRAM and does not allocate into LLC (Park et al., 12 Jun 2025).

This organization turns the LLC into a low-latency DMA buffer. When a NIC or NVMe SSD receives data, the device issues PCIe writes targeting LLC, the LLC allocates lines in designated DCA ways, and the CPU later reads these lines from LLC into its MLC, avoiding a DRAM access (Park et al., 12 Jun 2025). Coherence is maintained by treating DMA writes as coherent writes into LLC, and lines are written back to DRAM only when eventually evicted (Park et al., 12 Jun 2025). The LLC therefore plays two roles simultaneously: victim cache for CPU private L2 in a non-inclusive architecture, and first-class DMA buffer for I/O devices (Park et al., 12 Jun 2025).

The RDCA formulation generalizes this principle from local I/O to network receive processing. In that setting, the target is to “move memory out of the receiver host datapath and reserve a small area in the last level cache (LLC) for the RNIC to send data to, enabling remote hosts to directly access the receiver cache (RDCA)” (Li et al., 2022). Conceptually, the remote node sees an interface like RDMA, but the target is not DRAM; it is a reserved portion of the LLC. The RNIC writes incoming data into this LLC region and, in the normal case, never touches DRAM (Li et al., 2022). CPU and I/O devices such as SSDs then consume that cache-resident data directly. This suggests that “Direct Compute Access” is best understood not as a single ISA-level feature, but as a cache-centric data placement paradigm spanning coherent DMA, DDIO-style LLC injection, and receiver-side cache residency control.

2. Cache hierarchy semantics and non-inclusive LLC behavior

The most detailed microarchitectural account in the available material concerns recent Intel Xeon server processors using a non-inclusive LLC with an inclusive directory (Park et al., 12 Jun 2025). Reverse engineering reused in that work reports that each LLC slice has 11 data ways and a directory split into 11 “traditional” directory ways and 12 “extended” directory ways, with two directory ways shared between the two groups and 1:1 coupled with the two rightmost data ways (Park et al., 12 Jun 2025). Those two rightmost data ways are called inclusive ways, and they are the only LLC ways that can hold cache lines that are simultaneously present in the LLC and in the MLCs (Park et al., 12 Jun 2025). By contrast, standard ways hold LLC-exclusive lines only, while DCA ways are the dedicated target for DCA write-allocate (Park et al., 12 Jun 2025).

This coupling determines how DMA-written lines evolve under CPU access. DMA-written lines initially land in DCA ways as LLC-exclusive lines. When a core reads such a line into its MLC, the coherence state changes so that the line becomes shared and LLC-inclusive. Because only the inclusive ways can represent LLC-inclusive lines, the line is migrated from its original DCA way to one of the inclusive ways (Park et al., 12 Jun 2025). The paper identifies this migration as a hidden microarchitectural property rather than a documented architectural interface (Park et al., 12 Jun 2025).

The RDCA work approaches cache behavior from a systems rather than reverse-engineering perspective, but its assumptions are compatible with the same broad picture. Lamda reserves a small LLC region as a cache-resident buffer pool and uses Intel CAT and DDIO to ensure these local addresses map into the Lamda-reserved LLC region (Li et al., 2022). It also notes that CAT restricts allocation but not access, so instruction cache and other processes may still evict Lamda’s data; the implementation therefore slightly over-provisions the LLC ways, periodically touches random data in the pool to keep it hot, and disables prefetchers and some OS scheduling features that cause unwanted cache occupancy (Li et al., 2022). A plausible implication is that practical DCA deployment depends not only on whether I/O can land in LLC, but also on how precisely software can control residency, way allocation, and cross-client interference.

3. Contention mechanisms introduced by DCA

The Xeon study identifies two previously unrecognized LLC contentions triggered by emerging high-bandwidth I/O devices (Park et al., 12 Jun 2025). The first, denoted C1, arises because DMA-written cache lines in DCA ways are migrated to inclusive ways when accessed by CPU cores, unexpectedly contending with non-I/O cache lines within the inclusive ways (Park et al., 12 Jun 2025). Observation O1 is stated explicitly: when read by CPU cores, DMA-written I/O cache lines in DCA ways are migrated to inclusive ways, where I/O and non-I/O workloads contend (Park et al., 12 Jun 2025). This produces what the paper terms directory contention. In experiments with DPDK-T, which receives packets and touches them, and X-Mem, a cache-sensitive microbenchmark pinned to two ways using Intel CAT, the LLC miss rate of X-Mem spikes not only when it overlaps DCA ways and standard ways affected by DMA bloat, but also when it overlaps inclusive ways (Park et al., 12 Jun 2025). The inclusive-way spike disappears for DPDK-NT, which drops packets without touching them, and also disappears when DCA is disabled (Park et al., 12 Jun 2025). The contention is therefore specifically tied to DCA-written lines that are later read by CPU cores and migrated into inclusive ways.

The second contention, denoted C2, concerns storage versus network I/O in DCA ways (Park et al., 12 Jun 2025). Experiments with FIO and NVMe SSDs show that for block sizes greater than or equal to 32 KB at I/O depth 32, storage throughput is almost unchanged whether DCA is enabled or disabled, while memory bandwidth consumption remains high even when DCA is enabled because of DMA leak (Park et al., 12 Jun 2025). The DMA leak process is described as follows: the SSD DMA-writes a large block into DCA ways at high rate, subsequent DMA writes evict those cache lines before the CPU has time to consume the entire block, and later CPU reads fetch the data from DRAM, causing extra memory bandwidth consumption (Park et al., 12 Jun 2025). Storage transfers involve orders of magnitude more bytes per I/O than network packets, CPU processing begins only after the entire block is DMA’d, and larger data with more complex processing extends the consumption window (Park et al., 12 Jun 2025). When storage and network traffic co-run, both devices continuously DMA-write into the same DCA ways, and storage traffic with poor temporal locality evicts network packets before they are consumed (Park et al., 12 Jun 2025).

The RDCA paper describes an analogous problem at the scale of the receiver host datapath. It reports that high memory bandwidth consumption of modern distributed applications can lead to significant throughput drop and large increase of tail latency in high-speed RDMA networks because of contention of memory bandwidth between application processes and network processes (Li et al., 2022). Under high load, the RNIC cannot get enough DRAM bandwidth, its internal buffers fill, and packets are dropped inside the RNIC or backpressure is triggered (Li et al., 2022). The paper states that with 1 MB messages on both 25 Gbps and 100 Gbps setups, DDIO miss rate reaches approximately 100%, so every NIC write spills into DRAM, increasing rather than relieving DRAM pressure (Li et al., 2022). This is called the leaky DMA problem. Read together, the two studies suggest that DCA does not eliminate the memory system bottleneck automatically; rather, it relocates the bottleneck to a more complex interaction among LLC ways, directory semantics, DMA write allocation, and workload temporal locality.

4. RDCA and Lamda as a receiver-side realization of DCA

The RDCA work proposes Lamda, described as a receiver cache processing system that consumes a small amount of CPU cache to process received data from the network at line rate (Li et al., 2022). Lamda reserves a small region of LLC, for example 12 MB, as a cache-resident buffer pool and provides a cache-centric receive datapath of the form NIC to LLC to application or SSD (Li et al., 2022). Its three main components are a cache-resident buffer pool, a swift cache recycle controller, and a cache-pressure-aware escape controller (Li et al., 2022).

For small messages below 4 KB, Lamda uses SEND/RECEIVE with a Shared Receive Queue. Many QPs share one receive queue, and Lamda posts approximately 1K WQEs of 4 KB each into the SRQ for 32 QPs, giving an initial SRQ buffer of approximately 4 MB (Li et al., 2022). For large messages above 4 KB, Lamda receives data by issuing RDMA READs in a receiver-driven controlled fashion (Li et al., 2022). Large messages are fragmented into units no larger than 256 KB, the number of concurrent READ requests is bounded by a fixed-size sliding window such as 32, and a second window bounds the total bytes in flight, for example 8 MB (Li et al., 2022). Together, SRQ and READ windows provision a cache-resident pool of 4 MB for SRQ plus 8 MB for READ, ensuring by design that the aggregate footprint never exceeds 12 MB of LLC (Li et al., 2022).

The swift cache recycle controller minimizes post-RNIC residence time through a 4 KB slab allocator, shared cache for metadata and buffer release, parallelism via multithreading, and fine-grained pipelining (Li et al., 2022). Lamda and applications share an IPC region mapped to cache; Lamda writes buffer pointers there when data arrives, and applications notify completion through the same shared region without syscalls or extra memory copies (Li et al., 2022). The pipeline lets Lamda recycle a 4 KB slab as soon as it clears the get, process, and release or forward stages, rather than waiting for the entire message (Li et al., 2022). For storage workloads, CRC checks are offloaded on the NIC, and Lamda uses “huibuffer” for in-place serialization and deserialization without extra copying (Li et al., 2022).

The escape controller handles rare cases in which data remains in the cache-resident buffer longer than normal. It monitors free space in the cache pool and, as pressure increases, performs buffer replacement, CPU multithreaded memcopy to DRAM, or signals congestion by causing NIC firmware to emit CNPs via adjusted ECN thresholds (Li et al., 2022). The paper states that the additional DRAM bandwidth introduced by copying is bounded by an expression of the form extra bandwidth less than or equal to $\alpha \times$ network bandwidth, and gives the example that with a 100 Gbps NIC and $\alpha = 4\%$, this is at most 1 GB/s DRAM bandwidth (Li et al., 2022). This suggests that RDCA is not a claim that DRAM disappears universally, but that DRAM can be removed from the common-case critical path while bounded fallback mechanisms preserve correctness.

5. Runtime management and quality-of-service control

The Xeon study presents A4, a runtime LLC management framework designed to alleviate both C1 and C2 among diverse co-running workloads using a hidden knob and other hardware features implemented in those CPUs (Park et al., 12 Jun 2025). A4 is explicitly built on top of Intel CAT, programs way masks per class of service using intel-cmt-cat, leverages PCM for counters, and does not modify replacement policies or coherence behavior (Park et al., 12 Jun 2025). Its goal is to protect latency-sensitive high-priority workloads, particularly network services, from I/O-driven LLC contention while still allowing low-priority workloads to make use of residual LLC capacity (Park et al., 12 Jun 2025).

A4 maintains three logical zones over LLC ways: an HP Zone, an LP Zone, and a DCA Zone corresponding to DCA ways such as way0:1. It periodically samples hardware counters every 1 s, including LLC hits and misses per core, DCA hits and misses, PCIe read and write throughput per device, and DRAM bandwidth (Park et al., 12 Jun 2025). Every 10 s of stable operation, it temporarily reverts to the initial partition for 1 s to detect phase change (Park et al., 12 Jun 2025). Its policy for HP versus LP partitioning uses a threshold $T_1 = \text{HPW\_LLC\_HIT\_THR} = 20\%$ on relative drop in HPW LLC hit rate when shrinking the HP Zone, expanding LP allocation gradually until the hit-rate degradation exceeds the threshold (Park et al., 12 Jun 2025).

For C1, A4 avoids allocating inclusive ways to low-priority workloads. If I/O HPWs are present, the initial HP Zone is set to way[2:10], the DCA Zone is way[0:1], and the LP Zone starts at way[7:8], explicitly excluding inclusive ways way[9:10] from LP allocation (Park et al., 12 Jun 2025). This is motivated by observation O3 that overlapping inclusive ways with I/O workloads improves caching efficiency and reduces DMA bloat (Park et al., 12 Jun 2025). For C2, A4 detects storage-driven DMA leak using thresholds on DCA miss rate (DMALK_DCA_MS_THR, 40%), LLC miss rate for storage I/O (DMALK_LLC_MS_THR, 40%), and fraction of system I/O throughput due to storage devices (DMALK_IO_TP_THR, 35%) (Park et al., 12 Jun 2025). If the thresholds are exceeded, the storage workload is deemed a DCA antagonist (Park et al., 12 Jun 2025).

The hidden control that enables this policy is per-PCIe-port DCA control through undocumented fields in PCIe port register perfctrlsts_0 (Park et al., 12 Jun 2025). Setting NoSnoopOpWrEn disables “allocating writes” for that port, and clearing Use_Allocating_Flow_Wr prevents writes from using DCA flows (Park et al., 12 Jun 2025). This allows DCA to be disabled selectively for NVMe SSD ports while leaving it enabled for NIC ports (Park et al., 12 Jun 2025). A4 also labels workloads as antagonists if both MLC and LLC miss rates exceed ANT_CACHE_MISS_THR = 90\%, then progressively shrinks their allocations down to minimal trash ways, implementing pseudo LLC bypass (Park et al., 12 Jun 2025). In the RDCA setting, the same kind of runtime reasoning appears in different form: Lamda uses admission control based on expected throughput times expected post-RNIC timespan, controls the number and aggregate size of outstanding READs, and employs escape thresholds such as CACHE_SAFE, CACHE_DANGER, MEM_ESC, and CREDIT to prevent the cache-resident pool from becoming unstable (Li et al., 2022). Across both systems, DCA depends on active resource management rather than passive reliance on DDIO.

6. Empirical results, limitations, and broader implications

The A4 study reports that, overall, the framework improves the performance of latency-sensitive, high-priority workloads by 51% without notably compromising that of low-priority workloads (Park et al., 12 Jun 2025). In aggregate real-world experiments, HPW performance improves by 51% in an HPW-heavy mix and by 47% in an LPW-heavy mix relative to Default; total workload performance improves by 22% and 33%, respectively (Park et al., 12 Jun 2025). HPW LLC hit rate increases by 79% in the HPW-heavy mix and by 68% in the LPW-heavy mix, while overall LLC hit rate increases by 83% and 63% (Park et al., 12 Jun 2025). Under heavy storage I/O and mixed SPEC and Redis, the most complete A4 variant reduces Fastclick average network latency by approximately 58% versus Default and increases throughput by approximately 5% at the largest block size (Park et al., 12 Jun 2025). A4 also reduces overall DRAM read bandwidth by 11% versus Default despite 5% higher I/O throughput (Park et al., 12 Jun 2025). The daemon overhead is reported as less than 800 $\mu$s per iteration (Park et al., 12 Jun 2025).

The RDCA study reports that Lamda improves network throughput by 4.7% with zero memory bandwidth consumption on storage nodes, and improves network throughput by up 17% and 45% for large block size and small size under memory bandwidth pressure, respectively (Li et al., 2022). In a two-host testbed, with 256 KB messages under DRAM saturation, throughput improves by up to 1.96$\times$ versus DDIO baseline on a 100 Gbps setup and up to 1.54$\times$ on a 25 Gbps setup (Li et al., 2022). Average latency is reduced by 46.4% for 4 KB messages and by 74.8% for 256 KB messages on the PFC-enabled 25 Gbps network, with similar or larger reductions on the PFC-free 100 Gbps network (Li et al., 2022). DDIO “hit throughput” improves by up to 56.6$\times$ at 100 Gbps and 26.7$\times$ at 25 Gbps for 256 KB messages, while PCIe outbound stalled writes are reduced by up to 99.1% and 96.4%, respectively (Li et al., 2022). In production distributed storage, throughput improvement reaches up to 2.11$\times$, P999 latency for 256 KB messages is reduced by 86.4%, and DRAM bandwidth in the storage node datapath decreases by approximately 89% on average, with residual DRAM use below 0.5 GB/s coming almost entirely from escape mechanisms (Li et al., 2022). For HPC workloads, Lamda reduces latency by up to 35.1% for all-to-all and 25% for all-gather, with 5.5% for all-reduce (Li et al., 2022).

These results support several constrained conclusions. First, DCA is crucial for low-latency network processing, but can be largely useless for large-block storage I/O under deep queues because storage throughput may remain almost unchanged while cache churn and DMA leak remain high (Park et al., 12 Jun 2025). Second, DDIO-style cache injection by itself is passive; without explicit control of residency, concurrency, and eviction domains, DCA can degenerate into leaky DMA and intensify LLC contention rather than reduce it (Li et al., 2022). Third, non-inclusive LLCs with inclusive directories create subtle cross-way interactions, especially migration from DCA ways to inclusive ways, that are not exposed by public documentation but materially affect isolation and QoS (Park et al., 12 Jun 2025). Finally, the platform specificity is explicit: the inclusive-way behavior and per-port DCA registers are Skylake/Xeon-specific and undocumented, while CAT-based cache reservation restricts allocation rather than access and therefore requires software workarounds (Park et al., 12 Jun 2025, Li et al., 2022). A plausible implication is that broader adoption of Direct Compute Access would benefit from architected per-device controls, explicit cache-as-I/O interfaces, and hardware support for cache-aware QoS, rather than relying on undocumented knobs and reverse-engineered microarchitectural behavior.