Unlimited-Working-Set In-Cache RX Path

• The paper demonstrates that explicit cacheline invalidation in the RX pipeline overcomes DRAM bottlenecks, enabling full line-rate packet processing for arbitrarily large working sets.
• It outlines a method where packet data is DMA-ed into the SmartNIC's cache, processed on an Arm core, and then transferred directly to the host without incurring costly DRAM traffic.
• The study quantitatively shows that FlexiNS sustains full line-rate performance even when the RX working set exceeds the cache capacity, contrasting sharply with traditional SmartNIC designs.

Unlimited-working-set in-cache processing RX path refers to a set of SmartNIC design and runtime mechanisms that enable full line-rate receive (RX) path packet processing, independent of DRAM bandwidth or working set size, by keeping all active RX buffer data in the SmartNIC's last-level cache (LLC) and promptly invalidating buffer cachelines after use. This approach, as realized in FlexiNS, eliminates DRAM bottlenecks for software network stacks executing on SmartNICs, thereby supporting arbitrarily large RX rings or descriptor sets without throughput degradation (Chen et al., 25 Apr 2025).

1. Motivation and Technical Challenges

Traditional SmartNIC-centric stacks place incoming packets into the SmartNIC’s DRAM, which is then accessed by an onboard CPU (such as Arm) for protocol processing. However, SmartNIC DRAM bandwidth is typically much lower than modern network line rates; for example, the Nvidia BlueField-3 offers 480–700 Gbps DRAM bandwidth versus 2×400 Gbps network bandwidth. Each packet may traverse SmartNIC DRAM multiple times for RX: network → DRAM → CPU → DMA → host, quickly overwhelming available DRAM bandwidth as the RX working set grows. When the working set (total RX buffers) exceeds LLC size—common with e.g., 16 RX queues × 512 descriptors × 4 KiB = 32 MB—DRAM becomes a fundamental throughput limiter.

The objective is thus the realization of a receive path that sustains wire-rate packet processing regardless of RX ring size, exploiting the SmartNIC’s cache to maximum effect and bypassing SmartNIC DRAM for post-processed RX data.

2. Architectural Principles and Mechanisms

Conventional DDIO Limitation

Direct Data I/O (DDIO), as implemented by Intel platforms, provides a partial solution: network DMAs write incoming packets directly to the CPU or SmartNIC’s LLC, bypassing DRAM. This is effective only if the entire RX buffer working set can reside in the LLC. When the set exceeds cache capacity, cacheline thrashing and forced evictions drive data to DRAM, degrading throughput.

Unlimited-working-set In-Cache RX Core Insight

The key insight underpinning unlimited-working-set in-cache processing is that RX packet data, after being processed and transferred to the host (via DMA), is no longer needed on the SmartNIC—the corresponding cachelines can be invalidated and immediately reused for subsequent packets. If processing promptly invalidates post-transfer RX cachelines using hardware-supported cache-invalidate primitives, processed RX buffers never cause DRAM writes, even as working set size grows without bound.

Procedural sequence:

1. NIC hardware receives a packet, DMA’s payload into SmartNIC LLC (via DDIO).
2. Onboard Arm core executes protocol (header) processing solely within cache.
3. Arm core initiates a DMA transfer of the packet payload to the host.
4. Immediately after transfer, Arm invokes a hardware cacheline-invalidate instruction (not a generic ARM dcimvac but a platform-specific flush/invalidate) on cachelines holding the RX buffer.
5. Subsequent incoming packets can DMA into these now-invalidated cachelines. No SmartNIC DRAM traffic is incurred for packet data in this path.

This pipeline is structured as a streaming microarchitecture, with the only requirement for line-rate absorption being that

$$\text{LLC size} > \text{Network Rate (bytes/sec)} \times \text{Average RX Processing Latency (sec)}$$

thus enabling the cache to absorb in-flight RX buffers during bursts. For example, at 400 Gbps and 10 μs processing latency, only 500 KB of cache can maintain continuous line-rate reception, a requirement readily met by modern SmartNICs.

3. Detailed RX Path Execution and Algorithms

The unlimited-working-set RX processing pipeline divides into four principal stages:

1. DMA into LLC: Packet ingress via network hardware is DMA’d directly into cache, not DRAM.
2. Protocol Processing: In-cache Arm CPU logic analyzes packet headers and, if needed, prepares an ACK.
3. DMA to Host: Payload is forwarded to the host via direct DMA (zero-copy transfer).
4. Explicit Invalidation: Arm software issues specific cache-invalidate instructions to mark post-transfer RX buffer cachelines as free; hardware avoids writing these invalidated lines to SmartNIC DRAM on eviction.

RX buffer organization aligns buffers to LLC cacheline boundaries (commonly 64B) to ensure safe invalidation. All processing, including pipeline and batch work, can be performed entirely in software using standard C/C++ for maximum programmability; explicit cache management leverages only the NIC platform’s hardware APIs.

RX Path Stage	Operation	DRAM Traffic (RX)
DMA to LLC	Packet data to cache	No
Protocol proc.	Header handled in-cache	No
DMA to Host	Payload direct to host	No
Cache Invalidate	Lines marked for reuse, not written back	No

4. Performance, Evaluation, and Comparative Analysis

Empirically, when the RX working set exceeds LLC size (e.g., >16 MB), conventional SmartNIC RX throughput degrades by factors of 1.6–2.9x compared to FlexiNS. In contrast, FlexiNS sustains full line-rate (demonstrated up to 48 MB working set) with <0.8 GB/s SmartNIC DRAM traffic, i.e., essentially all RX packet handling is "in-cache." This behavior contrasts sharply with both native SmartNIC software (which saturates DRAM as working set grows) and DDIO-based hosts (which depend on LLC capacity for performance).

Feature	FlexiNS Unlimited-In-Cache RX	Prior SmartNIC RX (Naïve)
RX Working Set > LLC	Yes (fully supported)	No (performance drops)
DRAM BW Used (RX)	<0.8 GB/s (<2%)	Grows with working set
Max Throughput (large set)	Line-rate	34–60% of line-rate
Programmability	High (C/C++ on Arm)	High, perf-limited
HW Requirements	HW invalidate API	No (but not practical)

A plausible implication is that in environments supporting explicit cacheline management, such as BlueField-3, RX pipeline design can decouple data-plane throughput from SmartNIC DRAM limitations. This enables use cases involving deep RX descriptor rings or large burst buffers without DRAM-induced head-of-line blocking.

5. Impact, Limitations, and Deployment Considerations

FlexiNS’s unlimited-working-set in-cache RX design allows SmartNIC software network stacks to scale RX throughput to line-rate under arbitrary working set sizes, facilitating fully programmable stacks on off-path SmartNICs. This in turn permits advanced transport logic and user-level protocol engines without recourse to hardware-only approaches.

Memory bandwidth is conserved for other tasks, reducing SmartNIC DRAM energy use and potentially improving device longevity. Furthermore, the RX path’s exclusive use of cache for packet handling supports robust burst absorption, a key requirement for cloud and disaggregated storage applications.

A limitation is that this approach depends on the presence of a hardware cache-invalidation API suitable for use in a streaming, fine-grained packet pipeline (as found in BlueField-3). If this cache management primitive is not available or performant, effectiveness may be compromised. Additionally, protocol processing must complete before cacheline invalidation, requiring careful pipeline management to avoid premature cache reuse.

6. Broader Context and Comparison with Adjacent Approaches

Stack Type	RX Path Data Handling	Memory Pressure	Programmability	Throughput (large set)
Linux/Microkernel	Host CPU memcopy from DRAM	High host CPU/DRAM	High	CPU/mem limited
HW Offload	ASIC direct DMA to host	None on SmartNIC	Low	Line-rate (for fixed HW)
Naïve SmartNIC	NIC→SmartNIC DRAM→Arm→Host	Very high DRAM	High	DRAM BW limited
FlexiNS	NIC→LLC [process+invalidate]	Near-zero	High	Line-rate (any set size)

The distinctive feature of FlexiNS versus traditional or hardware-offloaded stacks is the decoupling of RX throughput from DRAM, while maintaining full software programmability on the Arm core. This suggests that large-scale, software-defined SmartNIC applications (block storage disaggregation, KV cache transfer, flexible transport protocol logic) can be constructed without transmission-path hardware bottlenecks even at high line rates (Chen et al., 25 Apr 2025).

7. Formulaic Guidelines and Practical Example

The general cache sizing requirement is:

$$\text{Required LLC size} > \text{Network line rate (bytes/sec)} \times \text{Average RX packet processing latency (sec)}$$

Example: For 400 Gbps (50 GB/s) and 10 μs processing latency,

$$50\,\mathrm{GB/s} \times 10\,\mu\textrm{s} = 500\,\mathrm{KB}$$

This result indicates that modern SmartNICs with multi-megabyte LLCs are amply provisioned for unlimited-working-set in-cache RX adoption in line-rate operation.

8. Summary

Unlimited-working-set in-cache processing RX path, as exemplified by FlexiNS, relies on explicit hardware cacheline invalidation and careful pipeline design to efficiently process packet payloads without incurring SmartNIC DRAM bottlenecks. This method enables software-based RX logic to retain line-rate performance irrespective of RX working set size and provides a foundation for scalable, programmable network stacks in SmartNIC environments (Chen et al., 25 Apr 2025).