RoCE BALBOA: FPGA RDMA Stack

• RoCE BALBOA is an open, RoCE v2-compatible RDMA stack designed for FPGA-based SmartNICs that supports 100G throughput and programmable in-line services.
• It integrates host and GPU direct DMA capabilities with AXI-stream interface controls for customizable packet processing, security, and compute offload.
• The design demonstrates near-line rate performance, reliable connection semantics, and versatility in deployment with commodity RNICs and datacenter switches.

to=arxiv_search.query 彩票总代理 ส่งเงินบาทไทยjson {"query":"\"RoCE BALBOA\" OR (Heer et al., 27 Jul 2025)","max_results":5,"sort_by":"relevance"} to=arxiv_search.query 񎔎ని ็ตทรูjson {"query":"(Heer et al., 27 Jul 2025)","max_results":5,"sort_by":"relevance"} to=arxiv_search.query 天天中彩票提现json {"query":"(Weintraub et al., 2 Jul 2025) OR (Borrill, 5 Mar 2026)","max_results":5,"sort_by":"relevance"} RoCE BALBOA is an open-source, RoCE v2-compatible, 100G-capable RDMA stack for FPGA-based SmartNICs and accelerators that is intended to be fully interoperable with commodity RNICs and datacenter switches while exposing a programmable, high-throughput datapath for security services, inspection, and application-specific compute at line rate. It is positioned as a production-ready, standards-compliant RoCE v2 transport in an extensible form, with deployment on AMD Alveo U55C FPGAs and mixed operation with Mellanox/NVIDIA ConnectX-5 NICs across commodity Ethernet fabrics such as Cisco Nexus 9000 switches. The design targets researchers and practitioners who require host-bypassing RDMA semantics, direct DMA to host or GPU memory, and architectural control over packet processing, scheduling, retransmission, and offload insertion that is not available in closed commercial NIC stacks (Heer et al., 27 Jul 2025).

1. Scope, goals, and system position

BALBOA is released under an open-source license with hardware, software, and a testbench ecosystem, and it has been deployed on AMD Alveo U55C FPGAs and ported to multiple shells, including Coyote v2 and ONIC/RecoNIC. Its stated scope is not merely functional RoCE connectivity, but a foundation for building SmartNICs and NIC-attached accelerators that operate directly on network data streams. In that role, it combines standards compliance with explicit module boundaries on AXI-stream interfaces so that user-defined services can be inserted either in the datapath or alongside it without breaking RoCE compliance or 100G operation (Heer et al., 27 Jul 2025).

The system is explicitly differentiated from commercial NIC and DPU stacks by openness and modifiability. Commercial products are described as closed, with programmable cores that often operate off the datapath and face cache or memory bottlenecks and limited line-rate coverage. BALBOA instead exposes control over packet, classifier, scheduler, DMA, and QP/CQ/WQE handling, which makes it suitable for prototyping protocol extensions, infrastructure services, and deep compute pipelines. This suggests that BALBOA occupies a middle ground between FPGA-only research RDMA prototypes and datacenter-deployable NICs: it retains interoperability with commodity switching and RNIC ecosystems while allowing modifications inside the transport datapath itself (Heer et al., 27 Jul 2025).

A central architectural consequence of this positioning is support for DMA to host memory and GPU memory, including direct-to-GPU data paths in the style of GPUDirect RDMA. The stack is therefore oriented not only toward conventional host memory transport but also toward accelerator-centric dataflows in which the NIC or SmartNIC performs computation before placing data directly into GPU memory (Heer et al., 27 Jul 2025).

2. Transport model, verbs, and reliability mechanisms

BALBOA implements RoCE v2 in Reliable Connection mode over IPv4, with parsing and generation of UDP, IP, and InfiniBand transport headers. The operations it targets are one-sided verbs—specifically RDMA WRITE, RDMA READ and READ RESPONSE, and RDMA ACK/AETH processing—while atomics are not implemented and two-sided SEND/RECV are not the focus of the design. This choice aligns the transport with cloud and disaggregated-system usage patterns in which minimal host involvement and compatibility with stream-oriented offloads are primary requirements (Heer et al., 27 Jul 2025).

Per-QP state is maintained in BRAM-based tables that hold connection parameters, expected PSNs, and MSN sequencing for multi-message transfers. Default capacity is 500 QPs, configurable according to resource budgets. Commands arrive on a control stream per QP and payloads on a data stream; on receive, RETH and AETH decoding drive DMA to host or GPU memory, while completions are generated upon ACK reception or READ RESPONSE handling and surfaced to software through the Coyote thread API. The software model therefore resembles conventional InfiniBand verbs usage, including completion polling such as checkCompleted(LOCAL_WRITE) (Heer et al., 27 Jul 2025).

Memory registration and zero-copy semantics are implemented through a cThread abstraction that allocates host RDMA buffers, exchanges QP information out-of-band over TCP, and returns a pointer for zero-copy DMA. Scatter-gather entries permit multi-offset placement and burst handling. DMA engines write directly to host RAM or GPU memory without CPU copying. For retransmission, outgoing payloads are retained in card memory until acknowledged, specifically in a dedicated HBM channel, which avoids PCIe re-fetch during recovery (Heer et al., 27 Jul 2025).

Reliability is based on per-QP PSN sequencing, duplicate and out-of-order dropping, a transport timer for timeout detection, and retransmission under RC semantics. Flow control is ACK-clocked: each transmitted request consumes credit from a per-QP budget, and incoming ACKs replenish that budget. Receive acceptance is additionally gated by host or GPU-side queueing capacity; if credits are insufficient for the current packet, the packet is dropped locally to avoid stalling, and sender-side RC retransmission performs recovery. The current release does not implement explicit ECN/DCQCN state machines or TIMELY; the CMAC IP supports PFC, and the stack is structured to be extendable to DCQCN or TIMELY, but ACK-clocked crediting is the default (Heer et al., 27 Jul 2025).

The transport design is quantified in the paper with simple capacity models. Buffering and credit provisioning are framed through the bandwidth–delay product,

$BDP = C \times RTT,$

with the example that at $C = 100$ Gbps and $RTT \sim 10\ \mu s$, the BDP is approximately $125$ kB, so per-QP credits and retransmit buffers should provision at least the BDP to maintain a full pipe without drops. Throughput and latency are similarly summarized as

$$T_{pipeline} = \min_i T_i,\qquad L_{total} = \sum_i L_i,$$

which matches the empirical conclusion that end-to-end throughput is determined by the slowest stage and that latency is dominated by packet processing rather than by iCRC generation (Heer et al., 27 Jul 2025).

3. Datapath architecture and hardware realization

BALBOA is organized as a set of AXI-Stream data and control planes, which makes the design portable across FPGA shells. On the receive side, the parse/classify pipeline strips IP, UDP, and IB headers and evaluates them via FSMs; BTH decoding extracts opcode, QPN, and PSN, while RETH supplies address, rkey, and length, and ACK processing produces completion events. On the transmit side, commands and per-QP state, including MSN and PSN, generate RETH and BTH headers; payload merging is interleaved with header formation; UDP and IP headers are prepended; and iCRC is computed before transmission to the CMAC (Heer et al., 27 Jul 2025).

The internal datapath is 512 bits wide at 250 MHz, corresponding to 128 Gbps of internal bandwidth and thus providing headroom for 100G operation even with header and iCRC overhead. Critical datapaths are designed for initiation interval $II=1$. Retransmission support is integrated through arbitration and stream multiplexing of WRITE and READ-RESPONSE payloads from the host into a single stream, mirrored to HBM for later release or reload. On-chip BRAM stores connection, state, and MSN tables and small buffers such as flow-control counters, whereas HBM is reserved for retransmission payloads to avoid PCIe roundtrips (Heer et al., 27 Jul 2025).

Packet compliance includes full parse and generation of IPv4 and UDP headers, InfiniBand BTH, RETH, and AETH/ACK processing, and line-rate iCRC. The iCRC engine is implemented as optimized parallel CRC pipelines tuned to common AXI beat sizes, specifically 512-bit full beats, 320-bit partial beats, and a multi-stage 32-bit path, leveraging RoCE’s 4-byte alignment guarantees. iCRC is inserted after IB, UDP, and IP header formation, and final Ethernet framing is emitted by the 100G CMAC per IEEE 802.3 (Heer et al., 27 Jul 2025).

A concise post-route resource summary for the AMD Alveo U55C is given below.

Component	LUTs	Other resources
Whole BALBOA	~43.7k	101 BRAM, ~103k FF
Packet processing	~29.9k	66 BRAM, ~75k FF
ICRC	~12.5k	—
Flow control	~787	—
Retransmission logic	~587	34 BRAM
AES ECB core	~65.7k	—
ML DPI	~54.4k	—

These figures indicate that the transport itself occupies a modest fraction of the U55C fabric—reported as 3.4% LUTs, 5.1% BRAM, and 4.0% FF for the whole BALBOA design—while service modules such as AES and ML DPI can be significantly more resource-intensive than the core transport logic (Heer et al., 27 Jul 2025).

4. Service insertion model and programmability

A defining feature of BALBOA is the distinction between two service integration modes. On-datapath services are in-line modules placed before or after packet processing on AXI streams and operate on payloads at line rate; the AES example belongs to this class. Parallel-path services receive a multiplexed copy of packets or payloads in parallel to the main pipeline and return decisions or metadata to control interfaces; ML-based DPI is the canonical example. The paper states that the latency of such parallel-path services is hidden by packet-processing latency so that they do not stall the datapath (Heer et al., 27 Jul 2025).

The developer workflow is based on attaching HLS, VHDL, or Verilog blocks to AXI-stream interfaces and using provided control buses to access QP state or annotate commands. An example annotation is a “malicious” flag returned by a DPI module. A simulation framework is included to validate offloads against BALBOA’s behavior, and a 100G packet sniffer can produce PCAP traces for debugging. At the software boundary, the C++ cThread API abstracts QP creation, RDMA buffer registration, and verbs-like operations such as invoke(REMOTE_RDMA_WRITE, &sg), while offloads are associated with threads or QPs via user logic slots in the shell (Heer et al., 27 Jul 2025).

This modularity is important because BALBOA’s programmability is not limited to side-channel software control; it is integrated into the packet and DMA path itself. A plausible implication is that BALBOA can be used to study how transport semantics, security functions, and application-specific compute interact when all three share the same timing and buffering constraints, rather than being composed as separate subsystems. That emphasis distinguishes it from programmability models in which custom logic executes after packet reception or outside the line-rate datapath (Heer et al., 27 Jul 2025).

5. Deployment characteristics, interoperability, and measured behavior

The reported deployment uses mixed endpoints consisting of AMD Alveo U55C FPGAs and NVIDIA/Mellanox ConnectX-5 NICs connected by 100G Ethernet through Cisco Nexus 9000 switches, with both one-tier and two-tier fat-tree topologies tested. Network parameters include an MTU of 4 KB, the minimum IEEE 802.3-compliant CMAC inter-packet gap, and standard RoCE v2 UDP/IP encapsulation with IB headers. The software stack combines BALBOA on the FPGA, a lib-based C++ cThread verbs API on the host, out-of-band QP exchange over TCP, and standard Mellanox RoCE endpoints on the peer side. Both BALBOA↔BALBOA and BALBOA↔ConnectX-5 flows are reported to operate correctly (Heer et al., 27 Jul 2025).

Throughput measurements show that BALBOA reaches 100G line rate for RDMA WRITE at buffer sizes of at least 32 KB. RDMA READ reaches near-line rate, with a moderate dip at some mid-range sizes due to READ-response overheads before saturating again at larger sizes. For small buffers, ConnectX-5 exhibits lower latency because of higher internal clocks, but the gap shrinks as message sizes grow, and BALBOA’s tail latency at P95 is described as negligible and comparable in stability to ASIC NICs. A measured first-byte-in to first-byte-out breakdown attributes most latency to packet processing, with iCRC contributing only a small share and arbitration before or after the stack forming the next-largest component (Heer et al., 27 Jul 2025).

Multi-QP scaling is handled by batched transmission across multiple queue pairs, with arbiters reported to distribute link capacity evenly across flows. The default design scales to 500 QPs, configurable per deployment. CPU utilization is reduced because data movement and inserted services bypass the host CPU; the AES and preprocessing demonstrations are specifically cited as showing large reductions in host CPU consumption relative to CPU-side implementations (Heer et al., 27 Jul 2025).

The use-case evaluations provide the clearest illustration of BALBOA’s intended role. In the AES encryption example, an on-datapath AES ECB pipeline adds approximately 10 cycles at 250 MHz, or about 25 ns, per payload segment. Keys are exchanged out-of-band during QP setup and stored per QP. BALBOA sustains 100G with AES enabled, whereas a 16-core AMD EPYC host using OpenSSL reaches only a fraction of 100G, peaking near 3600 MB/s for GB-sized buffers; a published BlueField-2 DPU IPsec offload figure of about 16 Gb/s is cited as a contrast point (Heer et al., 27 Jul 2025).

In the ML-based deep packet inspection example, the classifier is a fully-connected ternary neural network compiled with hls4ml and trained to distinguish malicious executables from CSV, PNG, and TXT payloads. Reported detection is 97.83% when the payload is entirely malicious and 89.35% when malicious content is partially embedded. Inference latency is approximately 44 ns per AXI beat and is reported as fully hidden by the packet pipeline, with no measurable effect on throughput or latency; the module flags payloads and raises a user interrupt for policy handling (Heer et al., 27 Jul 2025).

The line-rate compute-offload example targets preprocessing for recommender systems using the stateless operators Neg2Zero, Logarithm, and Modulus from Meta’s DLRM preprocessing toolkit, each implemented with $II=1$ on 64-byte streaming beats. The operators are placed on the receive datapath so that data is transformed as it arrives from the network and then DMA’d directly into AMD MI210 GPU memory without touching host RAM. In the reported cluster, a PCIe switch caps NIC-to-GPU DMA at about 70 Gbps, or about 8500 MB/s. Against this constraint, CPU-based preprocessing on one core achieves about 700 MB/s, while BALBOA reaches about 8500 MB/s, corresponding to

$S \approx \frac{8500}{700} \approx 12.1\times.$

The same experiment reports latency savings of roughly 20–135 $\mu s$, depending on size, by avoiding host-memory staging and copy, as well as higher sustained feed rates to the GPU and complete removal of CPU cycles from the preprocessing path (Heer et al., 27 Jul 2025).

6. Limitations, comparisons, and broader RDMA context

BALBOA’s current limitations are explicitly enumerated. Atomics are not supported; full two-sided verbs are not implemented; DCQCN and TIMELY are not part of the current release; and virtualization features such as SR-IOV, per-tenant isolation, and programmable policies are absent. The default QP tables support about 500 QPs and can be enlarged with additional BRAM or URAM, but the paper notes that large-scale multi-tenant scenarios will require deeper table designs and isolation mechanisms. Security services are present as inserted modules, exemplified by AES and DPI, but formal key management remains out-of-band and in-NIC key stores are proposed as future work (Heer et al., 27 Jul 2025).

The comparison to related systems situates BALBOA as an open, interoperable, and transport-complete FPGA stack rather than a fabric-specific research prototype. Relative to ConnectX-5, the claim is not that BALBOA exceeds commercial RNIC performance, but that it matches line-rate behavior while permitting in-line services and GPU-direct pipelines. Relative to BlueField-class DPUs, the distinction is that BALBOA keeps services on the datapath in deep pipelines rather than relying on programmable cores that are often off-datapath. Relative to open-source FPGA systems such as StRoM, the differentiator is commodity RoCE v2 interoperability across switched fabrics; relative to closed or vendor IP such as Schelten et al., ETRNIC, or ERNIC, the differentiator is openness plus host and GPU DMA support. The discussion of P4-based pipelines and NetFPGA is similarly constrained: they are characterized as strong at L2–L4 programmability but not sufficient for full RoCE v2 transport compliance and line-rate iCRC, BTH, and RETH handling without substantial custom logic (Heer et al., 27 Jul 2025).

BALBOA also sits within a broader RDMA literature that addresses different problems. A subsequent paper on RDMA completion semantics argues that RDMA CQE semantics guarantee placement rather than application-level commitment and calls the conflation of the two the “completion fallacy”; that paper explicitly states that BALBOA is not mentioned and does not describe any Balboa mechanism for bridging delivery and commitment (Borrill, 5 Mar 2026). The relationship is therefore contextual rather than direct: BALBOA provides a standards-compliant RC transport with extensible services, but it does not claim to redefine RDMA completion semantics.

A second neighboring line of work argues for abandoning reliable RoCE-style transport assumptions in large-scale training and instead making the learning algorithm robust to packet loss over UDP or another unreliable datagram substrate. That framework removes ACKs, retransmissions, and PFC dependence by using unbiased gradient aggregation and bounded-drift parameter broadcasts under packet loss (Weintraub et al., 2 Jul 2025). BALBOA takes the opposite systems stance: it remains within Reliable Connection RoCE v2 and uses transport reliability plus programmable datapath services to enable SmartNIC and accelerator functions. This suggests that the contemporary design space has at least two active directions—service-enhanced reliable RDMA, represented by BALBOA, and algorithmically loss-tolerant best-effort training substrates—whose trade-offs are complementary rather than mutually exclusive (Heer et al., 27 Jul 2025).

In summary, RoCE BALBOA is an open, standards-compliant RoCE v2 RC stack for FPGA-based SmartNICs and accelerators that combines 100G-capable packet processing, per-QP reliability state, retransmission buffering in HBM, ACK-clocked flow control, and verbs-like software integration with a modular AXI-stream service plane. Its significance lies less in introducing a new transport semantic than in making a datacenter-interoperable RDMA implementation available in a form that can be modified, instrumented, and extended with line-rate security, inspection, and compute pipelines, including direct placement into GPU memory (Heer et al., 27 Jul 2025).