Blink Serving Architecture for LLM Inference

• Blink Serving Architecture is an end-to-end LLM inference system that offloads control and tokenization from the CPU using a SmartNIC and a persistent GPU kernel.
• It leverages zero-copy data movement and persistent CUDA graphs to minimize latency and maximize throughput even under heavy CPU interference.
• The architecture achieves significant improvements in tail latency and energy efficiency, demonstrating robust performance compared to conventional CPU-based stacks.

Blink Serving Architecture is an end-to-end LLM inference system that eliminates the host CPU from the steady-state inference path by redistributing orchestration and control responsibilities between a SmartNIC (DPU) and a GPU. This architecture is designed to address the limitations of conventional LLM serving stacks, which are vulnerable to CPU interference and suboptimal resource utilization due to their dependence on the CPU for token-level control and orchestration. Blink achieves high throughput, low tail latency, and stable performance under interference by offloading request handling and tokenization to the SmartNIC and utilizing a persistent GPU kernel for scheduling, batching, and key-value (KV) cache management—entirely bypassing the CPU after initialization (Siavashi et al., 8 Apr 2026).

1. System Components and Architecture

Blink's architecture consists of three primary hardware components:

• Host CPU: Engaged exclusively during an initialization phase to load model weights onto the GPU and capture CUDA computational graphs. Thereafter, it is removed from the steady-state data path.
• SmartNIC (BlueField-3 DPU): Responsible for running an OpenAI-compatible HTTP/SSE server, prompt tokenization, and management of a ring buffer in GPU HBM via one-sided RDMA operations. It polls for generated tokens and detokenizes output before streaming back to clients.
• GPU: Hosts a single persistent CUDA graph responsible for continuous batching, FFCS scheduling, on-device KV-cache indexing, and token sampling without CPU involvement.

The overall data path in steady-state operation is as follows:

1. HTTP request from client is received by the SmartNIC.
2. Input prompt is tokenized on the DPU (requiring 1–2 μs for short prompts, approximately 8–20× faster than CPU approaches).
3. SmartNIC front end allocates a slot in a ring buffer in GPU memory.
4. Tokens are transferred directly into GPU HBM using one-sided RDMA.
5. Slot metadata updates signal prefill readiness.
6. GPU-side persistent scheduler (256 threads) scans and claims work using atomic operations.
7. Scheduler launches the prefill CUDA graph device-side ("fire-and-forget").
8. Completion is detected by polling a per-slot shared memory flag.
9. Slot state transitions to decoding, and the decode CUDA graph executes, incorporating mechanisms for KV attention and top-p sampling.
10. Sampled tokens are written to a per-step output buffer.
11. DPU polls for completion, reads tokens, and detokenizes them.
12. Output is streamed to the client via Server-Sent Events.

This loop repeats per output token, with the host CPU uninvolved after initial setup (Siavashi et al., 8 Apr 2026).

2. SmartNIC Offload and Data Handling

SmartNIC offload is central to Blink’s design, moving all network, protocol parsing, tokenization, and metadata management to the DPU. Key features include:

• Protocol and Tokenization: The HTTP/SSE server and a custom merge-rule tokenizer—implemented as a flat hash table optimized for DPU ARM cores and L1D cache—operate fully on the DPU. NEON-accelerated regex pre-splitting and pre-allocated thread-local state are used for efficiency.
• GPU-Resident Ring Buffer: A circular buffer in GPU HBM holds up to 4,096 slots, each associated with input and output token arenas. Slot metadata (64 bytes per slot) are manipulated via one-sided RDMA (submit, poll, state transition) with no host DRAM involvement, and slot allocation uses an amortized O(1) free slot cache.
• Zero-Copy Data Movement: Prompts and generated tokens are sent directly between the network interface and GPU memory, never traversing host memory. This approach enables zero-copy transport, eliminating kernel interrupts and DRAM bottlenecks (Siavashi et al., 8 Apr 2026).

3. GPU-Resident Persistent Scheduler

The serving stack’s control plane resides entirely in a persistent CUDA graph running continuously on the GPU, leveraging a single thread block of 256 threads that never yields to the host. The scheduler executes the following logic:

• Ring slots are scanned in parallel for new prefill or decode work.
• Claims are made for pending slots via atomicCAS.
• For prefill states, the scheduler launches the prefill CUDA graph with appropriate batch/sequence length.
• For decode states, a decode CUDA graph is launched, attending to cached KV, computing next-token logits, and performing top-p sampling in-line.
• The scheduler polls for output completion and handles per-slot state transitions.

Continuous batching is applied using a first-come-first-served (FCFS) policy, with batch admission rules defined as:

$$\text{admit\_new} = (n_\mathrm{pending} > 0) \land (|\text{Batch}| < B_\mathrm{max}) \land (W - L \geq n_\mathrm{pending})$$

where $W = 120$ is the CUDA fire-and-forget launch budget and $L$ is the launches issued. When launches are exhausted, a tail launch resets $L$ to zero, amortizing this cost across 120 steps (less than 0.03 μs per step). The device-side "fire-and-forget" launches complete in approximately 2 μs, compared to 11–17 μs when orchestrated by the host, reclaiming up to 7.7 ms per 512-token sequence (Siavashi et al., 8 Apr 2026).

4. On-Device KV-Cache Management

Blink employs a paged KV cache architecture, in which key/value matrices for past tokens are stored in fixed-size HBM blocks. Management features include:

• Linear Arena: KV blocks are arranged linearly in HBM, with each ring slot maintaining its block index list.
• Write Policy: During prefill, blocks are appended to the block list.
• Zero-Copy Gathering: On decode, only the relevant KV blocks are gathered per the slot’s block list.
• Eviction: When HBM is full, blocks can be evicted according to LRU or FIFO policies. In practical experiments with 96 GB HBM, all active KV state is retained, avoiding frequent evictions.
• Cache Efficiency: Empirical hit rate satisfies $h \approx 1 - M/T$ with the miss ratio $M/T < 1\%$ pre-saturation.
• Bandwidth Considerations: KV operations scale with GPU memory bandwidth (up to approximately 1.5 TB/s on H100), with no CPU involvement (Siavashi et al., 8 Apr 2026).

5. Performance and Benchmarking

Blink’s end-to-end performance exhibits substantial improvements over state-of-the-art LLM serving stacks such as TensorRT-LLM, vLLM, and SGLang. Two principal tail-latency metrics are emphasized:

• P99 TTFT (99th Percentile Time to First Token):

$$\mathrm{TTFT_{99}} = \mathrm{Percentile}_{0.99}(t_\mathrm{first} - t_\mathrm{arrival})$$

• P99 TPOT (99th Percentile Time per Output Token):

$$\mathrm{TPOT_{99}} = \mathrm{Percentile}_{0.99}(t_i - t_{i-1})$$

Key empirical results across the Llama-3 8B, Phi-4 15B, Qwen-3 32B, and Qwen-3 30B-A3B models include:

Model	Blink P99 TTFT	Best Baseline P99 TTFT	Speedup
Llama-3 8B	653.8 ms	880.0 ms	1.35×
Phi-4 15B	1,109.4 ms	1,453.8 ms	1.31×
Qwen-3 32B	9,481.3 ms	9,621.4 ms	1.02×
Qwen-3 30B-A3B	1,397.5 ms	4,814.7 ms	3.45×

Maximum reductions observed: up to 8.47× lower P99 TTFT, 3.40× lower P99 TPOT, and up to 2.1× higher decode throughput. Under conditions of severe CPU interference (e.g., 45-thread pbzip2 plus 45-job Ninja build on 90 cores), baseline systems’ throughputs collapse to 32–64% of their isolated values and P99 TTFT inflates by 3–18×. In contrast, Blink maintains output within ±15% of isolated latency and retains ≥99% throughput (Siavashi et al., 8 Apr 2026).

6. Energy Per Token and Efficiency

Blink measures server-level energy efficiency using smart metering and onboard DPU readings. The energy per token is defined as:

$$E_\mathrm{tok} = \frac{P_\mathrm{avg} \times T_\mathrm{run}}{N_\mathrm{tokens}}$$

Observed energy per token values for Blink in isolation:

• Llama-3 8B: 363 mJ/token
• Qwen-3 32B: 1,306 mJ/token

These reflect 13.7–48.6% reductions compared to the next best system. Under interference, Blink’s energy per token is 423–1,584 mJ, whereas the best baseline ranges from 1,045–3,597 mJ—a savings of 41.4–70.7%. All systems draw similar wall power (1.1–1.4 kW), so energy improvements are attributed to higher sustained token throughput (Siavashi et al., 8 Apr 2026).

7. Significance and Implications

Blink demonstrates that eliminating the CPU from the steady-state inference path for LLM serving yields robust, high-throughput, and energy-efficient operation, especially under CPU contention. Offloading orchestration, tokenization, and memory management to the SmartNIC and GPU not only improves latency and throughput but also ensures predictable performance unaffected by host-side workloads. This suggests a plausible path forward for scalable datacenter LLM inference stacks, maximizing utilization of both compute and I/O resources, and motivating further research into end-to-end device-level orchestration paradigms (Siavashi et al., 8 Apr 2026).