---
title: Request-Level Batching (RLB)
url: https://www.emergentmind.com/topics/request-level-batching-rlb-6050dc4e-37f1-4845-ace7-bb4f1282fa08
type: topic
---

# Request-Level Batching (RLB)

Request-Level Batching (RLB) refers to the practice of aggregating complete inference requests—rather than partial computations such as graph nodes or individual tokens—into shared execution units for increased throughput, reduced resource overhead, and improved quality-of-service (QoS) under varied constraints. In modern AI systems, RLB has emerged as a critical scheduling abstraction in both production-grade large language model (LLM) serving [2512.16134, 2507.17120], cloud and edge inference engines [2010.13103, 2305.01519], adaptive fine-tuning infrastructures [2312.05677], and billion-scale recommender systems [2511.06077], superseding simpler batching strategies to address heterogeneity, scalability, and strict deadline requirements.

## 1. Principles and Definitions

RLB is characterized by grouping entire inference requests or training examples into a batch prior to shared computation over the model execution graph. Each request consists of an input (such as a prompt, history, or image), associated metadata (such as a target, user adapter, or SLO/SLA deadline), and is processed end-to-end through appropriate layers. This contrasts with static batching (fixed-size wait for requests) or token-level “continuous” batching, which combine partial computations but do not address holistic request-level resource amortization or scheduling [2507.17120, 2512.16134].

Formally, in the RLB paradigm:
- A batch $B = \{ r_1, r_2, ..., r_N \}$ is a set of independent requests, each with its own execution trajectory through the model.
- In LLM serving, RLB groups queries with different sequence lengths and arrival times, forming a schedule for simultaneous execution of their full prefill or decode phases [2512.16134, 2507.17120].
- In multitarget recommender systems, RLB enables shared encoding of repeated features (e.g., user history) across multiple targets [2511.06077].
- In adapter-based inference (e.g., FLoRA), RLB supports batching diverse requests each with their own low-rank adaptation weights [2312.05677].
- In SLA/SLO-constrained inference, RLB underpins adaptive admission control to balance deadline satisfaction and resource utilization [2010.13103, 2305.01519, 2506.19677].

## 2. Algorithmic Methodologies

### 2.1 Scheduling and Buffering

RLB systems typically introduce a scheduler-side buffer that temporarily holds requests and dynamically forms batches according to configurable windows, resource states, or pending deadlines [2512.16134]. For example, Staggered Batch Scheduling (SBS) buffers requests for a window $\Delta t$ to match device availability and then dispatches an optimal batch across distributed DP units, reducing queueing in hidden device buffers and improving both time-to-first-token (TTFT) and throughput [2512.16134]. In contrast, offline batch pipelines (e.g., BlendServe) may reorder requests according to resource profiles to maximize both prefix sharing and operator resource overlap [2411.16102].

### 2.2 Bin Packing and Load Balancing

Batch formation in RLB must account for heterogeneity such as sequence length (LLMs), memory usage (autoregressive models), priority (SLA/SLO), or adapter identity (personalized fine-tuning). Water-filling, longest-job-first, and prioritized scheduling algorithms are deployed to maximize resource occupancy and minimize straggler effects [2507.17120, 2512.16134, 2411.16102]. In distributed LLM serving, global allocation policies—such as Prioritized Batch Allocation and IQR-aware decode scheduling—dynamically bin-pack requests by DP unit state, available capacity, and cache pressure [2512.16134].

### 2.3 Adaptive and SLA-aware Policies

Where requests carry deadlines or SLOs, RLB includes runtime predictors to admit only those batches likely to finish within constraints. This involves slack estimators [2010.13103], reward-augmented reinforcement learning objectives [2305.01519], or explicit per-request feasibility models based on measured resource-speed curves (e.g., Universal Scalability Law for CodeLLMs) [2506.19677]. Node-granular batching (LazyBatching) allows fine-grained preemption, adaptively merging execution across subgraphs to maintain high throughput while upholding strict deadlines [2010.13103].

## 3. System Architectures and Implementation Patterns

### 3.1 Scheduler Layer Organization

RLB requires an orchestrated control plane managing request queues, resource monitors, and feedback from compute engines. Architectures such as SBS in LLM clusters utilize a three-plane design (Control, State, Resource) to synchronize buffer formation, dispatch triggers, and load allocation [2512.16134]. Edge deployments (BCEdge) instantiate an MDP-based DRL agent that dynamically selects batch size and concurrency for each scheduling epoch [2305.01519].

### 3.2 Memory-efficient Data Structuring

Bucket-based batchers (BucketServe) maintain disjoint batches (“buckets”) partitioned by sequence length, dynamically splitting/merging as workloads and memory availability fluctuate [2507.17120]. Jagged tensors and compacted record blocks are used for minimal padding overhead in multimodal and long-sequence pipelines [2511.06077]. In RLB-enabled recommender systems, collating all per-request targets into grouped micro-batches enables shared encoding and bandwidth amortization [2511.06077].

### 3.3 Adapter Batching for Personalized Inference

Standard low-rank adaptation mechanisms (LoRA) are batch-incompatible with heterogeneous adapters. Fast LoRA (FLoRA) reframes the forward pass via an elementwise Hadamard mask, allowing each request in the batch to carry its own $(B_i, A_i)$ adapter while sharing the expensive GEMM operations, resulting in 2–5$\times$ throughput and latency gains for small rank values [2312.05677].

## 4. Theoretical Models and Mathematical Formulations

The performance and feasibility of RLB is governed by explicit cost models:
- Service times and device queuing: Immediate dispatch gives average device queueing of $T/2$ per batch, while RLB with $N$ parallel engines reduces this to $T/(2N)$ plus scheduler wait, yielding lower TTFT if scheduler intervals are kept small [2512.16134].
- Memory fitting: For LLMs, the maximum batch size $N_{\max}$ given memory budget $M_{\text{safe}}$ is
  $$
  N_{\max} = \max \left\{ N : 2 L H D S_{\max} B_{\text{dtype}} N \leq M_{\text{safe}}\right\}
  $$
  where $L$, $H$, $D$ exponentiate transformer layers, head count, per-head dim, and $B_{\text{dtype}}$ is datatype size [2507.17120].
- Utility and reward functions: In edge settings, the log-utility function
  $$
  U(b, m_c) = \log\left(\frac{T_{t_i}(b, m_c)}{L_{t_i}(b, m_c)/(\sum_j \text{SLO}_j/m_c)}\right)
  $$
  is maximized via entropy-regularized RL to find the optimal $(b, m_c)$ within memory and SLO constraints [2305.01519].
- Admission control and SLA feasibility: Predictive models $f(L)$ for concurrent load allow per-request acceptance only if expected completion time under new concurrency meets deadline, optimizing “goodput” (percent fulfilling SLA) [2506.19677].

## 5. Empirical Results and Comparative Evaluations

### 5.1 LLM Serving and Throughput

- SBS achieves 30–40% lower TTFT and 15–22% higher throughput versus immediate dispatch, with chunk utilization improved from ~52% to nearly 89% [2512.16134].
- BucketServe demonstrates 3.58$\times$ higher token/sec throughput than UELLM and 1.31$\times$ DistServe, and can handle nearly double the request load at 80% SLO compliance, sustaining $>$80% GPU utilization [2507.17120].
- SABER yields up to 26% more SLA-compliant completions (“goodput”) than the best static setting, dropping end-to-end latency variability by 31–45% [2506.19677].

### 5.2 Resource-Optimized Training and Recommendation

- Request-Level Batching in billion-scale recommenders reduces history encoding compute and GPU memory by $\sim$8$\times$, cuts PS CPU usage by 50%, and doubles training throughput (2.2$\times$ over point-wise), with no loss in metric (AUC/NLL) [2511.06077].
- BlendServe outperforms SOTA offline LLM batchers (vLLM/SGLang/NanoFlow) by $\sim$20.8% in throughput, reaching 86.6% of the feasible optimum for joint compute/memory overlap, while maintaining $>$97% of prefix-sharing efficiency [2411.16102].

### 5.3 Personalization and Adapter Diversity

- FLoRA achieves 2–5$\times$ lower token latency and up to 3$\times$ higher throughput for small rank adapters in personalized LLM serving, matching LoRA’s accuracy across diverse code-generation and speech recognition tasks [2312.05677].

A representative summary of salient RLB gains is:

| System               | Relative Throughput | SLA/TTFT/SLO Gains  | Utilization   |
|----------------------|--------------------|---------------------|--------------|
| SBS (LLM) [2512.16134]      | +15–22%             | –30–40% TTFT         | 88% chunks   |
| BucketServe [2507.17120]    | +3.58x (offline)    | 1.93x more reqs @80% SLO | >80% GPU     |
| SABER [2506.19677]          | +26% goodput        | –45% latency var      | adaptive     |
| BCEdge [2305.01519]         | +37.6% utility      | SLO <5% up to 40 rps  | adaptive     |
| FLoRA [2312.05677]          | 2–5x (rank 1–4)     | ~0.5s/token latency   | N/A          |
| Douyin RLB [2511.06077]     | +2.2–5.1x (train)   | +3.35% finish rate    | ~8x longer   |

## 6. Applications, Limitations, and Extensions

RLB is fundamental to high-throughput, low-latency AI workloads with heterogeneous, bursty, or personalized request patterns. It underpins modern LLM serving pipelines, billion-scale personalization (video, recommendation), and edge inference with hard SLO/SLA guarantees. Key limitations include overheads from complex batch formation logic, challenges in predicting output-length distributions for balanced memory use [2411.16102], and potential under-utilization from overly conservative admission or slack estimation. Inference across highly dynamic graphs or with extreme deadline variance remains challenging for naive RLB schedulers [2010.13103]. Future directions include further integration of online learning for dynamic batch/scheduling policies [2305.01519], extension to mixture-of-adapters per-request [2312.05677], and incorporation of advanced attention/memory management schemes for even higher resource efficiency [2411.16102].

## 7. Comparison to Alternative Batching Paradigms

The table below contrasts RLB with alternative batching approaches:

| Paradigm                | Batch Unit         | Flexibility        | SLA/SLO Awareness | Padding Overhead | Personalization |
|-------------------------|-------------------|--------------------|-------------------|------------------|-----------------|
| Static Batching         | Fixed N requests  | Low                | None              | High (variable)  | No              |
| Token-level/Continuous  | Individual tokens | High (decode loop) | None              | Medium           | No              |
| Graph-level Batching    | All nodes/graph   | Moderate           | Low/explicit      | Medium–High      | No              |
| Request-Level Batching  | Entire request    | High               | Yes (via policy)  | Minimal (bucket) | Yes (adapters)  |

RLB uniquely enables resource-optimal, deadline-aware, and personalized servicing of heterogeneous and stateful inference workloads across cloud, edge, and offline/online use cases [2512.16134, 2507.17120, 2312.05677, 2511.06077, 2010.13103, 2305.01519, 2411.16102, 2506.19677].

Source: https://www.emergentmind.com/topics/request-level-batching-rlb-6050dc4e-37f1-4845-ace7-bb4f1282fa08