Fluid Batching in Adaptive Systems

• Fluid batching is a dynamic mechanism that adaptively determines batch composition based on real-time input patterns rather than fixed-size grouping.
• It applies to both deep neural network inference on edge NPUs and decentralized transaction systems, enhancing system utilization and reducing costs.
• Empirical evaluations show improved throughput and lower latency, with enhancements such as up to 6.7× tail-latency improvement on NPU platforms and significant gas savings in smart contracts.

Fluid batching refers to the continuous, adaptive grouping of computation or transactions, whereby the batching strategy eschews fixed-size or model-wide batch commitments in favor of dynamic, workload-driven or layer-specific decisions. Fluid batching spans multiple domains, including deep neural network (DNN) inference on edge neural processing units (NPUs) and cost-efficient transaction aggregation in decentralized systems, and fundamentally improves system utilization, latency, and cost-efficiency under stochastic or highly variable workloads. Two primary instances are established in the literature: NPU-based real-time streaming inference (Kouris et al., 2022) and smart contract batching in Ethereum (Wang et al., 2021).

1. Principles and Definition

Fluid batching is a runtime adaptation mechanism in which the batch size or composition is determined per-layer, per-window, or in response to real-time arrival patterns, rather than adhering to a pre-set batch size or static grouping. In the edge NPU context, the DMA (Direct Memory Access) engine supports per-layer decisions in terms of how samples are appended—either along row (R-batch), column (P-batch), or a hybrid axis within the matrix decomposition for GEMM (General Matrix Multiply) acceleration.

In decentralized transaction systems, fluid batching groups contract invocations arriving within a tunable time window or those that meet workload-driven thresholds, instead of triggering on reaching a rigid count. Both instantiations enable fluid trade-offs between latency and system efficiency while responding to external rate fluctuations and non-deterministic computation paths (Kouris et al., 2022, Wang et al., 2021).

2. Fluid Batching in Early-Exit Neural Network Inference

Under dynamic inference scenarios, especially with early-exit (EE) DNNs, the input stream exhibits Poisson arrival with highly variable batch occupancy—partly because a stochastic fraction $p_{\mathrm{exit}}^{(i)}$ of samples utilize intermediate exits, shrinking the active batch as execution proceeds. A fixed batch size cannot reclaim hardware utilization lost to this dynamic attrition; fluid batching, by contrast, allows the hardware–software stack to (1) preemptively refill freed slots at exit points and (2) efficiently remap tensor tiling for subsequent layers (Kouris et al., 2022).

The exit-aware, preemptive scheduler tracks the live batch size $B_{\mathrm{act}}$, and at each exit $e_i$, computes the slack for SLO (Service Level Objective) adherence. When conditions permit, new samples are appended to backfill slots, and the batch continues with optimal dimensions. Key metrics include:

• Service rate at batch size $b$: $\mu(b) = \frac{b}{T_{\mathrm{exec}}(b)}$.
• Probability a sample exits before layer $l$: $$P_{\mathrm{exit}(<l)} = 1 - \prod_{j:e_j<l} (1-p_{\mathrm{exit}}^{(j)})$$.
• Utilization: $$U(b) = \frac{\mathrm{achieved\;GOp/s}(b)}{\mathrm{peak\;GOp/s}}$$.

3. Hardware Architecture: Fluid Batching Engine and Stackable Processing Elements

To enable on-the-fly adaptation, the NPU employs a Fluid Batching Engine (FBE), utilizing a lookup table populated at design time for every layer $l$ and batch size $b$. This table stores tuples $B_{\mathrm{act}}$0 dictating the number of R-batch samples and the parallelism configuration variable $B_{\mathrm{act}}$1, which orchestrates Stackable Processing Elements (PEs). Stackable PEs provide the hardware flexibility to reshape tensor processing pipelines and matrix widths according to instantaneous batch characteristics, sustaining up to 95% of peak GEMM throughput for batch size $B_{\mathrm{act}}$2—a regime where traditional NPUs fall below 40% utilization (Kouris et al., 2022).

4. Fluid Batching in Transaction Batching and Ethereum Smart Contracts

In iBatch, fluid batching is realized by eschewing fixed-size batch triggers in favor of continuous policies, such as windowed and size-driven adaptive schemes (e.g., Wsec-MinX, Wsec-Top1, MinX, and 1block dynamic gas bidding). The system secures off-chain aggregation and stateless on-chain validation, using a two-round signing protocol and bytecode rewrites (no per-account counter), defending against relay misbehavior while maintaining compatibility.

Key formulae:

• Gas savings per invocation: $B_{\mathrm{act}}$3
• Ether savings per invocation: $B_{\mathrm{act}}$4
• Cost–latency threshold: batching is profitable iff $B_{\mathrm{act}}$5 for average data-words $B_{\mathrm{act}}$6 per call (Wang et al., 2021).

Empirical results demonstrate gas cost savings up to 59.1% and ether cost reductions up to 31.5% with negligible protocol overhead, provided the average batch size meets or exceeds five calls.

5. Performance Outcomes and Empirical Evaluation

On edge NPU platforms (Xilinx ZC706/ZCU104), fluid batching yields:

• Average latency reduced by $B_{\mathrm{act}}$7 over best fixed/adaptive batch baselines.
• Tail-latency SLO satisfaction improved by up to $B_{\mathrm{act}}$8, with outputs below tight deadlines scaling with load.
• Throughput sustained at 55 inferences/s (ZCU104) versus sub-20 inferences/s for baselines at maximum request rates.
• Utilization maintained at 80–95% across $B_{\mathrm{act}}$9, significantly outperforming state-of-the-art methods capped at 40–60% (Kouris et al., 2022).

For iBatch, across several decentralized applications (IDEX, BNB, Chainlink, Tether), empirical replay traces confirm the feasibility of fluid batching in real-world workloads, with overheads sufficiently negligible for $e_i$0.

6. Trade-offs, Limitations, and Extensions

Fluid batching creates pronounced benefits at small batch sizes, particularly by recuperating otherwise idle hardware resources and preventing latency inflation under bursty or sparse arrivals. The primary overheads arise from non-overlapped DMA for ingress data (<0.05% total latency) and increased (but bounded) scheduler complexity—substantially fewer invocations than prior layer-wise preemption schemes.

Limitations include an explicit focus on convolutional GEMM-based workloads; extending fluid batching to accommodate attention-based Transformer layers is an open problem. Exit policies have been static; runtime-adaptive threshold tuning could optimize for shifting workload and accuracy/latency curves. Cloud-scale scaling (e.g., on TPU v4-class accelerators) and multi-tenant processing represent further avenues.

In summary, fluid batching unifies a class of layer-wise or window-wise dynamic batching and scheduling mechanisms that, across distributed inference and transactional domains, provide significant gains in utilization, responsiveness, and economic efficiency under stochastic input patterns (Kouris et al., 2022, Wang et al., 2021).