CPU & GPU-Aware Micro-batching (CGAM)

• CPU and GPU-Aware Micro-batching (CGAM) is a set of techniques that dynamically sizes and schedules micro-batches to optimize performance in mixed CPU/GPU systems.
• It leverages online cost models to map operators to the most efficient device, balancing latency, throughput, and energy consumption across diverse workloads.
• CGAM employs adaptive batching, device-aware scheduling, and overlapped execution to maintain low latency and high throughput while minimizing energy use and data movement.

CPU and GPU-Aware Micro-batching (CGAM) is a suite of algorithmic and system design techniques intended to optimize end-to-end performance for data processing and machine learning workloads on heterogeneous clusters that integrate both CPU and GPU resources. CGAM addresses the divergent scaling behaviors, latency-throughput trade-offs, and energy inefficiencies that arise when CPU-bound and GPU-bound operations are intermixed in streaming, deep learning, graph analytics, and serverless DNN inference pipelines. Key mechanisms encompass dynamic micro-batch sizing, per-operator CPU/GPU mapping guided by online cost models, device-aware scheduling and load balancing, and resource-conscious analytical modeling of heterogeneous function provisioning.

1. Motivations and Sources of Heterogeneity

Contemporary cloud and edge inference systems, distributed streaming analytics platforms (e.g., Spark + RAPIDS), agentic AI frameworks embedding LLMs and external tools, and large-scale GNN training pipelines increasingly couple CPUs and GPUs in tightly integrated configurations. Several inefficiencies motivate the need for CPU/GPU-aware micro-batching:

• CPU–GPU Utilization Imbalance: In agentic AI and heterogeneous streaming, CPU tool processing (retrieval, summarization, web I/O) can dominate system latency (up to 90.6%), while GPUs remain underutilized or suffer from memory overprovisioning (Raj et al., 1 Nov 2025).
• Throughput Saturation and Energy Usage: As batch size grows, GPU throughput hits memory and bandwidth bottlenecks, while CPU energy use scales superlinearly with over-subscription, peaking at 44% of dynamic system energy for large batches (Raj et al., 1 Nov 2025).
• Tail and Median Latency Concerns: Large, unconditional batches raise P50 and P90 latency—problematic for applications with strict quality-of-service SLOs (Raj et al., 1 Nov 2025, Chen et al., 2024).
• Serverless and Multi-SLO Heterogeneity: In serverless DNN inference, low per-app arrival rates and heterogeneous SLOs render naive batching inefficient, and the optimal CPU/GPU function choice depends nontrivially on these variables (Chen et al., 2024).
• Data Movement and Compute Skew in Graph Analytics: For billion-node GNN training, PCIe-bound mini-batches and CPU sampling can limit GPU utilization; naive batching incurs prohibitive data movement (Dong et al., 2021).

This context underpins the development and adoption of CGAM methodologies for workload- and hardware-adaptive batch design and scheduling.

2. Core Algorithms and Mechanisms

CGAM is realized via several composable mechanisms:

2.1. Dynamic Micro-batch Admission and Sizing

Rather than statically fixing batch size or trigger intervals, CGAM computes the largest batch consistent with bounded per-request or per-window latency. Formally, for streaming or batch systems, the estimated worst-case batch latency is

$$\text{EstMaxLat}_i = \max_{j} [ \text{Buff}_{(i,j)} + \text{Part}_{(i,j)} / \text{AvgThPut}_{i-1} ],$$

admitting a micro-batch exactly when this value approaches a sliding window or tumbling window bound. This right-sizing is adaptive to observed traffic and system throughput, ensuring latency does not overshoot SLOs while exploiting available hardware parallelism (Lee et al., 2021).

2.2. Device-Aware Per-Operator Mapping

For each batch, CGAM dynamically maps logical operators (or stages) to CPU or GPU based on data partition size, device cost models, and transfer penalties. For operator $o$ and partition $j$:

$$\text{CPU}_{(i,j,o)} = \text{baseCost}_o \cdot (\text{Part}_{(i,j)} / \text{InfPT}_i), \quad \text{GPU}_{(i,j,o)} = \text{baseCost}_o \cdot (\text{InfPT}_i / \text{Part}_{(i,j)})$$

where $\text{InfPT}_i$ is a learned inflection point parameter representing the data size threshold for CPU/GPU preference, refined continuously via an online regression against observed throughput and latency (Lee et al., 2021).

2.3. Batching Cap Selection and Micro-batch Partitioning

In agentic and inference scenarios, the system empirically determines a batch size "cap" ($B_{\text{cap}}$): the largest batch size before throughput gain ratio $r(B) = T(B)/T(B/2) < \lambda$ (with $\lambda$ typically set to 1.1). Input batches above $B_{\text{cap}}$ are divided into micro-batches of size $\leq B_{\text{cap}}$, which are processed sequentially or with overlapped CPU/GPU stages (Raj et al., 1 Nov 2025). This micro-batching keeps throughput near its saturating peak while reducing median and tail latency.

2.4. Asynchronous, Overlapped Execution

CGAM executes CPU-bound and GPU-bound stages of different micro-batches in pipelined overlap to further reduce tail latency, enabled when the respective service times are comparable. This involves launching the CPU stage of micro-batch $o$0 in parallel with the GPU stage of micro-batch $o$1 (Raj et al., 1 Nov 2025).

2.5. Two-Stage Resource Grouping for Serverless Functions

For multi-SLO heterogeneous inference, CGAM (as realized in HarmonyBatch) employs a two-stage merging algorithm: (i) partition requests by SLO tightness and merge recursively into groups, then (ii) for each group, jointly optimize batch sizing, resource type (CPU vs. GPU), and provisioning by leveraging analytical latency and cost models. The cost optimization ensures batch feasibility, memory constraint, and SLO compliance (Chen et al., 2024).

3. Analytical and Mathematical Foundations

CGAM relies on explicit analytical modeling to guide decisions at every level:

• Latency Modeling: For batch $o$2, baseline system latency is $o$3; after partitioning into $o$4 micro-batches, $o$5 (Raj et al., 1 Nov 2025).
• Throughput and Energy: CGAM preserves throughput at the batch cap while reducing CPU dynamic energy, as concurrency and over-subscription are reduced (Raj et al., 1 Nov 2025).
• Device Preference Inflection Point: The online cost model maintains $o$6 as a linear function of previous throughput and latency statistics, converging rapidly to the load-balancing optimum (Lee et al., 2021).
• Serverless DNN Inference Cost and SLO: For each candidate grouping $o$7, average latency and worst-case latency for CPU and GPU provisioning are given by fitted exponential and linear models, with SLO constraints expressed as $o$8 for all requests $o$9. Cost is analytically minimized using resource and batch sizing theorems, subject to Poisson arrival-driven batch completion dynamics (Chen et al., 2024).
• Graph Analytics Sampling: For GNNs, micro-batching is realized via global neighborhood sampling with GPU-resident caches, reducing per-batch data movement by a factor of $j$0 with $j$1 the cache fraction and $j$2 the GNN depth (Dong et al., 2021).

4. Implementation Strategies and System Integration

CGAM techniques have been implemented and evaluated in diverse system architectures:

• Streaming Analytics: LMStream integrates dynamic admission and device mapping into Apache Spark + NVIDIA RAPIDS, partitioning batches by CPU core count and dispatching operators per cost-plan (Lee et al., 2021).
• Deep Learning Training: In fixed-size and adaptive Hogbatch SGD, the asynchronous coordinator allocates batches to CPU and GPU worker threads, maintaining model consistency via Hogwild updates and staleness control. Adaptive batch sizing tracks update counts for balanced convergence (Ma et al., 2020).
• Agentic AI and Orchestrator Pipelines: CGAM appears as a wrapper in orchestrators (e.g., LangChain Runnable.batch), intercepting and partitioning batches before CPU and GPU dispatch, with parameter selection determined by throughput profiling (Raj et al., 1 Nov 2025).
• Serverless Inference: HarmonyBatch profiles empirical latency/cost curves for both function types, then uses micro-batching and group merging logic to create cost-minimized, SLO-compliant scheduling plans (Chen et al., 2024).
• Graph Analytics (GNN): Global Neighborhood Sampling maintains a GPU cache, overlaps compute and communication via CUDA streams, and applies micro-batching at subgraph/data transfer level (Dong et al., 2021).

5. Empirical Evaluation and Comparative Results

Key empirical findings across representative workloads and platforms demonstrate the impact of CGAM approaches:

Workload / System	Latency Speedup	Throughput / Cost Impact	Notable Results
LMStream (streaming)	Up to 70.7% lower latency	Up to 1.74× higher throughput	Micro-batching + device mapping negligible (<1%) overhead
CPU+GPU Hogbatch SGD	2–5× faster to convergence	2–3× speedup over TensorFlow	Full CPU+GPU utilization; 90% loss in 1.1s vs. 3.2s (GPU)
Agentic AI (LangChain, etc.)	2.11× (P50), 1.33× (P90)	Maintains ≥97% peak throughput	CPU energy drops 1.5×; CPU share from 44% to ~30%
HarmonyBatch (serverless)	0% SLO violations	Up to 82.9% cost reduction	Reduces groups from 8→2–3; batch size grows 7→13
GNS GNN (billion-node)	2–14× faster/epoch	Convergence/accuracy preserved	80–90% less PCIe data; F1 scores maintained

All approaches preserve or improve statistical efficiency (convergence per update or epoch), resource utilization, and often reduce variance in update distribution between CPU and GPU components (Lee et al., 2021, Ma et al., 2020, Raj et al., 1 Nov 2025, Chen et al., 2024, Dong et al., 2021).

6. Domain-Specific Adaptations and Practical Guidelines

While the core paradigm generalizes, practical deployment entails domain-specific considerations and calibration:

• Profiling for Cap Selection: Users are advised to profile their hardware/software pipeline to empirically determine the batch cap ($j$3) and device mapping breakpoints (Raj et al., 1 Nov 2025, Chen et al., 2024).
• SLO and Poisson Arrivals: In serverless DNN platforms, accurate latency modeling is critical for SLO enforcement; HarmonyBatch leverages Poisson arrival assumptions for closed-form batch sizing (Chen et al., 2024).
• Shared vs. Isolated Memory: For CPU tool tasks, use multi-processing for isolated small tools but thread pools for shared-memory (FAISS, etc.) to maximize cache sharing (Raj et al., 1 Nov 2025).
• Energy-Aware Operation: In thermally constrained deployments, lower $j$4 can reduce energy at some throughput cost (Raj et al., 1 Nov 2025).
• Dealing with External/Irregular Workloads: Highly heterogeneous or external-API-bound requests may require additional, request-level cost estimation to retain micro-batch efficiency (Raj et al., 1 Nov 2025).
• Graph Dynamics: For dynamic graphs, cache refresh frequency and adaptive cache sizing must be tuned to balance memory and convergence stability (Dong et al., 2021).
• Limitations: Gains may be marginal when CPU tool cost is negligible, GPU memory is plentiful, or batch heterogeneity is extreme (Raj et al., 1 Nov 2025).

7. Theoretical Guarantees and Extensions

Analytical results establish strong theoretical properties for CGAM derivatives:

• Convergence under Mixed Updates: For SGD with asynchronous heterogeneous micro-batching, convergence rate matches $j$5; update sequencing yields efficient loss reduction (Ma et al., 2020).
• Optimal Resource Search: HarmonyBatch leverages unimodal cost curves and explicit CPU/GPU latency theorems, enabling efficient (rather than exhaustive) search over batch size and provisioning (Chen et al., 2024).
• Cache-Aware Sampling in GNN: Sampling variance with a global cache remains $j$6 for SGD, provided cache fraction $j$7 and refresh period $j$8 are chosen appropriately (Dong et al., 2021).

A plausible implication is that further generalization of the micro-batching paradigm to novel hardware or resource types (e.g., TPUs, FPGAs) is feasible by extending the analytical cost/performance models, online inflection point tuning, and batch cap calibration methodologies established in the literature.

---

References: (Lee et al., 2021, Ma et al., 2020, Raj et al., 1 Nov 2025, Dong et al., 2021, Chen et al., 2024)