DiveBatch: Adaptive Batching Methods

• DiveBatch is a functional term for methods that optimize batch composition, size, or execution order to improve training efficiency and model convergence.
• It includes techniques such as DPP-based mini-batch sampling, gradient-diversity aware batch-size adaptation, and dynamic auto-batching in computation graphs.
• These approaches reduce redundancy, lower gradient variance, and accelerate convergence across applications from machine learning to numerical simulations.

“DiveBatch” does not denote a single universally standardized algorithm in the arXiv literature. Instead, the name is used, or mapped retrospectively, across several lines of work concerned with batch construction, batch scheduling, or batch-size adaptation. In the DPP-based stochastic optimization literature, DiveBatch corresponds to Diversified Mini-Batch SGD (DM-SGD), where a Determinantal Point Process (DPP) or $k$-DPP samples repulsive mini-batches (Zhang et al., 2017). In later work, closely related labels refer to reference-model-free diversified subset selection during training (Hong et al., 2024), gradient-diversity-aware adaptation of the mini-batch size itself (Chen et al., 19 Sep 2025), dynamic or automatic batching of computation graphs (Neubig et al., 2017), and diversified batched decision making in Bayesian optimization and offline reinforcement learning (Azimi et al., 2011). Taken together, these usages suggest an umbrella notion: batch formation is no longer treated as a passive systems detail, but as an explicit optimization object.

1. Naming and conceptual scope

The available usages of “DiveBatch” span several technically distinct problem settings. In some cases the name is explicit; in others it is a descriptive label attached to a method whose published name differs.

Usage	Core mechanism	Representative source
Diversified mini-batch SGD	DPP/$k$-DPP sampling over data indices	DM-SGD (Zhang et al., 2017)
Diversified batch selection	Group-wise orthogonalized representativeness	DivBS (Hong et al., 2024)
Adaptive batch-size SGD	Batch size proportional to estimated gradient diversity	DIVEBATCH (Chen et al., 19 Sep 2025)
Automatic batching	Runtime grouping of identical operations in dynamic graphs	DyNet auto-batching (Neubig et al., 2017)
Dynamic batch BO	Adaptive parallel query count based on near-independence	Dynamic Batch BO (Azimi et al., 2011)
Diversified batched BO	DPP-reweighted Thompson sampling	DPP-BBO / DPP-TS (Nava et al., 2021)
Safe diversified batch RL	Diverse policy ensembles under support and safety constraints	Safe Diversified MBPS (Najib et al., 2024)

This multiplicity matters because the common intuition—“improve batching by injecting structure”—masks fundamentally different objects of control. In DM-SGD and DivBS, the batch is a subset of training examples. In DIVEBATCH, the controlled variable is the batch size. In dynamic computation-graph systems, the “batch” is an execution schedule over graph nodes rather than a statistical sample. In Bayesian optimization and offline RL, the batch is a parallel set of decisions or policies. This suggests that “DiveBatch” is best understood functionally rather than nominally: it refers to methods that make batch composition, size, or execution order adaptive and non-uniform, rather than fixed and uniform (Zhang et al., 2017, Hong et al., 2024, Chen et al., 19 Sep 2025, Neubig et al., 2017, Azimi et al., 2011, Nava et al., 2021, Najib et al., 2024).

2. DPP-based diversified mini-batching

In the terminology of “Determinantal Point Processes for Mini-Batch Diversification,” DiveBatch corresponds to Diversified Mini-Batch SGD (DM-SGD), which replaces uniform mini-batch sampling with DPP-based sampling (Zhang et al., 2017). The method starts from a positive semidefinite kernel $L \in \mathbb{R}^{n \times n}$ encoding similarity among the $n$ training examples. For an $L$-ensemble DPP,

$P(S) = \frac{\det(L_S)}{\det(L + I)},$

and for a fixed-size $k$-DPP,

$P(S) = \frac{\det(L_S)}{e_k(L)}, \qquad |S| = k.$

Here $L_S$ is the principal submatrix indexed by $S$, and $k$0 is the $k$1-th elementary symmetric polynomial of the eigenvalues of $k$2.

The operational idea is repulsion. DPPs give low probability to subsets containing redundant points and higher probability to subsets containing diverse points. In the unconditioned DPP, the marginal kernel is $k$3 and the inclusion covariance satisfies

$k$4

so sampling one index decreases the probability of sampling similar indices. The fixed-size $k$5-DPP does not preserve negative dependence for every pair in the same strict form, but the practical behavior remains repulsive for nearby or very similar items. This directly targets the failure mode of classical uniformly sampled SGD on imbalanced or near-duplicate data: redundant samples co-occur, gradients become positively correlated, and the mini-batch estimator becomes unnecessarily noisy.

The gradient analysis in DM-SGD is stated in terms of the empirical objective

$k$6

and the mini-batch estimator

$k$7

With $k$8 the inclusion indicator, $k$9, and

$L \in \mathbb{R}^{n \times n}$0

the variance decomposition for

$L \in \mathbb{R}^{n \times n}$1

is

$L \in \mathbb{R}^{n \times n}$2

The sufficient condition

$L \in \mathbb{R}^{n \times n}$3

implies variance reduction relative to uniform sampling. The interpretation is explicit: when similar samples produce positively correlated gradients, a repulsive sampler makes $L \in \mathbb{R}^{n \times n}$4 negative on those pairs, so the covariance term subtracts variance rather than adding it.

Kernel construction determines what “diversity” means. The paper uses linear kernels $L \in \mathbb{R}^{n \times n}$5, cosine kernels, and RBF kernels. For text, tf-idf vectors with an annealed linear kernel $L \in \mathbb{R}^{n \times n}$6 with $L \in \mathbb{R}^{n \times n}$7 were used; for images, label-feature blends such as $L \in \mathbb{R}^{n \times n}$8 and $L \in \mathbb{R}^{n \times n}$9 allow an explicit trade-off between class balance and intra-class feature diversity. Uniform sampling is recovered when $n$0 is proportional to the identity, and biased stratified sampling is recovered from a block-diagonal $n$1 with all-ones within strata and zeros across strata. The method therefore generalizes stratified sampling beyond discrete bins.

The optimized objective can be viewed as a diversified risk,

$n$2

which is biased relative to the original empirical risk. If unbiasedness is required, importance weighting yields

$n$3

and

$n$4

The paper notes, however, that the biased diversified risk often performs better on imbalanced data.

The reported empirical profile is correspondingly mixed across unsupervised and supervised settings. On Reuters R8, DM-SVI improved average per-class accuracy from $n$5 to $n$6 and total accuracy from $n$7 to $n$8. On Oxford 102 Flowers, the best DM-SGD result at $n$9 reached $L$0 versus a baseline of $L$1. On balanced MNIST, benefits remained visible through lower variance and improved learning curves, indicating that the method is not restricted to overt class imbalance (Zhang et al., 2017).

3. Reference-model-free diversified batch selection

A second, distinct DiveBatch usage is Diversified Batch Selection (DivBS), described in “Diversified Batch Selection for Training Acceleration” and also referred to there as DiveBatch (Hong et al., 2024). Here the setting is online batch selection: at each training step, one draws a large candidate batch $L$2 of size $L$3 and selects a smaller subset $L$4 of size $L$5 for the optimizer step. The paper’s criticism of prior reference-model-free methods is that sample-wise top-$L$6 scoring ignores redundancy: multiple high-scoring but near-duplicate samples can be selected simultaneously.

DivBS addresses that failure mode by replacing additive sample-wise scoring with a group-wise orthogonalized representativeness objective. Let $L$7 be the selection features, taken in the paper to be last-layer per-sample gradients. For a subset $L$8, let $L$9 denote the set of orthonormal bases of the span of $P(S) = \frac{\det(L_S)}{\det(L + I)},$0. The representativeness objective is

$P(S) = \frac{\det(L_S)}{\det(L + I)},$1

A key simplification in the paper is

$P(S) = \frac{\det(L_S)}{\det(L + I)},$2

Thus the objective measures the Euclidean norm of the projection of the batch-sum vector onto the subspace spanned by the selected examples, scaled by the number of orthogonal directions. Redundancy is explicitly removed by orthogonalization.

The associated auxiliary function

$P(S) = \frac{\det(L_S)}{\det(L + I)},$3

is normalized, monotone, and submodular. Consequently, greedy maximization under a cardinality constraint attains the classical $P(S) = \frac{\det(L_S)}{\det(L + I)},$4 approximation guarantee. The theory-friendly greedy rule orthogonalizes every candidate by Gram–Schmidt residualization,

$P(S) = \frac{\det(L_S)}{\det(L + I)},$5

and selects the candidate maximizing $P(S) = \frac{\det(L_S)}{\det(L + I)},$6, where $P(S) = \frac{\det(L_S)}{\det(L + I)},$7.

The practical DivBS algorithm avoids per-candidate orthogonalization at each step. It instead selects

$P(S) = \frac{\det(L_S)}{\det(L + I)},$8

orthogonalizes only the chosen sample, and updates

$P(S) = \frac{\det(L_S)}{\det(L + I)},$9

This preserves the “explain the remaining batch signal” interpretation while reducing computational cost from the greedy procedure’s candidate-wise residualization to a substantially cheaper realization.

Empirically, DivBS is evaluated on CIFAR-10, CIFAR-100, Tiny ImageNet, CIFAR-100-LT, PASCAL VOC 2012 trainaug, Wikipedia cross-modal retrieval, and GPT-2 Medium with LoRA on the E2E NLG Challenge. At a $k$0 budget, CIFAR-100 accuracy rises from $k$1 under uniform selection to $k$2 under DivBS, close to the full-data result of $k$3. On CIFAR-10, the corresponding numbers are $k$4 for uniform and $k$5 for DivBS, with a full-data result of $k$6. On PASCAL VOC trainaug, DivBS reaches $k$7 mIoU at $k$8 budget versus $k$9 for uniform. The paper also reports that selection overhead is $P(S) = \frac{\det(L_S)}{e_k(L)}, \qquad |S| = k.$0 extra versus uniform sampling. These results place DiveBatch, in this sense, at the intersection of online subset selection, gradient-space coreset construction, and diversity-aware training acceleration (Hong et al., 2024).

4. Gradient-diversity-aware batch-size adaptation

A third usage is the explicitly named “DIVEBATCH: Accelerating Model Training Through Gradient-Diversity Aware Batch Size Adaptation” (Chen et al., 19 Sep 2025). Unlike DM-SGD or DivBS, this method does not alter which samples are chosen within a candidate set. It dynamically adjusts the mini-batch size itself in proportion to an estimate of gradient diversity.

The central quantity is

$P(S) = \frac{\det(L_S)}{e_k(L)}, \qquad |S| = k.$1

If individual gradients are aligned, the denominator is large and $P(S) = \frac{\det(L_S)}{e_k(L)}, \qquad |S| = k.$2 is small; if gradients are diverse or cancel, the denominator is small and $P(S) = \frac{\det(L_S)}{e_k(L)}, \qquad |S| = k.$3 is large. DiveBatch estimates this quantity at epoch boundaries by accumulating, over the batches of epoch $P(S) = \frac{\det(L_S)}{e_k(L)}, \qquad |S| = k.$4, the sum of per-sample gradient norm squares and the norm of the accumulated gradient sum:

$P(S) = \frac{\det(L_S)}{e_k(L)}, \qquad |S| = k.$5

The update rule is then

$P(S) = \frac{\det(L_S)}{e_k(L)}, \qquad |S| = k.$6

with scaling parameter $P(S) = \frac{\det(L_S)}{e_k(L)}, \qquad |S| = k.$7, small initial batch size $P(S) = \frac{\det(L_S)}{e_k(L)}, \qquad |S| = k.$8, and maximum batch size $P(S) = \frac{\det(L_S)}{e_k(L)}, \qquad |S| = k.$9.

The theoretical motivation is imported from convergence analyses in which the “safe/effective” mini-batch size scales like $L_S$0. The paper states that if $L_S$1, one mini-batch SGD step reduces the expected squared distance to the minimizer approximately $L_S$2 times more than one single-sample step, up to second-order terms controlled by

$L_S$3

The paper also presents standard convergence bounds under Lipschitz-gradient and bounded-variance assumptions, emphasizing that diversity directly modulates the variance/covariance structure of mini-batch gradients.

The algorithmic cost is substantial because exact implementation requires per-sample gradients. The reported average peak memory on CIFAR-10 is $L_S$4 MB for SGD$L_S$5, $L_S$6 MB for SGD$L_S$7, $L_S$8 MB for AdaBatch$L_S$9, and $S$0 MB for DiveBatch$S$1. The implementation uses Backpack-for-PyTorch for per-sample gradients, and the paper notes that micro-batching or proxy statistics could reduce cost, although the reported experiments use exact per-sample sums.

The empirical picture is explicitly framed as a speed–accuracy trade-off. On CIFAR-10, the time to reach $S$2 of final accuracy is $S$3 s for DiveBatch, versus $S$4 s for AdaBatch, $S$5 s for SGD$S$6, and $S$7 s for SGD$S$8; final validation accuracies are $S$9 for DiveBatch, $k$00 for AdaBatch, $k$01 for SGD$k$02, and $k$03 for SGD$k$04. On CIFAR-100, the corresponding times are $k$05 s, $k$06 s, $k$07 s, and $k$08 s, with final accuracies $k$09, $k$10, $k$11, and $k$12. On Tiny-ImageNet, DiveBatch reaches $k$13 final accuracy versus $k$14 for AdaBatch and $k$15 for SGD$k$16, but reduces time to near-final accuracy from $k$17 s to $k$18 s. The stated overall range is $k$19–$k$20 faster convergence than standard SGD and AdaBatch, with a slight trade-off in performance (Chen et al., 19 Sep 2025).

5. Dynamic and automatic batching of computation graphs

A different literature uses DiveBatch as a descriptive label for automatic or dynamic execution batching in neural computation graphs. In “On-the-fly Operation Batching in Dynamic Computation Graphs,” the problem is not statistical sampling but execution scheduling: automatically group semantically identical operations from different parts of a dynamic graph into one batched kernel, even when architectures involve variable-length sequences, trees, or input-dependent control flow (Neubig et al., 2017). DyNet accomplishes this by separating graph construction from execution via lazy evaluation, assigning each node a batching signature that encodes operator type, shapes, parameter identities, broadcasting semantics, and attributes, and then applying an agenda-based scheduler. The scheduler repeatedly selects a ready node, collects all ready nodes with the same signature, and executes them together. Exact optimal batching-aware scheduling is stated to be NP-hard, so the system uses heuristics based on average depth and operator cost class.

This runtime perspective yields large throughput improvements without developer-managed padding or masking. On a synthetic BiLSTM benchmark with fixed length $k$21 and batch size $k$22, CPU throughput improves from about $k$23 ms/sentence to about $k$24 ms/sentence under the agenda-based strategy, approximately an $k$25 speedup; GPU throughput improves from about $k$26 ms/sentence to about $k$27 ms/sentence, also approximately an $k$28 speedup. On more irregular tasks such as TreeLSTM and transition-based dependency parsing, substantial gains remain, although smaller for strongly sequential control flow (Neubig et al., 2017).

“Effective Approaches to Batch Parallelization for Dynamic Neural Network Architectures” presents a related but more manual formulation based on grouping identical module calls across a minibatch of dynamic graphs (Suarez et al., 2017). If each example has graph size at most $k$29 and module vocabulary size $k$30, standard topological batching yields at most $k$31 expensive module calls rather than $k$32 under naive per-example execution. For predetermined trees or DAGs with depth $k$33, improved depth-based batching yields at most $k$34 calls, and for a balanced tree this becomes $k$35. In sparsely gated mixture-of-experts layers, the approximate speedup is

$k$36

where $k$37 is the number of experts and each example activates $k$38 experts. The paper reports speedups of over $k$39 on modular program execution and up to $k$40 for sparsely gated MoE layers (Suarez et al., 2017).

ACRoBat pushes this line toward compiler-driven divergence-aware auto-batching (Fegade et al., 2023). It combines inline depth computation, program phases, ghost operations that delay premature dispatch across conditionals, grain-size coarsening of static subgraphs, tensor-dependent control-flow support via fibers, parameter-reuse inference, horizontal fusion, gather-operator fusion, and ahead-of-time host code generation. The grouping key is a tuple of operator type, depth, phase, reuse signature, and layout signature. The framework reports up to $k$41 better performance than DyNet on an Nvidia GeForce GPU, with substantial reductions in DFG construction, scheduling overhead, memory copies, and CUDA API time. In this dynamic-systems sense, DiveBatch names a family of execution planners whose objective is to recover batch parallelism from control-flow divergence rather than from data subsampling (Fegade et al., 2023).

6. Diversified batching in Bayesian optimization, offline RL, and distributed training

In Bayesian optimization, DiveBatch refers to selecting multiple parallel evaluations while staying close to a strong sequential acquisition rule. “Dynamic Batch Bayesian Optimization” adapts the batch size $k$42 at each BO iteration rather than fixing it (Azimi et al., 2011). The method is built around Expected Improvement (EI), the Gaussian-process posterior

$k$43

and the observation that a candidate can be added to the current batch if its posterior mean is nearly independent of the as-yet-unobserved outcomes of previously selected points. The decision rule is based on an outcome-independent bound on the expected absolute mean change,

$k$44

and a point is accepted into the current batch if this quantity is at most $k$45. Reported speedups are in the $k$46–$k$47 range with little impact on performance across eight benchmarks (Azimi et al., 2011).

A distinct BO formulation introduces DPPs directly into batched Thompson sampling. “Diversified Sampling for Batched Bayesian Optimization with Determinantal Point Processes” defines

$k$48

where $k$49 is the Thompson-sampling maximum distribution and the determinant term enforces mutual-information-based diversity (Nava et al., 2021). The MCMC acceptance probability simplifies to

$k$50

The paper proves a Bayesian simple regret bound tighter than that of classical batched TS through an explicit negative $k$51 term, reflecting a diversity bonus.

In offline reinforcement learning, the directly relevant name is not DiveBatch but iterative batch reinforcement learning via Safe Diversified Model-based Policy Search (Najib et al., 2024). The method trains an ensemble of $k$52 policies in a learned differentiable simulator, enforces support and safety via behavior regularization, likelihood-threshold penalties, or direct action constraints, and rewards diversity among trajectories through the minimum lock-step Euclidean distance,

$k$53

A representative objective is

$k$54

On the Industrial Benchmark, diversity accelerated cost reduction and lowered variance under both constrained-policy and soft-constraint variants.

A further batch-adaptation use appears in distributed deep learning. DBS dynamically adjusts each worker’s mini-batch size according to measured throughput while keeping the global batch size fixed (Ye et al., 2020). With worker throughputs $k$55, the proportional allocation rule is

$k$56

followed by integer rounding and dataset repartitioning through DynamicDatasetAdjust. The synchronized gradient aggregation remains

$k$57

so the statistical semantics are those of standard mini-batch SGD, while wall-clock efficiency improves on heterogeneous clusters.

7. Specialized domain-specific formulations

Some uses of DiveBatch are highly application-specific but retain the same structural theme: a bottleneck batch dimension is redesigned so that scalability is recovered without sacrificing the global coupling that makes the original problem hard.

In numerical PDE solving, the term is used for a batched GPU methodology in which many independent linear systems

$k$58

share the same banded left-hand side $k$59 (Carroll et al., 2021). The implementation stores only one global copy of the tridiagonal or pentadiagonal matrix, pre-factorizes it once, and solves all right-hand sides with an interleaved memory layout that yields coalesced accesses. Relative to cuThomasBatch and cuPentBatch, this reduces left-hand-side storage by about $k$60 for tridiagonal systems and about $k$61 for pentadiagonal systems, with reported speedups up to about $k$62–$k$63 for the tridiagonal case and about $k$64–$k$65 for the pentadiagonal case at large problem sizes. The paper demonstrates $k$66 independent one-dimensional Cahn–Hilliard simulations on a single GPU (Carroll et al., 2021).

In deep subspace clustering, a mini-batch strategy replaces full-batch self-expressive training with a memory bank of latent features (Jiang et al., 26 Jul 2025). Classical DSC requires

$k$67

with global coefficient matrix $k$68 and full latent matrix $k$69, which forces full-batch processing. The proposed batch-wise DSC (BDSC) stores a memory bank $k$70 and uses

$k$71

for the current batch. The AE-based objective becomes

$k$72

while the contrastive variant CLBDSC replaces reconstruction with an InfoNCE term that uses the bank as a source of negatives. On COIL100 with $k$73 inputs and a ResNet-18 encoder, the paper reports ACC/NMI of $k$74 for BDSC and $k$75 for CLBDSC; on ORL at $k$76, the corresponding numbers are $k$77 and $k$78 (Jiang et al., 26 Jul 2025).

These specialized uses underscore a broader pattern. Whether the obstacle is the $k$79 self-expressive matrix of subspace clustering, the repeated left-hand side of banded linear systems, the control-flow divergence of dynamic neural programs, or the redundancy of uniformly sampled gradients, DiveBatch-style methods intervene at the batch layer itself. The intervention may take the form of repulsive subset selection, orthogonalized representativeness, gradient-diversity-based resizing, execution-graph scheduling, safe diversified policy ensembles, or shared-structure amortization. What remains constant is the rejection of naive fixed batching in favor of batch policies that encode geometry, dependence, safety, or systems structure.