Distributed-Index Batching

• Distributed-index batching is a technique that forms batches using sample indices and realized training costs to synchronize operations across distributed workers.
• It leverages compact indirection and locality-aware execution to optimize resource utilization, ensuring controlled batching and reduced latency in complex ML and retrieval systems.
• Implementations report significant improvements, with throughput gains up to 13.1× and memory reductions of up to 89%, demonstrating its practical efficiency in large-scale settings.

Distributed-index-batching is used in several technically distinct but structurally related ways. In the most explicit formulation, it is “the practice of building batches from sample indices and their realized training costs at the point where those costs become accurately observable, then aligning the resulting variable-size group counts across data-parallel workers so that DDP step counts match.” Closely related work applies the same organizing idea to spatiotemporal windows assembled from compact start-index arrays, to storage-side planning from object placement metadata and archive membership, to per-hop batched remote expansions in a distributed DiskANN graph, and to composite-index or disjoint-recovery formulations in coding theory (Li et al., 18 Jun 2026, Ockerman et al., 15 Jul 2025, Aizman et al., 25 Feb 2026, Adams et al., 7 Sep 2025, Sadeghi et al., 2016, Lipmaa et al., 2014).

1. Terminological scope and research settings

The literature uses the term across ML systems, distributed retrieval, and coding theory. This suggests a shared emphasis on replacing many small, independent operations with index-driven batch formation, while preserving constraints such as DDP step alignment, deterministic output order, bounded per-server access, or low tail latency.

Setting	Index or metadata	Batched operation
LLM fine-tuning	sample indices and realized post-pipeline lengths	variable-size groups aligned across DDP ranks
ST-GNN training	start-index array over a time-series tensor	runtime snapshot assembly across multiple GPUs
Object storage	placement metadata and object-in-archive membership	one ordered streaming retrieval for many objects
Distributed ANN search	node IDs, shard mapping, and head-index seeds	per-hop remote expansions and per-host IO/compute
Full-text indexing	global routing trie and local Patricia tries	coalesced query routing and DRMA verification
Coding-theoretic settings	composite indices or disjoint recovery sets	load-balanced multi-item retrieval

In Online Dynamic Batching, the central indexed object is the post-pipeline sample length that becomes visible only after preprocessing, augmentation, templating, tokenization, and multimodal visual-token expansion. In PGT-I, the indexed object is the window start over a base tensor $X \in \mathbb{R}^{T \times N \times F}$. In GetBatch, the planner operates on consistent-hash ownership and archive operations. In DistributedANN, orchestration uses a sharded head index and key-to-shard routing over a unified DiskANN graph. In distributed index coding and linear batch codes, the “index” is algebraic: composite-message subsets, server groupings, or recovery supports (Li et al., 18 Jun 2026, Ockerman et al., 15 Jul 2025, Aizman et al., 25 Feb 2026, Adams et al., 7 Sep 2025, Liu et al., 2018, Lipmaa et al., 2014).

2. Recurrent design principles

A recurrent principle is compact indirection rather than eager materialization. ODB moves batch formation to the DataLoader boundary where true sample cost is observable, eliminating preprocessing- and augmentation-dependent length caches. PGT-I stores one copy of the raw spatiotemporal tensor plus compact indices that reference window starts. GetBatch treats placement metadata as the planning substrate for a single multi-object request. DistributedANN stores a single logical DiskANN graph as key-value entries, then uses an in-memory “head” ANN index to seed batched remote expansions (Li et al., 18 Jun 2026, Ockerman et al., 15 Jul 2025, Aizman et al., 25 Feb 2026, Adams et al., 7 Sep 2025).

A second principle is locality-aware execution. DistributedANN pushes node and neighbor scoring onto storage hosts and returns only IDs and scores. GetBatch coalesces requests per owning node and per archive, activates those nodes as senders, and uses a single Designated Target to assemble an ordered stream. The distributed full-text index routes through a replicated global trie and delegates substring verification to local Patricia tries with DRMA. Outback similarly separates compute-heavy bucket-location logic at compute nodes from compute-light dereferencing at memory nodes, so the memory node avoids hash computation, probing, and key comparison on reads (Fischer et al., 2016, Liu et al., 13 Feb 2025).

A third principle is controlled batching rather than maximally large batching. DistributedANN explicitly uses batching only where it reduces round-trips or amortizes overhead, and keeps per-host batch size $b$ small—typically $b=1$—to control latency and tail behavior. ODB bounds in-flight views by an outstanding depth $$D = \max(\text{prefetch\_factor} \times \text{num\_workers}, \text{buffer\_size})$$. GetBatch couples ordered streaming with admission control, HTTP 429 rejection under memory pressure, and CPU/disk throttling. The distributed full-text index models batch time as amortizing startup latency $L$ over $q$ queries (Adams et al., 7 Sep 2025, Li et al., 18 Jun 2026, Aizman et al., 25 Feb 2026, Fischer et al., 2016).

3. Runtime batch formation in distributed learning

In ODB, the observability problem is fundamental: offline samplers cannot see the quantity that determines padding, memory use, and GPU saturation unless they maintain a preprocessing- and augmentation-dependent length cache. ODB therefore turns DataLoader workers into single-sample producers and inserts a dedicated collate process that drains worker queues, sorts by realized length, greedily forms variable-size batches under a token budget, and aligns the resulting group counts across ranks with one Gloo all_gather. Dynamic local group size is chosen by

$$B(l)=\max\left(\left\lfloor L_{\max}/l \right\rfloor, 1\right),$$

so that shorter realized lengths produce larger groups. The alignment problem is formalized as the Distributed Group Alignment Problem, and the Max-Based Bidirectional Group Alignment protocol guarantees deadlock-free bounded termination, strict identity coverage in default join mode, and sample-quota closure in opt-in non-join mode. The implementation requires no model, optimizer, or attention-kernel changes, uses one small all_gather per round of about 128 KB at $W=8$ and buffer $=1024$, and reports overhead below $2\%$. Across public 2B/8B Qwen3-VL runs, ODB improves emitted-sample throughput by $b$0–$b$1 on single-node Full FT/LoRA and $b$2–$b$3 on two-node Full FT; on production MM-Mix it reaches $b$4, while remaining Standard-comparable in quality (Li et al., 18 Jun 2026).

PGT-I uses “index-batching” and “distributed-index-batching” in a more literal data-structure sense. Standard sliding window analysis pre-materializes every overlapping input-output pair, yielding

$b$5

whereas index-batching stores one copy of the data plus start indices,

$b$6

On PeMS, an $b$7 GB raw dataset becomes $b$8 GB after SWA; index-batching instead stores a base tensor and assembles $b$9 and $b=1$0 at runtime. Distributed-index-batching extends this across multiple GPUs without graph partitioning: each worker holds a local copy of $b=1$1, global shuffling is synchronized by the same seed, only gradients are all-reduced, and batch data are never communicated during training. The paper reports the first-ever training of an ST-GNN on the entire PeMS dataset without graph partitioning, peak memory reduction by up to $b=1$2, and speedup by up to $b=1$3 over standard DDP with $b=1$4 GPUs. It also reports $b=1$5 total runtime speedup and $b=1$6 training-only speedup versus single-GPU, and a single-GPU GPU-index-batching configuration with $b=1$7 GB CPU memory, $b=1$8 GB GPU memory, and $b=1$9 end-to-end runtime improvement (Ockerman et al., 15 Jul 2025).

4. Retrieval and search over distributed indexes

DistributedANN applies batching to graph traversal over a single 50 billion vector DiskANN index spread across over a thousand machines. The system stores graph nodes as key-value entries, adds a dense, sharded, in-memory head index built from the top layers of the unified graph, and pushes near-data scoring onto storage hosts. Query ingress searches the head index, initializes result and candidate heaps, then executes $$D = \max(\text{prefetch\_factor} \times \text{num\_workers}, \text{buffer\_size})$$0 hop iterations in which the orchestrator takes the best $$D = \max(\text{prefetch\_factor} \times \text{num\_workers}, \text{buffer\_size})$$1 candidate keys, groups them per shard, and issues node-scoring RPCs concurrently. The design explicitly distinguishes per-hop batching of remote expansions from per-host IO/compute batching; because the shard-to-$$D = \max(\text{prefetch\_factor} \times \text{num\_workers}, \text{buffer\_size})$$2 ratio is high, per-host batch size is typically $$D = \max(\text{prefetch\_factor} \times \text{num\_workers}, \text{buffer\_size})$$3. On a 50B-vector Bing slice with parameters $$D = \max(\text{prefetch\_factor} \times \text{num\_workers}, \text{buffer\_size})$$4, $$D = \max(\text{prefetch\_factor} \times \text{num\_workers}, \text{buffer\_size})$$5, $$D = \max(\text{prefetch\_factor} \times \text{num\_workers}, \text{buffer\_size})$$6, $$D = \max(\text{prefetch\_factor} \times \text{num\_workers}, \text{buffer\_size})$$7, and $$D = \max(\text{prefetch\_factor} \times \text{num\_workers}, \text{buffer\_size})$$8, the reported production system delivers more than $$D = \max(\text{prefetch\_factor} \times \text{num\_workers}, \text{buffer\_size})$$9 QPS, $L$0 ms median latency, $L$1 ms p99, Recall@5 $L$2, Recall@200 $L$3, IO per query $L$4, and network per query $L$5 MiB. The paper attributes the efficiency gain over clustered partitioning to unified graph traversal, flexible IO allocation, near-data scoring, and random sharding (Adams et al., 7 Sep 2025).

GetBatch makes batch retrieval a storage-native operation. A single HTTP GET with a JSON body lists many items; the store resolves ownership, coalesces entries per sender and per archive, and a Designated Target assembles a strict-order streaming TAR response. The semantics are deterministic, support continue-on-error placeholders, and preserve output order equal to request order. In a synthetic benchmark on a 16-node cluster, GetBatch improves throughput by up to $L$6 for 10 KiB objects, $L$7 at 100 KiB, and $L$8 at 1 MiB. In a production speech training workload, it reduces batch P95 latency from $L$9 ms to $q$0 ms, batch P99 latency from $q$1 ms to $q$2 ms, and per-object P99 latency from $q$3 ms to $q$4 ms (Aizman et al., 25 Feb 2026).

The distributed full-text index of Fischer, Kurpicz, and Sanders treats batching as simultaneous processing of $q$5 queries per processing element. Its two-level structure consists of a replicated global trie over boundary suffixes and a local Patricia trie per PE. Query routing uses the replicated global trie to determine an interval of responsible PEs, messages are coalesced by destination PE, and DRMA retrieves substrings for verification. The batch model is

$q$6

and the paper reports counting queries up to $q$7 times faster than the distributed suffix array while using about the same space. Its succinct LOUDS variant uses only one third of the memory compared with the non-succinct variant at the expense of about $q$8 slower query times (Fischer et al., 2016).

Outback is a contrasting case. Its prototype does not employ request batching in experiments in order to keep latency low, but its separated perfect hashing index still instantiates a related design logic: compute nodes precompute exact bucket and slot using Othello arrays and per-bucket seeds, while memory nodes perform direct slot and KV reads in one round trip. The paper reports higher throughput than state-of-the-art one-sided and two-sided RDMA-based in-memory KVS by $q$9–$$B(l)=\max\left(\left\lfloor L_{\max}/l \right\rfloor, 1\right),$$0 (Liu et al., 13 Feb 2025).

5. Information-theoretic and coding formulations

In distributed index coding, batching becomes a question of how many receivers can be served by coded summaries emitted from multiple servers. The general model has $$B(l)=\max\left(\left\lfloor L_{\max}/l \right\rfloor, 1\right),$$1 servers, one for each nonempty subset $$B(l)=\max\left(\left\lfloor L_{\max}/l \right\rfloor, 1\right),$$2, with server $$B(l)=\max\left(\left\lfloor L_{\max}/l \right\rfloor, 1\right),$$3 holding messages $$B(l)=\max\left(\left\lfloor L_{\max}/l \right\rfloor, 1\right),$$4 and broadcasting over a link of capacity $$B(l)=\max\left(\left\lfloor L_{\max}/l \right\rfloor, 1\right),$$5. The outer bound is expressed through a polymatroidal set function, and the generalized MAIS bound gives

$$B(l)=\max\left(\left\lfloor L_{\max}/l \right\rfloor, 1\right),$$6

for acyclic $$B(l)=\max\left(\left\lfloor L_{\max}/l \right\rfloor, 1\right),$$7. The inner bounds are distributed composite coding schemes: server-by-server coding, joint coding with the same decoding sets across all servers, and the most general scheme with different decoding sets across server groupings. For the $$B(l)=\max\left(\left\lfloor L_{\max}/l \right\rfloor, 1\right),$$8 instance $$B(l)=\max\left(\left\lfloor L_{\max}/l \right\rfloor, 1\right),$$9 with symmetric capacities $W=8$0, the more coordinated schemes achieve sum rate $W=8$1, whereas the server-by-server construction is limited to $W=8$2. For the $W=8$3 instance $W=8$4 with $W=8$5, the fixed-decoding-set scheme yields sum rate below $W=8$6, while grouping-dependent decoding achieves $W=8$7 and matches the outer bound. The strengthened capacity-theorem literature goes further: distributed composite coding with flexible decoding configurations and enhanced fractional rate allocations matches outer bounds for the sum-rate in all $W=8$8 non-isomorphic four-message problems with equal link capacities, using outer bounds that incorporate general groupings and $W=8$9-separation (Sadeghi et al., 2016, Liu et al., 2018).

Linear batch codes express the same idea as load-balanced algebraic retrieval. An $=1024$0 batch code stores $=1024$1 coded symbols across $=1024$2 buckets so that any multiset of $=1024$3 requested indices can be recovered by reading at most $=1024$4 symbols from each bucket. In the binary special case $=1024$5 and $=1024$6, a linear batch code is denoted $=1024$7, with generator matrix $=1024$8. The central theorem states that the generator matrix of a binary linear batch code is also a generator matrix of a classical binary linear error-correcting code $=1024$9 with minimum distance $2\%$0. This imports standard bounds such as Singleton, Hamming, Plotkin, Gilbert–Varshamov, and Griesmer. The simplest example is the $2\%$1 code with generator

$2\%$2

which supports two disjoint recovery sets for repeated requests to the same source symbol (Lipmaa et al., 2014).

6. Trade-offs, limitations, and open directions

The dominant trade-offs differ by setting, but a common pattern is that index-driven batching exchanges one bottleneck for another. ODB removes the need for preprocessing-dependent length caches and preserves DDP step alignment without model-side changes, yet it introduces a collate-side control protocol, exact token-level loss scaling when post-alignment counts differ, and a termination distinction between default join mode and opt-in non-join mode. The paper reports that default join mode gives strict identity coverage and per-iteration zero-discard with negligible throughput cost, with mean delta about $2\%$3 and range $2\%$4 to $2\%$5 across studied cells (Li et al., 18 Jun 2026).

DistributedANN makes the space, bandwidth, and latency costs explicit. Compressed-vector duplication yields a space amplification factor of about $2\%$6 for $2\%$7, $2\%$8, $2\%$9, and 8-byte IDs; network per query rises to about $b$00 MiB from $b$01 MiB in the clustered baseline; and latency increases from $b$02 ms to $b$03 ms at p50 and from $b$04 ms to $b$05 ms at p99. The system nevertheless improves throughput and recall, and recall degrades gracefully under partial failures—for example, Recall@5 drops from $b$06 to $b$07 at a $b$08 per-host failure rate (Adams et al., 7 Sep 2025).

PGT-I’s local-replication regime avoids data communication but does not divide the compact dataset across workers: per-worker memory remains approximately the index-batching size, and global cluster memory is therefore about $b$09. GetBatch reduces request amplification but introduces a per-request serializer in the Designated Target; the paper notes that the first saturation point is DT disk, then memory and CPU for buffering and ordering, and that large-object workloads see diminishing gains. The distributed full-text index scales well under balanced load but remains sensitive to popular patterns, so virtualization and random document permutation are used to mitigate hot destinations (Ockerman et al., 15 Jul 2025, Aizman et al., 25 Feb 2026, Fischer et al., 2016).

Several cited systems state concrete next steps. DistributedANN identifies kernel-bypass networking and computational storage for node scoring as promising directions. PGT-I points to hybrid strategies combining index-batching with graph partitioning and to extensions for dynamic graphs with temporal signal. GetBatch identifies server-side shuffle modes, multi-DT or hierarchical assembly, and tighter integration with GPU preprocessing stacks. Distributed index coding remains open on tighter general outer bounds, graph-structural characterizations, and algorithmic selection of server groupings and decoding sets with provable guarantees (Adams et al., 7 Sep 2025, Ockerman et al., 15 Jul 2025, Aizman et al., 25 Feb 2026, Sadeghi et al., 2016).