Centralised Metaheuristic Load Balancer

• Centralised metaheuristic load balancer is a system architecture that gathers global state to solve assignment problems via iterative heuristic and stochastic search methods.
• It employs various techniques, including Greedy, Tabu Search, Simulated Annealing, Genetic Algorithms, and quantum annealing to address dynamic load and partitioning challenges.
• Empirical studies demonstrate its benefits in reducing straggler effects in cloud, HPC, and delayed-feedback systems, while highlighting trade-offs in scalability and model sensitivity.

Searching arXiv for the cited works to ground the article in current preprints. A centralised metaheuristic load balancer is a load-balancing architecture in which a single control point gathers system-wide state, formulates a global assignment or partitioning problem, and computes reassignments by heuristic, stochastic, or simulator-driven search rather than by static dispatch rules. In the recent literature, this pattern appears in cloud resource management with VM live migration, online token redistribution for distributed Diffusion Transformers training, quantum-annealing-based partitioning for adaptive mesh refinement and smoothed particle hydrodynamics, and partially observable queue routing under delayed acknowledgements (Sliwko, 22 Sep 2025, Sliwko et al., 13 Nov 2025, Zhang et al., 8 Aug 2025, Rathore et al., 2024, Tahir et al., 2021).

1. Conceptual scope and defining characteristics

Across the cited systems, centralisation means that the balancing decision is made from globally collected metadata rather than from purely local rules. In the cloud prototypes, a single scheduler queries nodes for current resource utilisation and resident services, maintains the current assignment, and issues migration commands (Sliwko, 22 Sep 2025). In KnapFormer, all GPUs in a balancing group perform an all-gather of length metadata on the CPU, after which a single CPU process solves a global assignment problem and broadcasts a routing plan (Zhang et al., 8 Aug 2025). In the quantum-annealing HPC setting, the balancer takes a global list of task weights, and, when needed, communication costs, formulates a small Ising/QUBO problem, minor-embeds it onto a QPU, samples many anneals, and returns the lowest-energy assignment (Rathore et al., 2024). In the delayed-feedback cluster model, the dispatcher maintains a belief state over all servers and selects the routing action at each arrival epoch by Monte Carlo tree search over a simulator (Tahir et al., 2021).

The “metaheuristic” aspect varies by domain. The cloud schedulers explicitly employ Greedy, Tabu Search, Simulated Annealing, and Genetic Algorithm variants (Sliwko, 22 Sep 2025, Sliwko et al., 13 Nov 2025). KnapFormer uses a three-pass greedy heuristic for a global multi-knapsack assignment (Zhang et al., 8 Aug 2025). The HPC partitioner compares Quantum Annealing against Steepest Descent and Simulated Annealing (Rathore et al., 2024). The delayed-feedback dispatcher is a related instance of centralised online search: it adapts POMCP with UCT selection and rollout simulation to the queuing simulator (Tahir et al., 2021). This suggests that, in practice, “centralised metaheuristic load balancer” denotes a family of globally informed optimisation loops rather than a single algorithmic template.

Setting	Central state	Optimisation target
Cloud service placement	Node capacities, task demands, migration costs	Stability and minimum cost
HPC partitioning	Task weights, edge-cut weights	Imbalance and communication
DiT training	Sequence lengths, workload estimates	Variance of per-GPU workloads
Delayed-feedback routing	Belief over queue states	Discounted return

2. Mathematical formulations

In the cloud formulation denoted the D-Resource System Optimisation Problem, physical nodes are $n=\{n_1,\dots,n_m\}$, tasks are $T=\{t_1,\dots,t_p\}$, and there are $d$ resource types $v=\{1,\dots,d\}$. A task-to-node assignment $u:T\to n$ induces node residual resources

$f_i(n)=a_i(n)-\sum_{t:u(t)=n} r_i(t),$

and an assignment is stable iff $f_i(n)\ge 0$ for all nodes and resources. Given an initial assignment $u_0$, the total System Transformation Cost is

$$\mathrm{STC}(u_0\to u_1)=\sum_{t\in T} c(t)\,[u_0(t)\neq u_1(t)],$$

and the optimisation target is a stable assignment $u^*$ with minimum STC (Sliwko, 22 Sep 2025). A closely related binary placement model uses decision variables $T=\{t_1,\dots,t_p\}$0, capacity constraints $T=\{t_1,\dots,t_p\}$1, assignment constraints $T=\{t_1,\dots,t_p\}$2, and a multi-term objective $T=\{t_1,\dots,t_p\}$3 (Sliwko et al., 13 Nov 2025).

For HPC partitioning, the central model is cast as Ising/QUBO optimisation. For adaptive mesh refinement, with indivisible patches and patch cell counts $T=\{t_1,\dots,t_p\}$4, the number-partitioning Hamiltonian is

$T=\{t_1,\dots,t_p\}$5

with $T=\{t_1,\dots,t_p\}$6 in the reported experiments (Rathore et al., 2024). For smoothed particle hydrodynamics, a graph $T=\{t_1,\dots,t_p\}$7 with node weights $T=\{t_1,\dots,t_p\}$8 and edge weights $T=\{t_1,\dots,t_p\}$9 yields the multi-objective energy

$d$0

where

$d$1

Here $d$2 is a load-balance penalty, $d$3 is an edge-cut penalty, and $d$4 tunes the relative weight of the two objectives (Rathore et al., 2024).

In KnapFormer, the balancing problem is a global multi-knapsack assignment on $d$5 sequences across $d$6 compute bags. If $d$7 is the estimated workload of sequence $d$8, $d$9 indicates assignment to bag $v=\{1,\dots,d\}$0, and $v=\{1,\dots,d\}$1, then the bag load is

$v=\{1,\dots,d\}$2

and a stated objective is to minimise the variance of per-GPU workloads: $v=\{1,\dots,d\}$3 subject to assignment and capacity constraints (Zhang et al., 8 Aug 2025). The semi-empirical workload model is

$v=\{1,\dots,d\}$4

with $v=\{1,\dots,d\}$5 fitted per GPU, for example $v=\{1,\dots,d\}$6 on H100 (Zhang et al., 8 Aug 2025).

In the delayed-feedback cluster model, the dispatcher–server system is a POMDP. Each server state is $v=\{1,\dots,d\}$7, where $v=\{1,\dots,d\}$8 is current queue length, $v=\{1,\dots,d\}$9 is the number of outstanding acknowledgements not yet observed, and $u:T\to n$0 is the number of acknowledgements observed in the last epoch. A common reward is

$u:T\to n$1

and the objective is to maximise discounted return $u:T\to n$2 (Tahir et al., 2021).

3. Centralised control architectures

The cloud scheduler is organised around two principal modules: an Event Monitor & Workload Database and an Optimizer. The monitor listens to task submit/cancel events, node up/down events, and VM-live-migration cost readings; it stores the current assignment $u:T\to n$3, node resource vectors $u:T\to n$4, and live usage $u:T\to n$5. The optimizer periodically, or upon overload, invokes a metaheuristic, reconstructs migration costs from the LMDT formula, and submits VM-migration commands through a hypervisor API or cloud-platform API (Sliwko, 22 Sep 2025). The reported data structures include a hash-map Task$u:T\to n$6Node, two $u:T\to n$7-vectors $u:T\to n$8 per node, and a sorted set or heap for overloaded nodes.

KnapFormer instantiates the same centralisation pattern in GPU training. Each iteration, GPUs measure sequence lengths and estimated per-sequence workload, all-gather length metadata on the CPU, and receive a centrally computed routing plan. A single all-to-all collective then moves packed token chunks to target devices; within each compute bag, sequences are split evenly into contiguous chunks and processed in parallel via DeepSpeed Ulysses; after forward and backward passes, a symmetric reverse-all-to-all restores the original ordering (Zhang et al., 8 Aug 2025). This is centralised at the metadata and planning level, but distributed at the data-movement and execution level.

The quantum-annealing implementation is likewise centralised, though the solver is hybrid CPU/QPU. The problem Hamiltonian

$u:T\to n$9

is derived from the Ising expansion, minor-embedded onto D-Wave Chimera or Pegasus via minorminer, and sampled over $f_i(n)=a_i(n)-\sum_{t:u(t)=n} r_i(t),$0 anneals, with the lowest-energy bit string chosen as the assignment (Rathore et al., 2024). The details emphasise precomputing five distinct embeddings to average out hardware biases and monitoring the chain-break fraction.

The delayed-feedback dispatcher is fully online. At each arrival epoch, belief is represented by $f_i(n)=a_i(n)-\sum_{t:u(t)=n} r_i(t),$1 particles, a search tree is built at the root belief node, UCT selection and rollout simulation are executed within a time budget approximately equal to the next-arrival interval, and the action $f_i(n)=a_i(n)-\sum_{t:u(t)=n} r_i(t),$2 is dispatched (Tahir et al., 2021). This architecture remains centralised because the policy search is performed at a single dispatcher using a global belief state.

4. Algorithmic families

The cloud literature presents the most explicit catalogue of centralised metaheuristics. Greedy starts from the current assignment and repeatedly moves a task from an overloaded node to the “best-fit” alternative node. Tabu Search evaluates all single-task reassignment neighbors while maintaining a tabu list of the last $f_i(n)=a_i(n)-\sum_{t:u(t)=n} r_i(t),$3 moves; the tuned parameters reported are $f_i(n)=a_i(n)-\sum_{t:u(t)=n} r_i(t),$4, $f_i(n)=a_i(n)-\sum_{t:u(t)=n} r_i(t),$5, and $f_i(n)=a_i(n)-\sum_{t:u(t)=n} r_i(t),$6 restarts. Simulated Annealing uses the same neighborhood and accepts uphill moves with probability $f_i(n)=a_i(n)-\sum_{t:u(t)=n} r_i(t),$7, cooling by $f_i(n)=a_i(n)-\sum_{t:u(t)=n} r_i(t),$8. Genetic Algorithm uses a population of assignments, standard crossover, mutation by random task reassignment, and elitist retention; one prototype uses $f_i(n)=a_i(n)-\sum_{t:u(t)=n} r_i(t),$9 and $f_i(n)\ge 0$0, while another reports effective ranges of $f_i(n)\ge 0$1, $f_i(n)\ge 0$2, and $f_i(n)\ge 0$3 (Sliwko, 22 Sep 2025, Sliwko et al., 13 Nov 2025).

A distinctive refinement is seeded GA. Instead of fully random initial populations, the GA is seeded with outputs of Greedy, Tabu Search, and Simulated Annealing, producing SGA-Greedy, SGA-TS, and SGA-SA (Sliwko et al., 13 Nov 2025). The reported rationale is to reduce the expensive “genetic drift” step while preserving high-quality diversity.

KnapFormer’s solver is not a GA or SA, but it is still a centralised heuristic search over a global assignment. Its three passes are: descending sort by workload and bag assignment by minimal current occupancy under capacity constraints; chunk partitioning within each bag; and routing-plan generation for one all-to-all (Zhang et al., 8 Aug 2025). The method relaxes the integrality of a knapsack solver in exchange for simplicity and speed.

The quantum-annealing protocol adds hardware-specific metaheuristic structure. Minor embedding maps logical qubits to chains of physical qubits; chain strength is set either by uniform torque-compensation or by manual tuning with multiplier $f_i(n)\ge 0$4; and anneal times are $f_i(n)\ge 0$5, with $f_i(n)\ge 0$6 anneals per call (Rathore et al., 2024). The primary obstacle to scalability is limited coupling on current quantum annealing hardware.

The delayed-feedback setting uses POMCP adapted to the queuing simulator, with UCT exploration constant $f_i(n)\ge 0$7, depth $f_i(n)\ge 0$8, and rollouts bounded by the inter-arrival interval (Tahir et al., 2021). Although framed as a policy-search algorithm rather than a classical metaheuristic, it occupies a similar operational niche: centralised stochastic search under incomplete information.

5. Empirical behaviour across application domains

The reported results show that centralised metaheuristic balancing is not tied to a single performance regime; its behaviour depends strongly on state observability, communication topology, and migration or embedding overhead.

Setting	Metric	Reported result
AMR, $f_i(n)\ge 0$9	Normalized maximum disparity	QA: $u_0$0; RR: $u_0$1
SPH, $u_0$2	Imbalance, cut-wt	QA: $u_0$3; METIS: $u_0$4
Delayed feedback, $u_0$5	Drop rate	POL $u_0$6; JMO $u_0$7; FI $u_0$8
KnapFormer, 32 H100	WIR, FBL, HFU	WIR reduced to $u_0$9; FBL $$\mathrm{STC}(u_0\to u_1)=\sum_{t\in T} c(t)\,[u_0(t)\neq u_1(t)],$$0; HFU $$\mathrm{STC}(u_0\to u_1)=\sum_{t\in T} c(t)\,[u_0(t)\neq u_1(t)],$$1
Cloud trace replay	Throughput, overload	MASB $$\mathrm{STC}(u_0\to u_1)=\sum_{t\in T} c(t)\,[u_0(t)\neq u_1(t)],$$2, $$\mathrm{STC}(u_0\to u_1)=\sum_{t\in T} c(t)\,[u_0(t)\neq u_1(t)],$$3 overload

In the AMR experiments, Quantum Annealing consistently beats Round Robin and is on par with Simulated Annealing and Steepest Descent for $$\mathrm{STC}(u_0\to u_1)=\sum_{t\in T} c(t)\,[u_0(t)\neq u_1(t)],$$4, except for a “kink” at $$\mathrm{STC}(u_0\to u_1)=\sum_{t\in T} c(t)\,[u_0(t)\neq u_1(t)],$$5 due to stochastic sampling. With $$\mathrm{STC}(u_0\to u_1)=\sum_{t\in T} c(t)\,[u_0(t)\neq u_1(t)],$$6 anneals, the QA best sample equals the SD best sample, while roughly $$\mathrm{STC}(u_0\to u_1)=\sum_{t\in T} c(t)\,[u_0(t)\neq u_1(t)],$$7 of samples beat RR (Rathore et al., 2024). For SPH, the reported outcome is stronger: both QA and METIS cut $$\mathrm{STC}(u_0\to u_1)=\sum_{t\in T} c(t)\,[u_0(t)\neq u_1(t)],$$8 edges, but QA achieves cut-weight $$\mathrm{STC}(u_0\to u_1)=\sum_{t\in T} c(t)\,[u_0(t)\neq u_1(t)],$$9 METIS cut-weight, node-imbalance $u^*$0 versus $u^*$1, and edge-cut $u^*$2 versus $u^*$3; the Pareto front obtained by varying $u^*$4 strictly dominates METIS, and approximately $u^*$5 of QA samples also Pareto-dominate METIS (Rathore et al., 2024).

In the delayed-feedback POMDP, the policy obtained by MCTS outperforms limited-information strategies such as JMO and JMO-E and nearly matches or overlaps full-information baselines such as JSQ and SED in several regimes. For $u^*$6, $u^*$7, $u^*$8, $u^*$9, $T=\{t_1,\dots,t_p\}$00, and $T=\{t_1,\dots,t_p\}$01, the reported drop rates are approximately $T=\{t_1,\dots,t_p\}$02 for POL, $T=\{t_1,\dots,t_p\}$03 for JMO, $T=\{t_1,\dots,t_p\}$04 for JMO-E, and below $T=\{t_1,\dots,t_p\}$05 for full-information policies (Tahir et al., 2021).

In distributed DiT training, the reported heterogeneity is extreme: without balancing, the Workload Imbalance Ratio ranges from $T=\{t_1,\dots,t_p\}$06 up to $T=\{t_1,\dots,t_p\}$07. KnapFormer with g8n4 bags reduces WIR to approximately $T=\{t_1,\dots,t_p\}$08, corresponding to less than $T=\{t_1,\dots,t_p\}$09 variance, and yields $T=\{t_1,\dots,t_p\}$10 to $T=\{t_1,\dots,t_p\}$11 speedup in end-to-end training workloads, including the example $T=\{t_1,\dots,t_p\}$12 for forward-backward latency in the hardest joint-pretraining case (Zhang et al., 8 Aug 2025). This suggests that when metadata are compact and re-sharding can be amortised by heavy computation, centralised balancing can substantially reduce straggler effects.

In large-scale cloud replay, MASB processes approximately $T=\{t_1,\dots,t_p\}$13 K tasks/min at $T=\{t_1,\dots,t_p\}$14 CPU, $T=\{t_1,\dots,t_p\}$15 memory, and $T=\{t_1,\dots,t_p\}$16 overload, fits an additional $T=\{t_1,\dots,t_p\}$17 tasks/min at $T=\{t_1,\dots,t_p\}$18 workload with $T=\{t_1,\dots,t_p\}$19 overload, and retains the full $T=\{t_1,\dots,t_p\}$20 K tasks/min under compaction from $T=\{t_1,\dots,t_p\}$21 K to $T=\{t_1,\dots,t_p\}$22 K nodes with $T=\{t_1,\dots,t_p\}$23 overload (Sliwko, 22 Sep 2025). A second cloud prototype reports that SGA-TS gives the lowest cost in fixed time, and in the largest scenario explores approximately $T=\{t_1,\dots,t_p\}$24 fewer candidate solutions than unseeded GA while finding a $T=\{t_1,\dots,t_p\}$25 better solution (Sliwko et al., 13 Nov 2025).

6. Limitations, trade-offs, and extensions

The central limitation of these systems is that global optimisation quality is purchased with centralised coordination overhead. In the cloud prototype, a single-head metaheuristic scheduler “chokes once the node-count $T=\{t_1,\dots,t_p\}$26 and task-count $T=\{t_1,\dots,t_p\}$27,” whereas offloading the logic to NAs gives linear scaling up to $T=\{t_1,\dots,t_p\}$28 K nodes (Sliwko, 22 Sep 2025). This directly challenges the misconception that a centralised metaheuristic load balancer is inherently scalable in all deployment regimes.

A second recurring limitation is model sensitivity. In the delayed-feedback system, performance depends on the accuracy of $T=\{t_1,\dots,t_p\}$29 and of arrival/service estimates; mis-specification leads to suboptimality, and long feedback delays with $T=\{t_1,\dots,t_p\}$30 require larger particle sets (Tahir et al., 2021). In KnapFormer, the compute-bag topology must be pre-specified, the greedy solver is not globally optimal, and irregular attention masks require more complex modelling (Zhang et al., 8 Aug 2025). In the cloud placement models, higher-quality allocations incur more migrations, while lower-quality allocations incur higher resource waste (Sliwko, 22 Sep 2025).

Quantum annealing adds hardware-specific constraints. Fully connected problems hit qubit-coupling limits, chain-break fraction rises sharply for $T=\{t_1,\dots,t_p\}$31 under default chain strengths, manual tuning has no a priori prescription, and current NISQ hardware limits dense graphs to approximately $T=\{t_1,\dots,t_p\}$32 (Rathore et al., 2024). The reported evidence therefore does not support a general claim that QA uniformly dominates classical load-balancing heuristics; on AMR it is competitive with SA and SD rather than decisively superior, whereas on the SPH multi-objective graph partitioning it is Pareto dominant over METIS (Rathore et al., 2024).

The cited extensions preserve the centralised optimisation loop while altering its solver or hardware substrate. Proposed directions include co-locating QPU with CPU in the same data-centre and using QA asynchronously at regular rebalancing intervals, replacing recursive two-way splits by a $T=\{t_1,\dots,t_p\}$33-way QUBO, employing machine learning to predict good $T=\{t_1,\dots,t_p\}$34, parallelising MCTS or offloading rollouts to GPU, incorporating more detailed delay models, exploring integer relaxations for the KnapFormer assignment, and extending the centralised knapsack formulation to Mixture-of-Experts routing, pipeline-parallel stage balancing, or multi-tenant inference serving (Rathore et al., 2024, Tahir et al., 2021, Zhang et al., 8 Aug 2025). A plausible implication is that the long-term research frontier is not the abandonment of centralisation, but the design of central controllers whose optimisation kernels are domain-specific, latency-aware, and hybrid across CPU, GPU, and QPU resources.