Service Balancing: Optimization in Heterogeneous Systems

• Service balancing is a framework for dynamically allocating and routing service demand based on load, capacity, and fairness in heterogeneous and resource-constrained systems.
• It integrates formulations like throughput maximization, fairness, and equalized workload, with applications ranging from cloud scheduling to network dispatch and distributed storage.
• Key methodologies include rank-based policies, approximate state estimation, and threshold control that ensure robust performance under uncertainty, migration, and temporal shifts.

Service balancing denotes a family of allocation, routing, scheduling, ranking, and placement problems in which service demand must be distributed so that operational objectives remain controlled under heterogeneity, uncertainty, and resource constraints. In the literature summarized here, the term spans real-time cloud service ranking with checkpoint-based migration (Belgaum et al., 2019), transport-layer dispatch that maximizes aggregated throughput and overall service utilization (Aghdai et al., 2018), communication-aware routing across parallel servers (Mendelson et al., 2022), location-allocation models that trade off service efficiency against spatial equality (Kong et al., 2023), and state-dependent threshold control in flexible server systems (Lu et al., 20 Jan 2026). The common structure is the replacement of naive equal splitting by policies that account for load, capacity, affinity, fairness, persistence of state, or robustness to perturbation.

1. Problem classes and objective functions

A central formulation treats service balancing as a ranking problem over candidate services. In checkpoint-based cloud scheduling, repeated invocation of several services to observe response times is described as time-consuming and as a waste of service invocations, and the proposed remedy is to predict service ranks from previously offered service values (Belgaum et al., 2019). The ranking mechanism uses a correspondence value

$CV(x,y)=\frac{a-b}{n(n-1)/2},$

a prefer value

$P(x,y)=S_x-S_y,$

and a priority value

$PV=\sum P(x,y).$

Only positive correspondence values are retained, and services are then ranked by priority, with implicit previously accessed services given more importance.

A second formulation treats service balancing as a throughput problem. In data-center transport-layer load balancing, the stated objective is to maximize the aggregated throughput $T^j$ of a VIP’s service instances, with overall service utilization measured by

$$\Omega^j = T^j / C^j, \qquad C^j=\sum_{i=1}^{N^j} C_i^j, \qquad T^j=\sum_{i=1}^{N^j} U_i^j C_i^j.$$

This formulation rejects equal-probability dispatch as a sufficient principle when flows are heavy-tailed and instances have different or time-varying capacities (Aghdai et al., 2018).

Other formulations replace throughput or rank with explicit fairness or equality terms. In extended $p$-median models for public services, the tradeoff is between total travel distance and total envy, with the combined objective

$$\Sigma_{i\in I}\Sigma_{j\in J} w_j d_{ij}x_{ij} +\beta \Sigma_{i\in I}\Sigma_{j\in J,\ d_{ij}>d^*} w_j(d_{ij}-d^*)x_{ij},$$

where the first term is interpreted as efficiency and the second as equality or inequity aversion (Kong et al., 2023). In distributed storage, dynamic service balancing is formalized through worst-case discrepancy after limited popularity-rank swaps: $$D^*(t,p)=\min_{(S_1,\dots,S_{2t})\in\mathcal{S}_t}\max_{I_{t,p}\in\mathcal{I}_{t,p}}D(S_1,\dots,S_{2t};I_{t,p};t),$$ so balance is measured by resilience of block-sum equality under perturbation (Sima et al., 2024).

These objective functions show that service balancing is not a single optimization criterion. Depending on the system, the balanced outcome may mean high-priority service reuse, high aggregate utilization, low queueing delay, bounded envy, equalized workload, or perturbation-resilient access discrepancy.

2. Queueing and mean-field foundations

For large parallel-server systems, service balancing is often analyzed through fluid, diffusion, or mean-field limits. Under $SQ(d)$ routing with general service distributions, queue lengths alone are not Markovian because future departures depend on service ages. A state descriptor therefore augments queue length with age-resolved survival terms: $$Z_\ell^{(N)}(t,r)=\frac{1}{N}\sum_{i\in S_\ell(t)} \frac{\overline G(\alpha_i^{(N)}(t)+r)}{\overline G(\alpha_i^{(N)}(t))}.$$ As $P(x,y)=S_x-S_y,$0, this converges to the unique solution of a hydrodynamic PDE system, which yields queue-length distributions, propagation of chaos, and mean virtual waiting time (Aghajani et al., 2015). In the exponential case, the PDE reduces to the classical supermarket-model ODEs, so the framework strictly generalizes the memoryless analysis.

The same line of work also shows that intuitive balancing rules need not be dynamically benign. In flexible many-server systems with multiple customer classes and server pools, the LQFS-LB rule is intended to minimize and perfectly balance server pool loads, yet local fluid instability can occur even when the positive-rate activities form a tree (Stolyar et al., 2010). The reported consequence is stronger than transient instability: diffusion-scaled stationary distributions may be nontight and may escape to infinity. This is a direct warning that static balance conditions do not by themselves imply stable dynamic balancing.

Affinity-constrained systems add another structural layer. When jobs have primary and secondary server sets, the policy studied in large-scale systems assigns a job to an idle primary server if possible, otherwise to an idle secondary server, and otherwise to the shortest primary queue. A coupling construction yields stochastic dominance relative to reference systems such as random assignment, $P(x,y)=S_x-S_y,$1, or $P(x,y)=S_x-S_y,$2, and a fluid limit for the symmetric combinatorial model exhibits fixed-point structure and bistability phenomena (Cardinaels et al., 2018). Service balancing in this setting is therefore not pure symmetry restoration; it is balancing under compatibility restrictions and unequal service rates $P(x,y)=S_x-S_y,$3.

These results establish a general principle: service balancing policies are meaningfully characterized not only by their nominal objectives, but by the state representations and limit dynamics needed to certify their stability, equilibria, and transient behavior.

3. Information structures, communication budgets, and threshold control

A recurrent issue is how much state information a balancing rule must obtain. The CARE framework makes this explicit through four components—Communication, Approximation, Resource allocation, and Environment—and replaces exact queue lengths $P(x,y)=S_x-S_y,$4 by approximations $P(x,y)=S_x-S_y,$5 (Mendelson et al., 2022). The approximation error satisfies

$P(x,y)=S_x-S_y,$6

The framework studies basic approximation, queue-length emulation, and MSR approximation, together with RT-$P(x,y)=S_x-S_y,$7, DT-$P(x,y)=S_x-S_y,$8, and ET-$P(x,y)=S_x-S_y,$9 communication protocols. A central guarantee is that for every $PV=\sum P(x,y).$0, there exists a design with

$PV=\sum P(x,y).$1

and, under exponential service times, maintaining approximation error $PV=\sum P(x,y).$2 requires communication frequency only $PV=\sum P(x,y).$3. Routing is then done by JSAQ, which sends each job to the server with the smallest approximate queue length.

Threshold policies provide a second low-overhead mechanism. In self-learning threshold-based load balancing, the intended balanced occupancy profile is that almost all pools have either $PV=\sum P(x,y).$4 or $PV=\sum P(x,y).$5 tasks, and the threshold $PV=\sum P(x,y).$6 is fluid-optimal when $PV=\sum P(x,y).$7 for all $PV=\sum P(x,y).$8 (Goldsztajn et al., 2020). The implementation uses green and yellow tokens, dispatcher memory of $PV=\sum P(x,y).$9 tokens, and at most two messages per task. When $T^j$0 is unknown, an online control rule updates the threshold using the same token information and eventually settles at an equilibrium threshold.

A related utility-maximization formulation orders server-pool coordinates by decreasing marginal utility $T^j$1 and studies two policies: Join the Largest Marginal Utility (JLMU) and Self-Learning Threshold Assignment (SLTA) (Goldsztajn et al., 2021). JLMU uses full occupancy information, while SLTA uses at most two bits per pool and a learned scalar index $T^j$2. Both achieve the stationary upper bound asymptotically in the large-scale regime.

Taken together, these results undermine the idea that high-performance service balancing requires exact, high-frequency global state. The queueing evidence instead supports approximate states, tokenized thresholds, and learned rank thresholds as asymptotically optimal or near-optimal information structures.

4. Capacity-aware dispatch and placement across networks

In networked systems, service balancing frequently becomes a capacity-aware dispatch problem. Spotlight implements load balancing at the network edge and uses Adaptive Weighted Flow Dispatching (AWFD), which derives weights from available capacity

$T^j$3

New flows are assigned in proportion to these weights while per-connection consistency is preserved by a connection table (Aghdai et al., 2018). In the reported experiments, AWFD with $T^j$4 and $T^j$5 ms updates achieves more than $T^j$6 Gbps in a testbed where ECMP and consistent hashing fail to reach $T^j$7 Gbps, and average flow completion time improves by $T^j$8 to $T^j$9 over ECMP. The architecture is explicitly distributed per VIP and uses in-band flow dispatching.

INCAB addresses the same transport-layer setting but emphasizes in-network congestion awareness and memory economy. Each DIP’s load level is defined as the utilization of its highest-utilized resource, and the design uses two fixed-size hash tables, a Bloom filter, and an ultra-compact false-positive table rather than host-level traffic redirection or full connection tracking (Aghdai et al., 2018). Host-level rerouting is reported to add as much as $$\Omega^j = T^j / C^j, \qquad C^j=\sum_{i=1}^{N^j} C_i^j, \qquad T^j=\sum_{i=1}^{N^j} U_i^j C_i^j.$$0 extra traffic to the underlying network, whereas INCAB improves average flow completion time by $$\Omega^j = T^j / C^j, \qquad C^j=\sum_{i=1}^{N^j} C_i^j, \qquad T^j=\sum_{i=1}^{N^j} U_i^j C_i^j.$$1 compared to stateless solutions.

NFV service function chain provisioning broadens the scope from dispatch to joint placement and routing. Stringer optimizes a weighted objective

$$\Omega^j = T^j / C^j, \qquad C^j=\sum_{i=1}^{N^j} C_i^j, \qquad T^j=\sum_{i=1}^{N^j} U_i^j C_i^j.$$2

thereby trading off maximum utilization $$\Omega^j = T^j / C^j, \qquad C^j=\sum_{i=1}^{N^j} C_i^j, \qquad T^j=\sum_{i=1}^{N^j} U_i^j C_i^j.$$3 against the number of used servers (Chua et al., 2016). A queueing-theoretic $$\Omega^j = T^j / C^j, \qquad C^j=\sum_{i=1}^{N^j} C_i^j, \qquad T^j=\sum_{i=1}^{N^j} U_i^j C_i^j.$$4 latency estimator is then used to evaluate end-to-end delay. The underlying claim is that maximum resource utilization is an effective proxy for latency because queueing delays and packet drops rise sharply as utilization approaches one.

In edge-to-cloud IoT placement, EPOS Fog formulates a decentralized multi-objective problem that combines utilization variance $$\Omega^j = T^j / C^j, \qquad C^j=\sum_{i=1}^{N^j} C_i^j, \qquad T^j=\sum_{i=1}^{N^j} U_i^j C_i^j.$$5 with local execution cost $$\Omega^j = T^j / C^j, \qquad C^j=\sum_{i=1}^{N^j} C_i^j, \qquad T^j=\sum_{i=1}^{N^j} U_i^j C_i^j.$$6 (Nezami et al., 2020). Agents generate local plans and cooperatively select them through I-EPOS. The reported outcomes are reductions in utilization variance of $$\Omega^j = T^j / C^j, \qquad C^j=\sum_{i=1}^{N^j} C_i^j, \qquad T^j=\sum_{i=1}^{N^j} U_i^j C_i^j.$$7 to $$\Omega^j = T^j / C^j, \qquad C^j=\sum_{i=1}^{N^j} C_i^j, \qquad T^j=\sum_{i=1}^{N^j} U_i^j C_i^j.$$8 compared with First Fit and service execution delay reductions of $$\Omega^j = T^j / C^j, \qquad C^j=\sum_{i=1}^{N^j} C_i^j, \qquad T^j=\sum_{i=1}^{N^j} U_i^j C_i^j.$$9 to $p$0.

Across these systems, service balancing is inseparable from state locality, per-flow consistency, programmable data planes, and explicit capacity modeling. Equal splitting remains a baseline, but the dominant research direction is weight, route, or place service using real-time capacity or utilization information.

5. Fairness, utility, and equality as balancing criteria

Not all service-balancing problems are throughput-centric. In heterogeneous infinite-server pools, service balancing can be posed as maximizing normalized aggregate utility

$p$1

with concave class-dependent utility functions of occupancy (Goldsztajn et al., 2021). The optimal fractional task assignment fills coordinates in order of decreasing marginal utility, so balancing is with respect to service quality rather than queue length alone.

Public-service location models treat equality explicitly. The minimum distance and envy location problem and its capacitated counterpart use total envy above a threshold distance $p$2 as a linear fairness penalty, and the combined objective is described as inequity-averse (Kong et al., 2023). Computationally, the reported tradeoff is small efficiency loss for substantial equality gain: for MDELP the average increase in mean distance is about $p$3 and the average decrease in standard deviation is about $p$4; for CMDELP the corresponding numbers are about $p$5 and $p$6. Equality is evaluated by standard deviation, mean absolute deviation, and Gini coefficient of customer travel distances.

Healthcare workforce assignment uses a still stricter probabilistic fairness notion. Total workload imbalance between any pair of workers is required to remain below a threshold $p$7 with high probability under uncertain service times, and the joint chance constraint is handled through a distributionally robust worst-case CVaR approximation (Nguyen, 2021). In the reported synthetic experiments, the DRO approach yields an average violation probability of $p$8, compared with $p$9 for an expected-value benchmark, while mean reward decreases from $$\Sigma_{i\in I}\Sigma_{j\in J} w_j d_{ij}x_{ij} +\beta \Sigma_{i\in I}\Sigma_{j\in J,\ d_{ij}>d^*} w_j(d_{ij}-d^*)x_{ij},$$0 to $$\Sigma_{i\in I}\Sigma_{j\in J} w_j d_{ij}x_{ij} +\beta \Sigma_{i\in I}\Sigma_{j\in J,\ d_{ij}>d^*} w_j(d_{ij}-d^*)x_{ij},$$1.

Wireless charging deployment offers an analogous multi-objective structure in a physical service field. The deployment score

$$\Sigma_{i\in I}\Sigma_{j\in J} w_j d_{ij}x_{ij} +\beta \Sigma_{i\in I}\Sigma_{j\in J,\ d_{ij}>d^*} w_j(d_{ij}-d^*)x_{ij},$$2

balances charging fairness, measured by the fraction of covered receivers, against a service-quality term weighted by low state of charge (Liu et al., 2020). GA-based and PSO-based deployments are reported to outperform uniform and random deployment, with numerical results showing roughly $$\Sigma_{i\in I}\Sigma_{j\in J} w_j d_{ij}x_{ij} +\beta \Sigma_{i\in I}\Sigma_{j\in J,\ d_{ij}>d^*} w_j(d_{ij}-d^*)x_{ij},$$3–$$\Sigma_{i\in I}\Sigma_{j\in J} w_j d_{ij}x_{ij} +\beta \Sigma_{i\in I}\Sigma_{j\in J,\ d_{ij}>d^*} w_j(d_{ij}-d^*)x_{ij},$$4 or more charging-efficiency improvement depending on the scenario.

These formulations show that service balancing frequently becomes an equality design problem. The relevant balance may concern travel distance, workload, marginal utility, or residual battery state rather than server occupancy or packet delay.

6. Robustness under migration, perturbation, and temporal imbalance

A substantial part of the literature studies balancing under state migration or time-varying disturbance. In checkpoint-based cloud scheduling, a checkpoint is a locally stable state or point-in-time copy used so that a job migrated between sub-clouds resumes from the same state rather than restarting (Belgaum et al., 2019). Load is checked at checkpoint time, resources are assigned to VMs without exceeding available capacity, and historical service values are reused to reduce response-time overhead and cloud resource wastage.

Dynamic distributed-storage formulations push robustness further by requiring balance to survive popularity-rank swaps. For magnitude-one swaps, recent work proves

$$\Sigma_{i\in I}\Sigma_{j\in J} w_j d_{ij}x_{ij} +\beta \Sigma_{i\in I}\Sigma_{j\in J,\ d_{ij}>d^*} w_j(d_{ij}-d^*)x_{ij},$$5

and, for $$\Sigma_{i\in I}\Sigma_{j\in J} w_j d_{ij}x_{ij} +\beta \Sigma_{i\in I}\Sigma_{j\in J,\ d_{ij}>d^*} w_j(d_{ij}-d^*)x_{ij},$$6, constructs defining sets with

$$\Sigma_{i\in I}\Sigma_{j\in J} w_j d_{ij}x_{ij} +\beta \Sigma_{i\in I}\Sigma_{j\in J,\ d_{ij}>d^*} w_j(d_{ij}-d^*)x_{ij},$$7

leaving only about a $$\Sigma_{i\in I}\Sigma_{j\in J} w_j d_{ij}x_{ij} +\beta \Sigma_{i\in I}\Sigma_{j\in J,\ d_{ij}>d^*} w_j(d_{ij}-d^*)x_{ij},$$8 multiplicative gap between lower and upper bounds (Sima et al., 2024). The analytical machinery is based on paired graphs that encode actual swaps and potential imbalance changes. Earlier formulations cast the same objective as preserving balance of companion-set sums under collections of disjoint adjacent swaps (Sima et al., 2023).

Temporal imbalance also appears in network design and energy systems. In multi-period service network design with excessive demand, the carrier is allowed three responses: demand shifting through earliness or tardiness penalties, temporary leasing of additional assets, and outsourcing to third-party services (Secerdin et al., 2019). In the reported computational study, demand shifting is used very frequently, outsourcing is used when distances are long or capacity is tight, and leasing becomes more attractive when its cost is reduced. In power balancing service, electric boiler systems with thermal storage are scheduled through a stochastic linear program with Copula-based temperature uncertainty and K-means scenario reduction to $$\Sigma_{i\in I}\Sigma_{j\in J} w_j d_{ij}x_{ij} +\beta \Sigma_{i\in I}\Sigma_{j\in J,\ d_{ij}>d^*} w_j(d_{ij}-d^*)x_{ij},$$9 scenarios (Liu et al., 2021). The reported mean operating cost is $$D^*(t,p)=\min_{(S_1,\dots,S_{2t})\in\mathcal{S}_t}\max_{I_{t,p}\in\mathcal{I}_{t,p}}D(S_1,\dots,S_{2t};I_{t,p};t),$$0 yuan for the stochastic model versus $$D^*(t,p)=\min_{(S_1,\dots,S_{2t})\in\mathcal{S}_t}\max_{I_{t,p}\in\mathcal{I}_{t,p}}D(S_1,\dots,S_{2t};I_{t,p};t),$$1 yuan for the deterministic model.

Threshold structure also appears in finite clearing systems with two service modes. In a system with flexible Type-I servers and dedicated Type-II servers, the optimal policy is characterized by queue- and occupancy-dependent thresholds that determine whether a job is processed independently or collaboratively (Lu et al., 20 Jan 2026). The proposed linear heuristics are reported to achieve costs within $$D^*(t,p)=\min_{(S_1,\dots,S_{2t})\in\mathcal{S}_t}\max_{I_{t,p}\in\mathcal{I}_{t,p}}D(S_1,\dots,S_{2t};I_{t,p};t),$$2 of the optimal policy on average and to outperform benchmark policies that can exceed $$D^*(t,p)=\min_{(S_1,\dots,S_{2t})\in\mathcal{S}_t}\max_{I_{t,p}\in\mathcal{I}_{t,p}}D(S_1,\dots,S_{2t};I_{t,p};t),$$3 of the optimal cost.

The unifying implication is that robust service balancing requires persistence of state, explicit handling of temporal overload, and policies whose decisions remain effective under migration, demand shifts, or local perturbations. In this sense, balance is not only a static allocation property but also a stability property of the service system under change.