TORTA: Temporal Optimal Resource Scheduling

• TORTA is a distributed GPU inference scheduler designed with a two-layer architecture that integrates temporal foresight into multi-slot resource allocation.
• The framework employs optimal transport and reinforcement learning in its macro-level regional scheduler and uses a micro-level server allocator for efficient task placement.
• Experimental results demonstrate that TORTA reduces average response time by up to 15%, improves load balance by approximately 4–5%, and reduces power cost by 10–15%.

Searching arXiv for the specified paper and closely related work on distributed GPU inference scheduling. TORTA, short for Temporal Optimal Resource scheduling via Two-layer Architecture, is a distributed GPU inference scheduler for LLM services operating across geographically distributed regions under heterogeneous GPU capacity, dynamic demand, and high migration or switching overhead. It is formulated as a temporally aware alternative to instantaneous-state-only schedulers, which optimize one time slot at a time from the current system state and therefore ignore how present allocations affect future queueing, load balance, switching cost, and power cost. The framework combines a macro-level regional scheduler with a micro-level server allocator and is designed to jointly optimize response time, load balance, power cost, and state-transition overhead across multiple time slots rather than myopically optimizing each slot independently (Du et al., 14 Jul 2025).

1. Problem formulation and motivation

TORTA addresses distributed GPU inference over $R$ geographical regions and $T$ discrete time slots. Each task $n_i^t$ arriving at slot $t$ has computational requirements $c_i^t$, memory requirements $m_i^t$, and deadline $d_i^t$. The system state includes server utilization, regional queue lengths, inter-region latency, and temporal or history features. Allocation is represented by a binary tensor $X_t$, and the optimization target is the expected cumulative cost over the full horizon (Du et al., 14 Jul 2025).

The end-to-end completion time is decomposed as

$$T_{completion}(i,r,s,t) = T_{queue}(r,s,t) + T_{compute}(i,r,s) + T_{network}(i,r),$$

and the total objective is

$$\begin{aligned} \min_{\pi} \mathbb{E} \Bigg[ & \sum_{t=1}^{T} \bigg( \sum_{i=1}^{N_t} T_{completion}(i,r,s,t) + \alpha \cdot C_{switch}^t + \beta \cdot C_{power}^t \bigg) \Bigg]. \end{aligned}$$

Here, switching cost is defined as

$T$0

which penalizes abrupt reshuffling between adjacent slots, and the power term is

$T$1

The motivation for this formulation is the paper’s critique of reactive schedulers. Three limitations are emphasized. First, such methods miss predictable traffic buildup and only scale after a surge arrives, producing queueing spikes and a “staircase” response as GPUs warm up. Second, independent optimization in each slot can induce oscillations between very different allocations in adjacent slots, increasing task migration and model switching overhead. Third, failure recovery becomes brittle because nearby regions may be overloaded immediately after a regional failure. The paper therefore argues that reactive methods have a performance ceiling determined by the best single-slot solution plus unavoidable switching overhead, whereas a temporally aware scheduler can strictly improve the overall horizon cost (Du et al., 14 Jul 2025).

2. Two-layer architecture and optimization decomposition

The central design choice in TORTA is a two-layer architecture that decomposes the global spatiotemporal scheduling problem into a macro-level regional allocator and a micro-level server allocator. The macro layer manages inter-region task distribution and load balancing across geographical regions. The micro layer handles intra-region task-to-server assignment and server activation, with the explicit aim of reducing latency and switching or migration cost within each regional cluster (Du et al., 14 Jul 2025).

This decomposition is presented as the mechanism that reduces the complexity of the original exponential-size global optimization while preserving temporal coupling across slots. The macro layer reasons over region-to-region routing matrices and temporal features, while the micro layer converts macro decisions into concrete server-level placements. A plausible implication is that the architecture separates slow-timescale geographical balancing from fast-timescale hardware-aware dispatch, but the paper’s formal claim is more specific: the decomposition preserves the relevant temporal structure while making optimization tractable.

The full workflow is given procedurally: normalize demand and capacity, solve optimal transport, row-normalize the transport plan, predict future demand, produce the macro allocation with the RL policy, then for each region activate servers, sort regional tasks by urgency, compute scores across servers, assign tasks greedily, and update the system state for the next slot. Training is also decomposed: the demand predictor is pre-trained, baseline constants $T$2, $T$3, and $T$4 are estimated, and constraint weights are adaptively increased if the performance condition is violated (Du et al., 14 Jul 2025).

3. Macro-level scheduler: optimal transport and reinforcement learning

At the macro level, TORTA combines optimal transport (OT) with reinforcement learning (RL). OT provides a baseline regional allocation by matching the normalized request distribution $T$5 to the normalized GPU capacity distribution $T$6: $T$7 A cost matrix $T$8 is defined by

$T$9

and the OT problem is

$n_i^t$0

subject to $n_i^t$1 and $n_i^t$2. The resulting $n_i^t$3 is row-normalized into routing probabilities $n_i^t$4, which serve as a baseline allocation reference (Du et al., 14 Jul 2025).

RL then learns a temporally smooth policy around the OT baseline. The state is

$n_i^t$5

where $n_i^t$6 denotes utilization, $n_i^t$7 queue lengths, $n_i^t$8 the latency matrix, $n_i^t$9 historical load and temporal features, $t$0 predicted future arrivals, and $t$1 the previous allocation matrix. The action $t$2 is a region-to-region routing matrix with row sums equal to 1. Future arrivals are forecast by a demand predictor trained offline from the past $t$3 slots.

The macro-level reward has three components: alignment with OT, temporal smoothness, and queue or load cost: $t$4 The intended terms are

$t$5

$t$6

and

$t$7

The RL policy is trained with PPO, using a standard clipped policy-gradient structure with advantage $t$8 and clipping parameter $t$9. The policy network outputs Beta-distribution parameters for allocation entries so that actions remain in $c_i^t$0. Additional constraint losses are introduced to keep the learned policy within a bounded region relative to the OT baseline and to ensure enough switching-cost improvement. In the notation given,

$c_i^t$1

where $c_i^t$2 constrains deviation from OT and $c_i^t$3 enforces a target switching-cost improvement. The appendix states the performance condition

$c_i^t$4

with $c_i^t$5 and $c_i^t$6 the Lipschitz constants of response-time and power-cost functions, $c_i^t$7 the baseline reactive switching cost, $c_i^t$8 the achieved switching-cost reduction factor, and $c_i^t$9 the OT-deviation bound. Under this inequality, the paper claims that TORTA strictly outperforms any single-timeslot reactive method over the full horizon (Du et al., 14 Jul 2025).

4. Micro-level allocator: server activation and task placement

The micro layer converts regional routing decisions into concrete task-to-server assignments inside each region. It decides how many servers to activate and which server receives each task, using hardware compatibility, current load, and locality or warm-up reuse to minimize latency and state-transition costs (Du et al., 14 Jul 2025).

The target number of active servers in region $m_i^t$0 at slot $m_i^t$1 is given by

$m_i^t$2

where $m_i^t$3 is the server count in region $m_i^t$4, $m_i^t$5 is the current queue length, $m_i^t$6 the predicted incoming demand, $m_i^t$7 the average per-server capacity, and $m_i^t$8 a safety factor. The paper’s discussion indicates that active capacity is sized from queue, predicted load, and a variability buffer, and that activation or deactivation is performed gradually to avoid excessive warm-up loss.

Task placement is then performed greedily using a score

$m_i^t$9

whose components correspond to hardware fit, load fit, and locality. The hardware-compatibility term is

$d_i^t$0

with $d_i^t$1. The load-compatibility term is

$d_i^t$2

which exponentially penalizes overloaded servers. The locality term is

$d_i^t$3

rewarding assignment to servers that recently handled similar tasks so as to exploit cache or warm-up effects and reduce switching overhead.

Tasks are processed by urgency and resource demand. If no server can accept a task immediately, it may be buffered and can trigger additional activation. This suggests that the micro layer functions as an error-correction mechanism for macro-level decisions, a reading consistent with the paper’s later observation that the micro layer can partially absorb macro-level prediction errors (Du et al., 14 Jul 2025).

5. Experimental evaluation and reported performance

The experimental setup simulates 6 hours of operation over 480 time slots, each lasting 45 seconds. Task processing times follow a uniform distribution to reflect heterogeneity. Evaluation uses four SNDlib network topologies: Abilene with 12 nodes and 25 ms latency, Polska with 12 nodes and 45 ms latency, Gabriel with 25 nodes and 80 ms latency, and Cost2 with 32 nodes and 150 ms latency. The GPU infrastructure is heterogeneous, including A100, H100, 4090, V100, and T4 servers, and tasks span compute-intensive, memory-intensive, and lightweight categories. The cost model incorporates real-world electricity pricing data. Baselines are SkyLB, SDIB, and Round-Robin (Du et al., 14 Jul 2025).

Three primary metrics are reported: average response time, load balancing, and total cost. Load balance is measured as the inverse of the coefficient of variation of server utilization: $d_i^t$4 with the paper also giving the expanded expression in terms of $d_i^t$5, $d_i^t$6, and the number of active servers $d_i^t$7. Prediction accuracy is defined as

$d_i^t$8

The reported topology-specific results are as follows.

Topology	Mean response time	Mean load-balance coefficient
Abilene	16.39 s	0.765
Cost2	19.31 s	0.743
Gabriel	17.58 s	0.755
Polska	19.19 s	0.745

The abstract states that TORTA reduces average inference response time by up to 15% relative to state-of-the-art baselines. The paper also reports load balancing that is approximately 4–5% better than the baselines and states that power cost is reduced by about 10–15%, with specific values including 11.1 K$d_i^t$9 on Gabriel. Overall total operational cost is reported to decrease by roughly 10–20%, combining lower electricity expense with reduced switching overhead (Du et al., 14 Jul 2025).

The response-time breakdown indicates that queueing delay remains roughly 0.3 to 1.1 seconds, which the authors attribute to proactive resource management. The gains are strongest on the more challenging topologies and smaller on Polska, where higher routing flexibility allows simpler methods to perform reasonably well. The paper also studies forecast quality: response time decreases from about 20.5 s at low prediction accuracy to about 17.5 s at high accuracy, and TORTA begins to outperform all baselines once prediction accuracy reaches around 0.5 (Du et al., 14 Jul 2025).

6. Significance, limitations, and interpretive context

The paper frames TORTA’s contributions in four parts: it identifies the weakness of reactive GPU schedulers as the lack of temporal foresight; it integrates OT and PPO into a multi-slot, temporally smooth regional scheduler; it adds a micro-level allocator based on hardware compatibility, load awareness, and locality; and it reports improved end-to-end LLM inference performance across several topologies while reducing operational cost and load imbalance (Du et al., 14 Jul 2025).

The significance of the framework lies in its explicit treatment of temporal coupling. Queueing cost, migration overhead, and power cost are not modeled as isolated slotwise effects but as horizon-level consequences of successive allocations. The use of the squared Frobenius distance $X_t$0 as switching cost makes this coupling operational rather than purely conceptual. Similarly, the inclusion of predicted future arrivals $X_t$1 in the RL state embeds short-horizon forecasting directly into the control loop.

The limitations are also explicit. The method depends on a demand predictor, and its gains improve with forecast quality; the study demonstrates robustness down to moderate prediction accuracy but does not guarantee performance under very poor prediction. Evaluation is simulation-based with adapted baselines rather than large-scale production deployment. The formulation is also simplified relative to production data centers, using a limited set of topologies, a simplified task mix, and an incomplete treatment of full multi-tenant, heterogeneous hardware, and pricing complexities. This suggests that the reported results establish a strong simulation-based case for temporally aware scheduling, while leaving open the question of deployment behavior under more heterogeneous operational constraints (Du et al., 14 Jul 2025).