Not Every Sync Is Safe: Calibrated DiLoCo Scheduling for Shared AI Infrastructure

Abstract: DiLoCo-style training reduces communication by letting learner islands train locally before occasional outer synchronization, making it attractive for fragmented industrial AI fleets where training shares hardware with latency-sensitive serving. The question for such fleets is when an outer merge is worth its system cost, and whether choosing \emph{which} windows to defer matters at all. Existing scheduling studies evaluate workload-aware policies against fixed-period baselines, but most omit the control that isolates timing from budget: matched random deferral, which inherits the controller's synchronization budget but is not itself deployable. This omission is consequential: across controlled stress tests and real vLLM sidecar replays, matched random ties or beats every forecast-free policy we test, so gains reported against weaker baselines cannot be attributed to window choice. We fill this gap with Workload-Aware DiLoCo (WA-DiLoCo), a score-based controller that weighs learner progress against fleet pressure, and a calibration protocol that determines when matched random can be beaten, then demonstrate that it can. In the bursty regime where calibration exposes request-overlap structure, adding a one-step EWMA burst forecast to the online controller beats matched random in real vLLM sidecar replay, reducing SLO violations from 6.54\% to 5.09\% (8 of 10 seeds, $p=0.021$); offline Calibrated-WA, a non-deployable bound, shows the remaining headroom at 4.45\% versus 6.26\%. The deployable lesson remains the protocol: report real-sidecar effect-size transfer, a no-sync load match, and a matched-random envelope before claiming serving-SLO improvement.

• The paper introduces a matched-random evaluation protocol showing that synchronization policies must outperform random placement of the same merge budget before claiming serving-SLO benefits.
• Calibration reveals that synthetic pressure effects do not transfer reliably to real sidecars, while overlap-aware forecasting reduces violations from 6.54% to 5.09% in non-periodic replay.
• Results show that most online workload-aware policies tie random deferral, whereas offline calibration excels in bursty workloads, highlighting the need for deployment-specific calibration and stronger forecasting.

Motivation and problem

DiLoCo-style training reduces communication by letting learner islands run many local optimization steps before an outer synchronization, making it attractive for industrial fleets where pretraining shares hardware with latency-sensitive serving. The paper observes that prior work on DiLoCo variants—streaming, decoupled, gossip-based, NoLoCo, and SparseLoCo—makes synchronization cheaper or less blocking, but none decides whether the current serving state makes a merge safe. In a shared fleet, an 8-second outer merge occupies host, network, and checkpoint resources; any request in flight during that window inherits the interference. A fixed interval ignores fleet state, a pressure gate may defer too aggressively, and the authors argue that no single policy should be expected to dominate across regimes.

The central contribution is not a new scheduler but an evaluation protocol: a decision procedure for when a training-side policy has earned a serving-SLO claim. Its key control is matched random deferral, a non-deployable baseline that inherits the evaluated scheduler's merge budget post hoc and randomizes only window placement. This separates how often to synchronize from which windows to choose. A scheduler that cannot beat its own budget placed at random has demonstrated budget control, not workload awareness.

Methodology

The paper formalizes synchronization as feasible placement within a corridor $H_{\min} \le g_k \le H_{\max}$ (128 to 512 inner steps in the deployed configuration), with all policies sharing a token-weighted outer merge so that differences are attributable to timing alone. The objective is minimizing the sync-excess SLO violation rate $V_{\mathrm{ex}}(S) = V(S) - V_{\mathrm{nosync}}$, which isolates avoidable serving damage from base load violations.

WA-DiLoCo is a score-based controller combining capped learner progress signals (loss reduction, token contribution) with fleet pressure terms (network, serving, checkpoint) and staleness, weighted so serving pressure carries the largest fleet-side weight (0.75). It synchronizes when the mean score exceeds a threshold within the corridor.

Calibrated overlap cost replaces pressure proxies with a direct serving-cost signal: the number of sidecar requests in flight during a candidate merge window, fit through a monotone calibration map $\hat\psi$ from overlap to measured sync-excess violations. Offline Calibrated-WA reselects lowest-overlap windows after observing the full trace (a diagnostic bound, not deployable); Online Calibrated-WA scores windows using only decision-time information; and an EWMA forecaster adds a one-step request-rate estimate to expected overlap.

The evaluation protocol

The protocol requires, before any serving-SLO claim: (1) real-sidecar effect-size transfer (p95 ratios, not merely pressure–latency correlation), (2) a no-sync load match to compute sync-excess violations, and (3) a matched-random envelope, defined as the expected violation rate over uniform draws from feasible schedules with the same merge count, with placement gain $G(\pi) = \mathrm{MR}(\pi) - \mathbb{E}[V(S_\pi)]$. A five-tier claim hierarchy enforces that passing one tier motivates but does not license the next claim. An isolation audit with GPU-state guards quarantines contaminated runs.

The calibration results are the paper's most cautionary finding: the synthetic stress harness produces a high/low pressure p95 ratio of 1.71–1.72, but ordinary Qwen2.5 vLLM sidecars from 0.5B to 7B remain nearly flat at 1.002–1.005, even when pressure–latency correlation is high. Effect size does not transfer from the harness to real serving. Only targeted regimes transfer: pressure-heavy serving reaches a real ratio of 1.546 (correlation 0.898), and sync-heavy serving reaches a synchronization ratio of 1.556, showing that synchronization-window interference transfers independently of pressure. Any policy claim made against the stress harness alone would therefore be unsupported in production-like serving.

Results across regimes

The regime map that emerges is deliberately mixed:

Regime	Outcome
Pressure-heavy	Saturated; all policies within 1.3pp ($V$ between 35.91% and 37.21%)
Steady sync-heavy	WA-DiLoCo beats fixed-$H$ and gate baselines ($p=0.031$) but ties matched random ($G=-0.08$pp, $p=0.56$)
Bursty	Offline Calibrated-WA wins: $V_{\mathrm{ex}}$ 4.45% vs. 6.26% MR envelope, all five seeds ($V_{\mathrm{ex}}(S) = V(S) - V_{\mathrm{nosync}}$0pp, $V_{\mathrm{ex}}(S) = V(S) - V_{\mathrm{nosync}}$1); online Calibrated-WA ties random ($V_{\mathrm{ex}}(S) = V(S) - V_{\mathrm{nosync}}$2)
Non-periodic (forecast)	EWMA forecaster beats matched random in real sidecar replay

The bursty result is the strongest claim: offline calibrated scheduling is near-optimal within the matched-budget class under the calibrated overlap model (within $V_{\mathrm{ex}}(S) = V(S) - V_{\mathrm{nosync}}$3 of the best feasible schedule), and the online variant degrades gracefully with $V_{\mathrm{ex}}(S) = V(S) - V_{\mathrm{nosync}}$4 forecast error. But the online Calibrated-WA controller itself does not significantly beat matched random, exposing a gap between signal validity and online exploitability. The paper fills that gap with the pre-specified EWMA forecaster, specified before either replay was run: on the overlap-calibrated proxy it wins 8 of 10 seeds ($V_{\mathrm{ex}}(S) = V(S) - V_{\mathrm{nosync}}$5), and replaying the same schedules through real Qwen2.5-1.5B vLLM sidecars at a matched budget of eight merges it reduces SLO violations from 6.54% to 5.09% ($V_{\mathrm{ex}}(S) = V(S) - V_{\mathrm{nosync}}$6), cutting sync-active requests from 286 to 220. At the replayed load (~4,300 requests per 600 s), the 1.46pp gain corresponds to roughly 60 fewer SLO violations per sidecar per ten minutes at identical training throughput.

Ablations attribute the controller's stress-harness robustness to combining progress and pressure: full WA reaches 2.10% SLO-like violations, while progress-only degrades to 26.64%. Train loss is insensitive to placement at matched tokens, confirming serving SLO as the operative axis.

Limitations

The authors are explicit that this is a stress-tested replay study, not a live production deployment. The forecaster result rests on one sidecar model, one non-periodic trace family, and ten seeds, and its no-sync runs are not load-matched, so it is reported head-to-head rather than as sync-excess SLO. Training scale is limited to SmolLM2-135M on a single 8×B200 node with a shorter 1.7B check; the MI355X/RCCL fabric test validates collectives but is not a policy replay. Network and checkpoint pressure inputs remain unvalidated harness signals, and the online Calibrated-WA controller remains an incomplete implementation of the calibrated idea. The 8-second merge duration is configuration-specific, so calibration must be repeated per deployment.

Conclusion

The paper's durable contribution is methodological: gains reported against fixed-period baselines cannot be attributed to window choice unless they survive matched random deferral, and synthetic pressure effects must demonstrate real-sidecar effect-size transfer before licensing production claims. Held to that bar, most forecast-free workload-aware scheduling fails to beat random placement of its own budget; only calibrated overlap cost with forecasting clears it, and only in the bursty regime. The open question the paper leaves is whether a deployable online controller can close the remaining gap to the offline bound (5.92% vs. 4.45% in bursty replay) without oracle trace access, and whether the calibration protocol generalizes beyond the tested sidecar configurations.