---
title: Multi-DC Optical Networks
url: https://www.emergentmind.com/topics/multi-datacenter-optical-networks
type: topic
---

# Multi-DC Optical Networks

Multi-Datacenter Optical Networks constitute the physical and algorithmic foundation for distributed machine learning training that spans geographically separated datacenters interconnected via high-capacity optical fiber networks. These systems are increasingly critical for scaling large language model (LLM) and deep neural network (DNN) training where hardware resources in a single facility are insufficient. Multi-DC optical networks introduce novel challenges in resource assignment, communication scheduling, and system optimization, necessitating frameworks that co-design pipeline-parallel training algorithms with real-time network state awareness, latency estimation, and traffic engineering. Below, key principles, frameworks, and results from recent advances such as CBA ("Communication-Bound-Aware Cross-Domain Resource Assignment for Pipeline-Parallel Distributed LLM Training in Dynamic Multi-DC Optical Networks" [2512.20080]) are summarized in rigorous detail alongside representative approaches.

## 1. Distributed Training over Multi-DC Optical Network Topologies

Multi-DC optical networks are typically abstracted as a graph $G = (V, E)$ where $V$ represents individual datacenters (DCs) and $E$ are fiber links supporting $W$ frequency slots per fiber (e.g., $|W| = 80$ slots, each $12.5$ GHz, NSFNET topology [2512.20080]). Each link $e$ maintains a binary frequency-slot occupancy vector $s_e[1..W]$ at time $t$. In pipeline-parallel (PP) distributed LLM training, $L$ layers are partitioned into $P$ stages $P_0, ..., P_{P-1}$ with each stage mapped to a GPU—often spread across multiple DCs. Each of $M$ micro-batches per iteration triggers $(P-1) \cdot M$ inter-DC transmission requests via dynamic optical network traffic, where link occupancy may overlap across requests due to temporal demand.

Key metrics are:
- **Per-iteration runtime** $T_{iter}$: wall-clock time from start of first forward to end of last backward micro-batch.
- **Bubble ratio** $R_{bubble}$: proportion of iteration time spent idling owing to communication delays.
- **Blocking probability** $p_{block}$: fraction of transmission requests that cannot be assigned a feasible path and frequency slot block, inducing delay or cancellation.

## 2. Communication-Aware Resource Assignment and Scheduling

Recent frameworks such as CBA [2512.20080] model PP training as a mixed-integer linear program (MILP) seeking to minimize $T_{iter}$ under multi-DC optical network constraints. Decision variables $x_{r,p,i,f}$ indicate the assignment of micro-batch transmission $r$ (corresponding to a stage-to-stage data movement) to optical path $p$ and contiguous frequency slot block $f$ on every link of $p$.

The communication latency for a request $r$ with payload $c$ traversing path $p$ and slot block $f$ is captured by the $\alpha$–$\beta$ model:
\[
T_{comm}(r) = \alpha_p + \beta_p \cdot c + \varepsilon_p(c)
\]
where $\alpha_p$, $\beta_p$ are path-specific offset/bandwidth parameters updated per iteration, and $\varepsilon_p(c)$ accounts for queuing delays.

Scheduling constraints rigorously ensure that no frequency slot on any link is double-booked and that frequency-slot block assignment remains contiguous on all links of a path.

## 3. Communication-Bound-Aware (CBA) Dynamic Resource Adaptation

The crux of CBA [2512.20080] is adaptive, cross-domain orchestration:
- **Detection of communication-bound tasks**: the orchestrator inspects the previous schedule $S_{j-1}$ to label any micro-batch computation as communication-bound if network delays exceed prior dependency completion ($cur.start\_time > prev.completion\_time + Latency\_DC\_connect$).
- **Dynamic frequency slot demand adjustment**: if a transmission was blocked last iteration, decrease its slot demand by one; if labeled communication-bound, increment by one (bounded system-wide) to secure wider spectrum and improve latency.
- **K-shortest-path search with contiguity-aware path selection**: for each transmission, the framework evaluates $K$ candidate paths and slot blocks, calculating a fitness score
\[
I(p_k, f) = \frac{C_{avail}(p_k, f)}{L(p_k)} \cdot (1 - \rho(p_k))
\]
where $C_{avail}$ is the contiguity index, $L(p_k)$ is path hop count, and $\rho(p_k)$ is current frequency slot usage fraction.

This heuristic approach enables real-time resource adaptation as network state and model demands evolve during training. No formal worst-case guarantee on solution approximation is provided.

## 4. Performance Characterization and Benchmarks

Experimental evaluation [2512.20080] utilizes NSFNET (14 nodes, 21 links; 80 FS/link, 12.5 GHz; 64-QAM modulation), placing GPUs randomly in six DCs for Llama 3 models (8B, 70B, 8 PP stages).
- **Baseline algorithms**: KSP-FF (K-shortest paths, first-fit assignment) and SD-FF (shortest-distance path, first-fit).
- **Key results (Llama 3 70B, GPipe, $M=128$ micro-batches):**

| Metric              | KSP-FF | SD-FF | CBA (Ours) |
|---------------------|--------|-------|------------|
| Iteration time (s)  | 102.4  | 98.7  | **68.0**   |
| Bubble ratio (%)    | 48.1   | 45.5  | **37.9**   |
| Blocking prob. (%)  | 17.3   | 15.9  | **13.8**   |

- **Improvements over best baseline**:
  - $31.25\%$ reduction in iteration time
  - $11.96\%$ decrease in bubble ratio
  - $13.20\%$ fewer blocking requests

CBA ablation studies show that disabling communication-bound task labeling or dynamic $\alpha$–$\beta$ latency updates leads to inferior bubble ratio and blocking probability.

## 5. Theoretical and Algorithmic Complexity

The per-iteration complexity of CBA [2512.20080] is $O((P-1)\cdot M \cdot K \cdot (E\log V + W))$:
- $K$-shortest-path: $O(K (E \log V))$ per request,
- Contiguity and fitness computation: $O(W)$ per path.

Given practical values (e.g., $P=8$, $M=128$, $K=4$, $W=80$), CBA remains computationally tractable even in large network topologies.

## 6. Broader Context, Synergies, and Related Multi-DC Frameworks

Multi-DC optical networking for distributed machine learning entails tight co-design between application-level pipeline-parallel training and optical network resource management. CrossPipe [2507.00217] generalizes multi-DC pipeline scheduling as a constraint optimization model, providing both CP solver and greedy near-optimal schedules, explicitly accounting for bandwidth and latency (via $\alpha$–$\beta$ model) and achieving up to $33.6\%$ reduction in training time compared to static schedules.

Alternate frameworks such as SPP [2204.10562], HelixPipe [2507.00394], TawPipe [2511.09741], and BaPipe [2012.12544] focus on device-level communication patterns, weight-passing schemes, and load-balanced stage partitioning, providing the necessary abstractions for scaling within or across DC boundaries.

CBA represents the state of the art in integrating pipeline-parallel task scheduling with real-time optical network state, adapting spectrum assignment dynamically, and maximizing utilization under stringent multi-DC constraints. Such communication-bound-aware resource assignment mechanisms are fundamental to the sustainable scaling of distributed LLM and DNN training workloads across geographically distributed datacenters.

Source: https://www.emergentmind.com/topics/multi-datacenter-optical-networks