---
title: Multi-plane Clos Topologies Explained
url: https://www.emergentmind.com/topics/multi-plane-clos-topologies
type: topic
---

# Multi-plane Clos Topologies Explained

Multi-plane Clos topologies are Clos-derived switching structures in which multiplicity is introduced either through several parallel middle-stage modules inside a classical three-stage Clos fabric or through several parallel Clos fabrics operated as distinct planes. Across the literature, both interpretations appear: classical analysis treats the middle-stage multiplicity \(m\) as the key internal resource in \(C(n,m,r)\)-type networks, while optical and AI-cluster work treats a plane as either a selectable middle-stage module or a complete parallel Clos subfabric fed by host bandwidth breakout. In both cases, the governing design question is the same: how many parallel internal resources are needed to obtain the desired combination of nonblocking behavior, scalability, path diversity, and resilience [1204.3180] [2302.06238] [2605.04333].

## 1. Architectural forms and notation

The canonical foundation is the classical three-stage Clos network. In the notation \(C(n_1,r_1,m,n_2,r_2)\), the first stage contains \(r_1\) input modules of size \(n_1\times m\), the middle stage contains \(m\) modules of size \(r_1\times r_2\), and the last stage contains \(r_2\) output modules of size \(m\times n_2\). The symmetric case is written \(C(n,m,r)\) with \(n_1=n_2=n\) and \(r_1=r_2=r\). In this formulation, \(m\) is the number of middle-stage modules, and it is the parameter most naturally interpreted as the internal plane count [1204.3180].

A closely related parameterization appears in Clos-inspired sparse neural layers, where a three-stage Clos is defined as a 5-tuple \((I,O,R_i,R_m,R_o)\). Here \(I\) is the number of inputs, \(O\) the number of outputs, \(R_i\) the number of input routers, \(R_m\) the number of middle routers, and \(R_o\) the number of output routers. The paper defines path diversity as
\[
D=R_m,
\]
so the number of parallel middle routers is directly the number of available routes between any input-output pair. This supports a plane interpretation in which each middle router is one internal plane choice, even though the paper itself uses the terms “middle routers” or “modules,” not “planes” [1802.03885].

Optical realizations retain the same three-stage structure. In the Clos-ROADM architecture, the network “consists of three switch stages, i.e., input, middle, and output stages,” and neighboring stages are “interconnected by a fully connected network using short-reach fibers.” For a Clos-ROADM with \(D\) directional degrees and \(L\) fiber degrees, the design is denoted \(v(M,L,D)\), where \(M\) is the number of middle-stage switching elements. This again makes the middle-stage multiplicity the central internal resource, with every ingress module connected to every middle module and every middle module connected to every egress module [2302.06238].

In datacenter settings, the same structure appears in folded form. The literature explicitly states that a Spine-Leaf network is “essentially a (folded) Clos network,” with the Leaf layer corresponding to the stacked ingress and egress stages and the Spine layer corresponding to the middle stage. A stricter multi-plane usage then emerges in recent AI-cluster deployments, where a plane is a complete parallel Clos network and each host’s NIC bandwidth is broken out across several lower-rate links, one per plane [2302.06238] [2605.04333].

## 2. Nonblocking regimes and plane-count dimensioning

The most developed analytical treatment sizes the number of middle modules by the nonblocking criterion. In the classical symmetric Clos \(C(n,m,r)\), strict-sense nonblocking is obtained with
\[
m\ge 2n-1.
\]
For \(C(n,m,2)\), wide-sense nonblocking under the policy “reuse a busy middle crossbar whenever possible” requires
\[
m\ge \left\lfloor \frac{3n}{2}\right\rfloor.
\]
For multirate wide-sense nonblocking Clos networks, the paper derives the sufficient condition
\[
m \ge 5.6355n+4.
\]
These results are obtained through an LP-duality framework whose generic form is
\[
\max\{ c^T x \mid Ax\le b,\ x\ge 0\}
\]
and
\[
\min\{ b^T y \mid A^T y \ge c,\ y\ge 0\},
\]
with weak duality used as the certificate that one more middle-stage resource than the worst-case obstruction is sufficient [1204.3180].

The optical literature uses an analogous but application-specific parameterization. For the Clos-ROADM \(v(M,L,D)\), the “spatially strictly non-blocking condition for the Clos-ROADM is
\[
M > 2 \cdot L - 1,
\]
so we take \(M = 2 \cdot L\),”
while setting \(M=L\) corresponds to “the condition of a reconfigurable non-blocking Clos-ROADM.” In multi-plane language, the distinction is explicit: a larger number of middle-stage modules yields stricter nonblocking guarantees, while a smaller number may suffice for rearrangement or low blocking under dynamic traffic [2302.06238].

A useful cross-domain summary is given below.

| Context | Plane-count parameter | Condition or consequence |
|---|---:|---|
| Symmetric Clos \(C(n,m,r)\) | \(m\) | \(m\ge 2n-1\) for strict-sense nonblocking |
| Clos \(C(n,m,2)\) | \(m\) | \(m\ge \left\lfloor \frac{3n}{2}\right\rfloor\) for wide-sense nonblocking |
| Multirate Clos \(C(n,m,r)\) | \(m\) | \(m \ge 5.6355n+4\) for multirate WSNB |
| Clos-ROADM \(v(M,L,D)\) | \(M\) | \(M=L\) for reconfigurable non-blocking |
| Clos-ROADM \(v(M,L,D)\) | \(M\) | \(M>2L-1\) for spatially strictly non-blocking |
| ClosNet layer \((I,O,R_i,R_m,R_o)\) | \(R_m\) | \(D=R_m\) path diversity |

The ClosNet formulation adds an orthogonal dimensioning rule: the parameter count of the three-stage layer is
\[
P = R_m(I + O + R_i R_o),
\]
so increasing the number of middle modules raises path diversity linearly but also raises the number of learned parameters linearly [1802.03885].

## 3. Scaling, complexity, and resource efficiency

One of the principal reasons for introducing planes or parallel middle-stage modules is to replace a fully connected internal backplane with a staged decomposition. This is especially explicit in high-degree ROADMs. In a Spanke-ROADM \(s(D,L)\), the total number of WSSs is
\[
2 \cdot D \cdot L,
\]
and the backplane fiber count is
\[
(D^2 - D)\cdot L^2.
\]
For the Clos-ROADM \(v(M,L,D)\), the total number of WSSs is
\[
2 \cdot D + M,
\]
and the backplane fiber count is
\[
2 \cdot L \cdot M.
\]
Under the strict nonblocking benchmark \(M=2L\), the asymptotic complexities are \(\mathcal{O}(L^2)\) for Clos-ROADM fiber complexity versus \(\mathcal{O}(D^2\cdot L^2)\) for Spanke-ROADM, and \(\mathcal{O}(D+L)\) for Clos-ROADM element complexity versus \(\mathcal{O}(D\cdot L)\) for Spanke-ROADM element complexity [2302.06238].

The concrete example \(s(10,10)\) versus \(v(M,10,10)\) makes the scaling effect particularly clear. The Spanke realization requires **200 \(1\times 90\) WSSs** and **9,000 fibers**. The comparable Clos realization requires **20 \(10\times M\) (or \(M\times 10\)) WSSs**, **\(M\) \(10\times10\) WSSs**, and **\(20M\) fibers**. At \(M=6\), where the blocking probability is already very close to the Spanke-ROADM in the reported simulation, the fiber count becomes **120 fibers**, which is highlighted as more than **98%** fiber savings [2302.06238].

Blocking analysis in the optical case also exposes the interaction between spatial and spectral diversity. The paper gives the Erlang-B lower-bound model
\[
EB(\rho,w)=\frac{\rho^w/w!}{\left(\sum_{k=0}^{w} \rho^k/k!\right)},
\]
\[
B_i = EB(\rho,w),
\]
\[
B_o = EB(\rho(1-B_i),w), \qquad B = 1-(1-B_i)(1-B_o),
\]
where \(\rho\) is traffic load and \(w\) the number of available wavelengths. The TWC-WSS Clos-ROADM and TWC-AWG-TWC Clos-ROADM approach this theoretical limit because wavelength conversion removes internal wavelength-continuity restrictions. A plausible implication is that plane multiplicity is most effective when spectral constraints do not negate the nominal spatial path diversity [2302.06238].

Datacenter scaling uses the same logic in folded form. When a \(v(M,L,D)\) Clos is folded into Spine-Leaf form, each Leaf switch grows from size
\[
L \times M
\]
to
\[
(L+M)\times(L+M).
\]
More recent AI-networking work pushes this idea further by splitting an **800 Gb/s NIC** into **8 × 100 Gb/s** and building **eight parallel 100 Gb/s Clos planes**. With **51.2 Tb/s** switch silicon, this changes the effective switch port count from **64 ports at 800 Gb/s** to **512 ports at 100 Gb/s**, allowing a **two-tier** design in which each T0 switch has **256** ports downward and **256** upward, each T1 connects down to **512 T0 switches**, and the resulting network reaches **131,072 GPUs** [2605.04333].

The same flattening trend appears in recent multi-plane Fat-Tree comparisons. For a system of about **65K NICs**, the paper reports an **8-plane 2-layer Fat-Tree** with switch configuration \(512 \times 200\) Gbps, diameter \(d=2\), \(N_s=3{,}072\), \(N_o=2{,}097{,}152\) optical modules at 200G, and cost/NIC = \(\$5{,}075\), versus a **3-layer Fat-Tree** with \(d=4\), \(N_s=5{,}120\), \(N_o=393{,}126\) optical modules at 1.6T, and cost/NIC = \(\$10{,}323\). This suggests that multi-plane Clos is not only a control or resilience device; it is also a topology-flattening mechanism under high-radix breakout assumptions [2604.23519].

## 4. Path assignment, scheduling, and transport control

Once a multi-plane Clos is dimensioned, the control problem becomes the assignment of each connection to one internal resource among many equivalent or near-equivalent choices. In one-rate bidirectional Clos datacenter networks, the recent centralized model uses \(m\) low-level switches \(L_i\), \(n\) top-level switches \(T_i\), per-link capacities \(C_{i,j}\), demand matrix \(D_{j,k}\), and route-allocation tensors \(X_{i,j,k}\) and \(Y_{i,j,k}\). A connection between \(L_j\) and \(L_k\) traverses exactly one top-level switch and is represented as
\[
(T_i,L_j,L_k).
\]
Feasibility is defined by
\[
\forall i,j,\sum_{k=0}^{m-1}Y_{i,j,k}\le C_{i,j}
\]
and
\[
\forall j,k,\sum_{i=0}^{n-1}Y_{i,j,k}\ge D_{j,k}.
\]
Although the paper does not define planes explicitly, it states that the formulation can map naturally to a multi-plane deployment if each plane, or each optical circuit switch within a plane, is treated as one of the top-level switching resources [2507.12265].

Its main algorithm, FastReChain, handles route changes through replacement chains. The scheduling objective is to minimize
\[
\sum_{i,j,k}\left|X_{i,j,k}-Y_{i,j,k}\right|,
\]
that is, the total number of rearrangements between the current and new schemes. The method searches for short replacement chains with depth-first search and iterative deepening and accelerates the search through bitsets such as \(\mathrm{bitsetLT}\), \(\mathrm{bitsetLL}\), and \(\mathrm{bitsetLLT}\). The paper reports that, with bitset optimization, “the algorithm runs 1000 times faster,” and in dynamic scheduling the per-operation running times remain in the reported nanosecond range even with hundreds of top-level switches [2507.12265].

At the transport layer, multi-plane operation can be made explicit rather than abstracted. In production AI supercomputers, MRC and static SRv6 treat each entropy value as a direct encoding of one path on one plane. The sender builds an EV set, “typically **128 to 256 entries**,” chooses an equal number of EVs per plane, and sprays packets across them. The paper states that “all packets of a QP are sprayed across many paths on all planes in a multi-plane network,” and that “Each EV corresponds to a specific path on a specific network plane.” Static SRv6 then makes the EV-to-path mapping deterministic, so path identity is transport-visible and probeable [2605.04333].

These two control styles—centralized replacement-chain scheduling and end-host spraying across explicitly identified plane-local paths—address different operating regimes. The first is connection-level rescheduling in bufferless or optical Clos networks; the second is packet-level exploitation of a fixed large path set in synchronous AI training. Their common premise is that multi-plane Clos performance depends as much on plane-selection policy as on plane count.

## 5. Failure domains, graceful degradation, and operational behavior

The resilience case for multi-plane Clos is strongest where a plane is a complete parallel fabric rather than merely a middle-stage choice. In large AI clusters, each NIC’s total bandwidth is broken out across several planes, so a single fault removes only a fraction of total host bandwidth. The paper gives several explicit figures. Losing a **T0–T1** link reduces capacity from a node by about **3% in an 800 Gb/s plane**, versus about **0.4% in a 100 Gb/s plane**. Losing a **NIC–T0** link in an **8-plane** design costs **12%** of NIC bandwidth, but the training job can continue [2605.04333].

This smaller blast radius changes failure handling from fail-stop semantics toward graceful degradation. The production study reports that T0–T1 link failures and flaps can “largely be ignored.” In one **75K GPU** pretraining job, a T1 switch failure affected around a quarter of QPs and dropped around **580K packets**, yet after QPs mapped out the bad paths the throughput was “largely unaffected,” and when the switch actually rebooted there was **no impact**. In a **50K GPU** job on a **4-plane** system, a transceiver glitch flapping four links at once caused about **25% throughput reduction over the minute of flaps**, followed by immediate recovery, with no QP failure and no node eviction [2605.04333].

The same deployment also quantifies the latency and throughput regime achieved under normal operation. On Cluster B, **T0-local latency** is **\(5.09 \,\mu s\)** and **cross-T1 latency** is **\(6.54 \,\mu s\)**. Both **T0-local bandwidth** and **cross-T1 bandwidth** are approximately **770 Gb/s**, which is about **96% of theoretical peak**. At **42K GPUs**, NCCL over MRC reaches up to **92 GB/s per NIC** for large message sizes. These results combine topology and transport, but they show that a multi-plane two-tier Clos can maintain both high utilization and graceful degradation under ongoing failures [2605.04333].

This resilience model is not universal across the literature. The optical Clos-ROADM work, for example, explicitly notes that strict fault tolerance or explicit plane redundancy is not a major topic and does not present a formal resilience model, middle-stage protection scheme, or failure recovery method. A plausible implication is that multi-plane Clos has become operationally central first in large AI clusters and only secondarily in other Clos-derived domains [2302.06238].

## 6. Extensions, adjacent uses, and interpretive limits

The idea of a Clos with multiple parallel middle resources extends beyond packet or circuit switching. In ClosNets for DNN training, the topology is used as a predefined sparse connectivity pattern: routers become small fully connected learned subnetworks, while inter-router links are fixed scatters or permutations without weights. The key Clos properties retained are “full connectivity,” “shallowness,” “pre-determined connectivity,” “uniform and high path diversity,” and “an efficient hardware implementation.” The paper reports that Clos networks have comparable accuracy with the baseline networks while having **\(5.5\times\) less parameters**, and states that dense layer sizes can be reduced “by as much as an order of magnitude” without hurting model accuracy [1802.03885].

Not every architecture with high path diversity, however, is a multi-plane Clos. Recent work on Multi-Plane HyperX defines a plane as an **independent physical subnet/topology instance** and compares that model against **multi-plane Fat-Tree**. In a roughly **65K NIC** system, the paper reports an **8-plane 2-layer Fat-Tree** at \(\$5{,}075\) per NIC and an **8-plane 1D HyperX** at \(\$3{,}647\) per NIC, with the latter also reducing diameter from \(d=2\) to \(d=1\). This is a comparison against multi-plane Clos, not a reformulation of Clos itself [2604.23519].

A different neighboring direction is MRLS, a randomized two-level leaf-spine topology. That work states that the Fat-Tree is “essentially a multistage folded Clos network,” but MRLS is not presented as multiple parallel Clos planes. Its “multipass” property means repeated traversal of the same two-level fabric rather than traversal across distinct planes. The paper reports a **50% speedup against a Fat-Tree for an All2All collective comprising 100k endpoints**, but this should be understood as an alternative to conventional Clos regularity, not as a multi-plane Clos result [2605.26960].

A further interpretive limit appears in resource-centric topology analysis. One recent paper does not explicitly analyze Clos or multi-plane Clos fabrics, but it concludes that redundancy is most efficiently handled via parallel network instances “rather than intrinsic topological path diversity.” This provides indirect support for plane replication as a design pattern, while remaining outside formal Clos analysis [2601.19008].

The principal misconception, therefore, is to treat “multi-plane Clos” as a single, uniform object. The literature supports at least two technically distinct meanings: a three-stage Clos with several parallel middle-stage modules, and a set of several independent Clos fabrics used as explicit planes. The first meaning dominates classical nonblocking theory, optical ROADMs, and Clos-inspired sparse neural layers; the second dominates recent AI-supercomputer networking. The continuity between them lies in the role of the plane count as the decisive internal resource for path diversity, blocking behavior, and graceful degradation.

Source: https://www.emergentmind.com/topics/multi-plane-clos-topologies