---
title: 'TeleChat3-MoE: Scalable MoE LLM'
url: https://www.emergentmind.com/topics/telechat3-moe
type: topic
---

# TeleChat3-MoE: Scalable MoE LLM

TeleChat3-MoE is a series of large language models employing a Mixture-of-Experts (MoE) architecture, with parameter scales ranging from 105 billion to over one trillion. The models are trained end-to-end on Huawei’s Ascend NPU clusters and are underpinned by a systematized training infrastructure. The technical advancements facilitate reliable scaling to the frontier sizes required for state-of-the-art LLMs. TeleChat3-MoE integrates architectural innovations, comprehensive accuracy verification protocols, a suite of performance optimizations, and advanced parallelization methodologies to achieve near-linear scaling efficiency on clusters with thousands of devices [2512.24157].

## 1. Model Architecture

The core of TeleChat3-MoE is an MoE backbone within an encoder–decoder or decoder-only stack comprising $L \approx 45$–$60$ layers (depending on scale). Each layer incorporates Multi-Latent Attention (MLA) blocks, feed-forward networks (FFN), and a dedicated MoE block. MoE layers instantiate $E$ experts per layer (typically in the hundreds), but only the top $k=4$–8 experts are activated per token. A fallback shared expert is always available.

The expert routing employs a lightweight linear gating network $W_g \in \mathbb{R}^{E \times d}$, which calculates logits for expert selection from input representation $x \in \mathbb{R}^d$:
$$P_i(x) = \operatorname{softmax}_i(W_gx) = \frac{\exp(w_{g,i}^T x)}{\sum_{j=1}^E \exp(w_{g,j}^T x)}.$$
Top-$k$ routing enforces sparsity by zeroing out all but the $k$ highest routing probabilities. Expert parameters in a layer are estimated as:
$$P_{\text{layer}}^{\mathrm{MoE}} \approx E \cdot (d\,h_{ff} + h_{ff}\,d),$$
where $d$ is hidden size and $h_{ff}$ is the expert intermediate dimension. For non-MoE models, the corresponding dense FFN parameter count would be $2d\,d_{ff}$. The total model parameter count $P$ is
$$P \approx L \cdot [P_{\mathrm{MLA}}(d) + P_{\mathrm{MoE}}(d, h_{ff}, E)] + P_{\mathrm{embed}} + P_{\mathrm{head}},$$
with $P_{\mathrm{MoE}} = E \cdot (2d\,h_{ff})$, and $P_{\mathrm{MLA}} \approx 4d^2$ describing attention projections.

## 2. Model Scale and Memory Footprint

TeleChat3-MoE variants scale from $1.05 \times 10^{11}$ to over $1 \times 10^{12}$ parameters by adjusting:
- **Hidden size $d$:** $5,000$ to $20,000$
- **Layers $L$:** $45$ to $60+$
- **Experts per layer $E$:** Hundreds
- **Expert intermediate size $h_{ff}$:** Approximately $3$–$4$ times $d$

Memory usage per training run is decomposed as follows:
\[
\begin{align*}
M_{\text{params}} &= P \cdot S_p \\
M_{\text{optim}} &= k_{\mathrm{opt}} \cdot P \cdot S_p \\
M_{\text{activ}} &= B \cdot S \cdot d \cdot S_p \\
M_{\text{total}} &\approx (1 + k_{\mathrm{opt}}) P S_p + B S d S_p
\end{align*}
\]
where $S_p$ is bytes per parameter (2 for FP16, 4 for FP32), $k_{\text{opt}}$ is optimizer state multiplier (e.g. $2$ for Adam), $B$ is batch size, $S$ is sequence length.

## 3. Training Infrastructure and Accuracy Protocols

TeleChat3-MoE is trained on Ascend NPU clusters, leveraging both Atlas 900 and Ascend 910B hardware orchestrated via the MindSpore framework. Operator-level numerical verification is enforced by comparing to a golden CPU baseline in FP32/FP64. Tolerance $\Delta$ is set based on accumulation counts $N_{\text{acc}}$, with representative thresholds:
- $N_{\text{acc}} < 2{,}000$: $\Delta \approx 4 \times 10^{-3}$
- $N_{\text{acc}} < 1 \times 10^6$: $\Delta \approx 0.1$

Inputs are clipped to avoid division instability. End-to-end alignment involves systematic comparison of forward losses and backward gradients under cross-hardware and cross-parallelism settings, starting from single-device, tiny-model runs and scaling up; discrepancies are isolated by tensor dumping at specified checkpoints. Verification processes are outlined in Figure 1 and Table 3 of the source [2512.24157].

## 4. Parallelism and Work Scheduling Strategies

TeleChat3-MoE employs multi-dimensional parallelism configurations:
- **Data Parallelism (DP)**
- **Tensor Parallelism (TP)**
- **Pipeline Parallelism (PP)**
- **Sequence Parallelism (SP)**
- **Expert Parallelism (EP)**
- **Optimizer-state Parallelism (OP)**

Pipeline scheduling is optimized via interleaved layer allocation and a 1F1B execution schedule (one-forward/one-backward), which overlaps communication and computation. This arrangement yields a +10% throughput improvement over naive pipeline scheduling.

For very long sequences (up to 128,000 tokens), an attention-aware data scheduling approach sorts and shards samples by document length cost, balancing the sparse-attention computational load across devices.

Expert parallelism utilizes a hierarchical communication structure: inter-node AllGather, followed by local filtering and then intra-node All-to-All. This yields approximately 15% throughput gain at EP degree 16 compared to single global All-to-All. Communication overlap is achieved by partitioning batch and sequence dimensions to permit concurrent EP communication (RoCE AllGather + HCCS All-to-All) and FFN computation, reducing EP comm time from 30% to 5% of total.

Pipeline layout optimization is posed as an Integer Linear Programming (ILP) problem. Given a parallelism strategy (DP, TP, PP, EP, MB), ILP variables $x_{\text{layer, stage}}$ and recompute flags $r_i$ are selected to minimize:
$$T_{\text{total}} = T_{\text{compute}} + T_{\text{bubble}}(x) + T_{\text{comm}}(x)$$
subject to memory constraints $M_{\text{peak}}(x) \leq M_{\text{device}}^{\max}$. This reduces manual configuration from approximately 7 days to 0.5 days and is found to match or outperform expert-tuned baselines (39.97 ms vs. 40.08 ms step time on 4096 devices).

## 5. Performance Engineering and Optimizations

Hierarchical and overlapped expert parallelism communication provides up to 15% throughput improvement and cuts EP communication cost to 5% of total comm time.

DVM-based operator fusion consolidates "Cube" operators (GroupedMatMul) and "Vector" operators (such as Reshape, Cast) into fused composite operators. For instance, fusing a $[20 \times 5120 \times 3072]$ MatMul–Reshape–Cast sequence results in an 85% speedup.

Cluster-level optimizations include:
- **Host-bound regime:** CPU affinity and kernel isolation decrease per-node variance by 38% and increase mean throughput up to 10–15% for large configurations.
- **Device-bound regime:** Raising the firmware threshold for NPU idle mode increases throughput by 25–30% on 4096 NPUs.
- **IOMMU passthrough:** Enables additional 3–5% performance improvement through enhanced monitoring paths.

A summary of throughput improvements is given below:

| Optimization                     | Improvement          |
|----------------------------------|---------------------|
| Pipeline+1F1B                    | +10% throughput     |
| Hierarchical EP                  | +15% throughput     |
| EP overlap                       | EP comm 30%→5%      |
| Operator fusion                  | +85% (single-op)    |
| Cluster firmware tweaks          | +25–30% throughput  |
| Host isolation                   | +10–15% throughput  |
| IOMMU passthrough                | +3–5% throughput    |

## 6. Empirical Results and Scaling

TeleChat3-MoE demonstrates near-linear scaling from hundreds up to approximately 8192 NPUs, including training runs of over one trillion parameters. On a 4096-device run (438B model, $B_{\text{global}}=16,384$), example step times are:
- Expert-1: 40.08 ms
- Expert-2: 40.15 ms
- Tool-generated (ILP-optimized): 39.97 ms

End-to-end improvements against unoptimized baselines include:
- Pipeline+1F1B: +10% throughput
- Hierarchical EP: +15%
- EP overlap: EP comm reduced from 30% to 5% of total
- Operator fusion: +85% (single-op)
- Cluster firmware optimizations: +25–30%
- Host isolation: +10–15%
- IOMMU passthrough: +3–5%

These outcomes demonstrate the effectiveness of cohesive architectural, infrastructural, and parallelization design for MoE LLMs at trillion-parameter scale on specialized NPU clusters [2512.24157].

Source: https://www.emergentmind.com/topics/telechat3-moe