Training Report of TeleChat3-MoE

Abstract: TeleChat3-MoE is the latest series of TeleChat LLMs, featuring a Mixture-of-Experts (MoE) architecture with parameter counts ranging from 105 billion to over one trillion,trained end-to-end on Ascend NPU cluster. This technical report mainly presents the underlying training infrastructure that enables reliable and efficient scaling to frontier model sizes. We detail systematic methodologies for operator-level and end-to-end numerical accuracy verification, ensuring consistency across hardware platforms and distributed parallelism strategies. Furthermore, we introduce a suite of performance optimizations, including interleaved pipeline scheduling, attention-aware data scheduling for long-sequence training,hierarchical and overlapped communication for expert parallelism, and DVM-based operator fusion. A systematic parallelization framework, leveraging analytical estimation and integer linear programming, is also proposed to optimize multi-dimensional parallelism configurations. Additionally, we present methodological approaches to cluster-level optimizations, addressing host- and device-bound bottlenecks during large-scale training tasks. These infrastructure advancements yield significant throughput improvements and near-linear scaling on clusters comprising thousands of devices, providing a robust foundation for large-scale LLM development on hardware ecosystems.

• The paper presents a robust full-stack training infrastructure for trillion-scale MoE models, emphasizing aggressive sparsity and hardware-aware optimizations.
• The methodology incorporates interleaved pipeline scheduling, attention-aware data management, and hierarchical expert parallelism for superior throughput.
• Systematic numerical accuracy validation and automatic ILP-based parallelism ensure reproducible, efficient scaling across thousands of devices.

TeleChat3-MoE: Training Infrastructure and Optimization at Trillion-Scale

Model Design and Architectural Refinements

The TeleChat3-MoE model series encompasses parameter counts from 105B to over 1T, leveraging a Mixture-of-Experts (MoE) architecture specifically tuned for hardware affinity and distributed scaling. Notable architectural features include Multi-Latent Attention (MLA), which compresses KV vectors into lower-dimensional latent representations, substantially reducing KV cache footprint and elevating compute-to-memory access ratios for long-context inference. The adoption of a shallow-and-wide topology—with fewer layers but expanded hidden dimensions—further boosts per-layer arithmetic intensity and mitigates pipeline bubble overhead during distributed training. Aggressive MoE sparsity (top-4 to top-8 expert routing with one shared expert) enables low activation ratios, reducing memory footprint and maximizing device concurrency, especially in inference scenarios. These choices support efficient scaling from modest clusters for the 105B model up to 8192 devices for trillion-parameter variants.

Systematic Numerical Accuracy Verification

A comprehensive framework for numerical accuracy verification is implemented, starting with operator-wise checks. Systematic baselining utilizes high-precision CPU reference implementations, with strict tolerance thresholds dynamically set against the accumulation count (e.g., 0.0001 for float32 under 2,000 accumulations, 0.004 for float16). Inputs are rigorously clipped to avoid extreme values that would trigger error amplification in division-heavy operators, ensuring that operator-level discrepancies are tightly localized and controlled.

End-to-end model accuracy alignment is performed during cross-hardware migration and parallelism strategy transitions, employing progressive scaling verification and layer-wise tensor dumps. This workflow efficiently isolates divergence in loss or gradient trajectories attributable to back-end-specific handling (e.g., optimizer implementation inconsistencies or framework-specific randomness). Precise numerical equivalence is validated across parallelism strategies (DP, TP, PP, SP, EP), with gating clusters used to systematically compare activations and optimizer states, ensuring reproducibility and training reliability at scale.

Training Framework Performance Optimization

Extensive optimizations target the MindSpore training framework for maximal throughput and utilization:

• Interleaved Pipeline Scheduling & 1F1B Overlap: Moving beyond contiguous layer assignment, pipeline interleaving distributes layers in a non-contiguous fashion, minimizing pipeline bubble overhead and enabling computation-communication overlap. The 1F1B strategy, overlapping forward and backward micro-batches, improves E2E training throughput by ~10% over conventional pipeline approaches.
• Attention-Aware Data Scheduling: For long sequences (128K tokens), an attention-aware micro-batch scheduler balances computation by redistributing samples based on sub-sequence length, mitigating device idle time and improving sparse attention efficiency.
• Hierarchical Expert Parallelism Communication: Hierarchical AllGather followed by local filtering and intra-node All-to-All significantly reduces redundant communication in MoE training, achieving ~15% higher throughput under typical EP degrees.
• Communication Overlapping for Expert Parallelism: Multi-dimensional data partitioning enables overlapping between EP communication and computation, reducing EP comm time from 30% to 5% of total comm overhead.
• DVM-Based Operator Fusion: Automated fusion of memory-bound and compute-bound operators (Vector and Cube class) enhances compute unit utilization and reduces kernel launches, yielding up to 85% speedup on large fused operator sequences.

Systematic Parallelization Framework

The manual exploration of multi-dimensional parallelism (DP, TP, PP, VPP, SP, EP, OP) is replaced by an analytic toolchain incorporating integer linear programming (ILP). This framework parses model configs, generates and ranks candidate strategies with symbolic memory and performance estimation, then applies ILP to optimize pipeline stage assignment, interleaving, and recomputation under memory constraints. Tuning duration is reduced from seven days to 0.5 days, with throughput on 4096 NPUs matching or exceeding manual expert-designed baselines. The tool enables broader and deeper search of the parallelism configuration space, resulting in more balanced and memory-efficient schedules.

Cluster-Level Engineering and Firmware Optimization

Cluster-level optimizations address both host- and device-bound bottlenecks:

• Host-Bound Issues: Through resource isolation (CPU affinity, partitioning, kernel-level control), fluctuations induced by monitoring and process contention are suppressed—variance reduced by 38%, throughput gains of up to 15% on large clusters.
• Device-Bound Issues: Firmware-level tuning circumvents the Ascend NPU’s mis-triggered idle frequency scaling for short-duration operators, restoring chip clock under active workloads and delivering 25–30% throughput improvements.
• Monitoring-Induced Overheads: Transition to passthrough IOMMU configs eliminates query-induced host-device latency spikes, yielding an additional 3–5% throughput gain.

This methodological approach ensures reliable scaling and consistent performance across diverse hardware ecosystems.

Empirical Results and Contradictory Claims

The infrastructure delivers near-linear scaling across thousands of devices for both moderate (105B) and extreme (1T+) parameter models. Strong numerical results include:

• 85% operator-level speedup from DVM-based fusion on GroupedMatMul-Reshape-Cast sequences
• EP communication time reduction from 30% to 5% of total comm under overlapping
• Sustainable training throughput improvements of 10–30% after cluster-level optimizations
• Automated parallelization achieving step times indistinguishable from or better than manual baseline (39,969 ms vs. 40,076/40,147 ms on 4096 devices)

The TeleChat3-MoE design demonstrates that aggressive MoE sparsity, combined with infrastructural sophistication, can outperform lower-sparsity MoE models on end-to-end throughput and utilization—even at extreme parameter scales. This directly contradicts traditional assumptions that only moderate-sparsity regimes are feasible at such scales.

Implications and Future Directions

The work establishes a robust training solution tailored to large NPU clusters, advancing infrastructure reproducibility, engineering efficiency, and utilization. Open sourcing both models and infrastructure positions the community for rapid scaling and benchmarking beyond the current trillion-parameter frontier. The adoption of automatic parallelization tooling and systematic accuracy verification is likely to become standard practice in the development of even larger MoE-based architectures.

Practical implications include reduction of tuning effort, improvement of cluster stability, and enablement of stable training runs at record-breaking scales. For theory, the findings suggest new opportunities in sparse expert model optimization, distributed communication design, and hardware-aware transformer architectures.

Future research may further exploit the synergies of even higher sparsity MoE, longer context attention mechanisms, adaptive operator fusion schemes, and real-time firmware controls. These directions point toward robust, generalizable, and energy-efficient LLMs deployable on heterogeneous clusters worldwide.

Conclusion

TeleChat3-MoE’s training infrastructure demonstrates a mature, full-stack approach to scaling LLMs with MoE at unprecedented parameter counts. Through rigorous accuracy validation, advanced training framework optimizations, systematic parallelism strategy generation, and methodical cluster engineering, the work delivers a reproducible and efficient foundation for future large-scale LLM research. The open-sourced release invites further exploration and accelerated progress in both practical deployment and theoretical understanding of distributed MoE architectures.

Reference: "Training Report of TeleChat3-MoE" (2512.24157)