---
title: 'Ling-1T: Trillion-Param MoE LLM & Ling Adder'
url: https://www.emergentmind.com/topics/ling-1t
type: topic
---

# Ling-1T: Trillion-Param MoE LLM & Ling Adder

Ling-1T denotes two distinct lines of high-efficiency, large-scale AI systems: (1) Mixture-of-Experts (MoE) Large Language Models (LLMs) exemplified by Ling-Lite, Ling-Plus, and Ling-1T, and (2) circuit-level binary adders, specifically the high-speed “Ling Adder.” Both branches share a focus on minimizing latency and maximizing computational efficiency through architectural innovation; however, their domains and technical mechanisms are entirely orthogonal—LLMs vs. hardware adders.

## 1. Mixture-of-Experts Ling-1T Language Models: Definition and Architecture

Ling-1T, as introduced in "Every Activation Boosted: Scaling General Reasoner to 1 Trillion Open Language Foundation" [2510.22115], is a trillion-parameter reasoning LLM built on the Mixture-of-Experts (MoE) paradigm. The core architectural components are:

- **Experts per Layer:** 256 routed experts per layer; only 8 (plus 1 shared expert) are active per token, yielding a ~3.5% activation ratio.
- **Activated Parameters:** Of 1 trillion parameters, only 51B are used per inference token, enabling very high compute sparsity.
- **Layer Configuration:** Initial 4 dense layers for stability and improved routing, followed by MoE layers.
- **Attention Block:** Grouped-query attention (GQA), partial rotary position embeddings (first 64 dims), SwiGLU activations, pre-layer RMSNorm, QKNorm normalizations.
- **Tokenization:** 156K BBPE byte-level vocabulary supporting extensive multilingual alignment.

MoE operation is expressed as:

\[
\mathbf{p}_t = \mathrm{Softmax}(\mathrm{R}(\mathbf{h}_t)),
\qquad
\mathbf{o}_t = \sum_{i \in \mathrm{Topk}(\mathbf{p}_t)} \mathbf{p}_{t,i}\mathrm{E}_i(\mathbf{h}_t)
\]

Ling models (Ling-Lite, Ling-Plus) scale from 16.8B to 290B parameters, retaining strong efficiency and competitive accuracy by sparse routing and fine-grained expert specialization [2503.05139].

## 2. Efficiency, Scaling Laws, and Training Pipeline

### Sparsity-Driven Efficiency Leverage

The Ling-1T model empirically achieves >7× active-compute efficiency compared to dense architectures. The efficiency leverage (EL) scaling law for MoE is formalized as:

\[
\text{EL}(A, G, C) = \hat A^{\alpha + \gamma(\log G)^2 + \beta \log G}
\]

where \(A\) is the activation ratio, \(G\) is expert granularity, and \(C\) is the compute budget in FLOPs. With Ling-1T’s (A ≈ 3.5%, G=8), this predicts the empirically validated efficiency advantage.

### Data Mixtures and Reasoning Optimization

- **Pre-training Data:** Heavy infusion of task-specialized mathematical and code reasoning data (Ling Math, Ling Code datasets), which increase from 32% to 46% of the corpus during training.
- **Multi-stage Curriculum:** General pre-training (20T tokens), then mid-training with extended contexts (up to 128K tokens) and explicit Chain-of-Thought (CoT) samples.
- **MTP (Multi-Token Prediction):** Auxiliary head/loss for predicting multiple tokens, boosting reasoning accuracy (loss weight 0.1).
- **DFT/Evo-CoT:** Decoupled fine-tuning and evolutionary RL (Evo-CoT), dual-mode supervised fine-tuning for instant responses and in-depth reasoning.

- **FP8 Deep Quantization:** All activations and gradients in FP8 (per-channel statistics), trading a <0.25% loss in accuracy for a 15% gain in hardware throughput and reduced memory.

## 3. System and Infrastructure Co-Design

### Distributed Training and Elastic Resource Utilization

- **EDiT:** Elastic distributed local SGD with time-based synchronization and gradient penalty to eliminate stragglers and anomalies, delivering up to 66.1% speedup over standard synchronous training [2503.05139].
- **Custom File Caching (PCache):** All-flash distributed caching, user-space FUSE, distributed checkpoint writing; linear scaling of throughput with accelerator count.
- **Cross-Cluster Sync (Babel):** Aggressive metadata prefetching and sampled CRC verification, reducing large-scale initialization overheads (e.g., 190M files from >6 hours to ≈10 minutes).
- **XPUTimer:** Ultra-light runtime tracer for anomaly/bottleneck detection, O(1) memory per step.

### Knowledge Graph Data for Tool Use

- **Synthetic Data:** 14 knowledge graph subgraph patterns, first-order logic expansions, >30K instruction templates including real and synthetic API tasks.
- **Tool Use Training:** Reasoned chaining, multi-hop tool selection, argument generation in realistic agent scenarios; Ling-Plus achieves benchmark-leading scores on function-calling, chaining, and external API benchmarks.

## 4. Empirical Results and Benchmarks

Ling-1T attains state-of-the-art accuracy per FLOP across math, code, logic, and knowledge benchmarks:

| Model         | MATH | HumanEval | ToolBench | Long-Context Retrieval |
|---------------|------|-----------|-----------|-----------------------|
| Ling-Lite     | 73   | 83        | Best/Near | 64K tokens            |
| Ling-Plus     | 79   | Best      | Best      | 64K tokens            |
| Ling-1T       | SOTA | SOTA      | SOTA      | 128K tokens           |

- **Reasoning:** Leaderboards on MATH, CollegeMath, MinervaMath, HumanEval, MultiPL-E, OptiBench, AIME24/25, ARC-e/c.
- **Efficiency:** Matches or exceeds models with equivalent activated FLOPs that require dense 1T-parameter computation.
- **Safety:** Balanced helpfulness/harmlessness; competitive scores in refusal and safety tuning.

## 5. Hardware and Cost Implications

- **Commodity Accelerators:** Designed and validated for training on hardware <120–370 TFLOPS, 64–96GB RAM, rather than premium H100/H800; democratizing trillion-parameter scaling.
- **Cost Savings:** Training Ling-Plus (290B) on lower-spec hardware reduces costs by ≈20% (e.g., from 6.35M RMB to 5.08M RMB per 1T tokens).

## 6. The Ling-1T Adder: Hardware Perspective

In computer arithmetic, Ling-1T also refers to a hardware architecture for binary addition—specifically, the high-speed binary Ling adder [1908.09297]. Principal features:

- **Carry Computation:** Employs adjacent bits (\(g_i, p_i, d_i\)), where \(g_i = a_i \cdot b_i\), \(p_i = a_i + b_i\), \(d_i = a_i \oplus b_i\). Ling adder introduces “half-sum” bits for logic minimization.
- **Logic Depth:** Reduces logic levels for 4-bit addition (Ling: 4; CLA: 5), improving propagation speed.
- **Ripple Reduction:** Minimizes dependency on previous carries, thus lowering cumulative delay for wide adders.
- **Complexity:** Circuit complexity grows for large n; practical for moderate-bit-width, less favorable for very high-bit VLSI where other adders may dominate.

## 7. Summary Table: Ling-1T Key Specifications

| Attribute                 | Value                         |
|---------------------------|------------------------------|
| Total Params (LLM)        | 1T                           |
| Active per Forward        | 51B                          |
| MoE Experts per Layer     | 256 (8+1 routed)             |
| Efficiency Leverage       | >7× dense                    |
| Precision                 | FP8 end-to-end               |
| Context Window (max)      | 128K tokens                  |
| Hardware Cost Advantage   | ≈20% savings (sub-premium)   |
| Benchmark Leadership      | Math, Code, Logic, Tool Use  |
| Ling Adder Logic Levels   | 4 (vs. 5 for CLA)            |

---

## 8. Impact, Implications, and Future Directions

Ling-1T (LLM) establishes a new Pareto frontier, representing the highest known reasoning accuracy per computational cost at the trillion-parameter scale. The open foundation, co-designed scalable architecture, and tooling for democratized training underpin a robust blueprint for next-generation “thinking” AI (e.g., future Ring series).

The Ling adder, by contrast, contributes to hardware arithmetic design, particularly in contexts demanding minimized addition latency and where gate-count increases can be accepted for speed.

Both lines demonstrate scalable efficiency gains by architectural, routing, and data innovation—whether in digital logic or language model design. The Ling-1T family serves as a reproducible base for high-efficiency language reasoning and for high-speed hardware addition circuits.

Source: https://www.emergentmind.com/topics/ling-1t