Intern-S1-Pro: Trillion-Param MoE Model

• Intern-S1-Pro is a multimodal foundation model featuring a one-trillion-parameter Mixture-of-Experts architecture that integrates language, vision, and time-series data.
• It employs advanced training and inference engines (XTuner and LMDeploy) with FP8 mixed-precision techniques to ensure stable, large-scale reinforcement learning.
• The model achieves best-in-class performance across over 100 scientific and generalist benchmarks, driving innovation in multimodal and agentic reasoning.

Intern-S1-Pro is a scientific multimodal foundation model characterized by a one-trillion-parameter Mixture-of-Experts (MoE) architecture that integrates language, vision, and time-series modalities. It is explicitly designed for high-fidelity scientific reasoning as well as generalist intelligence, supporting over a hundred specialized benchmarks spanning chemistry, materials science, life sciences, earth sciences, and agentic multimodal tasks. Intern-S1-Pro advances the state of the art through innovations in extremely large-scale model and infrastructure design, enabling reinforcement learning and inference consistency at unprecedented scale (Zou et al., 26 Mar 2026).

1. Model Architecture and Parameter Scaling

Intern-S1-Pro extends the architecture of its predecessor, Intern-S1, by scaling its Mixture-of-Experts capacity to the trillion-parameter regime. The model backbone is based on the Transformer architecture, employing self-attention of the standard form: $$\mathrm{Attention}(X) = \mathrm{softmax}(QK^T/\sqrt{d})\,V$$ Non-MoE layers use standard feed-forward blocks, while MoE layers employ $E$ experts per layer, with Top-8 routing ($K=8$). Each token is routed to the top $K$ experts based on a learned router: $z = W_r x, \quad p = \mathrm{softmax}(z)$

$S = \mathrm{TopK}(p, K)$

$$\hat{y} = \sum_{i\in S} \left( \frac{p_i}{\sum_{j\in S}p_j} \right)\,E_i(x)$$

Routing uses a Straight-Through Estimator to propagate gradients to all experts and enable stable load balance. Experts are grouped (EP8), with each group supplying its top-1 expert per token for device-level balancing. The vision encoder is implemented as a native Vision Transformer, pre-trained contrastively on a corpus of ~300 million image–text pairs, and projected into a shared embedding space. Positional encoding employs a truncated Fourier series to jointly capture discrete and spectral positional information.

A time-series encoder with adaptive subsampling reduces input frame length to $O(10^3–10^4)$, with Transformer layers modeling dependencies across the sequence.

The total parameter count is approximately $1.0\times10^{12}$, achieved predominantly through expansion of MoE expert capacity. Training throughput is only moderately impacted ($\sim20\%$ reduction) due to group routing and expert parallelism. The model’s scaling strategy is informed by Kaplan et al.'s scaling laws, supporting >100 task domains without catastrophic interference.

2. Training and Inference Infrastructure: XTuner and LMDeploy

Trillion-scale model training with Reinforcement Learning from Human Feedback (RLHF) and policy gradient methods requires high aggregate throughput and strict numerical alignment between training and inference. Intern-S1-Pro employs two tightly integrated systems:

• XTuner is the training engine. Expert Parallelism (EP8) shards experts across 8 GPUs using grouped routing for balanced utilization. Mixed-precision Automatic Mixed Precision (AMP) is implemented for FP8 or BF16; custom kernel replacements for operations such as RMSNorm, router softmax, and embeddings ensure FP8/BF16 convergence without distributional drift between training and deployment. Rollout router replay records expert indices during policy rollouts and deterministically replays them during the backward pass for routing consistency.
• LMDeploy is the inference engine, mirroring XTuner’s quantization kernels (FP8), providing low-latency rollout inference. Object-based communication for routing traces removes HTTP overhead, and operator-by-operator numerical alignment keeps log-probability KL divergence below $10^{-3}$ between inference and training (after alignment steps).

Precision-consistency is ensured via FP8 quantization of expert linear layers, BF16 elsewhere, and FP32 at the language head. Kernel re-implementation for critical operations eliminates numerical mismatches arising in native frameworks.

3. Reinforcement Learning Algorithms and Loss Schemes

The model’s post-training optimization is based on a composite REINFORCE-style policy gradient objective with dual importance sampling and conservative masking: $$\mathcal{L}(\theta) = -\mathbb{E}_{x\sim D, \{y_i\}\sim\pi_{\theta_\text{rollout}}(\cdot|x)} \left[\frac{1}{G}\sum_{i=1}^G \frac{1}{|y_i|}\sum_{t=1}^{|y_i|} \mathrm{sg}(M(\rho_{i,t};\alpha,\beta)\cdot r_{i,t})\,\hat{A}_{i,t}\, \log\pi_\theta(y_{i,t}|x, y_{i,<t}) \right]$$ with $M(\rho;\alpha,\beta)$ masking high-discrepancy tokens and $\hat{A}_{i,t}$ a leave-one-out advantage baseline. The algorithm combines on-policy and off-policy corrections (IcePop), masking for stability, and end-to-end FP8 precision compatibility.

This scheme is implemented for all RL optimization steps at full trillion-parameter scale. FP8 mixed-precision training matches BF16 baselines within ±0.5 points on held-out validation during RL.

4. Empirical Performance across Scientific and Generalist Benchmarks

Intern-S1-Pro is evaluated on OpenCompass, VLMEvalKit, and AgentCompass using both “thinking” (sampling, $T=0.8$, $p=0.95$) and “non-thinking” (greedy) decoding. Evaluation covers scientific reasoning, multimodal perception, time-series inference, and agentic control. On SciReasoner, the model achieves 55.5% (versus 11.9% for Qwen3-235B and 13.6% for GPT-5.2), 52.7% on SFE, and 72.8% on MatBench. In bioinformatics (Biology-Instruction), it reaches 52.5%, outperforming prior models by wide margins.

General task benchmarks include MMMU-Pro (72.8%), MMLU-Pro (86.6%), and AIME-2025 (93.1%). On the SciTS time-series subset, F1 scores are substantially improved over prior art (e.g., 98.0 on ASU01, cf. 67.2 for GPT-4.1-mini).

A representative table of performance is as follows (excerpted):

Task	Intern-S1-Pro	Qwen3-235B	GPT-5.2	Gemini-3-Pro
SciReasoner	55.5	11.9	13.6	-
SFE	52.7	41.4	-	58.9
MatBench	72.8	49.7	-	-
MicroVQA	63.3	53.8	-	69.0

This suggests that the model delivers best-in-class performance for scientific tasks aggregated at the benchmark level, while retaining top-tier or near-best generalist capability.

5. Agentic, Multimodal, and Workflow Capabilities

Intern-S1-Pro demonstrates advanced agentic planning and multi-turn reasoning abilities in the context of tool use. Evaluations on AgentCompass verify capacity for web-based search (GAIA, 77.4%), dual-control simulation (τ²-Bench, 80.9%), and GUI grounding (ScreenSpot V2, 93.6%). Qualitative assessments include iterative experimental design, interpreting complex scientific figures, and generating executable scripts for multi-stage scientific workflows.

The model’s Fourier Position Encoding and native visual backbone allow for >10 point improvements in complex figure understanding (MicroVQA, MSEarth), indicating precise integration across vision, text, and quantitative data streams.

6. Scaling, Ablation Studies, and Limitations

Ablation studies in the paper establish that group-routing initialization (preserving Top-2 per group) is critical for stable expert allocation, avoiding up to 20-point performance drops observed with naïve Top-1..Top-8 assignment. The Straight-Through Estimator for routing expedites convergence and improves device utilization over standard Top-K routing.

Precision ablations demonstrate that the FP8 pipeline (for expert layers) preserves task accuracy relative to BF16.

Limitations include temporary initial expert homogenization (resolved with further training), significant infrastructure complexity (potentially impeding reproducibility), and open challenges in certain scientific frontiers (e.g., real-time control, high-throughput operations). Negative transfer between scientific and general data is mitigated through System-Prompt Isolation, though further refinement is possible.

A plausible implication is that this integration of MoE scaling, RL-facilitating infrastructure, and multimodal agentic paradigms will drive subsequent advances in both general and scientific foundation models. The architecture provides a reference point for combining broad scientific coverage with best-in-class general intelligence in the open domain (Zou et al., 26 Mar 2026).