TeleChat2.5: Fast Domain-Specialized Transformer

• TeleChat2.5 is a dense Transformer model that advances rapid inference using continual pretraining on domain-specific math and coding datasets.
• It integrates innovations like RMSNorm, SwiGLU, RoPE, and GQA to enhance training stability and efficiency across 35B and 115B parameter scales.
• Its multi-stage training combining supervised fine-tuning and reinforcement learning delivers state-of-the-art performance in latency-sensitive applications.

TeleChat2.5 is a high-parameter, dense Transformer-based LLM within the TeleChat series, distinguished by its emphasis on rapid inference and improved performance in mathematical and coding tasks. Evolving directly from TeleChat2, TeleChat2.5 achieves substantial performance enhancements not through sweeping architectural revisions but through strategic innovations in the training pipeline—most notably, a continual pretraining stage on domain-specialized datasets and a robust reinforcement learning regime tailored to code and mathematical reasoning. The model is publically released in multiple parameter scales (35B and 115B), with the flagship 115B configuration designed to deliver state-of-the-art performance in reasoning-dense and latency-sensitive applications (Wang et al., 24 Jul 2025).

1. Architectural Features and Innovations

TeleChat2.5 is based on a dense Transformer architecture, closely related to TeleChat2 and the T1 variant. The primary architecture leverages several incremental improvements designed for stability, efficiency, and long-context reasoning:

• Pre-Norm Transformer with RMSNorm: Employs Root Mean Square Layer Normalization (RMSNorm) for enhanced training stability.
• SwiGLU Activation: Utilizes the SwiGLU activation function, following advances in Gated Linear Unit (GLU) variants.
• Rotary Positional Embeddings (RoPE): Incorporates RoPE, with an increased base frequency, which facilitates the processing of extended context windows (up to 256K tokens with specialized schedules).
• Grouped Query Attention (GQA): For the 115B parameter variant, GQA replaces standard multi-head attention, yielding both training acceleration and improved inference-time efficiency, especially for key-value cache management.

These modifications, though modest relative to wholesale architectural overhauls, are critical for scalability and efficient deployment in large-parameter regimes.

2. Training Pipeline and Optimization Strategies

TeleChat2.5 follows a multi-stage training protocol combining massive-scale pretraining with nuanced post-training procedures:

Pretraining:

• Trained on a corpus comprising up to 10 trillion tokens, meticulously filtered for high quality and diversity.
• Curriculum learning is applied, with gradual extension of context windows (the "long-context annealing" stage), allowing the model to process and utilize very long-range dependencies.

Continual Pretraining:

• Distinguishes TeleChat2.5 from its predecessors by introducing continual pretraining on domain-specific datasets immediately following generic corpus pretraining. This data upsampling process preferentially exposes the model to mathematical problems, coding scenarios, and specialized technical domains.

Post-Training:

• Supervised Fine-Tuning (SFT): Instructional data spanning a breadth of tasks (math, programming, reasoning) is used. Data mixing ratios are iteratively adjusted using the update rule

$$r_i^{(t+1)} = r_i^{(t)} \cdot \kappa^{(s_i^{(t)} - \overline{s}^{(t)})/\mu}$$

where $r_i^{(t)}$ is the sampling ratio for subset $i$ at stage $t$, $s_i^{(t)}$ is the step of minimum validation perplexity, $\overline{s}^{(t)}$ is the weighted average, and $\kappa$, $\mu$ are tuned per dataset.

• Direct Preference Optimization (DPO): Post-SFT, the model is aligned with human preferences using paired datasets that distinguish model outputs via preference margin scoring.
• Reinforcement Learning (RL): Finally, RL is deployed to optimize Math and Code outputs. For mathematics, the RL reward function is based on correctness (using automated solvers such as math_equal). For coding, the RL regime involves test case verification in a sandboxed environment, with explicit negative rewards for format inconsistencies or tool-call errors.

This staged optimization pipeline enables TeleChat2.5 to achieve marked gains in both speed and accuracy, particularly in specialized domains.

3. Domain-Specific Data Sources

TeleChat2.5 is continually pretrained and post-trained with high-quality, domain-specific content. Key sources include:

• Mathematics: Datasets such as OpenR1-Math-220k and synthetic K-12/competition-level math problems, subjected to filtering and correctness validation through automated solution checkers.
• Programming: Realistic code examples with execution feedback and unit tests, ensuring coverage of both natural language programming instructions and code outputs.
• Specialized Texts: Finance, healthcare, and technical instruction datasets, selected to foster robust LLMing in enterprise, scientific, and technical domains.

The upsampling of these domains in the continual pretraining stage enables model alignment with the syntactic and semantic patterns found in real technical workflows.

4. Benchmarks, Performance, and Trade-offs

Quantitative evaluation is performed across a range of established benchmarks:

• Mathematics (e.g., MATH, GSM8K): TeleChat2.5 demonstrates improved accuracy in non-chain-of-thought ("non-thinking") settings, outperforming models such as OpenAI’s o1-mini and GPT-4o by several percentage points.
• Code Generation (HumanEval, MBPP): Reinforcement learning and data upsampling facilitate both higher success rates on code generation tasks and improved formatting consistency.
• Inference Efficiency: The model is explicitly optimized for rapid inference ("non-thinking" mode), eschewing step-by-step reasoning traces (as present in chain-of-thought models) in favor of minimal-latency response.

Comparison Table: TeleChat2.5 vs T1

Model	Chain-of-Thought	Inference Speed	Domain Strengths
TeleChat2.5	No (“non-thinking”)	High (rapid, low-latency)	Code, Math (fast)
T1	Yes (“thinking”)	Lower (high-latency)	Chain-of-Thought, Math

This design trade-off makes TeleChat2.5 suitable for applications where rapid response and accuracy on specialized domains are critical.

5. Applications and Deployment Contexts

TeleChat2.5’s performance and design profile make it apt for a spectrum of real-world deployments:

• Real-time Digital Assistants: The low-latency, instruction-following ability is suited for interactive chatbot systems.
• Code Generation and Debugging Tools: Fast, accurate code outputs, validated through RL, aid in development and educational scenarios.
• Mathematical Tutors and Research Assistants: The blend of precise mathematical modeling and rapid turnaround supports advanced tutoring and applied mathematical research.
• Enterprise Document Understanding: The extended context capacities (256K tokens) enable analysis of long-form documents common in legal, financial, and scientific practice.

A plausible implication is that TeleChat2.5, by focusing on rapid inference without substantive chain-of-thought traces, is particularly valuable in situations where user experience and throughput are paramount, whereas its sibling T1 may suit use cases prioritizing deep, stepwise reasoning.

6. Extended Functionality and Interoperability

TeleChat2.5 is made available in both 35B and 115B parameter scales, supporting a breadth of research and development applications. The architectural compatibility with open standards (e.g., AngularJS component integration in earlier communication tools) supports scalable deployment and ease of integration with data pipelines and third-party interfaces.

This interoperability suggests a path for further incorporation of conversational analytics tools (such as those derived from the Chat-Bot-Kit paradigm (Sugisaki, 2019)), conversational mode customization, and real-time analytics for deeper system evaluation.

7. Position within the TeleChat Model Family

TeleChat2.5 occupies a position within the TeleChat ecosystem as a model optimizing for speed and accuracy in domain specialization, differentiated from T1, which targets chain-of-thought depth and long-form reasoning (Wang et al., 24 Jul 2025). Both model families demonstrate notable gains over their predecessor (TeleChat), with T1-115B reportedly surpassing proprietary models like o1-mini and GPT-4o. TeleChat2.5 is thus positioned as the model of choice for latency-sensitive, high-accuracy, domain-specific NLP applications within corporate, scientific, and educational environments.