Youtu-LLM: Unlocking the Native Agentic Potential for Lightweight Large Language Models (2512.24618v1)

Abstract: We introduce Youtu-LLM, a lightweight yet powerful LLM that harmonizes high computational efficiency with native agentic intelligence. Unlike typical small models that rely on distillation, Youtu-LLM (1.96B) is pre-trained from scratch to systematically cultivate reasoning and planning capabilities. The key technical advancements are as follows: (1) Compact Architecture with Long-Context Support: Built on a dense Multi-Latent Attention (MLA) architecture with a novel STEM-oriented vocabulary, Youtu-LLM supports a 128k context window. This design enables robust long-context reasoning and state tracking within a minimal memory footprint, making it ideal for long-horizon agent and reasoning tasks. (2) Principled "Commonsense-STEM-Agent" Curriculum: We curated a massive corpus of approximately 11T tokens and implemented a multi-stage training strategy. By progressively shifting the pre-training data distribution from general commonsense to complex STEM and agentic tasks, we ensure the model acquires deep cognitive abilities rather than superficial alignment. (3) Scalable Agentic Mid-training: Specifically for the agentic mid-training, we employ diverse data construction schemes to synthesize rich and varied trajectories across math, coding, and tool-use domains. This high-quality data enables the model to internalize planning and reflection behaviors effectively. Extensive evaluations show that Youtu-LLM sets a new state-of-the-art for sub-2B LLMs. On general benchmarks, it achieves competitive performance against larger models, while on agent-specific tasks, it significantly surpasses existing SOTA baselines, demonstrating that lightweight models can possess strong intrinsic agentic capabilities.

• The paper introduces a novel 1.96B LLM pre-trained from scratch with built-in agentic signals to enhance long-horizon reasoning and planning.
• It leverages a dense Multi-Latent Attention mechanism and a STEM-optimized, multi-stage tokenizer to efficiently manage long-context sequences up to 128k tokens.
• Empirical evaluations on STEM, coding, and agentic benchmarks demonstrate competitive performance against larger models, underscoring practical scalability.

Unlocking Native Agentic Capabilities in Lightweight LLMs: An Expert Review of Youtu-LLM (2512.24618)

Introduction

Youtu-LLM is a 1.96B parameter LLM architected to provide intrinsic agentic capabilities in a sub-2B footprint. Diverging from prevailing approaches that rely predominantly on post-hoc distillation or instruction tuning, Youtu-LLM is explicitly pre-trained from scratch with agentic signals woven into its foundational learning stages. The model leverages a dense Multi-Latent Attention (MLA) architecture, a STEM-optimized, multi-stage tokenizer, and a robust long-context capacity of 128k tokens. The training corpus comprises approximately 11T tokens—curated to progressively transition from general commonsense, through STEM domains, to rich agentic data, effectively enabling long-horizon reasoning, robust planning, and adaptive interaction. This technical review synthesizes the architectural choices, data engineering, mid-training strategies, and empirical results, providing critical analysis of its implications for agentic AI research.

Architectural Innovations

Youtu-LLM’s architecture is rooted in dense MLA, which offers substantial memory and compute savings while preserving attention expressiveness in constrained parameter regimes. In contrast to Mixture-of-Experts (MoE) paradigms, which introduce IO overheads detrimental for on-device applications, MLA achieves high KV cache compression and leverages large projection matrices for enhanced attention (2512.24618). The model incorporates a STEM-centric tokenizer, built via an iterative, multi-stage process that demonstrates improved compression and coverage on scientific/coding tasks relative to standard baselines like Llama3 and Qwen3.

Figure 1: Parameter–performance scaling of base and instruct models on agentic benchmarks; Youtu-LLM achieves strong agentic performance at low parameter count.

Architectural configuration includes 32 layers, 2048 hidden size, tied embeddings, and a vocabulary of 128,256. Comparative scaling studies confirm that MLA surpasses GQA in accuracy across multi-lingual and STEM benchmarks for lightweight models.

Training Data Engineering

General Pre-Training Data

Corpus design prioritizes breadth, quality, and STEM-centric augmentation. The pipeline amalgamates 10T+ tokens from web, encyclopedic, STEM, and coding sources, meticulously filtered via quality scoring models and domain classifiers. Advanced deduplication and decontamination algorithms (e.g., MinHash-LSH, Aho-Corasick) are employed to mitigate test leakage and distribution shift.

Agentic Trajectory Data

200B+ tokens of trajectory data are constructed to systematically inject agentic behaviors into pre-training. Five principal categories are synthesized:

• Agentic-CoT: Raw chain-of-thought traces are decomposed into disciplined reasoning steps (analysis, plan, action, reflection, summary), minimizing redundancy and enhancing logical coherence.

  Figure 2: Agentic thinking paradigm enforces a stepwise reasoning architecture, mitigating overthinking and promoting strategic reflection.

• Math: Agentic mathematical trajectories are constructed via an atomic ability taxonomy, encapsulating skill decomposition, planning, execution, and self-reflection, fostering interpretable mathematical cognition.

  Figure 3: Modular math agent framework enabled for trajectory construction covering atomic abilities.

• Code: End-to-end agent traces for environment setup, issue fixing, and tool integration are generated and rigorously filtered. Trajectory branching utilizes success/failure annotation and controlled rollout, optimizing action diversity and robustness.

  Figure 4: Code trajectory synthesis pipeline–scaling tasks, contexts, and actions via branching strategies.

• Deep Research (DR): Closed-ended (multi-hop QA) and open-ended (report synthesis) trajectories are sourced and synthesized, including forward/inverse trajectory reconstruction from high-quality artifacts.

  Figure 5: Closed-ended DR trajectory synthesis; high-verifiability QA pipeline.

  

  Figure 6: Open-ended DR report generation with iterative research workflow.

  

  Figure 7: Inverse synthesis pipeline for open-ended DR; reference-based graph expansion.

• Tool-use/Planning: Multi-turn tool-use trajectories are formally constructed from atomic skill graphs and adversarial agent simulation. Negative augmentation and multi-level verification reinforce coverage and realism.

  Figure 8: Construction strategy for tool-use and planning data synthesizes complex multi-tool interaction.

Multi-Stage Pre-Training Paradigm

Youtu-LLM proceeds through four progressive stages:

1. Stage 1: Commonsense-focused training on 8.16T tokens, initializing basic world knowledge.
2. Stage 2: STEM/coding-rich batch (60% domain mix), leveraging targeted data for scientific/coding reasoning.
3. Stage 3: Long-context extension with learning rate decay and context length scaling, stabilizing complex reasoning patterns.
4. Stage 4: Agentic mid-training (60% agentic trajectories), further annealing learning rate and refining long-horizon planning/feedback behaviors.

   Figure 9: Pre-training data recipe and corresponding learning rate scheduler; increasing data quality correlates with staged progression.

Supervised Fine-Tuning and RL Alignment

Supervised fine-tuning is organized into a two-stage process: (I) reasoning SFT focusing on mathematics, code, scientific reasoning, and agentic tasks; (II) general SFT expanding task diversity while integrating “think” and “non-think” modes via control tokens.

Figure 10: Data engineering workflow for the high-quality SFT pipeline.

Reinforcement learning aligns outputs under heterogeneous reward signals, including mathematical correctness, code execution, instruction adherence, and robust safety modeling. Special attention is paid to on-policy stability, sampling-KL drift constraints, FP16 precision advantages over BF16, and repetitive pattern suppression.

Figure 11: Training dynamics with FP16 sampling; empirical accuracy validated on internal benchmarks.

Empirical Results and Agentic Benchmarking

General Capabilities

Across 16 major open benchmarks spanning commonsense, STEM, coding, and long-context reasoning, Youtu-LLM consistently outperforms likewise-scaled baselines (Qwen3-1.7B, SmolLM3-3B), frequently matching or exceeding larger models such as Qwen3-4B and Llama3-8B [Table: baseline_for_base, general_main_results_instruct]. Notable results include 77.6% on GSM8K (math), 64.6% on HumanEval (code), and competitive scores on MLQA-Zh and MMLU-ProX-Zh (multi-lingual, high-difficulty QA).

Agentic Benchmarks

Youtu-LLM demonstrates marked agentic superiority versus similarly-sized models on APTBench (code, DR, math, tool), and achieves competitive performance even against substantially larger models.

Figure 12: Agentic mid-training scaling yields significant APTBench gains with increasing trajectory token volume.

Ablation studies systematically decompose the impact of each trajectory type. Agentic-CoT trajectories provide maximal uplift in planning, mathematical trajectories enhance reasoning and execution, DR trajectories contribute to atomic citation grounding, and code/tool trajectories boost action reliability.

Figure 13: Pass@k performance on SWE-Bench-Verified with Openhands scaffold; agentic mid-training yields substantial performance gains.

Comprehensive agentic evaluation on benchmarks such as GAIA, xbench, SWE-Bench-Verified, and BFCL-V3 supports the conclusion that agentic mid-training is fundamental for native agentic intelligence in lightweight LLMs.

Implications and Future Directions

The findings indicate that systematic injection of agentic signals during pre-training leads to scalable, robust agentic capability, even in models constrained to 2B parameters—a result which directly challenges the prevailing assumption that agenticity is solely a function of scale or external augmentation. The techniques for trajectory synthesis, atomic ability decomposition, and multi-stage curriculum design offer replicable scaffolding for future agentic model construction.

Key stability observations—such as precision-type selection and sampling policy constraints in RL alignment—present critical guidance for applying deep pre-training to practical agentic deployments.

Nevertheless, bottlenecks remain in bridging the gap to massive proprietary LLMs, handling the curse of depth in trajectory-based reasoning (Sun et al., 9 Feb 2025), and extending agentic intelligence to multimodal domains, real-world tool ecosystems, and world modeling. Efficient architecture variants, e.g., Diffusion LLMs, and improved trajectory learning strategies are worthwhile research trajectories.

Conclusion

Youtu-LLM establishes that lightweight LLMs, when architected and trained with agentic principles and structured trajectory data, can achieve high intrinsic agentic intelligence, rivaling or surpassing much larger models on agentic tasks. The work demonstrates the operational feasibility of agentic mid-training and systematic trajectory injection at scale for lightweight models. The implications for system deployment, resource efficiency, and generalized agentic behavior are nontrivial. Going forward, augmentations across multimodal integration, execution simulation, and scalable architecture optimization are essential to further advance agentic AI grounded in compact LLMs.