ZGCM-1: AI Model for Mathematical Reasoning
- ZGCM-1 is a 7.39B-parameter dense decoder-only Transformer designed for mathematical reasoning and agentic search, capable of handling up to 256K tokens and employing hybrid attention mechanisms.
- The model uses a 5:1 gated sliding-window/global attention architecture, FP8 computation, Muon optimization, and progressive context training, achieving significant memory efficiency with a 6.4× reduction in KV cache size for long contexts.
- ZGCM-1 has demonstrated strong performance in mathematical reasoning benchmarks such as MATH-500 (97.13%) and AIME 2026 (75.0%) but shows limitations in general software engineering and factual reliability tasks.
ZGCM-1 is a fully open, 7.39-billion-parameter dense decoder-only Transformer developed by Zhongguancun Academy and the Zhongguancun Institute of Artificial Intelligence. It is designed for mathematical reasoning and agentic search through the combination of long internal deliberation, external tool use, long-context processing up to 256K tokens, and efficient training and inference. The reported system couples a compact parametric model with web search, page inspection, terminal interaction, binary-analysis tools, hybrid sparse attention, FP8 computation, Muon optimization, progressive context training, and agentic supervision (He et al., 11 Sep 2026).
1. Design objectives and system profile
ZGCM-1 is based on the premise that compact models cannot passively memorize the open web because of limited static parametric capacity. Its proposed alternative is a dual-engine architecture in which deliberate internal thinking is combined with active external information seeking. The model can conduct extended chain-of-thought-style reasoning, query external tools, process contexts up to 256K tokens, and use environment feedback during multi-step interaction.
The model is a dense decoder-only Transformer with approximately 7.39B parameters, 32 Transformer layers, a hidden size of 4,096, a SwiGLU intermediate size of 11,008, 32 query heads, 8 key/value heads, and a head dimension of 128. It uses grouped-query attention, RMSNorm with QK normalization, SwiGLU feed-forward layers, partial RoPE with a rotary fraction of 0.33, and a hybrid local/global causal-attention schedule.
| Component | Configuration | Component | Configuration |
|---|---|---|---|
| Parameters | 7.39B | Layers | 32 |
| Hidden size | 4,096 | Intermediate size | 11,008 |
| Query heads | 32 | Key/value heads | 8 |
| Head dimension | 128 | Maximum context | 256K tokens |
| Normalization | RMSNorm | Activation | SwiGLU |
| Positional encoding | Partial RoPE | Rotary fraction | 0.33 |
The long-context configuration uses a RoPE base of 10M at 256K context, while the standard 16K and 64K configurations use a base of 5M. The released model is intended to operate in both think and no-think modes, allowing extended reasoning when computation is available and concise direct responses when latency or output length is constrained.
The reported evaluations include 75.0% on AIME 2026, 97.13% on MATH-500, 70.42% on HMMT 2025, 63.09% on WebWalkerQA, 19.43% on BrowseComp, and 62.0% on Binary Function Search. The paper also reports an approximately 4.2× improvement in 16K pre-training time-to-loss relative to an OLMo-3-style 7B BF16/AdamW baseline. These results are reported under the paper’s stated protocols and do not establish universal superiority over frontier models.
2. Hybrid attention and long-context operation
Of the 32 layers, 27 use gated sliding-window attention and 5 use full-context causal attention. The global layers are placed at layers 6, 12, 18, 24, and 30 using one-based indexing. This produces a 5 local layers : 1 global layer schedule, with two trailing local layers.
Each sliding-window layer attends only to a 128-token window, whereas global-attention layers attend over the entire available context. The design combines local computational efficiency with periodic global information propagation. A normalized hidden state is processed by a gated sliding-window module whose attention output is multiplied by a learned sigmoid gate before the output projection. The gate provides content-dependent control over local-attention contributions; global-attention layers do not use this local output gate.
The principal memory advantage arises from retaining full key/value histories only in the five global layers. The 27 sliding-window layers retain only 128 tokens. The reported per-token KV-cache footprint is 128 KiB for full attention and 20 KiB for ZGCM-1’s hybrid attention. At 256K context, the reported KV-cache requirements are:
| Architecture | Parameters | KV cache at 256K |
|---|---|---|
| Full attention, 32 global layers | 7.39B | 32.0 GiB |
| GDN linear-attention baseline | 7.47B | 7.0 GiB |
| ZGCM-1 hybrid SWA 5:1 | 7.39B | 5.0 GiB |
The hybrid architecture therefore reports a 6.4× reduction in KV-cache size relative to full attention. Its speedup over full attention increases with context length, from 1.13× at 4K to 3.94× at 256K. Because autoregressive decoding is often memory-bandwidth-bound, the reduced KV cache is particularly relevant to long reasoning and multi-step agentic search.
Architecture ablations trained configurations for 10B tokens at sequence length 4,096 on eight H100 GPUs. SWA 5:1 achieved a tail loss of 1.93 and throughput of 9,566 tokens/s/GPU. SWA 3:1 achieved a slightly lower tail loss of 1.92 but lower throughput. Full attention also achieved a tail loss of 1.93 with lower throughput, while MLA provided no corresponding loss improvement and achieved 7,645 tokens/s/GPU. The selected SWA 5:1 configuration therefore reflects a throughput–loss trade-off rather than a claim that it minimizes loss in isolation.
3. Pre-training, optimization, and computational efficiency
Training consists of general pre-training, long-context mid-training, and post-training. General pre-training contains approximately 4.19T tokens across two stages: approximately 0.99T tokens in Stage 1 and 3.20T tokens in Stage 2. Mid-training uses approximately 600.51B sampled tokens with a 16K → 64K → 256K context curriculum. Post-training combines supervised fine-tuning with think/no-think and agentic data, followed by mixed reinforcement learning using GRPO.
The corpus includes curated English web data, controlled Chinese web data, educational PDFs, OCR-derived scientific material, arXiv papers and LaTeX, open-source code, mathematics corpora, textbooks, and specialized reasoning data. Processing includes source-specific quality filtering, language identification, document normalization, repository and artifact cleaning, formula-preserving mathematical normalization, versioned shards and manifests, cross-stage deduplication, and GLM-5.1 tokenizer-based tokenization. Stage 2 increases the relative proportion of code, mathematics, and specialized reasoning while retaining broad academic and web coverage.
Matrix parameters use Muon, while scalar parameters use Adam. The reported Muon settings include momentum 0.9, five Newton–Schulz iterations, spectral scaling, learning rate , weight decay 0.1, and gradient clipping 1.0. Matrix multiplications use Transformer Engine hybrid FP8, with E4M3 for forward computation and E5M2 for backward computation. Delayed scaling updates scaling factors from the maximum absolute value over a 1,024-step history.
TWEO is used as an activation regularizer to suppress extreme intermediate values that could destabilize FP8 training. The paper identifies activation outlier suppression combined with delayed scaling as the basis for stable FP8 pre-training while retaining approximately 60% BF16-equivalent MFU. The 16K production run reports approximately 585 model TFLOP/s/GPU and approximately 60% BF16-equivalent MFU.
The main 16K parallelism configuration is , with global batch size 768 and micro-batch size 1. Long-context configurations use tensor, pipeline, context, and data parallelism adjusted to maintain approximately 12.6M tokens per optimizer step. At 256K context, direct training of 30B tokens achieved a final loss of 1.19, whereas staged training with 10B tokens at 64K followed by 20B tokens at 256K achieved 1.16.
The reported approximately 4.2× time-to-loss improvement is attributed to four estimated factors: 1.4× from SWA 5:1 throughput, 1.5× from FP8 and system configuration, 1.8× from Muon step-to-loss efficiency, and 1.1× from Pre-LN data efficiency. The combined estimate is a composite rather than a single fully isolated factorial experiment.
4. Curriculum, MDP mid-training, and agentic supervision
ZGCM-1 uses multiple forms of curriculum. Stage 1 applies lexical-complexity ordering to general-language text, while code and mathematics are not sorted by the same metric because lexical complexity poorly correlates with algorithmic or mathematical difficulty. In a 10B-token 7B probe, coding BPB improved from 1.99 with random ordering to 0.81 with curriculum ordering, and mathematics BPB improved from 0.97 to 0.94. General-domain BPB worsened by approximately 0.03–0.09.
Mid-training uses a deduplicated candidate pool of approximately 2.86T tokens and 5.3B records. The sampled 600.51B-token schedule allocates 180B tokens to the 16K stage, 240B to the 64K stage, and 180.51B to the 256K stage. Longer stages retain substantial shorter-context replay rather than replacing it. Code remains close to 20%, mathematics remains close to 20%, knowledge increases from 10.50% to 14.23%, agentic data increases from 1.50% to 3.30%, and pre-training replay decreases from 13.00% to 9.00%.
Interaction traces are reformulated as Markov Decision Processes. A state contains the task description, interaction history, and current environment feedback. An action can be a natural-language reasoning step, search call, page visit, terminal command, code edit, tool invocation, final answer, or termination action. The environment returns a new state containing the resulting tool observation and updated history.
This formulation supplements full-trajectory causal modeling with state-conditioned next-action supervision. The overall mid-training objective combines ordinary full-sequence causal language modeling with denser action-level examples. The approach is intended to reduce sparse supervision in which only an entire interaction trajectory receives a success or failure label.
The reasoning pool contains approximately 35M examples and 38B tokens. Rule-based cleaning retained 34.7M examples, while 201K valuable questions with poor original traces were reconstructed or rewritten by teacher models. Examples are categorized according to self-containment, evidence for the answer, reasoning coherence, parsing validity, duplication, leakage risk, and trainability.
The agentic corpus includes general interaction traces, software-engineering trajectories, tool calls, environment observations, planning, and iterative refinement. The final software-engineering mid-training corpus contains approximately 129K records and 5.9B tokens. Execution-grounded traces use actual test outcomes and interaction results, whereas execution-free traces are judged using structural, patch, shell/edit, log, and behavioral evidence.
5. Post-training, tool use, and empirical performance
The supervised fine-tuning corpus contains 4,921,933 examples, of which 96.46% are general data and 3.54% are agentic data. The data includes instruction following, knowledge, mathematics, science, code, dialogue, reasoning, deep research, software engineering, and terminal interaction. Preprocessing includes message and tool-schema normalization, structural-validity checks, correctness checks, tool-turn consistency checks, benchmark decontamination, packed sequences, and assistant-only loss masks.
A quality-focused filtering ablation reduced the dataset to approximately 1.145M examples and improved the six-benchmark mean to 68.83, compared with 67.78 for approximately 2.08M minimally processed examples. Improvements were especially concentrated in BBH, where the increase was 10.10 points. The paper interprets this result as evidence that data quality can matter more than raw SFT volume.
SFT deliberately combines think examples containing explicit reasoning trajectories with no-think examples containing concise direct responses. In an observational checkpoint comparison, mixed SFT improved AIME 2025 from 10.00 to 43.33, MBPP+ from 55.03 to 75.93, and BBH from 58.49 to 69.48, while MMLU declined from 72.63 to 70.03 and HumanEval+ from 73.78 to 66.46. The comparison is not a perfectly controlled causal ablation.
The released model uses a 256K SFT variant trained over 19.46B packed tokens. Post-training also includes mixed RL for mathematics, code, and general capabilities. Mathematical rewards use binary answer correctness, code rewards use the fraction of executable tests passed, and general-task rewards use task-specific correctness or instruction-following criteria. Invalid or truncated answers receive no positive reward. GRPO uses group-relative advantages, and groups with zero reward variance are dynamically replaced.
In web research, the model uses a thinking-enabled ReAct policy with at most 64 search-and-read steps. The tool stack consists of Serper search, Jina Search fallback, and Jina Reader page extraction. ZGCM-1 reports 63.09 on WebWalkerQA, 19.43 on BrowseComp, and 42.52 on GAIA text-only. It exceeds Qwen3-235B-A22B on WebWalkerQA and BrowseComp in the reported table but trails it on GAIA. Web scores depend on search date, provider, page availability, extraction quality, summarization, judge model, retry policy, and environment configuration.
Binary Function Search uses a Ghidra-backed environment. The model receives a behavioral description of a function in a stripped ELF binary and must return its exact function-entry ELF virtual address. Available tools include functions, function, decompile, search, refs, and listing. On a 50-task evaluation, ZGCM-1 solved 31 tasks for an accuracy of 62%, matching Kimi-K2 and exceeding Qwen3-8B’s 12%, while remaining below DeepSeek-V4 Flash and Qwen3.5-397B-A17B at 76%.
The reported general reasoning results include:
| Benchmark | ZGCM-1-7B |
|---|---|
| MATH-500 | 97.13% |
| AIME 2024 | 80.62% |
| AIME 2025 | 73.33% |
| AIME 2026 | 75.00% |
| HMMT 2025 | 70.42% |
| HumanEval+ | 90.24% |
| MBPP+ | 63.23% |
| LiveCodeBench v6 | 46.86% |
| MMLU | 73.88% |
| GPQA-Diamond | 47.87% |
| IFEval | 75.42% |
The model does not dominate every task. The paper reports weaker performance in some general-knowledge and code comparisons, and no-think evaluation demonstrates a trade-off between deliberate reasoning and efficient direct response.
6. AI-native development, limitations, and release
ZGCM-1 was developed through an AI-native R&D workflow in which agent swarms support data engineering, architecture design, learning-algorithm design, experimentation and monitoring, infrastructure engineering, evaluation, and deployment engineering. Agents were used to write data-cleaning scripts, inspect distributions, launch and diagnose jobs, adjust configurations, profile storage and data-transfer bottlenecks, run experiments, execute evaluation suites, and maintain deployment workflows.
The Atomic Capability Evaluation suite contains 18 categories, 183 atomic capabilities, and 2,503 probes. Its strongest reported categories for ZGCM-1 are mathematics at 98.18%, planning at 97.78%, logic at 96.08%, causal reasoning at 96.00%, and reading at 95.04%. The weakest are instruction following at 49.74%, truthfulness at 68.91%, code at 72.24%, and language at 79.75%. ACE is presented as a diagnostic tool rather than a comprehensive measure of end-to-end model quality.
The paper describes autonomy as task-dependent. Experimentation and monitoring and deployment engineering were assigned L4, architecture design and learning-algorithm design were assigned L2, and most remaining categories were assigned L3. Only one of 99 individual ratings was L5, and no task was assigned L5. Thus the reported workflow supports substantial operational autonomy but not autonomous scientific objective formation.
The principal limitations concern parametric knowledge, instruction following, general software and terminal agency, protocol brittleness, evaluation methodology, and safety. At 7.39B parameters, the model has substantially less static memory than very large models and is expected to trail frontier systems on closed-book factual recall without retrieval or tools. Long reasoning supervision can induce verbosity and harm strict instruction adherence. On SWE-bench Verified Mini50, the model reports 2/50, or 4.0%, and on Terminal-Bench 2.0, 2/89, or 2.25%.
Agent performance depends strongly on exact tool schemas, message roles, tool-call and observation pairing, environment feedback, search API availability, observation formatting, context limits, and timeout limits. Other reliability concerns include hallucinated search conclusions, overconfident tool use, incorrect binary or code analysis, unreliable terminal actions, incomplete uncertainty calibration, and risks from executing model-generated commands. The ACE truthfulness score of 68.91% indicates meaningful reliability limitations.
The project states that it releases pre-training, mid-training, and post-training weights; intermediate checkpoints; training code; configurations; per-stage data; data recipes; evaluation harnesses; W&B logs; and telemetry and experiment records. The cited resources are the code repository at https://github.com/zgcagi/ZGCM-1, the model repository at https://huggingface.co/zgcagi/ZGCM-1-7B, and the data repository at https://huggingface.co/datasets/zgcagi/ZGCM-1-Data.
ZGCM-1 is therefore best characterized as a demonstration of compact-model engineering based on the combination of a 5:1 gated sliding-window/global attention architecture, FP8 and TWEO training, Muon optimization, progressive long-context curriculum, MDP-style action supervision, quality-focused post-training, and external tool use. The evidence is strongest for architectural and systems efficiency, mathematical reasoning at 7B scale, and selected structured tool environments. It is weaker for unrestricted software engineering, terminal agency, factual reliability, and universal replacement of frontier-scale models.