JT-Math-8B: Math-Specialized 8B LLM Family

• The paper presents a capability-staged family of math-specialized language models, including base, instruct, and thinking variants, each optimized for distinct mathematical reasoning tasks.
• The paper details a three-stage continual pretraining pipeline on a 210B-token math corpus, enhancing both concise response generation and long-horizon deliberative problem solving.
• The paper employs dual post-training tracks with supervised fine-tuning and GRPO-based reinforcement learning to refine model performance on competition-level challenges.

Searching arXiv for the primary JT-Math-8B paper and closely related supporting work. arXiv query: JT-Math-8B / JT-Math mathematical reasoning 8B JT-Math-8B is a family of open-source 8B-parameter LLMs specialized for mathematical reasoning, comprising base, instruct, and thinking variants built through a systematic, multi-stage optimization framework. The family is presented as targeting two distinct operating regimes—direct concise mathematical response generation and long-horizon deliberative problem solving—while grounding both in a specialized continual pretraining process centered on a 210B-token mathematical corpus, staged supervised fine-tuning, and GRPO-based reinforcement learning (Hao et al., 26 Jul 2025).

1. Model family, scope, and architectural lineage

JT-Math-8B is defined as a series of three related models. JT-Math-8B-Base is the pretrained foundation model specialized for mathematics. JT-Math-8B-Instruct is a post-trained variant optimized for clear, concise, direct responses and shorter reasoning traces. JT-Math-8B-Thinking is a post-trained variant optimized for deep, long-chain mathematical reasoning, including long deliberative traces and extended-context problem solving (Hao et al., 26 Jul 2025).

The model family is initialized from JT-Coder-8B-Base. That upstream base is stated to use a Qwen2.5-compatible tokenizer and architecture, so the safe characterization is that JT-Math-8B is an 8B Qwen2.5-architecture-compatible math-specialized model family adapted from a coder-oriented predecessor rather than a model trained from scratch on a wholly new architecture (Hao et al., 26 Jul 2025). Base and instruct training commonly use 8,192 tokens of context, while the thinking pipeline extends to 32,768 tokens. In the long-context extension stage, the sequence length is increased to 32,768 with RoPE base $= 500{,}000$ (Hao et al., 26 Jul 2025).

The paper is explicit about several omissions. It does not specify the exact transformer depth, hidden size, number of heads, FFN size, attention variant, tokenizer vocabulary size, positional interpolation or rope-scaling formula, hardware stack, total FLOPs, or exact checkpointing and parallelism implementation. This makes the framework conceptually clear while leaving important low-level reproduction details unspecified (Hao et al., 26 Jul 2025).

2. Corpus construction and data validation

The pretraining substrate is the JT-Math Corpus, a 210B-token continual pretraining corpus intended to improve general mathematical understanding, mathematical reasoning, and long-context reasoning ability. Its top-level composition is given as 100B tokens of mathematical concept explanations, computational abilities, factual mathematical knowledge, and related papers, together with 110B tokens of mathematical understanding and reasoning, Chain-of-Thought, and Tool-integrated Reasoning. The broader mixture also includes code data across multiple programming paradigms and applications, selected via language-model filtering, as well as general data from high-quality web-scraped content, open-source books, specialized STEM texts, and industry application documents (Hao et al., 26 Jul 2025).

The data pipeline combines web-crawled mathematical data, open-source mathematical domain data such as exam questions, encyclopedias, reasoning data, and theorem-proving data, together with LLM-generated translations and synthetic data. A specific motivation is the asymmetry between English and Chinese math resources: the paper states that English open data is much richer than Chinese math data, so high-quality English datasets are translated into Chinese to create an initial Chinese mathematical dataset. Cleaning includes length-based filtering, specialized text processing, standardization for encoding consistency, and full-scale deduplication. For translated Chinese data, a three-step refinement process is applied: rule-based secondary filtering, cleaning, and LLM-based quality assessment (Hao et al., 26 Jul 2025).

A distinctive feature is the model-based validation loop built around a homologous JT-1.5B-base system. Mathematical data is categorized by language type, text data, instruction data, reasoning data, and proving data, and the proportion and total volume of each dimension are fixed during validation experiments. The process begins from a high-quality seed corpus formed from the top-scoring 20% under finemath-classifier. Candidate data is evaluated by replacing matched seed subsets, training a validation model with the same hyperparameters, and comparing downstream benchmark performance with the baseline model. If the validation model is better or equal on key metrics, the candidate data is accepted; otherwise, the scoring strategy is adjusted by changing classifier, using LLM scoring, using AutoDS, or raising score thresholds. If data remains unqualified after more than three rounds, it is discarded (Hao et al., 26 Jul 2025).

Long-context data is handled separately. In the final continual pretraining stage, the sequence length is extended from 8,192 to 32,768. Long-context mathematical data consists mainly of LLM-synthesized content and LLM-selected data. For data longer than 16,384 tokens, the paper states that multiple responses per question are generated using LLMs of varying sizes, and samples with pass rate in $[0.1, 0.9]$ are selected in order to exclude both trivial and nearly impossible long-context examples (Hao et al., 26 Jul 2025).

This emphasis on corpus fidelity and validation aligns with a broader data-centric line of math-LLM research. Independent work on Common Crawl-based mathematical corpora reports that preserving equations and code, normalizing mathematical notation, and filtering noisy pages can materially improve Nemotron-T 8B on MATH, MBPP+, MMLU, and MMLU-Stem, which suggests that JT-Math-8B’s attention to validation and curation is consistent with a wider empirical pattern at the 8B scale (Mahabadi et al., 20 Aug 2025).

3. Continual pretraining pipeline

JT-Math-8B uses a three-stage continual pretraining recipe. In Stage 1, labeled General Math Knowledge, training starts from JT-Coder-8B-Base and runs on 260B tokens with a data distribution of 70% JT-Math Corpus and 30% AlgebraicStack, code, and general knowledge. The sequence length is 8,192, the learning rate uses warmup for 2,000 steps and then remains constant at $3 \times 10^{-4}$, and the stated purpose is to build broad mathematical knowledge and foundational capability (Hao et al., 26 Jul 2025).

Stage 2, Reasoning and Thinking, changes the data mixture by using more instruction data that improved downstream performance under quality assessment and by including Long CoT data with sequence length 8,192. Optimization uses cosine scheduling with the learning rate decaying from $3 \times 10^{-4}$ to $3 \times 10^{-5}$. This stage is presented as improving reasoning and deep thinking behavior rather than merely increasing domain coverage (Hao et al., 26 Jul 2025).

Stage 3, Long Context Extension, shifts the model to long reasoning math and general data, with a long-context corpus in which 50% of text is over 8,192 tokens. Sequence length is extended to 32,768, RoPE base is set to 500,000, and the learning rate is $3 \times 10^{-5}$. The stated purpose is long-horizon reasoning and long dependency adaptation (Hao et al., 26 Jul 2025).

The structure of these stages is important because the paper does not treat mathematical reasoning as a monolithic capability. Instead, it organizes training as a progression from mathematical knowledge acquisition to reasoning-style adaptation and only then to long-context competence. A plausible implication is that the framework treats context extension as a semantic training problem, not only as a positional-encoding problem: long-context data and long-context optimization are introduced after the model has already been pushed toward mathematical reasoning behavior (Hao et al., 26 Jul 2025).

4. Supervised fine-tuning and reinforcement-learning specialization

Post-training begins with supervised fine-tuning and then diverges into separate instruct and thinking paths. The shared SFT curation pipeline consists of data quality assessment, targeted data synthesis, and rule-based plus model-based filtering. The paper states that broad initial collections are reduced into several hundred thousand high-quality samples, emphasizing “quality over quantity.” Short CoT sources include NuminaMath, JiuZhang, ScaleQuest, and AceMath; Long CoT sources include OpenThoughts3, AM-Thinking-v1, OpenMathReasoning, Mixture-of-Thoughts, and DeepMath (Hao et al., 26 Jul 2025).

To address difficult benchmarks such as AIME and OlympiadBench, the authors synthesize additional hard-problem data by generating new solutions using a stronger model and by rewriting and condensing DeepSeek-R1 responses into concise, high-quality Short CoT examples. Newly synthesized samples are filtered with math_verify after answer extraction. For Short CoT data, model-based filtering uses Qwen2.5-Math-RM-72B, retaining only the top 10% of samples with the threshold set at the 0.9 quantile. Because reward score negatively correlates with response length, filtering is performed within response-length bins of 128 tokens. The paper states that this reduces dataset size by 90% while improving performance (Hao et al., 26 Jul 2025).

The SFT recipes differ substantially across the two post-training branches. The instruct model is trained on Short CoT data with AdamW, learning rate $3 \times 10^{-6}$, batch size 128, cosine scheduler, no warmup, and context length 8,192. The thinking model is trained on Long CoT data with AdamW, learning rate $8 \times 10^{-5}$, batch size 128, cosine scheduler, no warmup, and context length 32,768. The paper specifically notes that the higher learning rate is important for shifting behavior toward long step-by-step reasoning patterns; lower learning rates can produce similar stylistic outputs but much lower accuracy (Hao et al., 26 Jul 2025).

Reinforcement learning is GRPO-based in both branches and is described as strictly on-policy. Fresh rollouts are generated for each training batch; this is said to improve gradient consistency and reduce bias from stale samples. The paper also mentions static difficulty-filtered data sampling with GRPO, reduced KL regularization to allow broader exploration of new solution strategies, and hyperparameter search over response length, rollout count, learning rate, and clip ratio, although the chosen clip-ratio value is not reported (Hao et al., 26 Jul 2025).

For the instruct-model RL phase, the reward is explicitly binary,

$$r = \begin{cases} +1, & \text{if the answer is correct},\ -1, & \text{if the answer is incorrect}. \end{cases}$$

Correctness is validated through symbolic validation, character-level validation, and math-verify validation. Before RL training, 16 rollouts per query are used to estimate model accuracy, and queries with estimated correctness rate exactly 0 or exactly 1 are removed as too difficult or too easy. The instruct RL configuration uses context length 8,192, KL coefficient $1 \times 10^{-3}$, learning rate $[0.1, 0.9]$0, 16 rollouts per query, batch size 256, and generation temperature 1.2 (Hao et al., 26 Jul 2025).

For the thinking-model RL phase, the paper introduces a multi-stage curriculum over context length. Training starts from a strong SFT model and proceeds through 8K RL, then 16K RL, then 32K RL, with each checkpoint initializing the next stage. The rationale given is efficiency and stability: beginning directly at 32K would be expensive and unstable. The long-CoT RL configuration uses KL coefficient $[0.1, 0.9]$1, learning rate $[0.1, 0.9]$2, 16 rollouts per query, batch size 32, and generation temperature 1.2. For the thinking-model filtering stage, the paper specifies that queries with estimated score exactly 1 are removed, while the treatment of exact-0 cases is not fully specified (Hao et al., 26 Jul 2025).

Notably, the paper does not provide the formal GRPO objective, policy loss, advantage estimator, clipping equation, or explicit KL-regularized objective. It also does not describe process rewards, step-level verifier rewards, or intermediate CoT supervision during RL. The RL account is therefore operationally detailed but not fully formalized (Hao et al., 26 Jul 2025).

5. Benchmarking, evaluation protocol, and reported performance

The evaluation suite includes GSM8K, MATH / MATH-500, AIME 2024, AIME 2025, OlympiadBench, AMC 2023, CNMO 2024, and CMath. The paper states that all pretraining and post-training data are decontaminated using 10-gram filtering, with any training instance containing a 10-gram present in key benchmark datasets removed (Hao et al., 26 Jul 2025).

The three variants are evaluated under different protocols. The base model uses few-shot CoT prompting on GSM8K, MATH, and CMath. Instruct models use greedy decoding, with max output length 8,192 for JT-Math-8B-Instruct, 16,384 for GPT-4o, and 4,096 for DeepSeek-Math-7B-Instruct and Qwen2.5-Math-7B-Instruct. Reasoning models use sampling; JT-Math-8B-Thinking is evaluated with max generation length 32,768, temperature 0.65, and metric average@8, while DeepSeek-R1-Distill-Qwen-7B uses temperature 0.6 and o1-mini-128k uses its default temperature (Hao et al., 26 Jul 2025).

Variant	Evaluation summary	Reported comparative result
JT-Math-8B-Base	GSM8K 87.5, MATH 60.1, CMath 90.1, average 79.2	Best average in its base-model table
JT-Math-8B-Instruct	MATH-500 90.00, AIME24 37.50, AIME25 34.17, CNMO24 69.44, average 63.74	Best average in its instruct-model table
JT-Math-8B-Thinking	MATH-500 93.80, AIME24 69.17, AIME25 58.75, CMath 93.99, average 77.68	Best average in its reasoning-model table

The base-model table positions JT-Math-8B-Base against Qwen2.5-Base-32B, Llama-3.1-Base-405B, DeepSeek-Math-Base-7B, DeepSeek-Coder-V2-Lite-Base, and Qwen2.5-Math-7B. JT-Math-8B-Base does not lead GSM8K, but it has the best MATH score among the listed models and the best CMath score, giving the best average score overall in that table. This indicates that the reported pretraining recipe strengthens formal mathematical reasoning and Chinese mathematical understanding more than grade-school arithmetic performance (Hao et al., 26 Jul 2025).

The instruct-model table compares JT-Math-8B-Instruct with DeepSeek-Math-7B-Instruct, Qwen2.5-Math-7B-Instruct, and GPT-4o. The reported averages are 21.19, 47.12, 39.62, and 63.74 respectively, with JT-Math-8B-Instruct leading on MATH-500, AIME24, AIME25, OlympiadBench, AMC23, and CNMO24, while Qwen2.5-Math-7B-Instruct has a slightly higher CMath score, 92.53 versus 91.44 (Hao et al., 26 Jul 2025).

The strongest empirical claim is attached to JT-Math-8B-Thinking. In the reasoning-model comparison, the reported averages are 69.61 for DeepSeek-R1-Distill-Qwen-7B, 69.40 for o1-mini-128k, and 77.68 for JT-Math-8B-Thinking. The paper reports that JT-Math-8B-Thinking leads across all listed benchmarks in its table, including MATH-500, AIME24, AIME25, OlympiadBench, AMC23, CNMO24, and CMath (Hao et al., 26 Jul 2025).

6. Interpretation, research context, and limitations

JT-Math-8B is best understood as a capability-staged math-model family rather than as a single checkpoint. The base model establishes mathematical priors through a specialized, quality-controlled corpus and staged continual pretraining; the instruct model specializes this base for concise instruction-following answers; and the thinking model further specializes it for long deliberative reasoning through Long CoT SFT and a multi-stage RL curriculum (Hao et al., 26 Jul 2025).

Several aspects of the paper indicate that data curation is treated as a first-class modeling variable. The iterative use of finemath-classifier, LLM-based evaluation, JT-1.5B-base validation models, top-20% seed selection, and rejection of data after more than three failed rounds of validation collectively place corpus construction near the center of the framework. Independent evidence from large-scale Common Crawl math corpus research strengthens the plausibility of that emphasis: a high-quality extraction-and-cleaning pipeline for mathematical web data has been shown to improve an 8B model by $[0.1, 0.9]$3 to $[0.1, 0.9]$4 on MATH and $[0.1, 0.9]$5 to $[0.1, 0.9]$6 on MBPP+ over strong baselines, while also improving MMLU and MMLU-Stem (Mahabadi et al., 20 Aug 2025). This suggests that JT-Math-8B’s validation-heavy corpus design is part of a broader empirical shift toward quality-preserving scale in mathematical pretraining.

The paper also frames the instruct and thinking variants as different deployment modes. JT-Math-8B-Instruct is associated with greedy decoding, concise answer generation, and fast inference on simpler tasks. JT-Math-8B-Thinking is associated with sampled decoding, long reasoning traces, average@8 evaluation, and higher suitability for difficult competition-style problems, with correspondingly greater latency and cost. The distinction is architectural only in a loose sense; the stronger separation is behavioral and training-regime-specific (Hao et al., 26 Jul 2025).

The limitations are explicit. Training is focused primarily on mathematical tasks, so general natural language understanding and broader task handling are described as comparatively limited. The paper also notes a data-scale gap versus frontier models trained on trillions of tokens. Additional practical caveats follow from the evaluation protocol and disclosure level: the strongest reported gains for the thinking model depend on long generation budgets and average@8 rather than single-sample low-latency inference, and many systems-level details—particularly the exact GRPO objective and infrastructure configuration—are omitted (Hao et al., 26 Jul 2025).

Within the contemporary landscape of open mathematical LLMs, JT-Math-8B therefore occupies a specific position: it is an 8B family built around staged capability acquisition, with strong reported performance on competition-level mathematics and multilingual mathematical benchmarks, and with its most distinctive technical claims concentrated in data validation, dual post-training paths, and curriculum-based long-context reasoning (Hao et al., 26 Jul 2025).