InfiAlign: Efficient LLM Alignment
- The paper presents a novel framework that aligns LLMs for stronger reasoning while using just 12% of conventional alignment data.
- It integrates supervised fine-tuning with Direct Preference Optimization through a robust, multidimensional data selection pipeline focusing on diversity, difficulty, and quality.
- Evaluations reveal significant improvements on mathematical benchmarks, demonstrating efficient data usage and targeted preference alignment.
InfiAlign is a post-training framework for aligning LLMs toward stronger reasoning with reduced data and computational cost. It integrates supervised fine-tuning (SFT) with Direct Preference Optimization (DPO), and places a robust data selection pipeline at the center of the alignment process: open-source reasoning corpora are filtered through multidimensional metrics for diversity, difficulty, and quality before being used for post-training. Applied to Qwen2.5-Math-7B-Base, the framework yields models that match or surpass DeepSeek-R1-Distill-Qwen-7B while using about 12% of that model’s alignment data, and the DPO stage adds an average improvement of 3.89% on AIME 24/25 benchmarks (Cai et al., 7 Aug 2025).
1. Problem setting and design objective
InfiAlign addresses a specific inefficiency in reasoning-oriented LLM post-training. Standard alignment pipelines for mathematical, coding, and scientific reasoning typically rely on large SFT corpora distilled from strong teachers such as DeepSeek-R1 and QwQ-32B, sometimes followed by RLHF or DPO on large preference datasets. The paper identifies four connected limitations: data inefficiency, compute cost from long chain-of-thought sequences, heuristic and non-generalizable data selection, and rigid distillation pipelines tied to specific teachers or domains.
The framework therefore targets sample efficiency rather than raw scale. The central claim is not that reasoning performance can be obtained from arbitrarily small corpora, but that much of the alignment data used in existing pipelines is redundant for small and medium models. InfiAlign operationalizes this claim by replacing heuristic filtering with an automated selection procedure over more than 10 million open-source QA pairs, then combining compact SFT with a lightweight DPO phase. This suggests that the framework’s main contribution lies in the organization and prioritization of alignment data rather than in any architectural change to the base model.
2. Overall framework and pipeline structure
The pipeline begins from large open reasoning corpora spanning math, code, science, and general reasoning. The data sources listed include OpenThoughts-114K, OpenThoughts3-1.2M, AM-DeepSeek-R1-Distilled-1.4M, data-ablation-full59K (s1), NuminaMath-CoT, OpenCodeReasoning, Llama-Nemotron post-training data, Mixture-of-Thoughts, OpenScience, and other corpora, for a total of more than 10 million QA pairs. Data are normalized into QA format, restricted to English, and stripped of incomplete items. When chain-of-thought is missing, solutions are generated using strong teachers such as DeepSeek-Distill and Qwen3.
The alignment workflow then proceeds in three steps. First, a multidimensional selection stage constructs compact high-value SFT corpora. Second, a two-stage SFT curriculum trains the model on progressively broader reasoning domains. Third, a DPO stage refines preferences using verified teacher solutions as positives and incorrect SFT outputs as negatives. The released model family consists of InfiAlign-Qwen-7B-SFT-92K, InfiAlign-Qwen-7B-SFT-165K, InfiAlign-Qwen-7B-DPO-9K, and InfiAlign-Qwen-7B-DPO-10K (Cai et al., 7 Aug 2025).
The base model is Qwen2.5-Math-7B-Base. No architectural modifications are described; the method is a full-parameter post-training recipe for a decoder-only Transformer. A plausible implication is that InfiAlign is intended to be portable across backbones, but the reported experiments are limited to the Qwen2.5-Math-7B setting and a small scaling study on 32B models.
3. Data selection pipeline
The data selection pipeline is the defining component of InfiAlign. It uses three dimensions.
First, diversity is enforced at two levels. Domain-level diversity is obtained by LLM-based classification into math, code, science, and general reasoning, with finer subcategories such as Algebra, Geometry, Analysis, Number Theory, DP, Graph, Simulation, molecular biology, quantum mechanics, logic, STEM, and humanities. Semantic diversity is computed by encoding instructions with Alibaba-NLP gte-base-en-v1.5, clustering them with K-means, and sampling uniformly across clusters. The domain and semantic views are then merged, followed by deduplication via 20-gram overlap.
Second, difficulty is approximated by response length. For each QA pair, the token length of the answer is computed, and within each domain or semantic cluster the pipeline prioritizes longer solutions. The paper treats response length as an empirically correlated proxy for difficulty, especially in math and code. It does not provide a causal account of why longer reasoning is better training data, and the authors explicitly note that surface features such as length and reasoning markers have not received a full causal analysis.
Third, quality is enforced by a combination of format checks, verifiers, regeneration, and decontamination. Rule-based checks include the presence of \boxed{} for final math answers. Domain-specific verification uses MathVerify and similar tools for math, sandbox execution for code, and LLM-based scoring for open-ended tasks. Failed items may be regenerated and re-verified for up to 8 iterations; persistent failures are dropped. Decontamination uses high n-gram overlap with benchmarks, with 15-gram overlap and embedding cosine similarity greater than 0.9 treated as near-duplicate contamination.
The resulting SFT sets contain 92K or 165K QA pairs selected from more than 10M raw examples. The SFT mixing ratio is Math : Code : Science = 4 : 4 : 3. Math and code are emphasized because they show strong cross-domain transfer and benefit more from scaling in data quality and length. This suggests that InfiAlign treats domain composition as part of alignment design, not merely as corpus availability.
4. Supervised fine-tuning stage
The SFT phase uses standard token-level cross-entropy over answer tokens, with the selected dataset as the training distribution. Two datasets are constructed: InfiAlign-SFT-92K and InfiAlign-SFT-165K. The curriculum is explicitly two-stage.
In Stage 1, training emphasizes relatively simple math and code data, comprising 70% of the phase, with the goal of stabilizing learning around structured reasoning patterns such as algorithmic reasoning and simple proofs. In Stage 2, the full corpus is introduced, including harder math and code as well as more open-ended science and general reasoning tasks. Stage-1 data are retained to preserve distribution continuity and prevent catastrophic forgetting.
Training details are fixed across the 92K and 165K variants: 5 epochs, batch size 16, learning rate , mixed-precision training, and 8 × NVIDIA H800 GPUs. The paper makes no mention of LoRA; the procedure is full-parameter fine-tuning. The reported outcome is that InfiAlign-Qwen-7B-SFT-92K reaches performance comparable to DeepSeek-R1-Distill-Qwen-7B despite using 92K rather than 800K examples, and scaling to 165K examples yields further gains, especially on AIME24 and GPQA (Cai et al., 7 Aug 2025).
5. Preference alignment with DPO
After SFT, InfiAlign applies a lightweight DPO stage. Preference pairs are built from OpenMathReasoning, Mixture-of-Thoughts, and OpenScience. Problems are first decontaminated against evaluation benchmarks and deduplicated with respect to SFT data. They are then labeled by domain and category using Qwen2.5-32B-Instruct, and the pipeline selects problems with the longest verified solutions within each category as the hardest examples.
Negative responses are produced by the SFT model itself. For math and science, Qwen2.5-32B-Instruct grades SFT outputs as correct or incorrect; for code, correctness is determined by sandbox execution. Among incorrect SFT responses, the framework selects false samples with the longest reasoning, balanced per category. Positive responses are the verified teacher solutions already present in the source datasets, drawn from strong models such as DeepSeek-R1 and QwQ-32B.
Two DPO datasets are constructed. InfiAlign-DPO-9K contains 4K math, 3K code, and 2K science pairs; InfiAlign-DPO-10K contains 3.5K math, 3.5K code, and 3K science pairs. Training uses 360-LLaMA-Factory with sequence parallelism, 16 × H800 GPUs, 3 epochs, batch size 16, learning rate , cosine schedule with warmup ratio 0.1, sequence parallelism factor 4, and .
The gains are concentrated in mathematics. InfiAlign-Qwen-7B-DPO-9K improves AIME 2024 from 56.46 to 61.04, while InfiAlign-Qwen-7B-DPO-10K improves AIME 2025 from 42.19 to 47.45 and MATH500 from 92.70 to 93.45. The paper attributes this primarily to preference data focused on hard, long-CoT problems and to the use of stronger teacher solutions as positives (Cai et al., 7 Aug 2025).
6. Evaluation, ablations, and limitations
Evaluation spans six benchmarks: AIME 2024, AIME 2025, MATH500, GPQA-Diamond, MMLU-Pro, and LiveCodeBench v5. Unified evaluation uses temperature 0.6, top-p 0.95, and max_tokens 32,768, with avg@64 for AIME 24/25, avg@4 for MATH500, avg@8 for GPQA and LCB-v5, and pass@1 for MMLU-Pro.
| Model | Data size | Avg |
|---|---|---|
| DeepSeek-Distill-Qwen-7B | 800K | 54.43 |
| InfiAlign-SFT-92K | 92K | 54.70 |
| InfiAlign-DPO-9K | 9K | 54.94 |
| InfiAlign-SFT-165K | 165K | 57.52 |
| InfiAlign-DPO-10K | 10K | 57.20 |
The central comparative result is the sample-efficiency claim: InfiAlign-SFT-92K slightly exceeds DeepSeek-Distill-Qwen-7B in average score, 54.70 versus 54.43, using about 12% as many alignment samples. Scaling from 92K to 165K improves the average from 54.70 to 57.52, with especially large gains on AIME24 and GPQA. DPO improves mathematical reasoning further, but its effect is not uniformly positive across all tasks; for example, InfiAlign-DPO-10K improves AIME25 and MATH500 while reducing GPQA, MMLU-Pro, and LCB-v5 relative to InfiAlign-SFT-165K. This makes the DPO stage a targeted refinement rather than a universal boost.
The ablation studies support the data-selection thesis. On general reasoning data, all subsets are fixed at 17.1K examples, and length + dual diversity outperforms random sampling, dual diversity alone, and length alone. In science-domain sampling, diversity is particularly important for GPQA because of diverse subfields, while length-only helps MMLU-Pro. In math-domain sampling, hard long-answer subsets dominate easy subsets, and mixing NuminaMath with s1 through the dual-heuristic strategy gives the best balance across AIME and GPQA. On 32B models with 1K samples, InfiAlign-1K slightly exceeds s1K-QwQ, and the paper reports that QwQ responses are 20% longer and contain 78% more reasoning markers, again reinforcing response length as a practical proxy for richer reasoning.
The paper also states several limitations. Metric tuning across domains remains hand-designed, and response length plus current LLM-based scoring rules may require adjustment for domains such as law or medicine. Surface proxies correlate with performance, but a full causal analysis is absent. Base-model dependence is untested beyond Qwen2.5-Math-7B and a limited 32B study. The authors also note risks associated with reasoning-strong LLMs, including automated high-stakes exam cheating and code exploitation if mis-prompted. Future work is directed toward learned quality metrics, extension to additional domains, larger backbones, dynamic curricula, and integration with other preference-learning methods such as RM+RLHF and pairwise versus listwise approaches (Cai et al., 7 Aug 2025).
