Introspective X Training: Feedback Conditioning Improves Scaling Across all LLM Training Stages

Abstract: We tackle the question of how to scale more efficiently across the many, ever-growing stages of current LLM training pipelines. Our guiding intuition stems from the fact that the dynamics of later stages of the pipeline, e.g. post-training, can be used to inform earlier stages such as pre-training. To this end, we propose Introspective Training (or IXT), inspired by offline reward-conditioned reinforcement learning and applicable to any stage of training. IXT uses a thinking reward model to annotate data with natural language critique based feedback, enabling quality aware training from the earliest stages of the pipeline. Models are then trained by prefix-conditioning the data with the generated feedback -- ensuring that not all tokens are treated equally starting much earlier in training than usual. Comprehensive experiments on 7.5-12B transformer-based dense LLMs trained from scratch all the way up to 18 Trillion tokens seen show that our method: bends scaling curves resulting in up to 2.8x more compute efficiency generally; and reaches performance levels unachievable for models trained otherwise in domains such as math and code.

• The paper demonstrates that feedback conditioning during all LLM training stages improves compute efficiency, achieving up to 2.8× FLOP gains.
• It integrates templated quality tokens and natural language critiques as conditioning prefixes, refining performance in reasoning and coding tasks.
• The approach advances domain specialization and test-time control, challenging traditional pretraining followed by alignment paradigms.

Introspective X Training: Unified Feedback Conditioning for Improved LLM Scaling Efficiency

Overview

The paper "Introspective X Training: Feedback Conditioning Improves Scaling Across all LLM Training Stages" (2605.20285) introduces Introspective Training (X), a unified framework that directly incorporates quality-aware feedback into LLM (LM) training across all standard pipeline phases: pretraining, continued pretraining (CPT), and supervised fine-tuning (SFT). The approach operationalizes natural language or tokenized feedback, generated through rubric-based annotation by a judge LLM, as a conditioning prefix—replacing the conventional paradigm of exclusively post-hoc data curation and reward-driven post-training alignment. X is inspired by reward-conditioned offline reinforcement learning, but remains compatible with the standard next-token prediction objective, integrating feedback as a document prefix for conditioning throughout the entire training stack.

Figure 1: X applies feedback conditioning at all training stages. A judge LLM annotates documents on multiple axes, producing scores and a natural language critique; training is performed with this feedback prepended, improving efficiency.

Methodology: Data Annotation and Feedback Conditioning

Annotation Pipeline

For every document, a domain-agnostic rubric is used to guide evaluation along axes such as writing style, expertise, educational value, fact density/accuracy, and efficiency. A judge LLM (Qwen3-30B-A3B) assigns a score (1-5 per axis, plus overall) and produces a 1-2 sentence critique summarizing document quality. Annotations can be generated in a pointwise regime for broad corpora or using pairwise comparisons and a Bradley-Terry aggregation for curated sets (such as SFT mixtures with low score variance).

Conditioning Strategies

Two main mechanisms instantiate the feedback prefix during training:

• Templated Quality Tokens ($\phi_\text{tok}$): Translates overall score into a discrete quality label (e.g., [high], [low]) which is tokenized and prepended to the input.
• Natural Language Critiques ($\phi_\text{crit}$): Prepends the full rubric-based summary critique, allowing richer and finely-controllable feedback signals.

Crucially, the entire concatenated sequence, including the feedback prefix, is used for next-token prediction with no masking. This explicit association between feedback and data is preserved throughout all pipeline phases: from scratch pretraining, through continual domain-specific CPT, and into SFT.

Empirical Evaluation

Experiments involve Transformer-based LLMs (7.5B–12B params), trained on large corpora (up to 18T tokens) encompassing mathematics (CraneMath), code (Swallow Code), and general-domain blends (Dolmino). Evaluation is broad: reasoning, code generation, and general knowledge (benchmarks include GSM8k, MATH, HumanEval, MBPP, MMLU, ARC-Challenge, etc.).

Impact on Scaling Efficiency

X consistently shifts scaling curves upward relative to unconditioned next-token prediction (NTP) at equal compute, delivering up to 2.8× FLOP efficiency gains (all training and annotation compute accounted).

Figure 2: FLOP-scaling for X vs NTP on Dolmino. X delivers higher accuracy at equal compute, especially on math and code; coding requires 1.8–3.1× more compute under NTP to reach X performance.

On both pretraining from scratch and CPT, X “bends” the scaling law: reasoning-heavy domains (math/code) show the largest absolute improvements (e.g., GSM8k: 35.8→50.2, MATH: 42.2→53.9, HumanEval: 32.8→37.7, MBPP: 50.2→58.4). General tasks (MMLU and related) also show consistent, if smaller, gains. X persists in outperforming baseline NTP models even deep into training (12T–18T checkpoints): X applied at 12T can achieve HumanEval scores higher than the 18T unconditioned baseline.

Domain Specialization Versatility

X is shown to be an efficient pathway for fast domain specialization; models initialized from early checkpoints (e.g., 95B tokens) and further trained with X conditioning achieve non-transient and persistent boosts which persist even after extensive further pretraining. Utilizing specific subset annotation (where only select corpus segments, such as math and code, are feedback-conditioned) achieves comparable or greater improvements than full-blend annotation—while requiring annotation of only ∼15% of the data. Gains in reasoning/coding also spill over to general tasks.

Feedback Modality: Templated Tokens vs Natural-Language Critiques

Both forms of conditioning drive performance gains, but critique-based feedback improves further over tokens by an average of +2.6 points on aggregate, and is especially valuable for inference-time control and human steerability. Best performance, however, is rubric- and domain-dependent. Critique effectiveness is limited primarily by annotation calibration in non-textual domains (e.g., code correctness vs style).

Data Quality Distribution and Error Analysis

Score distributions in specialized datasets (Figure 3) are heavily skewed (e.g., only ~3.5% of SwallowCode in [high]). Conditioning on the upper tiers leads to varying code generation behaviors: with functionally correct code but variable adherence to test harness name requirements, which may artifactual degrade raw pass@1 scores due to evaluation details rather than true model errors.

Figure 3: SwallowCode pointwise score distribution; most documents fall in tiers 1–4, with tier 5 sparsity affecting coded response diversity and performance.

Analysis of Data Filtering vs. Tagging

Filtering (keeping only top-tier data) underperforms relative to conditioning: including tokens for lower-quality tiers while exposing their presence (rather than removing low-quality data) sustains learning and avoids degradation at high compute regimes. This demonstrates the value of explicit quality signals throughout training, rather than aggressive data curation or filtering alone.

Theoretical and Practical Implications

Theoretically, X challenges the traditional pre-align-post linearity of LLM pipelines. It demonstrates empirically that post-training supervision signals (including reward models and feedback) are expressively useful even if applied to pretraining. Unlike preference-conditioning that targets alignment, feedback-conditioning enables quality-aware optimization for general capability. The framework further validates the hypothesis that explicit token-level quality signals, maintained throughout autoregressive training, enable the model to "internalize" reward functions as part of general representation learning.

Practically, this translates into substantial computational efficiency improvements: less compute is needed for reaching target capabilities in high-value domains. The data annotation process is accounted as a one-time cost, and its amortization is justified by persistent downstream gains. Furthermore, X’s conditioning schema supports test-time steering: models can be biased toward different quality characteristics on demand, facilitating new control paradigms.

Limitations and Future Directions

Calibration of the annotation rubric is critical: general-purpose feedback may misalign for specialized domains such as code or scientific reasoning. The static (precomputed) nature of annotations introduces the risk that feedback loses discriminative power as the model advances; an online (iterative) annotation scheme synchronized with model capabilities represents a clear next step. Ablations across feedback granularity, mixture strategies, and hybrid conditioning signals may further clarify theoretical boundaries.

Conclusion

Introspective X Training is demonstrated as a unified, compatible, and practically efficient feedback conditioning paradigm for LLMs. By systematically integrating rubric-based critique and quality signals across the full training stack, X achieves significant scaling-law improvements, accelerates domain specialization, and supports expressive, controllable generation without intrusive objective modifications. It provides a strong foundation for future LLM training pipelines—where annotation, feedback, and alignment signals are designed as first-class citizens from pretraining onward.

Figure 4: FLOP-scaling on MATH and GSM8k for varying quality tiers; conditioning across the full data quality spectrum consistently outperforms strict filtering of low-quality data.