- The paper introduces IAR, a three-stage method that uses structured document reconstruction, answer-only QA fine-tuning, and checkpoint merging to internalize knowledge without retrieval.
- IAR outperforms Vanilla SFT on all four metrics in 7 of 8 settings, averaging 3.6 percentage points higher domain accuracy and 12.1 points better mean general performance.
- The results show IAR offers a strong domain-primary operating point, but performance depends on the model, corpus, and training setup, with limitations including single-run experiments and judge-mediated evaluation.
Problem setting and motivation
The paper studies document knowledge internalization: given a bounded corpus and document-derived questions, an instruction-tuned model must answer held-out questions without receiving retrieved passages at inference time. This setting is deliberately stricter than retrieval-augmented generation, and the authors motivate it by deployments where retrieval is unavailable or undesirable for latency, privacy, or evaluation reasons. The central tension is well known: supervised fine-tuning (SFT) on QA pairs teaches the test-time interface but supervises only the facts the QA generator selected, while continued pretraining (CPT) provides denser document exposure without teaching QA behavior, and both can cause catastrophic forgetting of general instruction-following ability.
The authors frame the problem as a domain–general operating-point frontier rather than a single-metric comparison, evaluating retrieval-free domain accuracy jointly with IFEval, MMLU, and MSBench as general-capability guardrails. This framing drives the entire experimental design.
The IAR framework
IAR decomposes internalization into three stages with separable hypotheses. Inject converts source documents into instruction-conditioned supervised reconstruction tasks—continuation from a document prefix, rewrite from a generated summary, outline, or knowledge skeleton, and instruction-formatted full-document reconstruction—using assistant-target loss with prompt tokens masked. The authors are careful to distinguish this from raw CPT: the objective retains an explicit prompt/target boundary rather than modeling the raw document stream. Align applies answer-only QA fine-tuning from the injected checkpoint rather than from the original instruction model. Recover performs post-hoc model merging between the domain-adapted checkpoint and the base instruction model, drawing from a fixed grid of SLERP, task arithmetic, TIES, and DARE operators.
A notable methodological contribution is the Recover selection protocol. Candidates are filtered by a domain-primary rule on the validation split: domain accuracy must be within one percentage point of Vanilla SFT, mean general performance must not fall below Vanilla SFT, and at least two of three general metrics must be within tolerance. Selection is completed before any held-out test evaluation, so the reported test frontiers diagnose rather than select operating points. The authors also explicitly state that IAR introduces no new merge algorithm or reconstruction loss family; the increment is the controlled empirical decomposition itself.
Main results
Across two corpora (CC with 14,258 training QA pairs, CCI with 10,926) and four model families (Llama-3.2-3B, Phi-4-mini, Qwen3-4B, SmolLM3-3B), IAR improves over Vanilla SFT on all four reported metrics in 7 of 8 settings, with average gains of 3.6 percentage points in domain QA accuracy and 12.1 points in mean general performance. The strongest single result is Qwen3-4B on CC: 50.5% domain accuracy versus 42.4% for Vanilla SFT, while improving all three general metrics.
The paper is commendably candid about boundary cases. Phi on CCI shows IAR improving IFEval and MSBench while slightly reducing domain accuracy and MMLU relative to Vanilla SFT, and the row is retained in the main table rather than relegated. Qwen3-4B on CCI starts from an unusually high 70.6% base domain accuracy; a post-hoc bits-per-byte diagnostic shows Qwen assigns lower conditional BPB than all three peer models on identical CCI source continuations, with all paired bootstrap intervals excluding zero. The authors correctly decline to over-interpret this: the diagnostic does not establish contamination, memorization, or causality, and Phi's BPB/QA ordering contradicts a simple likelihood-accuracy account.
The extended CC stress test qualifies the headline claim. SDFT is the strongest domain baseline for Llama, and LoRA and FAPM often win individual general metrics—but FAPM at sparsity 0.9 sacrifices substantial domain accuracy (e.g., Llama CC drops from 35.5% to 22.3%), and LoRA under-internalizes the corpus (22.5% domain for Llama CC). IAR's advantage is therefore a strong domain-primary operating point, not uniform dominance. The Instruct-initialized CPT diagnostics further show that CPT is initialization-sensitive, with CC Llama Instruct-CPT reaching 38.7% domain accuracy—competitive with the best pre-recovery Inject+Align checkpoint—so no uniform ordering between structured Inject and raw-document CPT is established.
Token-budget control and stage-level evidence
The BudgetMatch ablation addresses the most important confound: whether IAR's gains merely reflect more training tokens. Matching QA-only token budgets to the Inject+Align pipeline (99.9–102.8% of realized IA token volume), BudgetMatch improves over Vanilla SFT on CC but not uniformly on CCI. Relative to BudgetMatch, IAR wins 14 of 16 metric comparisons, with higher domain accuracy in three of four settings and higher mean general performance in all four. The exception—CC Llama, where IAR trades 3.9 domain points for an 11.0-point mean general gain—shows the frontier interpretation is genuine rather than rhetorical.
Pre-recovery evidence supports the mechanism claim: the best Inject+Align checkpoint improves domain accuracy over Vanilla SFT in all eight settings (gains of +2.8 to +7.7 points on CC, +0.4 to +6.1 on CCI). However, no Inject recipe is uniformly best—Mixed 1:1:2 dominates for Llama and Phi, while Qwen3-4B CCI slightly favors reconstruction-only 1:0:0—so the useful mixture remains model- and corpus-dependent.
Scaling behavior
On CC, the Qwen3-8B/14B/32B ablation shows a repeated pattern: the selected TIES d=0.3 checkpoint stays within 1.1 points of Best IA domain accuracy while recovering 14.9–24.1 points of mean general performance, with improvements on every general benchmark rather than compensation by a single metric. Inject+Align's contribution does not scale away—relative to Vanilla SFT, post-recovery IAR retains domain gains of 8.1, 4.8, and 6.4 points at 8B, 14B, and 32B respectively. The authors correctly scope this claim: all three rows use CC and the same merge density, so this is a repeated within-family pattern, not a scaling law. Recovery is also partial—selected checkpoints retain 15.6–19.2 points more domain accuracy than the base instruction models but do not fully restore their general scores.
Limitations and open questions
The paper concedes several limitations that materially qualify the results. Every reported checkpoint is a single training run with fixed seeds; no multi-seed variance is estimated, and domain generation uses unseeded sampling with no explicit judge seeds, so the authors explicitly disclaim repeated-run robustness. Domain evaluation relies on an LLM-judge panel (gpt-oss-120b, MiniMax-M2.5, DeepSeek-V3 variants) treated as an auditable instrument rather than a gold label; the 242,255-record audit reports binary agreement of .848 and Cohen's κ of .691, which is adequate but not conclusive. The scaling-run training provenance is incomplete—the original Inject/Align arguments and logs for the 8B/14B/32B rows are absent from the archive, so the shared optimization table does not cover them. The main CPT+SFT baseline uses Base initialization while IAR uses Instruct initialization, so the main table does not isolate the Inject objective; the available Instruct-initialized diagnostics cover only three settings. Open questions include why the best Inject mixture varies by model family, whether the Qwen3 scaling pattern transfers to other corpora or merge densities, and whether the high-base-prior phenomenon observed for Qwen on CCI generalizes.
Conclusion
IAR contributes a controlled decomposition of retrieval-free document internalization into structured exposure, QA alignment, and post-hoc recovery, evaluated under a pre-registered validation-based operating-point selection rule. The empirical claims are stated with appropriate scope: staged exposure and recovery produce strong domain-primary operating points in most of eight main settings and across a Qwen3 scaling sweep, but the framework is setting-dependent rather than uniformly dominant, and the evidence base carries single-run, judge-mediated, and initialization-confounded caveats that the authors document explicitly.