CapTrack: LLM Forgetting Analysis
- CapTrack is a framework that defines forgetting as any systematic drift from a base model affecting capabilities, behaviors, or execution reliability.
- It utilizes a three-part behavioral taxonomy—CAN, WILL, and HOW—to distinguish between latent competence, default preferences, and execution reliability.
- Extensive evaluation across diverse models and post-training methods reveals that forgetting extends beyond accuracy loss, highlighting the need for capability-aware mitigation.
CapTrack is a capability-centric framework for analyzing forgetting in LLM post-training. It combines a behavioral taxonomy with an evaluation suite built on established benchmarks and targeted adaptations, and it reframes forgetting from a narrowly accuracy-centric loss of factual or parametric knowledge into systematic model drift from an out-of-the-box base model that degrades capabilities, default behaviors, or execution reliability. In the reported study, CapTrack is used for a large-scale empirical analysis across post-training algorithms, domains, and model families, including models up to 80B parameters, with results reported as relative deviation from the base model rather than as standalone task scores (Thede et al., 19 Feb 2026).
1. Conceptual basis and definition of forgetting
CapTrack is motivated by the claim that conventional forgetting analyses are too narrow for modern foundation models. In this formulation, a post-trained model may preserve benchmark accuracy while still becoming more refusal-prone, less verbose, less robust to prompt reformulations, worse at maintaining multi-turn commitments, or less reliable in structured output and citation behavior. CapTrack therefore defines forgetting as “any systematic drift from the base model that adversely affects capabilities, default behaviors, or execution reliability” (Thede et al., 19 Feb 2026).
A central methodological choice is to evaluate forgetting relative to the out-of-the-box (OOB) model. The framework measures relative deviation (%) from that baseline, so the primary object of analysis is drift rather than absolute performance. This makes CapTrack explicitly comparative across post-training stages such as IFT, DPO, and IFT+DPO, and it also allows changes that are not purely accuracy losses to become visible.
For interpretive convenience, CapTrack discretizes relative deviation into qualitative regimes. The reported bins are None: , Minor: to , Moderate: to , Major: to , and Catastrophic: . The paper states that this binning is for visualization only and that the substantive analyses use continuous metrics. It also emphasizes that drift is not always intrinsically undesirable: some deviations may reflect intentional alignment or specialization rather than harmful degradation. This suggests that CapTrack is designed as a diagnostic framework rather than a doctrine that treats every deviation from base-model behavior as failure.
2. Behavioral taxonomy: CAN, WILL, and HOW
The framework’s core conceptual contribution is a three-part taxonomy that decomposes LLM behavior into latent competence, default preferences, and execution reliability. This separates what a model can do, what it will do by default, and how it executes protocols and interaction rules (Thede et al., 19 Feb 2026).
| Group | Scope | Subcategories |
|---|---|---|
| CAN | Latent competence | C1 parametric knowledge & skills; C2 reasoning & problem solving; C3 contextual comprehension; C4 epistemic faithfulness & grounding; C5 robustness of competence |
| WILL | Default behavioral preferences | W1 willingness to answer; W2 helpfulness & informational scope; W3 style & level of elaboration |
| HOW | Protocol compliance & execution | H1 instruction following; H2 output-format fidelity; H3 tool/function use & integration; H4 multi-turn state & commitment keeping; H5 context-window operations; H6 citation & attribution mechanics |
Within CAN, CapTrack includes classical performance-oriented abilities such as factual knowledge and reasoning, but it extends them with robustness dimensions. In particular, C5 is subdivided into C5a prompt-form invariance, C5b domain-shift robustness, and C5c multilingual stability. This broadens competence beyond single-dataset accuracy and makes robustness itself a first-class capability.
Within WILL, the framework tracks default response tendencies that are often strongly shaped by alignment and preference optimization rather than by raw competence. These include benign refusal, unsafe compliance / refusal, underspecified prompt behavior, coverage, overreach, verbosity, hedging, and formatting usage. The distinction matters because post-training can change compliance and style even when latent task competence remains largely intact.
Within HOW, CapTrack isolates execution-layer reliability: instruction following, schema adherence, tool use, multi-turn follow-through, context-window operations, and citation mechanics. The taxonomy explicitly distinguishes citation & attribution mechanics from epistemic faithfulness & grounding. A model may therefore be grounded in source material yet still fail to present citations in the required format or attach them to the correct source spans.
3. Evaluation suite and benchmark design
CapTrack operationalizes its taxonomy through a curated suite of 30 benchmarks totaling about 14.5k evaluation samples. To make the suite practical, large benchmarks are subsampled using Scales++, and multiple datasets are adapted to isolate specific forms of capability drift rather than only raw accuracy (Thede et al., 19 Feb 2026).
For CAN, the suite includes MMLU-Pro, PopQA, GSM8K, LiveMathBench, HumanEval, and MBPP for C1; MATH and SuperGPQA for C2; HotpotQA and BoolQ for C3; RAGTruth and TruthfulQA for C4; and rephrased or shifted variants of MMLU-Pro, GSM8K, WinoGrande, HellaSwag, MGSM, and XTREME for C5. The reported metrics include accuracy, evidence hit rate, hallucination rate, grounding or faithfulness scores, reasoning quality, and number of reasoning steps. Reasoning quality is scored with an LLM judge over atomic criteria such as step validity, logical coherence, and intermediate consistency.
For WILL, the suite uses GSM8K for benign prompts, HarmBench for unsafe prompts, and RULER (4k cropped) for underspecified prompts under W1; RAGTruth and ELI5 for W2; and MT-Bench (turn 1) plus OASST1 for W3. Metrics include benign refusal rate, unsafe compliance rate, underspecified compliance rate, coverage, overreach, verbosity, hedging, and formatting usage. One notable adaptation truncates the final 100 tokens of RULER so that the explicit question is removed; the resulting prompt is underspecified, and the desired model behavior is to recognize the ambiguity rather than hallucinate or improperly refuse.
For HOW, CapTrack uses IFEval and FollowBench for H1; schema-wrapped MMLU-Pro and GSM8K for H2; BFCL and MNMS for H3; MT-Bench (turn 2), StructFlowBench, RULER (32k), and LongBench-V2 for H4; and HotpotQA plus QASPER for H6. Metrics include pass rate, constraint satisfaction, parse rate or schema pass rate, tool selection accuracy, argument accuracy, multi-turn tool-use score, follow-through on second-turn and long-context commitments, citation format correctness, and source accuracy.
The suite also incorporates targeted benchmark adaptations: rephrased prompts for prompt-form invariance, JSON schema wrapping to test output-format fidelity, underspecified RULER prompts, long-context restrictions for LongBench-V2 to keep context under about 125k tokens, language filtering for MGSM and XTREME, and English-only first-turn OASST1 for style analysis. Scoring combines rule-based evaluation with LLM-as-a-judge evaluation; for judge-based metrics, CapTrack decomposes assessments into atomic checks and selects GPT-4o-mini because it correlates well with much stronger judges while being cheaper and faster.
4. Experimental protocol and model coverage
The reported empirical study spans 7 instruction-tuned models, 3 model families—Qwen, LLaMA, and Gemma—and two domains, legal and medical. The model list includes Qwen 3 Next 80B, Qwen 3 14B, Qwen 3 4B, LLaMA 3.3 70B, LLaMA 3.1 8B, Gemma 3 12B, and Gemma 3 4B. All are already instruction-tuned checkpoints, reflecting the common third-party adaptation setting in which aligned models are further post-trained rather than trained from scratch (Thede et al., 19 Feb 2026).
CapTrack compares IFT, DPO, and IFT+DPO. In the reported setup, IFT is standard instruction fine-tuning on instruction-response pairs, whereas DPO is direct preference optimization on chosen/rejected pairs with respect to a frozen OOB reference model and preference strength parameter . Training uses a fixed protocol across runs: one epoch per post-training method, the same optimization setup across models and domains, no model-specific tuning, and three random seeds for evaluation stability.
The implementation stack uses AdamW, bfloat16, FlashAttention-2 where available, and vLLM for inference. The setup also includes long-context configurations, including RoPE scaling for Qwen3-14B to extend context. This fixed-protocol design is important interpretively: it suggests that the reported differences are intended to reflect post-training method and model-family effects rather than bespoke tuning advantages.
5. Empirical findings and mitigation analysis
The main empirical result is that forgetting in LLM post-training extends well beyond parametric knowledge loss. CapTrack reports drift across all three capability groups, with especially pronounced degradation in robustness and default behaviors. The paper highlights examples in which standard benchmarks such as MMLU-Pro or IFEval show little change while multilingual robustness or verbosity degrades sharply, indicating that accuracy-preserving post-training can still alter user-visible behavior substantially (Thede et al., 19 Feb 2026).
Across domains and model families, IFT produces the largest forgetting, DPO is more conservative, and DPO after IFT can partially recover some lost capabilities. The most fragile regions are often within WILL, especially verbosity, formatting, refusal behavior, and coverage, although multilingual robustness under CAN is also reported as especially vulnerable and can degrade catastrophically under IFT. HOW is described as comparatively more stable, but it still shows drift in areas such as instruction following, multi-turn consistency, and citation mechanics.
CapTrack also reports persistent model-family differences. LLaMA and Gemma tend to show stronger drift in competence and behavioral preferences, whereas Qwen is described as generally more stable overall, though still vulnerable in areas such as multilingual robustness and instruction following. By contrast, model size is not a reliable predictor of forgetting: the paper reports no consistent monotonic relationship between scale and stability, with only a few capabilities—such as citation source accuracy—showing a clearer size effect.
The mitigation analysis covers three families. Under data-centric mitigation, the paper compares domain-specific legal or medical post-training, general-purpose Tulu-based post-training, and replay versus no-replay mixtures; the result is that no consistent data strategy universally reduces forgetting. Under architectural mitigation, CapTrack examines model merging methods such as TIES and variants, reporting a clear stability-plasticity trade-off: more weight on the adapted model improves in-domain performance but increases forgetting, while more weight on the OOB model improves stability but reduces adaptation gains. Under regularization-based mitigation, the paper studies LoRA ranks and learning rates and again finds the same trade-off. The reported overall conclusion is that no universal mitigation emerges, and a plausible implication is that mitigation must be capability-aware rather than globally constrained.
6. Terminology, scope, and disambiguation
The exact title “CapTrack” refers to the LLM post-training forgetting framework described above (Thede et al., 19 Feb 2026). In adjacent literature, however, similar strings have been used informally for unrelated systems in object tracking, scientific tracking software, biomedical tracking, medical image guidance, and wireless sensing. This creates a naming ambiguity that is best resolved by domain and paper title rather than by the string alone.
Examples of distinct systems include “Compact Transformer Tracker with Correlative Masked Modeling”, whose tracker is CTTrack and argues that vanilla self-attention is sufficient for visual tracking (Song et al., 2023); “Cell as Point: One-Stage Framework for Efficient Cell Tracking”, whose method is CAP and treats cells as points in a one-stage transformer-based framework (Song et al., 2024); “Context-Aware Token Pruning and Discriminative Selective Attention for Transformer Tracking”, whose tracker is CPDATrack and performs context-aware token pruning plus discriminative selective attention (Kugarajeevan et al., 25 Nov 2025); “ConTrack: Contextual Transformer for Device Tracking in X-ray”, which addresses catheter-tip localization in fluoroscopy and angiography (Demoustier et al., 2023); “Tracking with A Common Tracking Software”, which presents Acts, an experiment-independent HEP tracking toolkit (Ai, 2020); and “CentiTrack: Towards Centimeter-Level Passive Gesture Tracking with Commodity WiFi”, which is a passive gesture tracking system based on commodity WiFi CSI (Han et al., 2020).
This distinction matters because the LLM CapTrack framework is not a tracker in the computer-vision or signal-processing sense. It is an evaluation framework for post-training-induced drift in foundation models. A common misconception is therefore to read the title as belonging to visual or physical tracking. The literature provided here indicates that such an interpretation would conflate unrelated research programs that merely share the substring “Track.”