---
title: 'CapTrack: LLM Forgetting Analysis'
url: https://www.emergentmind.com/topics/captrack
type: topic
---

# CapTrack: LLM Forgetting Analysis

CapTrack is a capability-centric framework for analyzing forgetting in LLM post-training. It combines a behavioral taxonomy with an evaluation suite built on established benchmarks and targeted adaptations, and it reframes forgetting from a narrowly accuracy-centric loss of factual or parametric knowledge into systematic model drift from an out-of-the-box base model that degrades capabilities, default behaviors, or execution reliability. In the reported study, CapTrack is used for a large-scale empirical analysis across post-training algorithms, domains, and model families, including models up to 80B parameters, with results reported as relative deviation from the base model rather than as standalone task scores [2603.06610].

## 1. Conceptual basis and definition of forgetting

CapTrack is motivated by the claim that conventional forgetting analyses are too narrow for modern foundation models. In this formulation, a post-trained model may preserve benchmark accuracy while still becoming more refusal-prone, less verbose, less robust to prompt reformulations, worse at maintaining multi-turn commitments, or less reliable in structured output and citation behavior. CapTrack therefore defines forgetting as “any systematic drift from the base model that adversely affects capabilities, default behaviors, or execution reliability” [2603.06610].

A central methodological choice is to evaluate forgetting **relative to the out-of-the-box (OOB) model**. The framework measures **relative deviation (%)** from that baseline, so the primary object of analysis is drift rather than absolute performance. This makes CapTrack explicitly comparative across post-training stages such as **IFT**, **DPO**, and **IFT+DPO**, and it also allows changes that are not purely accuracy losses to become visible.

For interpretive convenience, CapTrack discretizes relative deviation into qualitative regimes. The reported bins are **None: \(> -1\%\)**, **Minor: \(-1\%\) to \(-5\%\)**, **Moderate: \(-5\%\) to \(-10\%\)**, **Major: \(-10\%\) to \(-20\%\)**, and **Catastrophic: \(< -20\%\)**. The paper states that this binning is for visualization only and that the substantive analyses use continuous metrics. It also emphasizes that drift is not always intrinsically undesirable: some deviations may reflect intentional alignment or specialization rather than harmful degradation. This suggests that CapTrack is designed as a diagnostic framework rather than a doctrine that treats every deviation from base-model behavior as failure.

## 2. Behavioral taxonomy: CAN, WILL, and HOW

The framework’s core conceptual contribution is a three-part taxonomy that decomposes LLM behavior into latent competence, default preferences, and execution reliability. This separates what a model **can do**, what it **will do by default**, and how it **executes protocols and interaction rules** [2603.06610].

| Group | Scope | Subcategories |
|---|---|---|
| **CAN** | Latent competence | C1 parametric knowledge & skills; C2 reasoning & problem solving; C3 contextual comprehension; C4 epistemic faithfulness & grounding; C5 robustness of competence |
| **WILL** | Default behavioral preferences | W1 willingness to answer; W2 helpfulness & informational scope; W3 style & level of elaboration |
| **HOW** | Protocol compliance & execution | H1 instruction following; H2 output-format fidelity; H3 tool/function use & integration; H4 multi-turn state & commitment keeping; H5 context-window operations; H6 citation & attribution mechanics |

Within **CAN**, CapTrack includes classical performance-oriented abilities such as factual knowledge and reasoning, but it extends them with robustness dimensions. In particular, **C5** is subdivided into **C5a prompt-form invariance**, **C5b domain-shift robustness**, and **C5c multilingual stability**. This broadens competence beyond single-dataset accuracy and makes robustness itself a first-class capability.

Within **WILL**, the framework tracks default response tendencies that are often strongly shaped by alignment and preference optimization rather than by raw competence. These include **benign refusal**, **unsafe compliance / refusal**, **underspecified prompt behavior**, **coverage**, **overreach**, **verbosity**, **hedging**, and **formatting usage**. The distinction matters because post-training can change compliance and style even when latent task competence remains largely intact.

Within **HOW**, CapTrack isolates execution-layer reliability: instruction following, schema adherence, tool use, multi-turn follow-through, context-window operations, and citation mechanics. The taxonomy explicitly distinguishes **citation & attribution mechanics** from **epistemic faithfulness & grounding**. A model may therefore be grounded in source material yet still fail to present citations in the required format or attach them to the correct source spans.

## 3. Evaluation suite and benchmark design

CapTrack operationalizes its taxonomy through a curated suite of **30 benchmarks** totaling about **14.5k evaluation samples**. To make the suite practical, large benchmarks are subsampled using **Scales++**, and multiple datasets are adapted to isolate specific forms of capability drift rather than only raw accuracy [2603.06610].

For **CAN**, the suite includes **MMLU-Pro**, **PopQA**, **GSM8K**, **LiveMathBench**, **HumanEval**, and **MBPP** for **C1**; **MATH** and **SuperGPQA** for **C2**; **HotpotQA** and **BoolQ** for **C3**; **RAGTruth** and **TruthfulQA** for **C4**; and rephrased or shifted variants of **MMLU-Pro**, **GSM8K**, **WinoGrande**, **HellaSwag**, **MGSM**, and **XTREME** for **C5**. The reported metrics include accuracy, evidence hit rate, hallucination rate, grounding or faithfulness scores, reasoning quality, and number of reasoning steps. Reasoning quality is scored with an LLM judge over atomic criteria such as step validity, logical coherence, and intermediate consistency.

For **WILL**, the suite uses **GSM8K** for benign prompts, **HarmBench** for unsafe prompts, and **RULER (4k cropped)** for underspecified prompts under **W1**; **RAGTruth** and **ELI5** for **W2**; and **MT-Bench** (turn 1) plus **OASST1** for **W3**. Metrics include benign refusal rate, unsafe compliance rate, underspecified compliance rate, coverage, overreach, verbosity, hedging, and formatting usage. One notable adaptation truncates the final 100 tokens of **RULER** so that the explicit question is removed; the resulting prompt is underspecified, and the desired model behavior is to recognize the ambiguity rather than hallucinate or improperly refuse.

For **HOW**, CapTrack uses **IFEval** and **FollowBench** for **H1**; schema-wrapped **MMLU-Pro** and **GSM8K** for **H2**; **BFCL** and **MNMS** for **H3**; **MT-Bench** (turn 2), **StructFlowBench**, **RULER (32k)**, and **LongBench-V2** for **H4**; and **HotpotQA** plus **QASPER** for **H6**. Metrics include pass rate, constraint satisfaction, parse rate or schema pass rate, tool selection accuracy, argument accuracy, multi-turn tool-use score, follow-through on second-turn and long-context commitments, citation format correctness, and source accuracy.

The suite also incorporates targeted benchmark adaptations: **rephrased prompts** for prompt-form invariance, **JSON schema wrapping** to test output-format fidelity, **underspecified RULER** prompts, **long-context restrictions** for **LongBench-V2** to keep context under about 125k tokens, **language filtering** for **MGSM** and **XTREME**, and **English-only first-turn OASST1** for style analysis. Scoring combines rule-based evaluation with **LLM-as-a-judge** evaluation; for judge-based metrics, CapTrack decomposes assessments into atomic checks and selects **GPT-4o-mini** because it correlates well with much stronger judges while being cheaper and faster.

## 4. Experimental protocol and model coverage

The reported empirical study spans **7 instruction-tuned models**, **3 model families**—**Qwen**, **LLaMA**, and **Gemma**—and **two domains**, **legal** and **medical**. The model list includes **Qwen 3 Next 80B**, **Qwen 3 14B**, **Qwen 3 4B**, **LLaMA 3.3 70B**, **LLaMA 3.1 8B**, **Gemma 3 12B**, and **Gemma 3 4B**. All are already instruction-tuned checkpoints, reflecting the common third-party adaptation setting in which aligned models are further post-trained rather than trained from scratch [2603.06610].

CapTrack compares **IFT**, **DPO**, and **IFT+DPO**. In the reported setup, **IFT** is standard instruction fine-tuning on instruction-response pairs, whereas **DPO** is direct preference optimization on chosen/rejected pairs with respect to a frozen OOB reference model \(\theta_0\) and preference strength parameter \(\beta\). Training uses a fixed protocol across runs: **one epoch per post-training method**, **the same optimization setup across models and domains**, **no model-specific tuning**, and **three random seeds** for evaluation stability.

The implementation stack uses **AdamW**, **bfloat16**, **FlashAttention-2 where available**, and **vLLM for inference**. The setup also includes long-context configurations, including **RoPE scaling for Qwen3-14B** to extend context. This fixed-protocol design is important interpretively: it suggests that the reported differences are intended to reflect post-training method and model-family effects rather than bespoke tuning advantages.

## 5. Empirical findings and mitigation analysis

The main empirical result is that forgetting in LLM post-training extends well beyond parametric knowledge loss. CapTrack reports drift across all three capability groups, with especially pronounced degradation in **robustness** and **default behaviors**. The paper highlights examples in which standard benchmarks such as **MMLU-Pro** or **IFEval** show little change while multilingual robustness or verbosity degrades sharply, indicating that accuracy-preserving post-training can still alter user-visible behavior substantially [2603.06610].

Across domains and model families, **IFT produces the largest forgetting**, **DPO is more conservative**, and **DPO after IFT can partially recover some lost capabilities**. The most fragile regions are often within **WILL**, especially **verbosity**, **formatting**, **refusal behavior**, and **coverage**, although **multilingual robustness** under **CAN** is also reported as especially vulnerable and can degrade catastrophically under IFT. **HOW** is described as comparatively more stable, but it still shows drift in areas such as **instruction following**, **multi-turn consistency**, and **citation mechanics**.

CapTrack also reports persistent **model-family differences**. **LLaMA** and **Gemma** tend to show stronger drift in competence and behavioral preferences, whereas **Qwen** is described as generally more stable overall, though still vulnerable in areas such as multilingual robustness and instruction following. By contrast, **model size is not a reliable predictor of forgetting**: the paper reports no consistent monotonic relationship between scale and stability, with only a few capabilities—such as citation source accuracy—showing a clearer size effect.

The mitigation analysis covers three families. Under **data-centric mitigation**, the paper compares domain-specific legal or medical post-training, general-purpose **Tulu-based** post-training, and replay versus no-replay mixtures; the result is that **no consistent data strategy universally reduces forgetting**. Under **architectural mitigation**, CapTrack examines model merging methods such as **TIES** and variants, reporting a clear **stability-plasticity trade-off**: more weight on the adapted model improves in-domain performance but increases forgetting, while more weight on the OOB model improves stability but reduces adaptation gains. Under **regularization-based mitigation**, the paper studies **LoRA** ranks and learning rates and again finds the same trade-off. The reported overall conclusion is that **no universal mitigation emerges**, and a plausible implication is that mitigation must be capability-aware rather than globally constrained.

## 6. Terminology, scope, and disambiguation

The exact title **“CapTrack”** refers to the LLM post-training forgetting framework described above [2603.06610]. In adjacent literature, however, similar strings have been used informally for unrelated systems in object tracking, scientific tracking software, biomedical tracking, medical image guidance, and wireless sensing. This creates a naming ambiguity that is best resolved by domain and paper title rather than by the string alone.

Examples of distinct systems include **“Compact Transformer Tracker with Correlative Masked Modeling”**, whose tracker is **CTTrack** and argues that vanilla self-attention is sufficient for visual tracking [2301.10938]; **“Cell as Point: One-Stage Framework for Efficient Cell Tracking”**, whose method is **CAP** and treats cells as points in a one-stage transformer-based framework [2411.14833]; **“Context-Aware Token Pruning and Discriminative Selective Attention for Transformer Tracking”**, whose tracker is **CPDATrack** and performs context-aware token pruning plus discriminative selective attention [2511.19928]; **“ConTrack: Contextual Transformer for Device Tracking in X-ray”**, which addresses catheter-tip localization in fluoroscopy and angiography [2307.07541]; **“Tracking with A Common Tracking Software”**, which presents **Acts**, an experiment-independent HEP tracking toolkit [2007.01239]; and **“CentiTrack: Towards Centimeter-Level Passive Gesture Tracking with Commodity WiFi”**, which is a passive gesture tracking system based on commodity WiFi CSI [2011.06896].

This distinction matters because the LLM CapTrack framework is not a tracker in the computer-vision or signal-processing sense. It is an evaluation framework for **post-training-induced drift** in foundation models. A common misconception is therefore to read the title as belonging to visual or physical tracking. The literature provided here indicates that such an interpretation would conflate unrelated research programs that merely share the substring “Track.”

Source: https://www.emergentmind.com/topics/captrack