TrackStar: Scalable Training-Data Attribution

• TrackStar is a scalable training-data attribution method that uses compressed per-module gradients and empirical-Fisher preconditioning to measure individual influence.
• It employs a single-checkpoint, influence-function–style approach with block-wise random projections, enabling both exact and approximate retrieval in the Bergson library.
• The method has been applied to capability provenance and topic-specific attribution, highlighting its scalability while noting challenges with hyperparameter sensitivity and projection granularity.

TrackStar is a scalable, gradient-based training-data attribution method for estimating how strongly individual training examples support or suppress a model’s behavior at a fixed checkpoint. In the current literature, it is most fully specified through the open-source Bergson library, which implements TrackStar as a single-checkpoint, influence-function–style method based on compressed per-module gradients, empirical-Fisher preconditioning, and optional gradient normalization (Quirke et al., 10 Jun 2026). Recent studies have used it to map capability provenance in OLMo3-7B and to trace how coordinated Wikipedia edits by the Pro-Animal Wikipedians shape Llama-family models’ handling of animal welfare topics (Matlin et al., 17 Jun 2026, Brazilek et al., 30 Apr 2026). The label “TrackStar” also appears in unrelated technical contexts, including sports multi-object tracking, STAR-RIS-aided ISAC, and forward tracking at STAR, so the term is not globally univocal across research domains (Sun et al., 2024, Li et al., 7 Jan 2025, Brandenburg et al., 2024).

1. Formal framework

TrackStar is formulated against the classical influence-function background in which the effect of a training item $z$ on a query $q$ is approximated by

$$I(z \to q) \approx -\nabla_\theta \ell(q;\theta)^\top H^{-1}\nabla_\theta \ell(z;\theta),$$

with $H$ the Hessian of the empirical risk. Bergson describes TrackStar as preserving this inverse-Hessian–vector-product spirit while replacing the exact Hessian inverse with an empirical Fisher–based preconditioner computed in a compressed gradient space (Quirke et al., 10 Jun 2026).

The compression step is central. For matrix-valued parameters $W_\ell$, TrackStar applies a block-wise, double-sided random projection using Kronecker-factored Rademacher matrices. Bergson states the projection identity as

$$(A_\ell \otimes B_\ell)\,\mathrm{vec}\!\left(\frac{d\ell}{dW_\ell}\right) = \mathrm{vec}\!\left(B_\ell \left(\frac{d\ell}{dW_\ell}\right) A_\ell^\top\right),$$

and, using $g_\ell(z)=X_\ell G_\ell^\top$, gives the compressed per-layer gradient as

$$r_\ell(z)=\mathrm{vec}\!\left(B_\ell X_\ell (A_\ell G_\ell)^\top\right).$$

The model-level compressed gradient is then the concatenation over modules. Optional stabilizers include Adam-style second-moment normalization and unit-vector normalization after projection (Quirke et al., 10 Jun 2026).

A complementary formalization appears in the capability-provenance study on OLMo3-7B. There, the unpreconditioned TrackStar score for document $j$ and query $i$ is

$q$0

where $q$1 is the unit-normalized Rademacher-projected gradient at module $q$2. The reported preconditioned two-model score is

$q$3

with projected, unit-normalized, preconditioned gradients defined separately on the base-side corpus model and the instruct-side query model (Matlin et al., 17 Jun 2026). In both presentations, TrackStar is fundamentally a preconditioned similarity between query and training gradients.

2. Implementation in Bergson

Bergson provides the first open-source TrackStar implementation and treats it as a reusable pipeline for LLMs and massive training sets (Quirke et al., 10 Jun 2026). Its architecture centers on on-disk gradient stores, multi-node distributed execution, and a build/query separation that amortizes the expensive gradient-collection phase over many downstream attribution tasks. The library supports exact top-$q$4 retrieval via torch.topk, FAISS-based approximate nearest-neighbor retrieval, per-sequence or per-token attribution, and storage in chosen precision.

The implementation pipeline is explicit. In the build phase, Bergson samples projection matrices $q$5, streams the value dataset with length-aware batching, computes compressed gradients through backward hooks without materializing full parameter gradients, optionally normalizes them, and writes them to an on-disk store. It then forms empirical-Fisher autocorrelations in compressed space, supports a mixed Fisher

$q$6

and constructs a Tikhonov-regularized inverse

$q$7

In the query phase, the library computes compressed gradients for the query set, optionally aggregates them by mean or sum, and scores each stored value vector by a preconditioned inner product (Quirke et al., 10 Jun 2026).

Bergson emphasizes scale engineering. It supports DDP, FSDP2, and SimpleFSDP; uses adapter-space collection when PEFT or LoRA is available; and can stream scores on the fly instead of persisting the full store. The paper reports a gradient index size of approximately $q$8 KB per document and notes that, for $q$9 documents, the resulting store is approximately $$I(z \to q) \approx -\nabla_\theta \ell(q;\theta)^\top H^{-1}\nabla_\theta \ell(z;\theta),$$0 GB in the capability-provenance setup (Matlin et al., 17 Jun 2026). This design makes TrackStar practical not only for toy attribution settings but for corpus-scale inspection of pretraining data.

3. Corpus-level capability provenance

A major recent use of TrackStar is the study of capability provenance in OLMo3-7B, where the goal is not merely to rank individual documents but to identify which structured regions of the pretraining corpus support different capabilities (Matlin et al., 17 Jun 2026). The study applies TrackStar via Bergson to a stratified working set drawn from the de-duplicated Dolma3 mix. From the approximately $$I(z \to q) \approx -\nabla_\theta \ell(q;\theta)^\top H^{-1}\nabla_\theta \ell(z;\theta),$$1 billion Common Crawl and olmOCR documents that could be assigned WebOrganizer labels, the authors construct $$I(z \to q) \approx -\nabla_\theta \ell(q;\theta)^\top H^{-1}\nabla_\theta \ell(z;\theta),$$2 topic$$I(z \to q) \approx -\nabla_\theta \ell(q;\theta)^\top H^{-1}\nabla_\theta \ell(z;\theta),$$3format bins and sample up to $$I(z \to q) \approx -\nabla_\theta \ell(q;\theta)^\top H^{-1}\nabla_\theta \ell(z;\theta),$$4 documents per bin, yielding a working set of $$I(z \to q) \approx -\nabla_\theta \ell(q;\theta)^\top H^{-1}\nabla_\theta \ell(z;\theta),$$5 documents, or approximately $$I(z \to q) \approx -\nabla_\theta \ell(q;\theta)^\top H^{-1}\nabla_\theta \ell(z;\theta),$$6 billion tokens.

The key methodological move is aggregation before interpretation. For benchmark $$I(z \to q) \approx -\nabla_\theta \ell(q;\theta)^\top H^{-1}\nabla_\theta \ell(z;\theta),$$7 and bin $$I(z \to q) \approx -\nabla_\theta \ell(q;\theta)^\top H^{-1}\nabla_\theta \ell(z;\theta),$$8, the paper defines the bin-level mean influence as

$$I(z \to q) \approx -\nabla_\theta \ell(q;\theta)^\top H^{-1}\nabla_\theta \ell(z;\theta),$$9

These $H$0 bin means are then z-scored within benchmark, and benchmark contrasts are defined by $H$1. The design is a $H$2 contrast over domain and capability type: SocialIQA and MMLU Social Sciences for the social side, ARC-Challenge and MMLU STEM for the STEM side; SocialIQA and ARC-Challenge for reasoning, and the two MMLU slices for knowledge (Matlin et al., 17 Jun 2026).

The reported results show that social and STEM reasoning draw on qualitatively distinct corpus regions, and that the split is sharper at the reasoning level than at the knowledge level. SocialIQA is described as a structural outlier: its $H$3-bin influence profile correlates with the other benchmarks at $H$4, whereas the three comparison benchmarks correlate among themselves at $H$5 to $H$6. SocialIQA places its strongest positive mass in dialogic and interpersonal formats such as Customer Support, FAQ, and Q&A Forum, plus narrative topics such as Literature and Social Life, while ARC-Challenge and the MMLU variants are strongest in Documentation and Academic Writing and consistently negative on News Article. A particularly high-contrast cell is Literature $H$7 Customer Support, reported as $H$8 for SocialIQA and strongly negative for MMLU Social Sciences ($H$9) and MMLU STEM ($W_\ell$0) (Matlin et al., 17 Jun 2026).

The same study adds targeted machine unlearning as partial causal validation. Using NGDiff with LoRA on OLMo3-7B Base, it forgets high-attribution topic regions and measures accuracy damage

$W_\ell$1

For SocialIQA, influence-targeted forgetting degrades accuracy more than in-topic random forgetting, with a median paired difference of $W_\ell$2 and Wilcoxon BH-adjusted $W_\ell$3. The evidence is weaker or mixed for MMLU-STEM, MMLU Social Sciences, and ARC-Challenge (Matlin et al., 17 Jun 2026). This suggests that, in at least some settings, TrackStar’s aggregated attribution maps capture corpus regions with causal relevance to benchmark performance.

4. Wikipedia advocacy and topic-specific retrieval attribution

TrackStar also appears in a concrete case study of how a small set of Wikipedia edits can shape language-model behavior on a narrow topic (Brazilek et al., 30 Apr 2026). That study examines the Pro-Animal Wikipedians (PAW), who made $W_\ell$4 edits across $W_\ell$5 pages, and asks whether those edits measurably affect how models discuss animal welfare. For the TrackStar portion of the analysis, the authors use Bergson’s implementation on Llama 3.1 8B and construct a balanced, paired corpus of $W_\ell$6 Wikipedia sections drawn from $W_\ell$7 pages, comprising $W_\ell$8 animal-welfare (AW) sections and $W_\ell$9 within-article non-AW control sections. Queries are split into $$(A_\ell \otimes B_\ell)\,\mathrm{vec}\!\left(\frac{d\ell}{dW_\ell}\right) = \mathrm{vec}\!\left(B_\ell \left(\frac{d\ell}{dW_\ell}\right) A_\ell^\top\right),$$0 AW queries and $$(A_\ell \otimes B_\ell)\,\mathrm{vec}\!\left(\frac{d\ell}{dW_\ell}\right) = \mathrm{vec}\!\left(B_\ell \left(\frac{d\ell}{dW_\ell}\right) A_\ell^\top\right),$$1 general queries about the same entities.

In this study, TrackStar is defined operationally rather than by a closed-form score equation. The authors state that “TrackStar computes gradient similarity between training documents and queries to estimate retrieval-based attribution. For each query, it ranks every training document by how strongly the document’s training gradient aligns with the query’s gradient, producing a score that reflects semantic relevance as learned by the model” (Brazilek et al., 30 Apr 2026). The emphasis is therefore on retrieval-style gradient alignment rather than counterfactual removal.

The results show strong topic specificity. For AW queries, AW sections comprise $$(A_\ell \otimes B_\ell)\,\mathrm{vec}\!\left(\frac{d\ell}{dW_\ell}\right) = \mathrm{vec}\!\left(B_\ell \left(\frac{d\ell}{dW_\ell}\right) A_\ell^\top\right),$$2 of top-5 results, $$(A_\ell \otimes B_\ell)\,\mathrm{vec}\!\left(\frac{d\ell}{dW_\ell}\right) = \mathrm{vec}\!\left(B_\ell \left(\frac{d\ell}{dW_\ell}\right) A_\ell^\top\right),$$3 of top-10, and $$(A_\ell \otimes B_\ell)\,\mathrm{vec}\!\left(\frac{d\ell}{dW_\ell}\right) = \mathrm{vec}\!\left(B_\ell \left(\frac{d\ell}{dW_\ell}\right) A_\ell^\top\right),$$4 of top-15, all with $$(A_\ell \otimes B_\ell)\,\mathrm{vec}\!\left(\frac{d\ell}{dW_\ell}\right) = \mathrm{vec}\!\left(B_\ell \left(\frac{d\ell}{dW_\ell}\right) A_\ell^\top\right),$$5 against a $$(A_\ell \otimes B_\ell)\,\mathrm{vec}\!\left(\frac{d\ell}{dW_\ell}\right) = \mathrm{vec}\!\left(B_\ell \left(\frac{d\ell}{dW_\ell}\right) A_\ell^\top\right),$$6 baseline; in $$(A_\ell \otimes B_\ell)\,\mathrm{vec}\!\left(\frac{d\ell}{dW_\ell}\right) = \mathrm{vec}\!\left(B_\ell \left(\frac{d\ell}{dW_\ell}\right) A_\ell^\top\right),$$7 of $$(A_\ell \otimes B_\ell)\,\mathrm{vec}\!\left(\frac{d\ell}{dW_\ell}\right) = \mathrm{vec}\!\left(B_\ell \left(\frac{d\ell}{dW_\ell}\right) A_\ell^\top\right),$$8 queries, AW content dominates the top-5. Mean attribution scores are $$(A_\ell \otimes B_\ell)\,\mathrm{vec}\!\left(\frac{d\ell}{dW_\ell}\right) = \mathrm{vec}\!\left(B_\ell \left(\frac{d\ell}{dW_\ell}\right) A_\ell^\top\right),$$9 for AW sections and $g_\ell(z)=X_\ell G_\ell^\top$0 for control sections, with Mann–Whitney $g_\ell(z)=X_\ell G_\ell^\top$1, $g_\ell(z)=X_\ell G_\ell^\top$2. For general queries about the same entities, the corresponding top-$g_\ell(z)=X_\ell G_\ell^\top$3 shares—$g_\ell(z)=X_\ell G_\ell^\top$4, $g_\ell(z)=X_\ell G_\ell^\top$5, and $g_\ell(z)=X_\ell G_\ell^\top$6—do not significantly deviate from chance (Brazilek et al., 30 Apr 2026). The paper interprets this contrast as evidence that the model links PAW content specifically to animal welfare topics rather than to the entities in general.

The same paper triangulates TrackStar with MAGIC and a fine-tuning ablation. MAGIC counterfactual influence on Llama-3.2-1B yields an even sharper version of the same asymmetry: in every one of five random training-order seeds, the top-10 most influential documents for animal-welfare queries are all PAW edits, while general queries sit at chance. A separate fine-tuning ablation shows topic-selective perplexity gains: the PAW-trained model reduces perplexity on animal-welfare text from $g_\ell(z)=X_\ell G_\ell^\top$7 to $g_\ell(z)=X_\ell G_\ell^\top$8, while the control-trained model reduces perplexity on control text from $g_\ell(z)=X_\ell G_\ell^\top$9 to $$r_\ell(z)=\mathrm{vec}\!\left(B_\ell X_\ell (A_\ell G_\ell)^\top\right).$$0 (Brazilek et al., 30 Apr 2026). In this usage, TrackStar supplies the retrieval-style leg of a broader methodological triangulation.

5. Comparative position, strengths, and limitations

TrackStar occupies a specific point in the design space of data attribution methods. Relative to classical influence functions, it avoids explicit inverse-Hessian computation in the full parameter space. Relative to TracIn, it uses a single checkpoint and a preconditioned inner product in compressed space rather than summing raw gradient inner products across many checkpoints. Relative to more expensive methods such as MAGIC, it is retrieval-style and scalable rather than a direct counterfactual over the training process (Quirke et al., 10 Jun 2026, Brazilek et al., 30 Apr 2026).

The principal strength repeatedly emphasized in the literature is scalability. Bergson’s implementation supports billion-parameter models, on-disk stores, distributed sharding, and approximate nearest-neighbor querying; the capability-provenance study uses a $$r_\ell(z)=\mathrm{vec}\!\left(B_\ell X_\ell (A_\ell G_\ell)^\top\right).$$1 million-document working set; and the Wikipedia study explicitly notes TrackStar’s suitability for “billion-parameter models and very large corpora” (Quirke et al., 10 Jun 2026, Matlin et al., 17 Jun 2026, Brazilek et al., 30 Apr 2026). A second strength is interpretability under aggregation. The OLMo3 work argues that raw document-level scores are too noisy for capability discovery and that taxonomy-first aggregation is essential for obtaining stable, semantically named corpus regions (Matlin et al., 17 Jun 2026).

The main limitations are equally clear. Bergson notes the single-checkpoint approximation, sensitivity to projection hyperparameters, dependence on the empirical-Fisher surrogate, possible mismatch under out-of-distribution queries, and the granularity trade-off between per-token and per-sequence attribution (Quirke et al., 10 Jun 2026). The Wikipedia study adds a task-specific caution: TrackStar measures gradient similarity rather than causal removal effects or conversational outputs, so attribution should not be overread as a complete account of downstream chatbot behavior (Brazilek et al., 30 Apr 2026). This suggests that TrackStar is often strongest when used either as a scalable first-stage retrieval method or as one component in a triangulated analysis that includes counterfactual or behavioral validation.

6. Cross-domain reuse of the name

Outside training-data attribution, “TrackStar” is also used as a label in unrelated tracking systems. The usages below are technically distinct and should not be conflated with the Bergson method.

Paper	Domain	Use of “TrackStar”
(Sun et al., 2024)	Sports MOT	Base tracker refined by GTA
(Wang et al., 2022)	Sports MOT	Placeholder tracker that can adopt SportsTrack components
(Li et al., 7 Jan 2025)	ISAC/mmWave	“Tracking with STAR-RIS”
(Brandenburg et al., 2024)	STAR detector instrumentation	Forward tracking at STAR

In sports multi-object tracking, TrackStar appears as the initial online tracker to which GTA, an appearance-based global tracklet association stage, can be added as a plug-and-play refinement. GTA targets two failure modes—mix-up errors and cut-off errors—by splitting contaminated tracklets with DBSCAN over OSNet embeddings and then reconnecting fragments through a cosine-distance matrix with temporal exclusivity and sports-specific spatial constraints. The reported gains are substantial on SportsMOT and SoccerNet; for example, Deep-EIoU + GTA reaches HOTA $$r_\ell(z)=\mathrm{vec}\!\left(B_\ell X_\ell (A_\ell G_\ell)^\top\right).$$2 on SportsMOT and $$r_\ell(z)=\mathrm{vec}\!\left(B_\ell X_\ell (A_\ell G_\ell)^\top\right).$$3 on SoccerNet (Sun et al., 2024). A related sports-tracking paper uses TrackStar as the system into which SportsTrack’s components—YOLOX-based detection, three-stage Hungarian matching, crowded one-to-many correspondence, and edge-lost restoration—could be inserted wholesale (Wang et al., 2022).

In integrated sensing and communications, TrackStar is explicitly glossed as “tracking with STAR-RIS.” The system uses an active simultaneous transmission and reflection RIS to sense outdoor dynamic scatterers while providing indoor communication service. It combines STAR-RIS preamble sensing, path identification, a Gaussian mixture probability hypothesis density filter for multi-target tracking, and predictive beam design for the BS and RIS. In simulation, the scheme yields stable received SNR close to an oracle with perfect real-time CSI while reducing retraining overhead through mismatch detection and path-collision prediction (Li et al., 7 Jan 2025).

In STAR detector instrumentation, the term appears in connection with forward tracking rather than data attribution or wireless sensing. The relevant subsystem is the Forward Silicon Tracker, a three-disk, low-mass silicon strip detector covering $$r_\ell(z)=\mathrm{vec}\!\left(B_\ell X_\ell (A_\ell G_\ell)^\top\right).$$4 in a $$r_\ell(z)=\mathrm{vec}\!\left(B_\ell X_\ell (A_\ell G_\ell)^\top\right).$$5 T field and designed for charge-sign determination with transverse-momentum resolution better than $$r_\ell(z)=\mathrm{vec}\!\left(B_\ell X_\ell (A_\ell G_\ell)^\top\right).$$6 for $$r_\ell(z)=\mathrm{vec}\!\left(B_\ell X_\ell (A_\ell G_\ell)^\top\right).$$7 GeV/c. It uses Hamamatsu double-metal p-in-n strip sensors, Kapton-based hybrids, APV25-S1 readout, and NOVEC 7200 cooling at $$r_\ell(z)=\mathrm{vec}\!\left(B_\ell X_\ell (A_\ell G_\ell)^\top\right).$$8C (Brandenburg et al., 2024). This suggests that “TrackStar” functions in the recent literature as a reusable label attached to several technically independent tracking problems.