--- title: 'TrackStar: Scalable Training-Data Attribution' url: https://www.emergentmind.com/topics/trackstar type: topic --- # TrackStar: Scalable Training-Data Attribution TrackStar is a scalable, gradient-based training-data attribution method for estimating how strongly individual training examples support or suppress a model’s behavior at a fixed checkpoint. In the current literature, it is most fully specified through the open-source Bergson library, which implements TrackStar as a single-checkpoint, influence-function–style method based on compressed per-module gradients, empirical-Fisher preconditioning, and optional gradient normalization [2606.11660]. Recent studies have used it to map capability provenance in OLMo3-7B and to trace how coordinated Wikipedia edits by the Pro-Animal Wikipedians shape Llama-family models’ handling of animal welfare topics [2606.19625][2606.24890]. The label “TrackStar” also appears in unrelated technical contexts, including sports multi-object tracking, STAR-RIS-aided ISAC, and forward tracking at STAR, so the term is not globally univocal across research domains [2411.08216][2501.03793][2407.09952]. ## 1. Formal framework TrackStar is formulated against the classical influence-function background in which the effect of a training item \(z\) on a query \(q\) is approximated by $$ I(z \to q) \approx -\nabla_\theta \ell(q;\theta)^\top H^{-1}\nabla_\theta \ell(z;\theta), $$ with \(H\) the Hessian of the empirical risk. Bergson describes TrackStar as preserving this inverse-Hessian–vector-product spirit while replacing the exact Hessian inverse with an empirical Fisher–based preconditioner computed in a compressed gradient space [2606.11660]. The compression step is central. For matrix-valued parameters \(W_\ell\), TrackStar applies a block-wise, double-sided random projection using Kronecker-factored Rademacher matrices. Bergson states the projection identity as $$ (A_\ell \otimes B_\ell)\,\mathrm{vec}\!\left(\frac{d\ell}{dW_\ell}\right) = \mathrm{vec}\!\left(B_\ell \left(\frac{d\ell}{dW_\ell}\right) A_\ell^\top\right), $$ and, using \(g_\ell(z)=X_\ell G_\ell^\top\), gives the compressed per-layer gradient as $$ r_\ell(z)=\mathrm{vec}\!\left(B_\ell X_\ell (A_\ell G_\ell)^\top\right). $$ The model-level compressed gradient is then the concatenation over modules. Optional stabilizers include Adam-style second-moment normalization and unit-vector normalization after projection [2606.11660]. A complementary formalization appears in the capability-provenance study on OLMo3-7B. There, the unpreconditioned TrackStar score for document \(j\) and query \(i\) is $$ s(j,i)=\sum_{k\in\mathcal K}\hat g_k(x_j)^\top \hat g_k(q_i), $$ where \(\hat g_k(\cdot)\) is the unit-normalized Rademacher-projected gradient at module \(k\). The reported preconditioned two-model score is $$ s(j,i)=\sum_k \tilde g^{\mathrm{base}}_k(x_j)^\top \tilde g^{\mathrm{inst}}_k(q_i), $$ with projected, unit-normalized, preconditioned gradients defined separately on the base-side corpus model and the instruct-side query model [2606.19625]. In both presentations, TrackStar is fundamentally a preconditioned similarity between query and training gradients. ## 2. Implementation in Bergson Bergson provides the first open-source TrackStar implementation and treats it as a reusable pipeline for large language models and massive training sets [2606.11660]. Its architecture centers on on-disk gradient stores, multi-node distributed execution, and a build/query separation that amortizes the expensive gradient-collection phase over many downstream attribution tasks. The library supports exact top-\(k\) retrieval via `torch.topk`, FAISS-based approximate nearest-neighbor retrieval, per-sequence or per-token attribution, and storage in chosen precision. The implementation pipeline is explicit. In the build phase, Bergson samples projection matrices \(\{A_\ell,B_\ell\}\), streams the value dataset with length-aware batching, computes compressed gradients through backward hooks without materializing full parameter gradients, optionally normalizes them, and writes them to an on-disk store. It then forms empirical-Fisher autocorrelations in compressed space, supports a mixed Fisher $$ \hat F_{\mathrm{mix}}(\alpha)=\alpha \hat F_{\mathrm{value}} + (1-\alpha)\hat F_{\mathrm{query}}, $$ and constructs a Tikhonov-regularized inverse $$ W_{\alpha,\lambda}=(\hat F_{\mathrm{mix}}(\alpha)+\lambda I)^{-1}. $$ In the query phase, the library computes compressed gradients for the query set, optionally aggregates them by mean or sum, and scores each stored value vector by a preconditioned inner product [2606.11660]. Bergson emphasizes scale engineering. It supports DDP, FSDP2, and SimpleFSDP; uses adapter-space collection when PEFT or LoRA is available; and can stream scores on the fly instead of persisting the full store. The paper reports a gradient index size of approximately \(8\) KB per document and notes that, for \(5{,}678{,}621\) documents, the resulting store is approximately \(44\) GB in the capability-provenance setup [2606.19625]. This design makes TrackStar practical not only for toy attribution settings but for corpus-scale inspection of pretraining data. ## 3. Corpus-level capability provenance A major recent use of TrackStar is the study of capability provenance in OLMo3-7B, where the goal is not merely to rank individual documents but to identify which structured regions of the pretraining corpus support different capabilities [2606.19625]. The study applies TrackStar via Bergson to a stratified working set drawn from the de-duplicated Dolma3 mix. From the approximately \(1.10\) billion Common Crawl and olmOCR documents that could be assigned WebOrganizer labels, the authors construct \(576\) topic\(\times\)format bins and sample up to \(10{,}000\) documents per bin, yielding a working set of \(5{,}678{,}621\) documents, or approximately \(10.5\) billion tokens. The key methodological move is aggregation before interpretation. For benchmark \(\beta\) and bin \(k\), the paper defines the bin-level mean influence as $$ \bar s_{\beta,k} = \frac{1}{|D_k|} \sum_{j\in D_k} \frac{1}{|Q_\beta|} \sum_{q\in Q_\beta} s(j,q). $$ These \(576\) bin means are then z-scored within benchmark, and benchmark contrasts are defined by \(\Delta z_k = z_\beta(k)-z_{\beta'}(k)\). The design is a \(2\times 2\) contrast over domain and capability type: SocialIQA and MMLU Social Sciences for the social side, ARC-Challenge and MMLU STEM for the STEM side; SocialIQA and ARC-Challenge for reasoning, and the two MMLU slices for knowledge [2606.19625]. The reported results show that social and STEM reasoning draw on qualitatively distinct corpus regions, and that the split is sharper at the reasoning level than at the knowledge level. SocialIQA is described as a structural outlier: its \(576\)-bin influence profile correlates with the other benchmarks at \(r \le 0.22\), whereas the three comparison benchmarks correlate among themselves at \(r=0.53\) to \(0.86\). SocialIQA places its strongest positive mass in dialogic and interpersonal formats such as Customer Support, FAQ, and Q\&A Forum, plus narrative topics such as Literature and Social Life, while ARC-Challenge and the MMLU variants are strongest in Documentation and Academic Writing and consistently negative on News Article. A particularly high-contrast cell is Literature \(\times\) Customer Support, reported as \(+16.0\) for SocialIQA and strongly negative for MMLU Social Sciences (\(-7.31\)) and MMLU STEM (\(-5.75\)) [2606.19625]. The same study adds targeted machine unlearning as partial causal validation. Using NGDiff with LoRA on OLMo3-7B Base, it forgets high-attribution topic regions and measures accuracy damage $$ \gamma = A_{\mathrm{baseline}} - A_{\mathrm{unlearned}}. $$ For SocialIQA, influence-targeted forgetting degrades accuracy more than in-topic random forgetting, with a median paired difference of \(+0.0160\) and Wilcoxon BH-adjusted \(p \approx 1.0\times 10^{-5}\). The evidence is weaker or mixed for MMLU-STEM, MMLU Social Sciences, and ARC-Challenge [2606.19625]. This suggests that, in at least some settings, TrackStar’s aggregated attribution maps capture corpus regions with causal relevance to benchmark performance. ## 4. Wikipedia advocacy and topic-specific retrieval attribution TrackStar also appears in a concrete case study of how a small set of Wikipedia edits can shape language-model behavior on a narrow topic [2606.24890]. That study examines the Pro-Animal Wikipedians (PAW), who made \(125\) edits across \(115\) pages, and asks whether those edits measurably affect how models discuss animal welfare. For the TrackStar portion of the analysis, the authors use Bergson’s implementation on Llama 3.1 8B and construct a balanced, paired corpus of \(72\) Wikipedia sections drawn from \(31\) pages, comprising \(36\) animal-welfare (AW) sections and \(36\) within-article non-AW control sections. Queries are split into \(80\) AW queries and \(90\) general queries about the same entities. In this study, TrackStar is defined operationally rather than by a closed-form score equation. The authors state that “TrackStar computes gradient similarity between training documents and queries to estimate retrieval-based attribution. For each query, it ranks every training document by how strongly the document’s training gradient aligns with the query’s gradient, producing a score that reflects semantic relevance as learned by the model” [2606.24890]. The emphasis is therefore on retrieval-style gradient alignment rather than counterfactual removal. The results show strong topic specificity. For AW queries, AW sections comprise \(68.0\%\) of top-5 results, \(66.6\%\) of top-10, and \(63.7\%\) of top-15, all with \(p<0.0001\) against a \(50\%\) baseline; in \(64\) of \(80\) queries, AW content dominates the top-5. Mean attribution scores are \(0.549\) for AW sections and \(0.444\) for control sections, with Mann–Whitney \(U\), \(p<0.0001\). For general queries about the same entities, the corresponding top-\(k\) shares—\(51.7\%\), \(49.4\%\), and \(48.3\%\)—do not significantly deviate from chance [2606.24890]. The paper interprets this contrast as evidence that the model links PAW content specifically to animal welfare topics rather than to the entities in general. The same paper triangulates TrackStar with MAGIC and a fine-tuning ablation. MAGIC counterfactual influence on Llama-3.2-1B yields an even sharper version of the same asymmetry: in every one of five random training-order seeds, the top-10 most influential documents for animal-welfare queries are all PAW edits, while general queries sit at chance. A separate fine-tuning ablation shows topic-selective perplexity gains: the PAW-trained model reduces perplexity on animal-welfare text from \(12.4\) to \(8.4\), while the control-trained model reduces perplexity on control text from \(16.1\) to \(11.4\) [2606.24890]. In this usage, TrackStar supplies the retrieval-style leg of a broader methodological triangulation. ## 5. Comparative position, strengths, and limitations TrackStar occupies a specific point in the design space of data attribution methods. Relative to classical influence functions, it avoids explicit inverse-Hessian computation in the full parameter space. Relative to TracIn, it uses a single checkpoint and a preconditioned inner product in compressed space rather than summing raw gradient inner products across many checkpoints. Relative to more expensive methods such as MAGIC, it is retrieval-style and scalable rather than a direct counterfactual over the training process [2606.11660][2606.24890]. The principal strength repeatedly emphasized in the literature is scalability. Bergson’s implementation supports billion-parameter models, on-disk stores, distributed sharding, and approximate nearest-neighbor querying; the capability-provenance study uses a \(5.68\) million-document working set; and the Wikipedia study explicitly notes TrackStar’s suitability for “billion-parameter models and very large corpora” [2606.11660][2606.19625][2606.24890]. A second strength is interpretability under aggregation. The OLMo3 work argues that raw document-level scores are too noisy for capability discovery and that taxonomy-first aggregation is essential for obtaining stable, semantically named corpus regions [2606.19625]. The main limitations are equally clear. Bergson notes the single-checkpoint approximation, sensitivity to projection hyperparameters, dependence on the empirical-Fisher surrogate, possible mismatch under out-of-distribution queries, and the granularity trade-off between per-token and per-sequence attribution [2606.11660]. The Wikipedia study adds a task-specific caution: TrackStar measures gradient similarity rather than causal removal effects or conversational outputs, so attribution should not be overread as a complete account of downstream chatbot behavior [2606.24890]. This suggests that TrackStar is often strongest when used either as a scalable first-stage retrieval method or as one component in a triangulated analysis that includes counterfactual or behavioral validation. ## 6. Cross-domain reuse of the name Outside training-data attribution, “TrackStar” is also used as a label in unrelated tracking systems. The usages below are technically distinct and should not be conflated with the Bergson method. | Paper | Domain | Use of “TrackStar” | |---|---|---| | [2411.08216] | Sports MOT | Base tracker refined by GTA | | [2211.07173] | Sports MOT | Placeholder tracker that can adopt SportsTrack components | | [2501.03793] | ISAC/mmWave | “Tracking with STAR-RIS” | | [2407.09952] | STAR detector instrumentation | Forward tracking at STAR | In sports multi-object tracking, TrackStar appears as the initial online tracker to which GTA, an appearance-based global tracklet association stage, can be added as a plug-and-play refinement. GTA targets two failure modes—mix-up errors and cut-off errors—by splitting contaminated tracklets with DBSCAN over OSNet embeddings and then reconnecting fragments through a cosine-distance matrix with temporal exclusivity and sports-specific spatial constraints. The reported gains are substantial on SportsMOT and SoccerNet; for example, Deep-EIoU + GTA reaches HOTA \(81.04\) on SportsMOT and \(83.11\) on SoccerNet [2411.08216]. A related sports-tracking paper uses TrackStar as the system into which SportsTrack’s components—YOLOX-based detection, three-stage Hungarian matching, crowded one-to-many correspondence, and edge-lost restoration—could be inserted wholesale [2211.07173]. In integrated sensing and communications, TrackStar is explicitly glossed as “tracking with STAR-RIS.” The system uses an active simultaneous transmission and reflection RIS to sense outdoor dynamic scatterers while providing indoor communication service. It combines STAR-RIS preamble sensing, path identification, a Gaussian mixture probability hypothesis density filter for multi-target tracking, and predictive beam design for the BS and RIS. In simulation, the scheme yields stable received SNR close to an oracle with perfect real-time CSI while reducing retraining overhead through mismatch detection and path-collision prediction [2501.03793]. In STAR detector instrumentation, the term appears in connection with forward tracking rather than data attribution or wireless sensing. The relevant subsystem is the Forward Silicon Tracker, a three-disk, low-mass silicon strip detector covering \(2.5<\eta<4\) in a \(0.5\) T field and designed for charge-sign determination with transverse-momentum resolution better than \(30\%\) for \(0.2