Which Models Are Our Models Built On? Auditing Invisible Dependencies in Modern LLMs

Abstract: Modern LLM training pipelines increasingly rely on other models to generate data, filter corpora, judge outputs, and guide development decisions. These dependencies are recursive: a model may depend on an upstream artifact whose own dependencies are documented only in separate releases and artifacts. As a result, the full dependency structure is fragmented across heterogeneous public artifacts, with complexity and recursive depth far outpacing humans' ability to trace. We introduce ModSleuth, an agentic system that recursively reconstructs LLM dependency graphs from public artifacts with source-grounded evidence. We find that the primary challenge is no longer information extraction, but defining what constitutes a dependency and reconciling artifact references across inconsistent documentation. We address these challenges through a formalization that distinguishes direct and indirect dependencies, represents heterogeneous pipeline roles through operation-centered relationships, and resolves artifact identities across names, versions, and repositories. Applying ModSleuth to four public-artifact-rich LLM releases, we recover 1,060 source-verified dependencies and construct large-scale dependency graphs of modern LLM development. These graphs reveal multi-hop license obligations, train-evaluation coupling, discrepancies between released and training-time artifacts, and documentation inconsistencies that would otherwise be difficult to uncover. We release ModSleuth and the resulting dependency graphs to support transparent analysis of the increasingly complex ecosystems underlying modern LLMs.

• The paper introduces ModSleuth, a novel system that audits LLMs by recursively tracing direct and indirect dependencies from diverse public artifacts.
• It formalizes dependency relationships using an identity lattice and an operation-centered approach to capture multi-hop interconnections in LLM pipelines.
• Quantitative evaluation reveals over 1,060 verified dependencies and highlights risks such as license propagation, contamination, and structural biases.

Auditing Invisible Dependencies in Modern LLMs: The ModSleuth System

Introduction and Context

The landscape of LLM development has rapidly evolved from straightforward web-scale pretraining and curated human annotation to intricate, multi-stage pipelines. These pipelines are now characterized by pervasive and recursive utilization of upstream models—not only for weight initialization, but for data generation, rewriting, filtering, evaluation, synthetic supervision, and meta-evaluation tasks. As a result, contemporary LLMs are not built in isolation; rather, they are now deeply embedded in a tangled ecosystem of models, datasets, and iterative processes. This paradigm shift introduces pressing concerns around provenance, license and terms-of-use compliance, contamination, evaluation-leakage, and broader governance challenges.

Despite their prevalence, these dependencies are often obscure, fragmented across varying public artifacts such as model cards, code repositories, technical reports, and dataset cards. Existing documentation and audit mechanisms (e.g., model/datasheets, ecosystem maps, parameter lineage inference) remain insufficiently expressive and often too “flat” to capture the multi-hop, operation-specific, and ambiguous nature of modern dependency relationships.

ModSleuth: Formalizing and Auditing Model Dependencies

Defining Recursive Dependency Tracing

ModSleuth addresses the core challenge of reconstructing LLM dependency graphs grounded in public evidence. At its foundation, it introduces rigorous formalisms to delineate and classify dependencies:

• Direct dependencies: Upstream artifacts that materially affect a target model’s weights or training data (e.g., through initialization, synthetic data generation, OCR, filtering). These are recursively defined—a direct dependency of a direct dependency is also classified as such.
• Indirect dependencies: Artifacts that inform development but do not directly impact training, such as evaluation judges, ablation variants, or explicit methodological adoptions.

Crucially, ModSleuth goes beyond the restrictive taxonomies of prior work by introducing operation-centered relationships and an open-vocabulary policy for operation types, preserving the fine-grained and unique character of real-world pipelines.

Artifact identities are managed via an identity lattice data structure, which supports resolution at varying levels of specificity (e.g., model family, checkpoint, stage). This design is essential due to inconsistent references across public sources and the frequent ambiguity of artifact names in code, docs, and cards.

ModSleuth System Architecture and Pipeline

ModSleuth operationalizes its approach via an agentic, staged pipeline leveraging contemporary agent systems (notably Claude Code):

1. Source Gathering: Collects all official public artifacts relevant to a target model—including technical reports, model/data cards, code repositories, and release blogs—forming the initial evidence set.
2. Entity Discovery and Resolution: Extracts mentions of upstream artifacts and resolves them into the identity lattice, capturing both specificity and residual ambiguity.
3. Dependency Construction and Reconciliation: Parses and maps dependencies as evidence-grounded edges, representing structured operations and merging or flagging inconsistent relationships across sources.
4. Recursive Expansion: Traces newly found upstream artifacts in the same manner, supporting BFS, DFS, or beam expansion modes, thereby recovering transitive multi-hop dependencies and building a unified, entity-resolved dependency graph.

   Figure 1: The ModSleuth pipeline extracts, resolves, and recursively traces dependencies from heterogeneous public artifacts to construct a unified, evidence-backed graph.

Quantitative Evaluation and Outcomes

The effectiveness of ModSleuth is evaluated against automated baselines including iterative prompting of GPT-5.4/5.5 Pro, Claude Code (single prompt), and ChatGPT Deep Research. Using four LLM releases with rich public documentation (Olmo 3, Nemotron 3 Super, DR Tulu, SmolLM3), the system’s recovery is assessed in three scopes: direct (depth-1), unbounded recursive, and BFS reachability.

Key findings include:

• ModSleuth produces markedly more verified dependencies compared to all baselines—over 1,060 verified relationships in unbounded mode, versus a maximum of 314 recovered with any baseline agent (see main text).
• Depth of recursion: Maximum observed dependency-chain depth reaches eight hops, indicating substantial transitive complexity.
• Scale: The global graph constructed from four targets includes 2,526 artifacts, 9,112 edges, and over 36,000 supporting evidence anchors.
• Role Type Breakdown: Direct dependencies (e.g., synthetic data generation, filtering, transformations) account for 72%, whereas indirect dependencies (evaluation, methodology) comprise 28%.
• Ecosystem Interconnectivity: For each model analyzed, over 75% of dependencies traced to external organizations, underlining the extent of cross-organizational entanglement.

Qualitative Audit Findings

By querying and analyzing the dependency graph, ModSleuth surfaces a range of audit findings that are otherwise inaccessible:

• Multi-hop hidden dependencies: Upstream models often exert influence via intermediate datasets, teacher models, or filters invisible to manual audit or standard reporting. For instance, DR Tulu’s SFT traces back through ScholarQA to Claude Sonnet 3.7; SmolLM3’s FineMath is indirectly shaped by Llama-licensed artifacts.

  Figure 2: Examples of complex multi-hop dependencies ModSleuth uncovers: SFT on DR Tulu depends recursively on Claude Sonnet; SmolLM3’s FineMath data pipeline traces to Llama artifacts; Olmo 3 both trains and evaluates on IFEval-derived data, with Qwen3 32B mediating both generation and evaluation.

• Structural train/eval coupling: Benchmark datasets feature both as training sources and evaluation targets, raising contamination and circularity concerns. For example, Olmo 3 leverages IFEval as both a data constraint source and benchmark.
• License pathway risks: Multi-hop synthetic data pipelines can silently propagate restrictive license terms (such as the Llama license prohibiting improvement of non-Meta models), which are not discoverable by inspecting released model licenses alone.
• Model-mediated data selection: Generator, filter, and judge roles are disproportionately concentrated in a small set of foundation models (notably Qwen, Llama, and GPT), whose synthetic outputs substantially steer the training content and evaluation pipelines of downstream releases.
• Code-level provenance and documentation: Many dependencies are only visible via config files, data-prep scripts, or YAMLs, revealing non-obvious influences and highlighting limitations in current documentation practices.
• Compliance and hygiene strategies: Examples identified include deliberate use of more permissive teachers in data generation, explicit filtering to avoid contaminating training data with evaluation benchmarks, and fine-grained decontamination documented in code rather than cards.

Implications and Future Directions

This work implies that programmatic, evidence-grounded auditing is now essential for transparency in LLM provenance. The findings underscore the inadequacy of “flat” documentation schemas and parameter-based ancestry inference for capturing the modern recursion and operational diversity of LLM pipelines.

Practical Implications:

• Model and dataset licenses, contamination risk, and provenance must be interpreted recursively—moving beyond surface-level inspection to multi-hop, operation-resolved graph tracing.
• Synthetic data and model-based evaluation are vehicles for structural bias, potential license violations, and unintentional “model collapse,” especially when upstream dependencies become dominant or cyclic.

Theoretical Implications and Future Development:

• Future governance, dataset curation, and model release processes will require machine-understandable, graph-centric disclosure and audit schemas.
• Fully tracing “true” (as opposed to reported) dependencies demands new research in behavioral inference, black-box testing, and potential access to internal data previously considered non-public.
• The ecosystem-level dominance of certain foundation model families in judge and generator roles signals both a consolidation of influence and a potential monoculture risk.

Conclusion

ModSleuth formalizes recursive LLM dependency tracing as a new audit task, providing the first scalable, agentic system capable of constructing evidence-backed, multi-hop dependency graphs from heterogeneous public artifacts. By operationalizing rigorous definitions of direct and indirect dependencies, operation-centered relationship modeling, and identity resolution, ModSleuth enables structured, query-driven audit and surfaces systemic risks hitherto inaccessible to manual or single-artifact methods. The results suggest that auditing and transparency mechanisms must evolve rapidly to keep pace with the accelerating complexity and recursion of the LLM ecosystem, and that explicit graph-centric disclosure will soon be essential for regulatory, scientific, and operational trust in foundation models.