LLM DNA: Tracing Model Evolution via Functional Representations (2509.24496v1)

Abstract: The explosive growth of LLMs has created a vast but opaque landscape: millions of models exist, yet their evolutionary relationships through fine-tuning, distillation, or adaptation are often undocumented or unclear, complicating LLM management. Existing methods are limited by task specificity, fixed model sets, or strict assumptions about tokenizers or architectures. Inspired by biological DNA, we address these limitations by mathematically defining LLM DNA as a low-dimensional, bi-Lipschitz representation of functional behavior. We prove that LLM DNA satisfies inheritance and genetic determinism properties and establish the existence of DNA. Building on this theory, we derive a general, scalable, training-free pipeline for DNA extraction. In experiments across 305 LLMs, DNA aligns with prior studies on limited subsets and achieves superior or competitive performance on specific tasks. Beyond these tasks, DNA comparisons uncover previously undocumented relationships among LLMs. We further construct the evolutionary tree of LLMs using phylogenetic algorithms, which align with shifts from encoder-decoder to decoder-only architectures, reflect temporal progression, and reveal distinct evolutionary speeds across LLM families.

• The paper introduces a formal definition of LLM DNA as a bi-Lipschitz functional embedding that captures the evolutionary relationships among LLMs.
• It presents a scalable, training-free pipeline using semantic embeddings and random projection to extract DNA vectors for diverse models.
• Empirical validation shows that DNA-based classifiers effectively detect model provenance and enable robust phylogenetic analysis.

Formalizing and Extracting LLM DNA: A Functional Approach to Model Provenance and Evolution

Motivation and Problem Statement

The rapid expansion of the LLM ecosystem has resulted in a proliferation of models with complex, often undocumented, evolutionary relationships due to fine-tuning, distillation, and adaptation. This opacity impedes critical tasks such as safety auditing, model governance, and systematic model management. Existing approaches to model provenance and representation are limited by task specificity, dependence on fixed model sets, or architectural constraints, and do not provide a principled, generalizable notion of model "DNA" that is intrinsic, stable, and agnostic to model internals.

Mathematical Definition of LLM DNA

The paper introduces a formal, functional definition of LLM DNA as a low-dimensional, bi-Lipschitz embedding of a model's functional behavior. An LLM is modeled as a function $f: \mathcal{S}_m \to \mathcal{O}$, mapping bounded-length input sequences to output logits. The function space $\mathcal{F}$ is shown to be a Hilbert space under a weighted $L_2$ metric over the input space, enabling the use of geometric and functional analytic tools.

LLM DNA is defined as a mapping $\tau: \mathcal{F} \to \mathbb{R}^L$ that is bi-Lipschitz with respect to the functional distance $d_H$ and the Euclidean distance in DNA space. This ensures that functionally similar models have proximate DNA vectors and vice versa, formalizing the properties of inheritance (small functional changes yield small DNA changes) and genetic determinism (small DNA distance implies functional similarity).

Existence and Construction via Johnson-Lindenstrauss

The existence of such a DNA embedding is established via the Johnson-Lindenstrauss (JL) lemma, which guarantees that for any finite set of $K$ LLMs, a random linear projection into $L = O(\epsilon^{-2} \log K)$ dimensions preserves pairwise distances up to a distortion parameter $\epsilon$. The bi-Lipschitz constants $c_1, c_2$ are directly related to the allowed distortion, and the construction is probabilistically optimal among linear methods.

Practical DNA Extraction Pipeline

The practical extraction of LLM DNA faces two challenges: (1) the infeasibility of evaluating models over the full combinatorial input space, and (2) the need for a semantic, model-agnostic output representation. The proposed pipeline addresses these as follows:

• Semantic-Aware Output Representation: Each LLM response to a prompt is embedded using a sentence embedding model (e.g., Qwen3-Embedding-8B), ensuring semantic similarity is captured and the method is agnostic to model internals or tokenizer.
• Stochastic Functional Distance: Instead of the full input space, a representative set of prompts is sampled from diverse real-world datasets. The concatenated semantic embeddings of model responses to these prompts form a high-dimensional functional representation.
• Random Projection: A fixed random Gaussian projection is applied to the concatenated embeddings, yielding the final low-dimensional DNA vector.

  Figure 1: Visualization of LLM DNA extraction workflow.

This pipeline is training-free, scalable, and applicable to both open- and closed-source models, as it only requires access to model outputs.

Empirical Validation and Applications

Relationship Detection and Provenance

The DNA representation is validated on 305 LLMs spanning diverse architectures and organizations. DNA-based SVM classifiers achieve an AUC of 0.957 in distinguishing correlated (evolutionarily related) from independent model pairs, significantly outperforming baselines. Notably, DNA uncovers previously undocumented relationships, as confirmed by manual inspection of model documentation.

Figure 2: Left: DNA distribution of LLMs, showing clear separation between correlated and independent models. Right: Mantel test demonstrating high stability of DNA distances across disjoint prompt sets ($\text{Pearson-R}=0.7797$).

Model Routing

DNA vectors, when used as frozen representations in a model routing task, outperform task-specific learned embeddings (EmbedLLM) despite not being trained on the routing dataset, indicating the generality and robustness of the DNA representation.

Stability Across Datasets

A Mantel test comparing DNA distances computed from disjoint prompt sets yields a Pearson correlation of 0.78 ($p < 0.001$), demonstrating that the DNA structure is stable and largely independent of the specific prompt distribution used for extraction.

Phylogenetic Analysis

By applying the Neighbor-Joining algorithm to DNA distances, the paper constructs a phylogenetic tree of LLMs that recovers known architectural and temporal evolutionary patterns, such as the shift from encoder-decoder to decoder-only models and the progression within model families (e.g., Llama 2 to Llama 3). The tree also reveals differing evolutionary rates across families, with some (e.g., Qwen, Gemma) evolving more rapidly than others.

Implementation Considerations

• Prompt Selection: The quality and diversity of the sampled prompt set directly affect the fidelity of the DNA representation. Empirical results suggest that 600 prompts from diverse benchmarks suffice for stable extraction.
• Embedding Model Choice: The sentence embedding model should be robust and high-capacity to ensure semantic fidelity. The pipeline is agnostic to the specific embedding model, but consistency across models is critical.
• Projection Dimension: The DNA dimension $L$ trades off between computational efficiency and fidelity. Theoretical guidance from the JL lemma and empirical validation should inform the choice.
• Scalability: The pipeline is highly parallelizable and can be applied to large model collections with modest computational resources, as it does not require model retraining or access to internal weights.

Implications and Future Directions

The formalization and practical extraction of LLM DNA provide a principled, scalable solution to model provenance, relationship detection, and evolutionary analysis in the LLM ecosystem. The approach is robust to model heterogeneity and agnostic to internal details, enabling applications in safety auditing, license compliance, and multi-agent system design.

Theoretically, the bi-Lipschitz functional embedding framework opens avenues for further paper of model space geometry, functional diversity, and the dynamics of model evolution. Practically, DNA-based phylogenetic trees can inform model selection, ensemble construction, and the detection of unauthorized model derivatives or backdoor propagation.

Future work may explore adaptive prompt selection for more fine-grained DNA extraction, integration with watermarking and fingerprinting techniques, and extension to multimodal or non-textual generative models. The DNA framework also provides a foundation for systematic benchmarking and governance of the rapidly expanding LLM landscape.

Conclusion

This work establishes a rigorous, generalizable framework for representing and analyzing the functional "DNA" of LLMs, bridging theoretical guarantees with a practical, scalable extraction pipeline. The resulting DNA vectors enable robust detection of model relationships, stable provenance analysis, and the construction of meaningful evolutionary trees, providing a new foundation for the management and paper of large-scale model ecosystems.