PhyloLM : Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks (2404.04671v3)

Abstract: This paper introduces PhyloLM, a method adapting phylogenetic algorithms to LLMs to explore whether and how they relate to each other and to predict their performance characteristics. Our method calculates a phylogenetic distance metrics based on the similarity of LLMs' output. The resulting metric is then used to construct dendrograms, which satisfactorily capture known relationships across a set of 111 open-source and 45 closed models. Furthermore, our phylogenetic distance predicts performance in standard benchmarks, thus demonstrating its functional validity and paving the way for a time and cost-effective estimation of LLM capabilities. To sum up, by translating population genetic concepts to machine learning, we propose and validate a tool to evaluate LLM development, relationships and capabilities, even in the absence of transparent training information.

• The paper introduces PhyloLM as a novel method applying phylogenetic algorithms to analyze fine-tuning relationships among LLMs and predict benchmark results.
• It utilizes dendrograms and phylogenetic distance metrics to systematically map the lineage of 99 LLMs, highlighting correlations between model ancestry and performance.
• The approach offers a cost-effective alternative to exhaustive benchmarking and opens new avenues for refining LLM taxonomy and AI evaluation methods.

Inferring the Phylogeny of LLMs and Predicting their Performances in Benchmarks

The paper presents a novel methodology, PhyloLM, that draws on principles from population genetics to examine the development and capabilities of LLMs. This approach utilizes phylogenetic algorithms to construct dendrograms that map out finetuning relationships among 77 open-source and 22 closed LLMs, further predicting their performance on benchmarks like MMLU and ARC. Through the phylogenetic distance metric, the method offers a cost-effective alternative for evaluating LLM capabilities, an asset particularly valuable given the opacity of training information and the resources required for extensive benchmarking.

Core Contributions

Several contributions distinguish this research. Firstly, the introduction of PhyloLM represents the application of a simplified phylogenetic algorithm to the LLM domain, establishing a conceptual bridge between machine learning and evolutionary genetics. Secondly, the paper meticulously explores the hyperparameter space, optimizing the tradeoff between prediction precision and computational efficiency. Results indicate that precision is less sensitive to genome size and number of token probes than previously assumed, elucidating cost-effective routes for model evaluation.

Additionally, the paper explores correlations between phylogenetic distance and LLM family origins, demonstrating alignment, particularly among open-access models where training information is more transparent. This framework extends beyond transparency, providing unique insights into performance and capabilities of proprietary models such as GPT-3.5 and GPT-4 families, despite restricted access to their training data.

Implications of the Research

The implications of this research are multi-faceted. From a practical standpoint, the ability to predict model performances without exhaustive benchmarks offers significant cost reductions for both academic and industrial actors in AI development. It also enables more informed selection processes for AI applications, optimized by understanding underlying model capabilities without needing direct performance evaluations on every new dataset or application domain.

Theoretically, transferring genetic concepts to AI enriches the analytical toolkit available for studying LLMs. It opens new pathways for understanding the interconnectedness between models, potentially leading to a refined taxonomy of LLMs that accounts for both architectural similarities and performance expectations. This methodological innovation could spur further research into other biological algorithms that might offer analogous insights within computer science.

Future Developments in AI

Looking forward, the PhyloLM approach may inspire the development of new algorithms extending beyond LLMs, including potential applications in convolutional neural networks and other AI systems which similarly lack transparent training documentation. There's also a path toward enhanced algorithmic refinement where genetic analogies could be adopted to visually map out evolutionary pathways of computational models, providing clearer lineage tracking within the swiftly advancing field of AI.

Further efforts could aim to refine the model's robustness, particularly across diverse genomes and broader AI model families. Additionally, the approach could be modified to predict non-benchmarkable capabilities such as creativity in language generation or adherence to ethical guidelines. This flexibility emphasizes the broad applicability of PhyloLM, highlighting its potential contribution to evolving AI governance and policy frameworks.

In conclusion, this paper contributes a valuable intersection of genetic methodologies and machine learning, offering both practical tools for evaluating LLMs and theoretical insights into their development. It lays the groundwork for continued exploration and application of biological analogies in AI research, representing an innovative step towards more holistic understanding and management of artificial intelligence capabilities.