Towards Imputation of Pre-Trained Language Model Metadata using Semantic Fingerprinting

Abstract: Pre-trained LLMs (PTLMs) hosted on platforms such as Hugging Face form complex lineage structures similar to software dependency graphs. However, unlike traditional software ecosystems, PTLM repositories often lack reliable provenance due to missing metadata, such as licenses, reuse methods, pipeline tags, model types, and training libraries. To address this gap, we introduce Semantic Fingerprinting (SemFin), a lightweight approach that combines Hugging Face (HF) configuration files with model repository tags to automatically impute missing model metadata fields and reconstruct model lineage chains. We evaluate SemFin on a large-scale dataset of 317,133 PTLMs. Our results show that configuration files typically encode the technical requirements necessary to instantiate and reuse models, enabling them to serve as a structural blueprint for model reuse, particularly for transformer-based architectures. By combining these configuration files with model repository tags, SemFin significantly outperforms the existing propagation-based imputation approaches, improving prediction accuracy by up to 31.4% and 26.6% compared to Graph Avg and Hub Avg baselines. Importantly, SemFin also imputes metadata for 16.6% of isolated models where propagation-based methods fail. Applying SemFin to impute missing reuse-method and license metadata for 167,089 unlabeled models reveals that traceable reuse method chains expand by 75.9% and license lineage chains by 53.6%, uncovering 86 previously invisible reuse method patterns, while the proportion of incompatible license patterns only increases from 34.8% to 36.8%. These findings demonstrate how automatically derived structural signals can support the automated construction of AI Bills of Materials (AIBOMs), helping transform metadata from an error-prone manual declaration into information inferred directly from model artifacts.

• The paper introduces Semantic Fingerprinting, which leverages configuration files and repository tags to accurately recover missing PTLM metadata with superior performance.
• It employs LightGBM and rigorous feature selection to predict key metadata fields, achieving near-perfect accuracy and full coverage for valid models.
• Empirical analysis reveals significant improvements in lineage recovery and license tracking, thereby reducing transparency debt in the AI model ecosystem.

Semantic Fingerprinting for Metadata Imputation in PTLM Repositories

Introduction

The accelerating growth of pre-trained LLMs (PTLMs), especially within centralized hubs such as Hugging Face, has led to an ecosystem characterized by intricate model lineages and widespread model reuse. However, unlike established software repositories equipped with automated manifest tracking, PTLM repositories rely predominantly on manual, frequently incomplete, or inconsistent metadata declarations. This results in substantial gaps in provenance, licensing, and reuse method metadata—collectively termed "transparency debt"—and severely hampers efforts to construct verifiable, AI-focused supply chain records such as AI Bills of Materials (AIBOMs).

This paper introduces Semantic Fingerprinting (SemFin), an artifact-driven approach to impute missing PTLM metadata by leveraging the technical signals present in model configuration files and repository tags. The method is empirically validated against state-of-the-art graph and hub-based baseline methods on a dataset comprising 317,133 PTLMs, revealing substantially improved performance, superior coverage, and deeper insights into the structural complexity of PTLM lineages.

Methodological Overview

SemFin constructs semantic fingerprints for each model by combining flattened, normalized configuration keys extracted from Hugging Face–compatible config.json files with the most prevalent repository tags. By carefully curating and filtering features—removing shared architectural "noise" and focusing on discriminative configuration keys—SemFin isolates lightweight signals predictive of missing target metadata.

A suite of machine learning models, primarily based on LightGBM, is employed to predict five core metadata fields: license, reuse method, pipeline tag, model type, and training library. The training pipeline includes robust cross-validation, strict token-based domain leakage prevention, and aggressive feature space reduction to ensure high generalizability and resistance to overfitting.

Comparison is made with state-of-the-art propagation-based imputation methods (Graph Avg and Hub Avg) that infer missing metadata via neighbor voting over explicit parent-child links. By contrast, SemFin's reliance on intrinsic model artifacts enables imputation regardless of graph connectivity and ensures 100% coverage for any model encapsulating a valid configuration.

Empirical Analysis of Configuration and Tag Evolution

Configuration files serve as both stable architectural cores and mutable blueprints for model specialization. The analysis of 150,951 parent-child pairs revealed that child models on Hugging Face inherit on average 98.2% of their parent’s configuration keys and 88.4% of repository tags, with targeted additions/removals distinguishing derivative models from unmodified copies. Key findings include:

• The presence or absence of specific configuration keys (e.g., quantization_config.bits, problem_type, dense_act_fn) is highly predictive of different reuse methods and tasks.
• A compact core of 68 invariant configuration keys is preserved across all reuse methods, while 324 keys exhibit significant change signals (value-level modifications, additions, removals).
• Each distinct reuse paradigm—Finetune, Quantization, Merging, PEFT, Distillation, Pruning, Deduplication—is characterized by a unique subspace of configuration keys, forming robust, method-specific fingerprints.

This structural inheritance and selective mutation validate the use of configuration key presence/absence for lightweight, architecture-agnostic metadata recovery.

SemFin: Design and Performance

The design of SemFin emphasizes:

• Noise reduction and strict feature selection tailored per reuse method to avoid long-tailed sparsity.
• TF-IDF vectorization and stratified sampling for classifier stability.
• Removal of any feature tokens matching target metadata to prevent trivial leakages.

Numerical Results:

• SemFin achieves near-perfect recovery for model_type (micro- and macro-accuracy ≥ 97%) and strong results for pipeline_tag (91–93%), consistently outperforming both Graph Avg and Hub Avg baselines, with accuracy improvements of up to 31.4% (reuse method), 22.8% (pipeline_tag), and 19.8% (license) over baselines.
• SemFin achieves 100% data coverage for models with valid configuration, solving coverage gaps left by graph-based approaches, which abstain when faced with sparsity or disconnected nodes.
• Statistical significance is demonstrated across all metadata fields (p < 0.001, large Cohen’s g), validating the methodological robustness of SemFin’s predictive advantage.

Impact of Metadata Imputation on Lineage and License Analysis

Application of SemFin to the unlabeled portion of the dataset fundamentally alters the revealed structure and complexity of the PTLM ecosystem:

• Lineage Recovery: Traceable reuse-method chains increase from 31,795 to 131,788 (a 4.1× increase), with unique reuse method patterns expanding from 76 to 155. 86 lineage patterns are shown to be completely invisible without automatic imputation.
• License Chains: License lineage chains increase from 60,965 to 131,356 after recovery, with unique license patterns growing from 250 to 419.
• Lineage Patterns: Single-step reuse patterns (e.g., Finetune, Quantization, Peft) overwhelmingly dominate after imputation, revealing that practitioners seldom compose complex, multi-step transformations.
• License Incompatibility: The proportion of incompatible license patterns remains relatively stable, increasing only modestly from 34.8% to 36.8%, with the majority of conflicts arising from non-commercial→commercial, AI-restricted→non-AI, and share-alike→other transitions.

These reconstructions demonstrate that missing metadata leads to "lineage mirages", distorting both the scale and risk profile of the ecosystem, and potentially obscuring critical legal risks.

Theoretical and Practical Implications

SemFin represents a significant methodological advance in automated provenance tracking for AI models, showing that configuration management artifacts—long established as foundational for software supply chain analysis—can serve an analogous role for PTLMs.

Practical Implications

• For Researchers: Enables ecosystem-scale, automated AIBOM generation and lineage analysis of provenance, reuse, and licensing, paving the way for future model composition analysis and bias propagation studies.
• For Platform Operators: Supports ingestion-time scanning to auto-populate metadata fields, detect inconsistencies, and audit supply chains with guarantees on coverage.
• For Practitioners: Provides instruments to track, audit, and ensure compliance for models across iterative reuse chains, reducing the risk of silent propagation of license/safety violations.

Theoretical Implications and Future Directions

• The empirical success of semantic fingerprinting motivates research into artifact-centric versioning, automated diffing of architectural evolution, and generalized model composition analysis analogous to Software Composition Analysis (SCA).
• There is a clear need for formal, machine-verifiable documentation standards for models (AIBOM), building not only on user-provided documentation but also directly on executable artifacts.
• As model composition (e.g., merging, cyclic reuse) grows in complexity, future research must consider non-linear, cyclic provenance graphs and provide platform-level support for structured, machine-readable model histories.

Conclusion

This work establishes semantic fingerprinting as a robust, scalable methodology for PTLM metadata imputation and lineage recovery, transforming incomplete, error-prone manual metadata into verifiable, artifact-derived provenance records. By demonstrating that configuration files and repository tags together encode sufficient signals for high-fidelity metadata recovery, the approach directly supports the development of next-generation supply chain governance, AIBOM standards, and automated ecosystem auditing in the era of increasingly complex model-based AI systems.