Pre-Trained Models (PTMs): Integration & Challenges

• Pre-trained Models (PTMs) are large-scale ML models that capture versatile representations from massive datasets, supporting a wide range of AI applications.
• They are integrated into open-source projects using static and dynamic loading patterns, with dependency graphs and centrality metrics outlining their core and peripheral roles.
• Maintenance challenges include inconsistent versioning, fragmented updates, and non-standardized testing, spurring research into semantic versioning and automated monitoring methods.

Pre-trained Models (PTMs) are large-scale machine learning models, frequently transformer-based, whose parameters are learned from massive corpora to capture general-purpose representations applicable across a broad range of tasks. PTMs are foundational to modern AI and software engineering workflows: they enable rapid prototyping and strong downstream performance via fine-tuning, parameter-efficient adaptation, or prompt-based usage. In open-source ecosystems, PTMs, often accessed through model hubs such as Hugging Face or PyTorch Hub, are integrated as dependencies that are loaded, maintained, tested, and evolved throughout the software project lifecycle.

1. PTM Integration and Usage Patterns in Open-Source Software

PTMs are generally incorporated into Python projects through standardized APIs such as transformers.AutoModel.from_pretrained and torch.hub.load. Integration follows two patterns:

• Static loading: Model names and parameters are hard-coded in the source code.
• Dynamic loading: Model identifiers and parameters are read from configuration files or environment variables.

The role of a PTM within a project is characterized structurally within the file-dependency graph:

• Core functionality: Loader files with high PageRank centrality, directly supporting primary workflows.
• Peripheral/optional: Invoked by a subset of modules (low centrality), frequently for add-on features or scripts.
• Disconnected/illustrative: Present solely for tutorials or proofs-of-concept, unreachable from primary logic.

Dependency structures are mapped through static analysis, with each loader situated in the repository's directed file graph (edges from A→B for imports), allowing computation of centrality metrics that inform the strategic footprint of PTMs within the codebase.

2. Maintenance Workflows and Model Evolution

Maintenance of PTMs as dependencies in OSS is substantially less standardized than traditional software libraries.

• Versioning remains inconsistent: there is no broad adoption of semantic-versioning schemas; instead, "model family" relationships (e.g., Llama 2 → Llama 3) are established through parent–child links in model hubs such as Hugging Face.
• Update mechanisms: Any replacement of a load site with a model from the same family is counted as an update. Longitudinal, commit-by-commit repository mining tracks replacements and removals.
• Dependency management: Some projects pin PTM versions; others allow implicit indirection through external configuration. Automated static analysis locates all loader sites, and issue-mining scripts extract PTM-related error and maintenance tickets.

Automated tools include:

• Static analysis parsers using PeaTMOSS signatures for loader detection.
• Issue-mining via keyword-driven and snowball sampling methods.
• Coverage measurement using Coverage.py to determine whether test suites actually execute PTM-loading code.
• Statistical modeling (Kaplan–Meier survival analysis) to estimate "lifespans" of PTMs in repositories.

3. Testing and Evaluation Methodologies

PTM testing coverage in OSS is heterogenous:

• Detection of test frameworks: Projects are filtered for recognized frameworks (pytest, unittest, nose); only those with passing automated tests are analyzed further.
• Coverage analysis: Coverage.py is injected to collect fine-grained data on which files and specific load sites are exercised.
• Classification of tests: Manual inspection classifies tests as unit, integration, performance, or stress, with special note of handling non-deterministic model outputs (such as controlling seeds and setting assertion tolerances).

No standardized ML benchmark is prescribed; instead, the paper proposes qualitative inspection and manual labeling of test coverage and functional validation at PTM integration points.

4. PTM Lifecycle Stages and Associated Challenges

Adopting, integrating, deploying, and evolving PTMs in open-source projects introduces new risks at each stage:

1. Adoption: Selecting appropriate PTMs is hindered by inconsistent or incomplete metadata documentation (e.g., missing licensing or provenance).
2. Integration (Fine-tuning): Codebases frequently mix static and dynamic load patterns, producing fragile and fragmented configurations.
3. Deployment: Discrepancies between model binaries in local development and those published on hubs, and large artifact sizes, complicate shipping and updating applications.
4. Monitoring (Usage): There is generally no systematic monitoring for concept drift or alerting for performance regressions driven by upstream model changes.
5. Evolution (Update/Removal): Lack of enforced semantic versioning impedes safe and visible updates; there are limited automated detection methods for identifying stale or replaceable models.

Lifecycle analysis exposes points where OSS projects are susceptible to technical debt and operational fragility due to PTM dependencies.

5. Planned Metrics and Quantitative Analyses

The proposed empirical paper will employ several operational metrics:

• Update frequency: $f_n = N_{\text{updates}}/T_{\text{project}}$, measuring how often models are swapped within the same family.
• Mean time to resolution (MTR): $$\text{MTR} = (1/|I|)\sum_{i\in I}(t_{\text{close},i} - t_{\text{open},i})$$, with $I$ the set of PTM-related issue tickets.
• PTM longevity/survival: Survival function $\hat S(t)$ is estimated using the Kaplan–Meier estimator to model duration until PTM removal.
• Issue type distributions: Counts and percentages of issues by category (versioning, dependencies, performance).

These quantitative methods will be applied to the PeaTMOSS dataset containing mappings from >15,000 repositories and >2,500 unique PTMs.

6. Proposed Recommendations and Best Practices

While longitudinal data is not yet available, anticipated recommendations are drawn from related work and the initial landscape assessment:

• Institute clear, semantic versioning (MAJOR.MINOR.PATCH) for all PTM releases, particularly weights.
• Require comprehensive, standardized metadata on model hubs: unambiguous licenses, dataset sources, and reproducible evaluation metrics.
• Embed PTM loader coverage into continuous integration, ensuring all sites importing/loading models are exercised by tests.
• Adopt or build dependency-update bots specialized in PTMs, with model-family awareness and safety checks.
• Regularly use file-graph centrality and PTM survival metrics to surface potentially obsolete or orphaned model code and trigger developer review.

Such practices would align PTM lifecycle management with best practices developed for traditional software dependencies.

7. Methodological Foundations

The empirical plan leverages multi-source mining:

• Static code analysis, guided by PeaTMOSS signatures, for PTM loader function mapping.
• Directed file-graph construction using import/dependency resolution and centrality measurement.
• Repository mining from July 2022–December 2024 to track PTM version updates and removals.
• Issue ticket analysis using stratified and snowball sampling, followed by open-coding and double annotation (with interrater agreement measured by Cohen’s κ).

Statistical analyses (Kaplan–Meier, frequency distributions) will be complemented by qualitative manual coding and categorization of usage and maintenance practices.

---

In summary, pre-trained models are now integral, but complex, dependencies in the open-source software lifecycle. Their integration, maintenance, testing, and evolution pose distinctive challenges compared to conventional software libraries, particularly due to poor versioning standards, fragmented documentation, and weak automation in monitoring and updating. The planned research program aims to generate actionable insights—semantic versioning, improved metadata, loader coverage, and specialized automation—that will promote more robust and sustainable PTM usage in open-source environments (Koohjani et al., 8 Apr 2025).