FAIRsoft Evaluator Overview

• FAIRsoft Evaluator is a metadata-driven tool that assesses and improves research software FAIRness using weighted indicators for findability, accessibility, interoperability, and reusability.
• It aggregates and normalizes metadata from sources like bio.tools, GitHub, and Europe PMC, enabling detailed software-level evaluations and enrichment.
• The tool supports a three-step workflow—metadata review, FAIR assessment, and export (including GitHub pull requests)—to streamline software quality improvement.

FAIRsoft Evaluator is the software-entry assessment and improvement component of the Software Observatory at OpenEBench. It is designed for software-level FAIRness assessment of research software, not for auditing algorithmic fairness in machine learning models. In its described workflow, it retrieves metadata from multiple sources, lets users review and complete that metadata, computes FAIRsoft scores with indicator-level results, and exports improved metadata in formats such as .CFF and maSMP; it can also generate GitHub pull requests. Within the broader Observatory, it functions as a developer-facing, metadata-driven instrument for evaluating and improving the Findable, Accessible, Interoperable, and Reusable properties of individual software entries (Pico et al., 7 Oct 2025).

1. Position within the Software Observatory

The FAIRsoft Evaluator is embedded in the Software Observatory at OpenEBench, a web platform for aggregating, normalizing, disambiguating, enriching, and analyzing research software metadata, with a current emphasis on Life Sciences research software. The Observatory operates at multiple levels of granularity: the entire software landscape, specific communities or projects, and individual software entries. Within that architecture, the FAIRsoft Evaluator occupies the software-entry level, whereas the surrounding Observatory exposes broader functions through components such as Trends, FAIR Scoreboard, and Data. The broader platform supports ecosystem-level trend analysis, source-completeness analysis, and aggregate FAIR statistics; the Evaluator is the part that enables software-level FAIRness assessment and improvement workflows (Pico et al., 7 Oct 2025).

This division of roles is important. The Evaluator is not presented as a generic metadata browser or a community-wide scoreboard. Its central purpose is more focused: to help a maintainer or curator inspect a specific software record, enrich its metadata, run a FAIRness assessment, and export the revised record in interoperable forms. This suggests a workflow in which large-scale metadata integration and individual remediation are tightly coupled, but not collapsed into the same interface (Pico et al., 7 Oct 2025).

2. FAIR model and indicator structure

The paper describes the Software Observatory as a practical implementation of FAIRsoft indicators, organized around the four standard FAIR dimensions: Findable, Accessible, Interoperable, and Reusable. The paper does not reproduce the complete FAIRsoft framework, but it explicitly names several indicators and their semantics. Under Findable, it identifies F2 with “rich metadata” and F3 with discoverability. Under Accessible, it explicitly names A1 as “Availability of a working version.” Under Interoperable, it names I1 as support for standard data formats and I2 as integration with other software. Under Reusable, it names R2 for licensing, R3 for contribution recognition or authorship, and R4 for versioning and version control (Pico et al., 7 Oct 2025).

The scoring model is weighted rather than uniform. The paper gives one explicit example: interoperability indicator I2 has a weight of 0.1, while I1 and dependency availability carry weights of 0.6 and 0.3, respectively. At the same time, the paper notes a presentation caveat: the current interface shows indicators with equal visual weight even though their computational weights differ. The paper does not provide a full aggregation formula in LaTeX, but it does make clear that the underlying assessment is not a simple unweighted checklist (Pico et al., 7 Oct 2025).

This weighted structure has methodological consequences. It means that the FAIRsoft Evaluator is not only checking for the presence or absence of metadata fields; it is applying a principle-level aggregation model in which some evidence types contribute more strongly than others. A plausible implication is that interpretation of results depends not only on which indicators are weak, but also on how strongly those indicators influence a principle-level score.

3. Metadata aggregation and evidence pipeline

The Evaluator depends on the Observatory’s metadata pipeline. The initial metadata sources are bio.tools, Bioconda, Bioconductor, Galaxy ToolShed, SourceForge with bioinformatics tags, and Galaxy Europe; GitHub metadata are additionally extracted by following links from primary records. Enrichment then draws on Europe PMC, Semantic Scholar, and direct service availability checks on software webpages. In the reported integrated dataset, the Observatory collected 65,942 initial metadata records and produced 45,334 final unique software records after normalization, disambiguation, and integration (Pico et al., 7 Oct 2025).

Normalization is explicit and domain-oriented. Input and output formats are cleaned and mapped to EDAM terms; license declarations are matched to SPDX License List identifiers using a curated synonym list; authors are heuristically classified as persons or organizations. The paper reports 7,005 normalized SPDX license declarations, covering 34.6% of license-bearing records, and 26,464 format values mapped across 3,931 records. It also reports 28,271 publications retrieved from Europe PMC, 4,403 publications with citations-per-year from Semantic Scholar, and 11,339 deployable tools subjected to service availability checks (Pico et al., 7 Oct 2025).

Identity resolution is handled through a layered process. Records are grouped if they share a normalized software name and type or at least one normalized repository or repository-like webpage link. Conflicts are detected when records grouped by name do not share repository evidence. Resolution combines a rescue heuristic, an LLM-based agreement proxy, and human review through structured GitHub issues, with decisions stored in a persistent conflict registry or block file. Quantitatively, the paper reports 555 conflict blocks, affecting about 1.2% of grouped records; 440 remained after rescue heuristics, and the LLM-based agreement proxy automated roughly 79% of all final integration decisions (Pico et al., 7 Oct 2025).

The Evaluator inherits both the strengths and the constraints of this pipeline. Because FAIRness assessment is metadata-driven, integration quality, normalization quality, and source coverage directly shape the evidence base from which indicator scores are computed.

4. Evaluation workflow and exported artifacts

The FAIRsoft Evaluator is described as a three-step workflow: metadata review and completion, FAIR assessment, and export of metadata. Metadata can be retrieved from GitHub, local files, or the Observatory database. After review and completion, the software entry can be assessed and the enriched metadata exported. The paper also states that the Evaluator can generate GitHub pull requests, making it part of an improvement loop rather than only a reporting interface (Pico et al., 7 Oct 2025).

Its outputs are more than a single aggregate score. The Evaluator exposes FAIRsoft scores, indicator-level results, and an interface for interpreting individual indicator outcomes. It also supports export in .CFF and maSMP. The latter is described as a structured JSON-LD profile compatible with CodeMeta, Bioschemas, and schema.org. This makes the Evaluator simultaneously an assessment tool and a metadata production tool: it not only identifies missing or weak evidence, but also helps produce machine-readable metadata suitable for reuse in external ecosystems (Pico et al., 7 Oct 2025).

The broader Observatory precomputes FAIRness scores, completeness indicators, and aggregation statistics for efficient rendering in dashboards, but the Evaluator’s interactive mode centers on software-specific remediation. This suggests a two-tier design: batch scoring for observatory-scale analysis and interactive reassessment for individual software maintenance.

5. Empirical behavior, case-study evidence, and limitations

The Proteomics case study provides the clearest empirical picture of how the Evaluator behaves on a curated collection. The ELIXIR Proteomics project collection contains 758 tools. Source provenance is dominated by bio.tools: 611 tools come exclusively from bio.tools, with additional contributions from Bioconda (100), Bioconductor (86), and GitHub (28). Metadata completeness is strong for descriptive fields—100% coverage for name, type, description, webpage, topics, and author—but weaker for engineering-oriented evidence: 89% have publication links, 86% documentation, only 10% testing information or explicit dependency declarations, and only 20% download links (Pico et al., 7 Oct 2025).

The FAIRness results are correspondingly uneven. Under Findable, 100% of tools achieved a perfect score on F2, reflecting rich metadata and EDAM annotation, and F3 is also described as strong. Under Accessible, A1 has a mean score of 0.14, driven by missing download links and installation documentation and limited presence in infrastructures such as Galaxy. Under Interoperable, I1 is above average but I2 is weak. Under Reusable, R3 authorship recognition reaches 100%, but licensing and versioning are markedly weaker: 70% of tools lack any declared license; among 229 tools (30%) with unambiguous licensing information, 203 are open source, including GPL (102), Apache (41), MIT (18), and Artistic (22); only 19% declare use of version control systems, and 30% provide evidence of software versioning (Pico et al., 7 Oct 2025).

The paper is explicit that such scores are metadata-dependent rather than direct measurements of intrinsic software quality. In the same Proteomics analysis, it notes that bio.tools does not support dedicated metadata fields for testing or dependencies, so low scores on related indicators may reflect registry schema limitations rather than actual absence of testing or dependency management. The paper also reports that about 44% of checked deployable-service URLs were unavailable, interpreting this as a structural sustainability problem in the software ecosystem. This suggests that FAIRsoft Evaluator results should be read as evidence-based FAIRness assessments over available metadata, not as exhaustive judgments about the software artifact itself (Pico et al., 7 Oct 2025).

6. Relation to adjacent evaluators and common misconceptions

The name “FAIRsoft Evaluator” can be misread as referring to a fairness-auditing toolkit for machine learning. The literature surveyed here makes the distinction sharp. FairX is an open-source Python-based benchmarking tool for model analysis under fairness, utility, explainability, and synthetic-data quality; it supports fair generative model training and evaluates outputs with metrics such as DPR, EOR, FTU, $\alpha$-precision, $\beta$-recall, and Authenticity. fairmetrics is an R package for group fairness evaluation in binary classification with a binary protected attribute, including bootstrap confidence intervals for multiple difference-based and ratio-based fairness criteria. Both are fairness evaluators in the algorithmic sense, but neither is a FAIR-principles compliance evaluator for research software (Sikder et al., 2024, Smith et al., 6 Jun 2025).

A more closely related research-software framework is FAIRSECO, which integrates FAIRness, software quality, and software impact by combining evidence from tools and services such as howfairis, Tortellini, SearchSECO, OpenAlex, and Semantic Scholar. FAIRSECO uses FAIRness as one component in a broader quality-and-impact model, whereas the FAIRsoft Evaluator is centered on software-level FAIRness assessment and metadata improvement inside a large observatory infrastructure (Deekshitha et al., 2024). Within that infrastructure, the FAIRsoft Evaluator differs from isolated checkers by coupling automated assessment with metadata aggregation, disambiguation, enrichment, community-level scoreboards, and export workflows (Pico et al., 7 Oct 2025).

This distinction clarifies the conceptual scope of the term. FAIRsoft Evaluator concerns the FAIRness of research software as a digital research object—its findability, accessibility, interoperability, and reusability as evidenced by metadata and related artifacts. It does not evaluate whether a classifier is demographically fair, whether an LLM is stereotype-free, or whether a recommendation model satisfies equal opportunity. In that respect, it belongs to the research-software observability and stewardship lineage rather than the algorithmic-fairness benchmarking lineage.