Software Observatory at OpenEBench
- Software Observatory at OpenEBench is a metadata-driven platform that consolidates and harmonizes research software records from diverse registries in the Life Sciences.
- It employs a robust ingestion and normalization pipeline to deduplicate entries, enrich metadata, and compute FAIRsoft scores for actionable insights.
- The platform leverages identity resolution and detailed FAIR assessments to support both large-scale ecosystem analysis and individual software remediation.
Searching arXiv for the cited OpenEBench and related observatory papers to ground the article. arxiv_search(query="(Pico et al., 29 May 2025)", max_results=5) Searching arXiv for the target paper and related observatory literature. {"query":"(Pico et al., 29 May 2025)","max_results":5} The Software Observatory at OpenEBench is a web platform for aggregating, harmonizing, analyzing, and assessing research software metadata in the Life Sciences. Built on top of OpenEBench, it consolidates fragmented metadata from multiple registries and repositories into a unified, deduplicated and enriched collection, computes trends from that collection, and evaluates software against FAIR principles for research software through FAIRsoft-based indicators (Pico et al., 7 Oct 2025). Its analytical reliability depends on a prior reference layer: identity resolution is the step through which heterogeneous records are assembled into a cohesive collection of research software, and that collection is the reference component of the observatory (Pico et al., 29 May 2025).
1. Purpose, analytical scope, and observatory model
The observatory was created in response to the fragmentation, redundancy, inconsistency, and incompleteness of research software metadata across registries, software repositories, packaging systems, and project webpages. In the Life Sciences, this fragmentation is especially acute because software is heavily used, often cited, and frequently embedded in workflows, yet the information needed to discover, compare, reproduce, or reuse it is distributed across sources such as bio.tools, Bioconda, Bioconductor, Galaxy registries, GitHub, and publication databases (Pico et al., 7 Oct 2025).
Its stated objective is not merely cataloguing. The platform turns distributed software descriptions into evidence about discoverability, accessibility, interoperability, reuse conditions, and community practices over time. In this setting, adherence to FAIR principles for research software is used as a proxy for understanding current trends in software development. The observatory does not treat FAIRness as a perfect measure of software quality in every engineering sense; rather, FAIR indicators provide a metadata-based lens on quality, maturity, usability, sustainability, and ecosystem health. This suggests an observatory model in which metadata is both operational infrastructure for software discovery and an empirical substrate for ecosystem-level analysis.
The platform supports drill-down across analytical scales. Users can inspect the entire integrated software landscape, narrow analysis to a specific community or project, and descend to an individual software entry for detailed FAIR assessment. The observatory therefore combines corpus-scale monitoring with software-level remediation.
2. Metadata ingestion, normalization, and integration pipeline
The observatory integrates metadata from several primary and secondary sources. The main registries ingested are bio.tools, Bioconda, Bioconductor, Galaxy ToolShed, SourceForge entries tagged for bioinformatics, and Galaxy Europe. GitHub repositories are mined by following links contained in those primary records, and publication metadata is retrieved from Europe PMC and Semantic Scholar. For deployable services such as web applications, REST services, SPARQL endpoints, SOAP services, workbenches, or suites, the platform adds service availability information derived from direct checks of the corresponding URLs (Pico et al., 7 Oct 2025).
| Source | Records |
|---|---|
| bio.tools | 30,063 |
| Bioconda | 8,846 |
| Galaxy ToolShed | 6,302 |
| Bioconductor | 3,470 |
| SourceForge | 2,989 |
| Galaxy Europe | 1,409 |
| GitHub-derived metadata | 3,852 |
The system initially collected 65,942 metadata records from the main Life Sciences registries. After normalization, disambiguation, and integration, this yielded 45,334 unique software records. The collected metadata includes names, software types, webpages, repository links, descriptions, topics, authors, documentation links, publication links, download links, version information, version control evidence, software formats, and licensing. The enrichment pipeline adds citation counts, abstracts, publication year, citations per year where available, and service responsiveness.
The workflow is modular. Metadata first enters a raw collection, then undergoes internal normalization and external enrichment. Internally, malformed fields are pruned, links are deduplicated, formatting is standardized, software formats are mapped to EDAM ontology terms, and license declarations are mapped to SPDX License List identifiers using a curated synonym list. The use of curated mappings is explicit: similarity-based string matching had produced unreliable results, so manual synonym mappings were preferred to reduce false positives. Normalized and enriched records are stored in an intermediate “Normalised” database that supports traceable and incremental updates. After integration and conflict resolution, unified entries are written into the final deduplicated “Merged” collection, which feeds the observatory visualizations and the FAIRsoft Evaluator.
Several quantitative normalization outcomes are reported. A total of 7,005 license declarations were standardized to SPDX identifiers, representing 34.6% of records that had license-bearing information. In addition, 26,464 format values across 3,931 records were mapped to EDAM terms, service availability was assessed for 11,339 deployable tools, publication metadata with total citation counts was retrieved from Europe PMC for 28,271 publications, and citations-per-year data was retrieved from Semantic Scholar for 4,403 publications.
3. Identity resolution as the observatory’s reference layer
Identity resolution is treated as infrastructure rather than cleanup. The observatory depends on a deduplicated, semantically resolved software corpus before it can compute reliable indicators about software quality, sustainability, visibility, and FAIR alignment. Without identity resolution, it becomes difficult to compute quality indicators, trace software usage, or enable consistent referencing; unresolved duplication also affects attribution and developer credit (Pico et al., 29 May 2025).
In operational terms, the current observatory contains around 45,000 unique research software records, with the number of conflicts ranging between 500 and 3,000 cases depending on pre-filtering assumptions. These conflicts constitute only 1–6% of all records, but they are described as the most time-consuming and impactful resolution challenges. The integration strategy is conservative. Software records are grouped into candidate blocks if they share either a normalized software name and type, or at least one normalized repository or repository-like webpage link, including domains such as GitHub, Bioconductor, or SourceForge. Link normalization uses domain-specific rules to collapse URL variants and preserve transitive linkage. Within each block, the system flags likely identity conflicts where records were grouped by name but do not share any repository or repository-like link. A “rescue heuristic” is then applied: if a disconnected record shares both a name and a source registry with an accepted group member, it may be promoted into the group (Pico et al., 7 Oct 2025).
The LLM-based disambiguation study defines identity resolution as a three-way classification problem over pairs of metadata records: “same,” “different,” or “unclear.” Models and human annotators were given name, description, repository URL, webpage URL, publication, and “authors, developers or maintainers,” together with cleaned contents of the referenced URLs. This makes the task semantic rather than purely lexical. The benchmark contains 100 manually annotated cases sampled from 555 conflicting pairs that remained after traditional methods and simplifying assumptions had filtered easier cases. The dataset is intentionally focused on difficult pairs rather than easy exact duplicates.
For production use, the key methodological contribution is an agreement-based proxy: a prediction was accepted automatically only when all top-performing LLMs agreed, and cases with disagreement were deferred to human review. In the evaluated study, five pairwise proxies were tested. Proxy I, combining Llama 4 Scout and Mixtral 8x22B, achieved accuracy $0.965$ with 95% confidence interval $0.930$–$1.000$, macro F1 $0.626$ with 95% confidence interval $0.581$–$0.667$, and issued verdicts for 86 out of 100 cases. Proxy V, combining Llama 4 Scout and Mixtral 8x7B, issued verdicts for 94 out of 100 cases, with accuracy $0.958$ and macro F1 $0.611$. All models and all proxies failed on the three “unclear” cases. In the observatory paper, the downstream integration outcome is reported more directly: 555 conflict blocks represented approximately 1.2% of all grouped metadata records, 440 conflict blocks remained after the rescue heuristic, and the LLM-based agreement proxy automated roughly 79% of all final integration decisions, with the remaining ambiguous cases manually reviewed (Pico et al., 7 Oct 2025).
A notable textual inconsistency is documented in the identity-resolution paper itself. Its discussion restates Proxy I as having macro F1-score $0.941$, which conflicts with the $0.626$ reported in the results section and figure-caption discussion. Because all proxies also failed on the unclear class, the lower value is more consistent with the rest of the paper. The inconsistency should therefore be noted when the benchmark is cited.
4. FAIRsoft assessment and user-facing components
The observatory interface is organized into four components, each corresponding to a distinct analytical function (Pico et al., 7 Oct 2025).
| Component | Function |
|---|---|
| Trends | Visual summaries of metadata attributes including licensing, versioning, and repository usage |
| FAIR Scoreboard | Aggregated FAIRsoft indicator scores across the dataset |
| Data | Metadata completeness statistics, source contributions, and integration coverage |
| FAIRsoft Evaluator | Software-level FAIRness assessment, metadata retrieval, editing, scoring, and export |
The “Trends” component summarizes metadata attributes and longitudinal changes over time. The “FAIR Scoreboard” displays aggregated FAIRsoft indicator scores and supports filtering by project or community; its charts can also be embedded elsewhere via iframes. The “Data” component reports completeness statistics, source provenance, and integration coverage. The “FAIRsoft Evaluator” supports the finest granularity of analysis by enabling metadata retrieval, editing, scoring, and export for individual software entries.
FAIR assessment is central to the platform. The observatory implements the FAIR principles for research software using the FAIRsoft framework and computes scores from metadata evidence. At ecosystem scale, FAIRsoft-related measures are computed automatically from the integrated metadata collection. At individual-tool scale, the process is semi-automated: users can supplement missing information manually, including testing infrastructure, access restrictions such as registration requirements, and other evidence not captured in upstream structured sources. This hybrid design is consequential because some FAIR-relevant evidence is absent from the schemas of the original registries.
The FAIRsoft Evaluator works in three broad stages: metadata review and completion, FAIR assessment, and export of improved metadata. Metadata can be retrieved from GitHub, local files, and the observatory database. After scoring, the tool can export metadata in .CFF and maSMP. The latter is described as a structured JSON-LD profile compatible with CodeMeta, Bioschemas, and schema.org. The evaluator also supports direct GitHub integration via automated pull requests, allowing metadata improvements to be pushed back into repositories.
The observatory paper gives one explicit weighting example within the FAIR model. For interoperability,
$0.930$0
where $0.930$1 is standard format support or related interoperability evidence, $0.930$2 denotes dependency availability, and $0.930$3 is integration with other software. The paper uses this example to identify a visualization limitation: the interface presents indicators with equal visual prominence even though their contributions to the aggregated FAIRness score are unequal.
5. Empirical profile of the Life Sciences software ecosystem
The observatory supports ecosystem diagnostics by aggregating metadata attributes, FAIRsoft scores, completeness measures, source provenance, and longitudinal changes over time. Its analytical output is therefore not limited to individual scores; it also characterizes structural properties of the integrated corpus (Pico et al., 7 Oct 2025).
One empirical signal concerns fragility of software availability. Service availability was assessed for 11,339 deployable tools, and around 44% of URLs associated with deployable services were unavailable. The authors interpret this as evidence of structural fragility in the research software ecosystem and as a constraint on sustainability, discoverability, and identity resolution. A plausible implication is that FAIR-oriented observability is partly limited by the volatility of upstream web resources.
The ELIXIR Proteomics case study illustrates the platform’s community-scale diagnostic role. This collection contained 758 tools linked through curated bio.tools metadata. Provenance analysis showed that most Proteomics tools came exclusively from bio.tools (611), with smaller numbers from Bioconda (100), Bioconductor (86), and GitHub (28). Metadata completeness was strong for name, type, description, webpage, topics, and author, all of which were universally present; publication links were present for 89% of tools, and documentation for 86%. By contrast, only 10% of tools reported testing information or explicit dependency declarations, and only 20% provided download links. The paper explicitly warns that such deficits may reflect source-schema limitations rather than true absence of engineering practice, because bio.tools does not currently support testing or dependency fields.
The FAIRsoft profile of the same community was differentiated rather than uniformly weak or strong. Findability was the strongest dimension: all tools achieved a perfect score on indicator $0.930$4, due to rich metadata and EDAM annotation. Accessibility was weak: indicator $0.930$5, “Availability of a working version,” had a low mean score of 0.14, largely because download links and installation documentation were often missing. Interoperability was moderate: support for standard data formats ($0.930$6) was relatively good, but integration with other software ($0.930$7) was weak because few tools exposed programmatic interfaces such as APIs or libraries, and Galaxy representation was limited. Reusability was mixed: all tools reported authorship, giving strong performance on $0.930$8, but licensing was a major weakness. Seventy percent of tools lacked any declared license; among the 229 tools with unambiguous license information, 203 were open source. Among these licensed tools, GPL was most common (102 tools), followed by Apache (41), Artistic (22), and MIT (18). Versioning and version control evidence was only average: 19% of tools declared use of version control systems such as Git, and 30% provided evidence of versioning.
These observations clarify the observatory’s interpretive stance. Low scores do not necessarily indicate poor software engineering in the narrow sense; they may indicate missing metadata, absent schema support in source registries, or lack of machine-readable exposure of evidence. The observatory therefore measures software as represented through metadata and enrichment, not software quality in isolation from its documented traces.
6. Limitations, contested points, and related observatory directions
The observatory papers are explicit about limitations. Metadata coverage is uneven and often constrained by the schemas of upstream registries; low FAIR scores may therefore reflect metadata absence rather than poor software practice. Author information remains under-modeled: there is no robust author disambiguation, ORCID use is inconsistent, and duplicated or variably formatted contributor identities cannot yet be resolved into collaboration networks. The current observatory does not mine rich metadata directly from README files, LICENSE files, or configuration files, even though those documents could expose dependencies, system requirements, usage details, or affiliations. Disambiguation presently targets only one conflict class—same-name records lacking a common repository—while more prevalent cases, such as different-name records sharing a repository, remain future work (Pico et al., 7 Oct 2025).
The identity-resolution benchmark adds further caveats. Its gold standard is small, skewed, and single-annotator; inter-annotator agreement was not measured. All LLMs and all agreement proxies failed on unclear cases, indicating that current instruction-tuned LLMs are poor at explicit abstention or uncertainty reporting in this domain. Performance depends not only on model capability but also on metadata presentation and web-content extraction quality. API-backed inference, unfixed seeds, provider variability, and the hardware and environmental burdens of larger models introduce additional deployment caveats. In an observatory context, false matches contaminate derived indicators, while false non-matches fragment a software identity across sources and undercount evidence of FAIRness (Pico et al., 29 May 2025).
Two adjacent observatory concepts clarify the broader design space without describing the OpenEBench platform itself. “Towards Observation Lakehouses: Living, Interactive Archives of Software Behavior” introduces an Observation Lakehouse as “a tall, append-only observations table storing every actuation (stimulus, response, context) and SQL queries that materialize SRC slices on demand.” OpenEBench is not mentioned explicitly there, so any linkage is inferential, but the work provides a plausible reference architecture for a behavioral evidence layer beneath a software observatory concerned with executions, benchmarking, or long-term retention of run-time observations (Kessel, 2 Dec 2025). By contrast, “Monitoring Data Distribution and Exploitation in a Global-Scale Microservice Artefact Observatory” proposes a federated research infrastructure with consensus voting among participants to establish and preserve ground truth about software artefacts. That work is framed around microservices rather than OpenEBench, but it offers a directly relevant governance pattern for distributed validation of observatory evidence (Gkikopoulos et al., 2020).
Taken together, these limitations and adjacent models delimit the observatory’s current meaning. The Software Observatory at OpenEBench is already a metadata-driven platform for Life Sciences research software that combines large-scale aggregation, normalization, enrichment, identity resolution, FAIRsoft-based scoring, and multi-scale dashboards. At the same time, its own papers make clear that observability remains contingent on schema coverage, persistent access to software resources, careful conflict resolution, and explicit human oversight in semantically ambiguous cases.