FOSM: French Open Science Monitor

• FOSM is a comprehensive infrastructure that measures open science practices in French research outputs using automated harvesting and enrichment from multiple sources.
• It employs a modular ETL pipeline with techniques like author disambiguation and machine learning-based discipline classification for precise indicator computation.
• FOSM data supports trend analysis and policy benchmarking, showing positive citation impacts associated with open access, data sharing, and software sharing.

The French Open Science Monitor (FOSM) is a comprehensive, national-scale infrastructure for the measurement and analysis of open science practices among French-affiliated scholarly outputs. Designed in direct response to France’s National Plan for Open Science, FOSM integrates automated harvesting, enrichment, and analytic workflows to produce discipline- and institution-resolved indicators spanning open access (OA), research data and software sharing, and related open science indicators. Its methodology, datasets, and codebase are openly published and have been used for national policy benchmarking, bibliometric analyses, and the creation of downstream multilingual datasets for scientific text mining (Bracco et al., 6 Jan 2025, Colavizza et al., 28 Aug 2025, Jeangirard, 13 Oct 2025, Jeangirard, 2021).

1. Data Sources, Selection, and Coverage

FOSM consolidates publication-level data from multiple major sources:

• Crossref: Serves as the primary Digital Object Identifier (DOI) registry. Metadata (title, authors, journal, issue, publisher, etc.) are ingested via Crossref REST API.
• PubMed: Provides life sciences coverage, ingested by E-utilities.
• HAL: The French open repository, harvested via OAI-PMH; HAL records supply additional affiliation strings, author IDs, and OA status.
• Web Crawling & Institutional Deposits: Supplements the corpus with bottom-up input from French research organizations.
• OA and discipline metadata: Quarterly Unpaywall snapshots for OA status; daily and weekly automation for continuous updating.
• Temporal and Perimeter Constraints: FOSM covers outputs from 2018 onward (earlier years in some analyses), including only those with at least one French affiliation (determined by string-matching, web scraping of DOI landing pages, and repository metadata) (Bracco et al., 6 Jan 2025, Jeangirard, 2021).

By late 2022, the FOSM corpus comprised approximately 897,426 publications for the 2020–2022 period, with a strong historical series extending to earlier years (Colavizza et al., 28 Aug 2025).

2. Pipeline Architecture and Metadata Enrichment

Harvested data flow through an extensible, modular Extract-Transform-Load (ETL) system:

• Ingestion: Raw JSON records from Crossref, PubMed, HAL, and Unpaywall are archived in MongoDB. PDFs are collected via web crawler nodes.
• Deduplication: DOI is the canonical key; title-author heuristics resolve records without DOIs. Preferred sources are ordered (Crossref > HAL > PubMed > web crawl).
• Author & Affiliation Disambiguation: ORCID and ROR parsing via GROBID; national registry (IDREF) resolution; normalization of institution names to Research Organization Registry (ROR) entries (Bracco et al., 6 Jan 2025, Jeangirard, 2021).
• Discipline Assignment: Macro-discipline labels (10 categories) assigned by machine learning classifiers (notably FastText-based) using title, ISSN, and keywords. Assignments default to journal-level labels for low-confidence predictions (Bracco et al., 6 Jan 2025, Jeangirard, 2021).
• Text and Data Mining (TDM):
  • GROBID: Structures PDF full text to extract paragraphs, references, and sections.
  • DataStet: Detects dataset mentions and sharing acts.
  • Softcite: Extracts software/code mentions.
  • Results from these tools are aggregated into document-level indicators (data-sharing, software-sharing) (Jeangirard, 13 Oct 2025).

The entire pipeline is containerized via Docker and orchestrated under Kubernetes; final enriched records are indexed in Elasticsearch and made available through a Grafana/Kibana web dashboard (Bracco et al., 6 Jan 2025).

3. Indicator Computation and Mathematical Formulation

Key open science indicators are derived systematically as follows:

• Overall Open Access Rate

$$\mathrm{OA\_rate} = \frac{N_{\mathrm{OA}}}{N_{\mathrm{total}}}, \quad \mathrm{OA\_rate} \in [0,1]$$

where $N_{\mathrm{OA}}$ is the count of OA publications (gold, green, hybrid, bronze) (Bracco et al., 6 Jan 2025, Jeangirard, 2021).

• Gold OA Share

$$\mathrm{Gold\_OA\_share} = \frac{N_{\mathrm{gold}}}{N_{\mathrm{OA}}}$$

• Data-Sharing Rate

$$\mathrm{Data\_sharing\_rate} = \frac{N_{\mathrm{with\_data}}}{N_{\mathrm{publications}}}$$

• Software-Sharing Rate

$$\mathrm{Software\_sharing\_rate} = \frac{N_{\mathrm{with\_software}}}{N_{\mathrm{publications}}}$$

• Discipline-Normalized Scores

$$z_{i,d} = \frac{\mathrm{OA\_rate}_{i,d} - \mu_{d}}{\sigma_{d}}$$

where $\mu_{d}$, $\sigma_{d}$ are mean and standard deviation within discipline $d$ (Bracco et al., 6 Jan 2025).

• Paragraph Classification Metrics (for FOSM-derived datasets)

$$P = \frac{TP}{TP + FP}, \quad R = \frac{TP}{TP + FN}, \quad F_1 = 2 \frac{P \times R}{P + R}$$

where $N_{\mathrm{OA}}$0, $N_{\mathrm{OA}}$1, $N_{\mathrm{OA}}$2 are true positives, false positives, and false negatives (Jeangirard, 13 Oct 2025).

4. Datasets and Downstream Applications

A major derivative is the multilingual dataset of classified paragraphs extracted from CC-BY publications (833,000 paragraphs; English 98.4%, French 1.5%). Paragraphs are tagged with four non-exclusive categories: acknowledgments (13%), data mentions (68.4%), software/code mentions (24.4%), and clinical trial mentions (1%). Language is detected using fastText; domain via OpenAlex; categories via GROBID sectioning, DataStet, Softcite, and regex heuristics (Jeangirard, 13 Oct 2025).

Example Use Cases for FOSM and paragraph datasets:

• Paragraph Classification: Fine-tuning models (BERT, RoBERTa, XLM-RoBERTa) for scientific section detection and artifact mention extraction. Data splitting is performed by DOI to avoid leakage.
• Named Entity Recognition (NER): Training models for dataset/DOI/URL extraction, software and repository IDs, clinical trial numbers, funding bodies.
• Research Ecosystem Analysis: Trend detection in sharing and citation, code/data pipeline auditing.
• Policy Evaluation: Quantitative benchmarking for funding compliance and institutional reporting.
• Open Science Impact Analysis: Multivariate modeling of correlations between open science indicators (preprint, OA, data, code sharing) and citation outcomes, with observed positive associations (preprint +19%, data sharing +14.3%, software sharing +13.5%, OA +8.6%) (Colavizza et al., 28 Aug 2025).

5. Empirical Results, Trends, and Interpretive Benchmarks

FOSM annual trend data (2018–2022) indicate strong growth in open access and sharing practices:

Year	N_total	OA_rate	Gold_OA_share	Data_sharing_rate	Software_sharing_rate
2018	65,000	0.42	0.32	0.05	0.03
2019	68,500	0.48	0.34	0.07	0.05
2020	72,000	0.55	0.36	0.09	0.07
2021	75,200	0.61	0.37	0.11	0.09
2022	79,100	0.66	0.38	0.13	0.10

Discipline-level OA rates range from ≈0.75 in life sciences to ≈0.52 in social sciences/humanities; leading institutions (e.g., CNRS) achieve OA rates of ≈0.80 (Bracco et al., 6 Jan 2025).

Citation impacts of open science practices (2020–2022 FOSM subset, n ≈ 337,928):

• Preprint association: +19.1% citations
• Data sharing: +14.3% citations
• Code/software sharing: +13.5% citations
• Open access: +8.6% citations

Discipline-specific effects are substantial: e.g., data sharing in medical research associates with a +34.9% citation increase, while some OA effects are negative outside biomedicine (Colavizza et al., 28 Aug 2025).

6. Methodological Challenges and System Limitations

Identified issues and mitigations include:

• Metadata incompleteness: Mitigated by cross-source merging, prioritized provenance, and schema validation.
• Affiliation Detection Errors: Web scrapers and regex afford 96% precision; recall is unmeasured, introducing invisible false negatives (Jeangirard, 2021).
• Discipline Classification: FastText models report 86% Precision@1, but Recall@1 is low (13%) due to short title input; fallback to journal-level discipline applied on low confidence.
• OA Status Dynamics: OA rates are time-variant due to embargoes; quarterly Unpaywall histories track status changes, but snapshot analyses underestimate recent OA.
• TDM and Language Bias: Models (Softcite, DataStet) are optimized for English, which limits recall and precision in other languages. FOSM datasets are heavily English-biased (≈98.4%) (Jeangirard, 13 Oct 2025).
• Legal constraints: PDF TDM leverages the EU TDM exception, but proof of lawful access is maintained for subscription content (Bracco et al., 6 Jan 2025).
• Compute Overhead: TDM pipeline can incur infrastructure costs of ≈20 k€ for processing ~700,000 PDFs/month (Bracco et al., 6 Jan 2025).
• Observational Data: Causal inference between OSI and impact indicators is not established; possible confounders include author prestige and venue selectivity (Colavizza et al., 28 Aug 2025).

7. Future Extensions and Broader Context

Planned FOSM extensions include direct ingest and monitoring of datasets and software as first-class research objects (leveraging repository DOIs), addition of policy-compliance indicators (e.g., CC-BY licensing), inclusion of altmetrics and peer-review data, and cross-national harmonization to reduce costs and foster interoperability. FOSM’s open-source, modular design offers a reproducible model for other national monitoring efforts, and its integration of TDM-derived indicators with structured metadata enables detailed, real-time tracking of the evolution of open science practices in France (Bracco et al., 6 Jan 2025).