ProMED: Surveillance, Benchmarking & Medical AI
- ProMED is a real-time, expert-curated outbreak reporting system that has disseminated global epidemic intelligence since 1994 via email, RSS, and web.
- Benchmark studies use ProMED as a 'silver standard' to evaluate automated outbreak alerting systems on metrics like sensitivity, specificity, and timeliness.
- Recent research repurposes the ProMed name for medical AI applications, distinguishing between outbreak surveillance and proactive clinical dialogue or multimodal EHR integration.
Searching arXiv for recent and foundational papers on ProMED and related usages of the term. ProMED most commonly denotes the Program for Monitoring Emerging Diseases and its ProMED-mail service: a freely available, 24/7 human network of expert volunteers operating under the International Society for Infectious Diseases, distributing expert-curated outbreak intelligence through email, RSS, web, and related channels. In the event-based surveillance literature, ProMED functions both as an operational source of epidemic intelligence and as an approximate gold standard or silver standard for evaluating automated outbreak alerting systems. More recent medical AI papers also reuse the name “ProMed” for unrelated model families, notably a reinforcement-learning framework for proactive medical LLMs and the ProMedTS multimodal EHR framework (Collier, 2011, Collier, 2011, Consoli et al., 2024, Ding et al., 19 Aug 2025, Niu et al., 19 Feb 2025).
1. Primary referent: ProMED-mail in outbreak surveillance
In the surveillance papers, ProMED is described as a program of the International Society for Infectious Diseases, staffed by expert volunteer reporters, supported by a staged editorial process, and distributed to about 40,000 subscribers via email, RSS, and web (Collier, 2011). A related description characterizes ProMED-mail as a freely available, 24/7 human network of expert volunteers whose moderators monitor global media and other sources, then subject reports to a staged review process before posting them to the mailing list and website (Collier, 2011). In a later LLM-oriented study, ProMED is presented as the largest publicly accessible outbreak-reporting system, launched in 1994, with more than 66,000 posts since 1994-08-19 (Consoli et al., 2024).
These descriptions situate ProMED within event-based surveillance rather than indicator-based surveillance. Its reports are openly accessible, broadly cover infectious diseases of interest, and are often timelier than official verification reports (Collier, 2011). The corpus spans common seasonal diseases and more sporadic or novel ones, including dengue, yellow fever, anthrax, plague, A(H1N1), and A(H5N1) (Collier, 2011). This suggests that ProMED’s analytical importance lies not only in rapid dissemination, but also in its combination of editorial filtering, global scope, and topic breadth.
A recurrent misconception is to treat ProMED as synonymous with formal case notification. The surveillance literature does not do so. Instead, it treats ProMED as an expert-curated signal that is highly valuable, globally relevant, and operationally useful, but not identical to laboratory-confirmed ground truth (Collier, 2011, Collier, 2011).
2. ProMED as benchmark and “silver standard” for automated alerting
Two foundational BioCaster studies use ProMED as the reference timeline for evaluating whether automated systems detect outbreak topics early enough. In the disease-news evaluation study, each qualifying ProMED-mail posting is treated as an alert that something unusual may be happening, after hand filtering to fit the paper’s case definition. A system alert is counted as a true alarm if it occurs within 7 days before and including a qualifying ProMED-mail report on the same disease-country topic; alerts outside that window are false alarms, and ProMED alert periods with no corresponding system alert are false negatives. Multiple system alerts inside a qualifying window count only once (Collier, 2011). The cross-lingual paper adopts the same core idea, explicitly calling ProMED a silver standard because true outbreak onset, duration, and spatial extent are hard to determine precisely, and similarly uses a tolerance window of 7 days prior to and including the ProMED report (Collier, 2011).
The evaluation task is not generic topic detection and tracking. It is specifically to determine whether a model can detect a country–disease event when it becomes unusual or bursty in the news, with performance judged by sensitivity, specificity, positive predictive value, negative predictive value, F1, alert burden, and timeliness relative to ProMED (Collier, 2011, Collier, 2011). In one study, the final test set contains 287 ProMED-mail postings, covering 18 outbreaks across 14 countries over a 366-day observation period, from 17 June 2008 to 17 June 2009; in another, the study evaluates 16 disease outbreak streams, with 153 ProMED events over roughly 129 days from January to May 2010 (Collier, 2011, Collier, 2011).
BioCaster’s mined signals are daily country-disease document counts extracted from online news and other feeds, then scored with EARS C2, EARS C3, W2, F-statistic, and EWMA, all using a 7-day baseline and a 2-day guard period (Collier, 2011). Thresholds were optimized manually on training outbreaks to maximize F1, with experimentally optimized thresholds of 0.3 (C3), 0.2 (C2), 0.2 (W2), 0.6 (F-statistic), and 2.0 (EWMA), together with a minimum standard deviation of 0.2 and purging of singleton daily counts (Collier, 2011).
| Algorithm | Aggregate result | Operational reading |
|---|---|---|
| C3 | sensitivity 0.78, specificity 0.95, PPV 0.49, NPV 0.99, F1 0.60, 7.65 alarms/100 days | Higher sensitivity, lower PPV |
| C2 | sensitivity 0.73, specificity 0.97, PPV 0.55, NPV 0.99, F1 0.63, 5.98 alarms/100 days | Close to ProMED alert volume |
| W2 | sensitivity 0.72, specificity 0.97, PPV 0.56, NPV 0.99, F1 0.63, 5.57 alarms/100 days | Best overall by F1 |
| F-statistic | sensitivity 0.80, specificity 0.91, PPV 0.45, NPV 0.98, F1 0.58, 13.77 alarms/100 days | Most sensitive, noisiest |
| EWMA | sensitivity 0.73, specificity 0.95, PPV 0.47, NPV 0.98, F1 0.58, 7.85 alarms/100 days | Lower practical balance |
Across the 18 outbreak datasets, W2 performed best overall by F1, with F1 = 0.63, narrowly ahead of C2, and C2 and W2 were closest to ProMED’s alert volume, whose mean rate was 4.36 alerts per 100 days (Collier, 2011). In the multilingual study, cross-lingual event capture improves sensitivity, F1, and timeliness in most models, although often at the cost of more false alarms; for example, global W2 sensitivity rose from 0.42 → 0.55, F1 from 0.49 → 0.54, and timeliness from 3.1 → 3.7 days earlier when moving from English-only to all-language news (Collier, 2011).
3. Methodological constraints of ProMED-based evaluation
The same literature is explicit that evaluation against ProMED is useful but nontrivial. A central limitation is that ProMED does not always behave like a pure first-detection stream. During outbreak tracking, it may post updates about case status, fatalities, control measures, or ongoing situation summaries, which weakens the assumption that every ProMED posting corresponds to initial outbreak detection inside a 7-day window (Collier, 2011). The cross-lingual study therefore excludes ProMED items outside the case definition using the International Health Regulations decision-tree instrument, including requests for information, reports mainly about control measures, and aggregated summaries not tied to specific outbreak events (Collier, 2011).
Several reporting artifacts are particularly important. Day-of-week effects are strong: BioCaster news counts were 1.37 documents/day on weekdays versus 0.49 on Saturdays and Sundays, while ProMED postings also showed day-of-week structure, with dengue postings biased toward Monday/Tuesday (Collier, 2011). Sudden reporting drops (“sinks”) can create weekend zero-count gaps followed by rebounds. Granularity at country level is another limitation, because many true alerts arise first at province or sub-country level, and country-level aggregation can hide local signals. False topic-location associations also occur; one example linked Senegal to yellow fever because of a WHO reference laboratory there, even though the actual outbreak was elsewhere (Collier, 2011).
The Cholera in Angola 2010 case study shows why alignment with ProMED can be awkward even when the underlying surveillance intuition is correct. BioCaster detected a Spanish report on 21/1/2010 about 31 cholera cases from Oct–Dec 2009, which was treated as a false positive because it was historical; it missed a ProMED report on 19/2/2010 on cholera in Bocoio; it detected a Spanish translation on 6/3/2010 of an Angop report later cited by ProMED on 19/3/2010, which the paper argues should really be considered a true positive; and earlier prevention-campaign articles created a spike that suppressed later alerts by distorting the moving baseline (Collier, 2011). The broader implication is that ProMED-based benchmarking captures operational relevance, but only imperfectly captures event identity, onset time, and discourse structure.
4. ProMED as an operational signal and historical sentinel
Beyond benchmarking, ProMED is used directly as a surveillance signal. In work on the 2014 West African Ebola epidemic, ProMED is treated as an Internet-based, event-driven reporting system for rapid global dissemination of outbreak information in humans and animals, and as an early-warning and verification layer rather than a primary case registry (Hossain et al., 2022). The paper emphasizes that a ProMED request for information (RFI) on an “undiagnosed viral haemorrhagic fever” in Guinea appeared on 19 March 2014, later aligning with Ebola confirmation reported on 22 March 2014, with confirmation from a laboratory in Lyons, France on 21 March (Hossain et al., 2022).
The study searched ProMED reports from 19 March 2014 to 15 October 2014, covering 31 weeks; from 272 ProMED reports, 240 were judged relevant and 32 reports were excluded because they were repeated summaries of earlier ProMED postings (Hossain et al., 2022). ProMED headlines and RFIs were then used to build a week-by-week timeline of outbreak awareness, response, and geographic spread, showing the transition from Guinea to Liberia and Sierra Leone within about week 2, Nigeria by week 19, the first U.S. patient by week 28, and a U.S. healthcare worker infected after caring for that patient by week 30 (Hossain et al., 2022).
This use of ProMED is analytically distinct from its role as a benchmark. Here it functions as the earliest digital sentinel in the outbreak narrative and as a bridge between community or local observations, formal public health response, and global awareness through digital networks (Hossain et al., 2022). The paper’s language of strong ties, weak ties, and alters frames ProMED as part of a broader information infrastructure in which both confirmed clinical transmission and early community signals matter for preparedness and response.
5. ProMED as unstructured corpus for epidemic information extraction
A more recent line of work treats ProMED not primarily as an alert stream but as a large, curated, and operationally important unstructured text corpus. In the LLM-based extraction study, ProMED and WHO Disease Outbreak News (DONs) are the main sources for event-based epidemic surveillance, with ProMED singled out as the largest publicly accessible outbreak-reporting system, broadly used by public health officials, researchers, clinicians, veterinarians, journalists, and the general public (Consoli et al., 2024). The central problem is that ProMED is highly informative but difficult to operationalize automatically because the key epidemiological variables needed for modelling and forecasting—disease / outbreak name, country / location, date, and case count—are embedded in unstructured natural language (Consoli et al., 2024).
The study evaluates EpiTator, several open-source LLMs, GPT-3.5-turbo-16k, GPT-4-32k, GPT-4-FewShots, and an Open-Ensemble on a gold-standard subset of the Incident Database (IDB) with 171 carefully selected samples from ProMED and WHO DONs (Consoli et al., 2024). The prompt schema instructs the model to extract only what is present, to avoid invention, to use None when absent, and to return the output as JSON (Consoli et al., 2024). Reported results show that LLMs are substantially better than classical NLP on several extraction subtasks, with GPT-4-32k best overall across tasks, GPT-3.5-turbo-16k strongest on outbreak-name extraction, and GPT-4-FewShots best on case-count extraction (Consoli et al., 2024).
This suggests a shift in how ProMED is being integrated into computational epidemiology. In the earlier BioCaster literature, ProMED is a reference stream against which automated news mining is judged. In the LLM literature, ProMED itself becomes a primary input whose latent epidemiological structure must be normalized into machine-usable variables for near-real-time surveillance, epidemic modelling, forecasting, and situational awareness (Consoli et al., 2024).
6. Later reuse of the name “ProMed” in medical AI
By 2025, the label ProMed also appears in the titles of unrelated medical AI systems. “ProMed: Shapley Information Gain Guided Reinforcement Learning for Proactive Medical LLMs” defines ProMed as a reinforcement-learning framework that shifts medical LLMs from a reactive paradigm to a proactive paradigm, in which the model asks clinically valuable follow-up questions before answering under partial information (Ding et al., 19 Aug 2025). Its core mechanism is the Shapley Information Gain (SIG) reward, integrated into a two-stage pipeline consisting of SIG-Guided Model Initialization with Monte Carlo Tree Search (MCTS) and SIG-Augmented Policy Optimization built on GRPO. On two newly curated partial-information medical benchmarks, the paper reports that ProMed outperforms state-of-the-art methods by an average of 6.29% and delivers a 54.45% gain over the reactive paradigm (Ding et al., 19 Aug 2025).
A second, related but distinct usage is “ProMedTS: A Self-Supervised, Prompt-Guided Multimodal Approach for Integrating Medical Text and Time Series”, which addresses multimodal EHR fusion rather than outbreak surveillance (Niu et al., 19 Feb 2025). ProMedTS combines medical notes, structured lab-test time series, and IQR-based anomaly captions through three modules—Time Series Prompt Embedding (TSPE), Multimodal Textual Information Fusion (MTIF), and Self-Supervised Learning (SSL)—with three losses: , , and (Niu et al., 19 Feb 2025). On MIMIC-III, ProMedTS reports Micro F1 63.67 and Macro F1 60.42, while ProMedTS* reports Micro F1 64.02 and Macro F1 60.78; on MIMIC-IV, the corresponding figures are 69.69 / 66.21 and 70.21 / 67.56 (Niu et al., 19 Feb 2025).
These later usages are nomenclaturally close but substantively different from ProMED-mail. One concerns proactive clinical dialogue policy learning; the other concerns prompt-guided multimodal diagnosis from EHR data. Neither is an outbreak-reporting network. For technical readers, distinguishing ProMED-mail from ProMed and ProMedTS is therefore necessary to avoid category errors in citation, benchmarking, and system comparison.
7. Significance and interpretive boundaries
Across the surveillance literature, ProMED-type expert moderation is presented as extremely valuable both as a benchmark and as a practical signal source (Collier, 2011). It provides a reasonably objective, openly available, globally relevant reference for evaluating automated systems, and its curated postings expose both the strengths and weaknesses of automated alerting methods (Collier, 2011). Cross-lingual aggregation improves sensitivity, F1, and timeliness in most models, which indicates that ProMED remains useful even when multilingual evidence complicates alignment and increases false alarms (Collier, 2011).
At the same time, the literature is consistent that ProMED is not perfect ground truth. It reflects human judgment, changing reporting objectives, geographic and language coverage constraints, and topic-location grounding errors (Collier, 2011, Collier, 2011). In the Ebola analysis, it is most valuable when treated as an early digital sentinel and verification layer within a broader multi-source surveillance ecology (Hossain et al., 2022). In LLM-based information extraction, its value derives from the richness of its expert-curated narrative reports, but that same richness makes automatic structuring difficult (Consoli et al., 2024).
The most stable synthesis is therefore dual. ProMED is, first, an operationally important event-based surveillance system whose posts can surface local outbreaks before they are fully recognized by formal systems. It is, second, an evaluative and data-generating substrate for computational epidemiology, where its strengths are inseparable from editorial judgment and reporting bias. This dual role explains why ProMED remains central in outbreak informatics even as the same name is repurposed for unrelated medical AI methods.