---
title: Early Risk Detection Error (ERDE)
url: https://www.emergentmind.com/topics/early-risk-detection-error-erde
type: topic
---

# Early Risk Detection Error (ERDE)

Early Risk Detection Error (ERDE) is a class of time-aware evaluation metrics explicitly designed to quantify both the accuracy and promptness of automated risk detection models when predicting adverse outcomes (e.g., mental health conditions) from temporal data streams such as social media posts or behavioral logs. ERDE emerged as the standard metric in the CLEF eRisk shared tasks and has since been widely adopted and generalized across sequential risk modeling domains, balancing penalties for late detection, incorrect detection, and missed detection in a single, parameterized framework [1911.06147][1804.07000][2106.16175][2410.17963][2505.11280][2605.23242].

## 1. Formal Definition and Variants of ERDE

The canonical ERDE metric (also denoted ERDE$_o$ or ERDE$_\theta$ depending on notation) assigns a per-user error defined by three components: (a) correctness of the detection (true/false positive/negative cases), (b) timeliness of positive detection relative to a deadline or grace parameter $o$ ($\theta$), and (c) application-specific unit costs for each case. The most common sigmoid-based form, as used in CLEF eRisk, is given by:

\[
\mathrm{ERDE}_o(d, k) = 
\begin{cases}
c_{fp}, & \text{if } d = p \land \mathrm{truth} = n \\
c_{fn}, & \text{if } d = n \land \mathrm{truth} = p \\
\ell_c(k)\cdot c_{tp}, & \text{if } d = p \land \mathrm{truth} = p \\
0, & \text{if } d = n \land \mathrm{truth} = n
\end{cases}
\]

where
\[
\ell_c(k) = 1 - \frac{1}{1 + \exp(k - o)}
\]
with $k$ the index at which a decision is made and $o$ the deadline parameter, typically set to $5$ or $50$ (number of posts/chunks) [1911.06147][1804.07000][2106.16175]. Unit costs $c_{fp}, c_{fn}, c_{tp}$ are usually set to $1$, but can be adjusted to reflect domain-specific trade-offs.

Linear and piecewise-linear variants exist, notably in recent BERT-based and synthetic benchmark evaluations, expressing delay penalty as $1 - \theta/d$ (for detection time $d > \theta$) or as $1 + \max(0, (d-k)/k)$ for integer cutoff $k$ [2410.17963][2106.16175][2605.23242].

ERDE$^%_o$ replaces the absolute count $k$ with a percentage of the user’s total data available, addressing biases in users with heterogeneous verbosity: 

\[
\ell^%_o(p) = 1 - \frac{1}{1 + \exp(p - o)}
\]
where $p = 100 \cdot k / n_d$ and $o$ is expressed as a percent threshold [1804.07000].

## 2. Intuitive Interpretation and Motivation

ERDE integrates three operational objectives:

- **Promptness**: Early, correct risk detection ("true positive" before the deadline) yields minimal or zero penalty; late detection is penalized increasingly as delay grows past $o$.
- **Specificity**: Any false positive (flag on a negative case) incurs a maximal penalty.
- **Missed Detection**: Failing to ever raise a positive decision for a true case (false negative) also receives the maximal penalty.

The latency cost $\ell_c(k)$ ensures that a correct prediction is not sufficient unless issued early; correctness is modulated by when the prediction is made. The deadline parameter $o$ encodes task-specific tolerance for evidence accumulation before full penalty is imposed: smaller $o$ enforces stricter earliness, while larger $o$ allows more leeway before delay costs are triggered [1911.06147][2410.17963][2505.11280].

## 3. Implementation Protocols and Evaluation Practice

In eRisk protocols and recent longitudinal evaluation frameworks (e.g., Cogniscope), a subject’s data is split into fixed-size temporal units (e.g., 10 "chunks" of posts per user). The system processes each unit sequentially, required to issue a binary decision (risk/no-risk) per subject, after which no further data from that user is ingested. ERDE is computed as the average per-user error across the test set:

- For CLEF eRisk, ERDE$_5$ and ERDE$_{50}$ are computed by setting $o=5$ or $o=50$, with the final score being the mean across users [1911.06147][2106.16175].
- In longitudinal benchmarks such as Cogniscope, for true positives, the penalty for late detection is linear with respect to the onset day and user-level grace window [2605.23242]:

\[
\mathrm{erde}_o(u)= 
\begin{cases} 
0, & y_u=1,\, t_u \le d_u \\ 
\min\left(1,\frac{t_u-d_u}{o}\right), & y_u=1,\, t_u>d_u \\ 
1, & y_u=1,\, t_u=\infty \\
1, & y_u=0,\, t_u<\infty \\
0, & y_u=0,\, t_u=\infty 
\end{cases}
\]

where $t_u$ is the first time the system alarmed, $d_u$ is ground-truth onset, and $o$ is the penalty window.

## 4. Comparative Analysis and Empirical Results

Empirical analyses demonstrate that enhancements in temporal modeling and context representation yield improved ERDE scores:

- **τ-SS3**—a text classifier integrating dynamic n-grams—achieves lower ERDE$_{50}$ compared to bag-of-words baselines on early depression/anorexia detection [1911.06147]. For example, ERDE$_{50}$ dropped from 8.12% (SS3) to 7.70% (τ-SS3) for eRisk 2017 depression, and to 6.17% on eRisk 2018 depression, setting state-of-the-art results.
- In benchmarks, transformer-based and time-aware models achieving earlier correct decisions consistently report lower ERDE than late-firing or conservatively thresholded models, even when raw F1 is similar [2505.11280][2410.17963][2106.16175].
- Use of ERDE$^%_o$ aligns system ranking more closely with intuitive early-detection behavior, especially when user post counts vary widely [1804.07000].

Notably, ERDE highlights the inherent trade-off: systems making aggressive early alarms risk high false positive penalty, while overly conservative systems incur steep delay or miss penalties.

## 5. Limitations, Modifications, and Ongoing Controversies

Critiques of ERDE focus on several systematic limitations:

- **Deadline/Parameter Sensitivity**: The deadline $o$ is task and dataset-dependent, requiring external calibration; varying $o$ can substantially alter relative model ranking [1911.06147][2410.17963].
- **Discrete Chunks vs. Proportional Data**: Original formulations penalize by count ($k$), leading to unfair assessments across users with heterogeneous data lengths. Proportional versions (ERDE$^%_o$) address this [1804.07000].
- **Unit Cost Uniformity**: In most evaluations $c_{tp}:c_{fp}:c_{fn}=1:1:1$, but domain mismatch between real-world consequences and these weights is noted. Some literature suggests increasing $c_{fp}$ to bias away from false alarms [2505.11280].
- **Late Decision Penalty**: In sigmoid-based ERDE, true positives made after $o$ incur penalties similar to false positives, sometimes under-rewarding models with moderate delay but high accuracy [1911.06147].
- **Complexity for Downstream Use**: The non-differentiable, piecewise nature of ERDE complicates its direct use as a training loss; however, recent work approximates ERDE with surrogate differentiable penalties in temporal fine-tuning of transformers [2505.11280][2410.17963].

## 6. Extensions and Related Metrics

Several alternatives build on or generalize ERDE:

- **Time-to-Detection (TTD)**: Average delay (in time units) between ground-truth onset and alarm, considering only detected positives, and disregarding false positives and missed cases [2605.23242].
- **F-latency**: Harmonic mean of precision and detection speed, often tracked alongside ERDE for model selection [2410.17963][2505.11280].
- **Ranking Metrics**: Precision@k, NDCG@k, used as complementary criteria to ERDE for systems designed for prioritized screening.
- **Sliding‐window Schemes and Delay Encodings**: Incorporation of explicit delay tokens into input representations and objective functions, allowing end-to-end optimization for early risk detection readiness [2505.11280][2410.17963].

## 7. Significance and Best Practices in ERDE-Optimized System Development

ERDE operationalizes the core requirement of timely and accurate intervention in longitudinal screening and monitoring, particularly in social or behavioral risk detection contexts. Key practices emerging from recent research include:

- Parameterizing and validating $o$ on held-out sets reflecting real-world timeliness demands.
- Encoding temporal delay into the input space when training temporal models.
- Employing proportional or percentage-based ERDE when subject activity levels are highly variable.
- Using ERDE, possibly in conjunction with TTD and F-latency, as early-stopping and model selection criteria.
- Calibrating error costs ($c_{fp}, c_{fn}$) to match deployment-specific risk tolerances and policy objectives [1911.06147][1804.07000][2505.11280][2410.17963].

ERDE and its extensions provide a rigorous, interpretable, and widely adopted standard for evaluating early risk detection systems under real-world constraints, and continue to shape the design and benchmarking of temporal models in health and safety contexts.

Source: https://www.emergentmind.com/topics/early-risk-detection-error-erde