Interpretation of performance declines after excluding potentially leaked cases

Determine whether the observed declines in predictive performance after excluding holdout observations potentially affected by unregistered-partnership data leakage reflect genuine leakage or the removal of an inherently more predictable subgroup.

Background

The paper identifies a possible source of outcome-related data leakage in the Dutch register data: household type in 2020 may have been partly inferred from future births for some individuals who had a child early in the 2021–2023 outcome period. The authors assess this possibility by excluding potentially affected observations from the holdout set and comparing model performance.

For several models, performance declines after this exclusion, but the authors note that partnered individuals may also be inherently easier to predict because they have more stable household characteristics and partner-level information. Consequently, the observed declines cannot be unambiguously attributed to leakage, leaving unresolved whether they reflect genuine leakage or subgroup-composition effects.