HIDeM: Regularized Illness-Death Model

• HIDeM is a statistical framework for modeling interval-censored illness-death events where death acts as a semi-competing terminal event.
• It integrates over latent illness times and applies transition-specific penalization for effective variable selection in high dimensions.
• The method uses a proximal gradient algorithm with structured tuning to enhance predictor recovery and improve prediction accuracy.

Searching arXiv for the specific term and closely related entries to ground the article in current papers. HIDeM is a statistical method and an R package for fitting regularized illness-death multi-state models when the primary nonterminal event is interval-censored and death acts as a semi-competing terminal event. It is designed for settings in which disease status is assessed only at scheduled visits, so exact onset time is unknown, while death may occur between visits and may preclude any diagnosis. In that setting, HIDeM combines an illness-death likelihood that integrates over latent illness times with transition-specific penalization for high-dimensional covariate selection, and it is implemented as the R package HIDeM in the paper "A regularized multi-state model for covariate selection with interval-censored survival data" (Bercu et al., 22 Aug 2025).

1. Definition and scope

HIDeM addresses a specific inferential problem: the event of interest is observed only intermittently, the exact onset time is not available, and death may occur before the event is ever diagnosed. The motivating examples are aging and chronic-disease cohorts, especially dementia cohorts, where disease status is checked at visits while death time is usually known exactly or subject only to right-censoring (Bercu et al., 22 Aug 2025).

The method is built on the classical illness-death model. Each subject starts in state $0$ (“healthy”), may move to state $1$ (“ill/diseased”), and may subsequently move to state $2$ (“dead”); alternatively, death may occur directly from the healthy state. The three transitions are therefore $0\to1$, $0\to2$, and $1\to2$. This is the canonical semi-competing-risks structure because illness is nonterminal whereas death is terminal (Bercu et al., 22 Aug 2025).

The central methodological objective is not only estimation under interval censoring, but also transition-specific variable selection in high dimension. In HIDeM, a covariate may be selected for the illness transition but not for direct death, or vice versa, rather than being forced into a single global effect structure across all transitions (Bercu et al., 22 Aug 2025).

2. Illness-death formulation and interval-censored likelihood

For subject $i$, the unobserved illness and death times are denoted $T_i^I$ and $T_i^D$, and the observed data are written as

$$D_i = (V_{0i}, L_i, R_i, \delta_i^I, T_i, \delta_i^D),$$

where $1$0 is the entry visit time, $1$1 is the last visit at which the subject was observed healthy, $1$2 is the diagnosis visit time if diagnosed, $1$3 indicates disease diagnosis, $1$4 is the minimum of death time and administrative end of follow-up, and $1$5 indicates death during follow-up (Bercu et al., 22 Aug 2025).

HIDeM uses proportional transition intensities. For transition $1$6,

$1$7

equivalently

$1$8

The cumulative transition intensities are

$1$9

As presented, this is a Markov-type proportional intensity specification in current time $2$0 (Bercu et al., 22 Aug 2025).

The technical core is the subject-specific likelihood under interval-censored illness diagnosis. For diagnosed subjects, illness occurred at an unknown time $2$1, so the likelihood integrates over all such onset times. For subjects never diagnosed before censoring or death, the likelihood is a sum over two latent paths: remaining healthy until $2$2, or becoming ill at some unobserved time $2$3 and then surviving in state 1 until $2$4 or dying there (Bercu et al., 22 Aug 2025).

For individuals healthy through $2$5, diagnosed at $2$6, and then censored or dead at $2$7, the contribution is

$2$8

For subjects healthy through $2$9, never diagnosed, and then censored or dead at $0\to1$0, the contribution is

$0\to1$1

The full likelihood and log-likelihood are

$0\to1$2

When age is used as the time scale, left truncation is handled by dividing each individual contribution by the probability of remaining alive and healthy until entry: $0\to1$3 This allows delayed entry in cohorts where subjects must be alive and disease-free at enrollment (Bercu et al., 22 Aug 2025).

A practical implication is that HIDeM explicitly distinguishes a subject who truly remained healthy from a subject who became ill and died before diagnosis. This suggests why simpler models that ignore interval censoring can attribute mortality-associated predictors to disease onset.

3. Penalization and transition-specific covariate selection

The regression parameters are

$0\to1$4

with one coefficient vector per transition. HIDeM estimates them by maximizing a penalized log-likelihood: $0\to1$5

The penalty decomposes by transition: $0\to1$6 with

$0\to1$7

Here $0\to1$8 controls the lasso–ridge mixture, and $0\to1$9 are transition-specific penalty parameters (Bercu et al., 22 Aug 2025).

This transition-specific construction is one of the defining features of HIDeM. It allows different degrees of sparsity across the three transitions and makes it possible to recover predictors that act on only one pathway. The paper also states that the package implements MCP and SCAD penalties as alternatives, although the main methodological development is given for elastic net (Bercu et al., 22 Aug 2025).

4. Optimization algorithm and baseline intensity models

For fixed tuning parameters, HIDeM uses a proximal gradient hybrid algorithm. The inner loop alternates between a proximal coordinate gradient descent update for the regression coefficients $0\to2$0 and a Marquardt-Levenberg / Newton-like update for the baseline intensity parameters $0\to2$1 (Bercu et al., 22 Aug 2025).

For a single coefficient $0\to2$2, with the other coefficients fixed, the update is based on a second-order Taylor expansion of the log-likelihood. If $0\to2$3 belongs to transition $0\to2$4, the coordinate update has the soft-thresholding form

$0\to2$5

where $0\to2$6 is the soft-threshold operator (Bercu et al., 22 Aug 2025).

If the opposite Hessian is not positive definite, the paper states that it is inflated using the Marquardt-Levenberg idea for numerical stability. The baseline parameters are then updated by maximizing the ordinary log-likelihood with $0\to2$7 fixed (Bercu et al., 22 Aug 2025).

The inner loop stops when both parameter stability and relative penalized-objective stability criteria are met: $0\to2$8 and

$0\to2$9

Default tolerances are $1\to2$0 for both, although the simulations used $1\to2$1 and $1\to2$2 (Bercu et al., 22 Aug 2025).

The baseline transition intensities may be parameterized in two ways:

Baseline model	Formula	Role
Weibull	$1\to2$3	Parsimonious parametric baseline
M-spline	$1\to2$4	Flexible smooth baseline

In simulations, the M-spline knots were placed at $1\to2$5 years; in the real-data application, at $1\to2$6 years (Bercu et al., 22 Aug 2025).

The package uses analytical gradients and Hessians, detailed in the Web Appendix of the paper. This is important because the likelihood contains integrals over latent illness times and is more intricate than standard Cox-type likelihoods (Bercu et al., 22 Aug 2025).

5. Tuning, post-selection refitting, and prediction

Tuning in HIDeM is based on BIC, not cross-validation. The selected hyperparameters minimize

$1\to2$7

where $1\to2$8 is the active set of nonzero coefficients (Bercu et al., 22 Aug 2025).

Because a full four-dimensional grid search over $1\to2$9 is computationally costly, the paper uses a structured preselection strategy. It considers $i$0; for each transition and each $i$1, 20 values of $i$2 are evaluated in a transition-specific model; the 3 best values are retained by BIC; these are then combined to form $i$3 candidate hyperparameter combinations for the full model (Bercu et al., 22 Aug 2025).

After variable selection, HIDeM performs an unpenalized refit on the selected active set to reduce shrinkage bias. The refitted parameters $i$4 and $i$5 are obtained by ordinary maximum likelihood using Marquardt-Levenberg, with covariance taken from the inverse negative Hessian (Bercu et al., 22 Aug 2025).

The paper explicitly notes that this post-selection inference is naive in the sense that it ignores selection uncertainty, so the resulting variance estimates are likely underestimated. That limitation is part of the method’s stated scope rather than an omission (Bercu et al., 22 Aug 2025).

For prediction, the paper focuses on the cumulative probability of illness before the first event time $i$6: $i$7 This quantity is used in the simulation study to assess prediction error (Bercu et al., 22 Aug 2025).

6. Empirical evaluation and applied use

The simulation study in the HIDeM paper used training and test samples of 2,000 subjects each, with follow-up to 18 years. Visit schedules were either frequent (every 2.5 years, up to 8 visits) or sparse (every 4.5 years, up to 5 visits), with uniform jitter between 0 and $i$8 years and 5% noninformative dropout at each visit. There were 50 standardized Gaussian predictors, either independent or group-Toeplitz correlated with correlation $i$9 (Bercu et al., 22 Aug 2025).

The compared methods were:

• Regularized IDM-ICT: the proposed interval-censored illness-death model.
• Oracle IDM-ICT: unpenalized illness-death model using only true relevant covariates.
• Regularized IDM-TT: the same regularized illness-death idea using true illness times.
• Regularized PHM: a semi-competing-risk proportional hazards model ignoring interval censoring, using midpoint imputation and heuristic handling of deaths after the last healthy visit (Bercu et al., 22 Aug 2025).

The main findings are that the proposed regularized interval-censored illness-death model performed very well in variable selection and was close to the oracle and true-time illness-death models. For the $T_i^I$0 transition, true illness predictors were consistently selected, while predictors associated only with death or irrelevant variables were rarely selected, often below 5% (Bercu et al., 22 Aug 2025).

By contrast, the regularized PHM that ignored interval censoring often selected incorrect predictors for illness. In Scenario B, variables associated with death after illness were incorrectly selected for illness with frequencies roughly 35%–75%; in Scenario C, these false selections reached up to 90% (Bercu et al., 22 Aug 2025). This supports the paper’s central claim that ignoring interval censoring can cause mortality-related predictors to “bleed” into the illness model.

Prediction was evaluated using

$T_i^I$1

Under high event rates and frequent visits (Scenario B), mean MSEP was approximately $T_i^I$2 for Oracle IDM-ICT, $T_i^I$3 for Regularized IDM-ICT, and $T_i^I$4 for Regularized PHM. Under high event rates and sparse visits (Scenario C), the gap widened further, while under low event rates and frequent visits (Scenario A), the simpler PHM performed similarly (Bercu et al., 22 Aug 2025).

The applied illustration used the Three-City (3C) Study, with 1,979 subjects from Bordeaux and Dijon who had baseline MRI and complete baseline data. Follow-up visits occurred every 2 or 3 years, up to 20 years. Observed outcomes were 198 dementia diagnoses, 89 deaths after dementia, 403 deaths without dementia diagnosis, and 1,378 subjects alive and dementia-free at the end of follow-up (Bercu et al., 22 Aug 2025).

Starting from 27 baseline predictors, the regularized illness-death model with interval-censored illness time selected, for the $T_i^I$5 transition, age at entry, all 5 cognitive tests, hippocampus volume, and medial temporal lobe volume. For the $T_i^I$6 transition, selected predictors included male sex, older age, higher number of medications, and higher basal ganglia PVS count. For $T_i^I$7, only TIV was selected (Bercu et al., 22 Aug 2025).

The authors report that in this application both the regularized illness-death model and the regularized PHM selected the same predictors and produced similar transition intensity ratios, which they interpret as resembling Scenario A from the simulations, where interval censoring was less consequential (Bercu et al., 22 Aug 2025).

7. Disambiguation, limitations, and relation to similarly named work

The term HIDeM in current arXiv usage refers, in the evidence provided here, to the high-dimensional regularized illness-death modeling framework and R package introduced in 2025 (Bercu et al., 22 Aug 2025). Several near-matching acronyms refer to unrelated topics and should be distinguished carefully.

Most notably, HMID-Net is a hyperbolic vision-language pretraining method called Hyperbolic Masked Image and Distillation Network, and the paper explicitly states that it is not presented as “HIDeM” and contains no direct mention of the string “HIDeM” (Wang et al., 13 Jul 2025). Likewise, the papers "Image data hiding with multi-scale autoencoder network" and "Hiding Data Hiding" do not introduce a system named HIDeM, although they are conceptually related to steganography and hidden communication (Huang et al., 2022, Wu et al., 2021). In wireless communications, the acronym HMIM denotes hierarchical mode-based index modulation, not HIDeM (Huang et al., 14 Jan 2025). In gait recognition, HID refers to Human Identification at a Distance, again a different term (Ma et al., 7 Feb 2026).

Within its own domain, HIDeM has several explicitly stated limitations. The proposed framework handles time-fixed covariates only; it assumes proportional transition intensities; and its post-selection standard errors ignore model-selection uncertainty and are therefore likely too small (Bercu et al., 22 Aug 2025). The paper also makes clear that computational cost is nontrivial because each likelihood contribution may involve integration over latent illness times and because tuning requires a structured grid search.

A plausible implication is that HIDeM is most valuable when interval censoring is substantial, visit spacing is sparse, and death before diagnosis is common. Conversely, when visit intervals are short and event rates are low, simpler semi-competing-risk models may yield similar practical conclusions, as observed in both Scenario A and the Three-City application (Bercu et al., 22 Aug 2025).

The package is identified in the paper as HIDeM: In high dimension, estimation of Smooth Hazard Models for Interval-Censored Data with Applications to Survival and Illness-Death Models, and the cited GitHub repository is https://github.com/arianebercu/HIDeM (Bercu et al., 22 Aug 2025). The paper does not provide package function names or an API specification, so those details remain outside the documented methodological description.

In summary, HIDeM is a regularized interval-censored illness-death modeling framework whose defining contribution is the integration of latent illness-time likelihood construction with transition-specific high-dimensional variable selection. Its methodological significance lies in correcting a specific failure mode of simpler competing-risk analyses: the misattribution of death-related covariates to disease onset when illness can occur unobserved before death (Bercu et al., 22 Aug 2025).