AnnotateMissense: a genome-wide annotation and benchmarking framework for missense pathogenicity prediction

Abstract: Missense variant interpretation remains challenging because pathogenicity depends on heterogeneous evidence from population frequency, evolutionary conservation, transcript context, amino acid substitution severity, prior pathogenicity predictors and protein-language-model-derived features. We present AnnotateMissense, a scalable annotation, benchmarking and genome-wide prediction framework for missense variant interpretation. AnnotateMissense integrates hg38 missense variants derived from dbNSFP v5.1 with ANNOVAR annotations, dbNSFP transcript/protein descriptors, AlphaMissense scores, ESM-derived features, conservation metrics, population-frequency variables, established pathogenicity predictors and engineered amino acid/codon-context features. Using 132,714 ClinVar-labelled missense variants, we benchmarked machine-learning and deep-learning models under controlled feature configurations. The full 303-feature benchmark set achieved the strongest performance with XGBoost, reaching mean MCC = 0.9411 and ROC-AUC = 0.9950 across stratified five-fold cross-validation. Restricted naive and location-oriented feature sets achieved lower best MCC values of 0.4989 and 0.5113, respectively. Circularity-controlled ablations showed that removing prior-predictor, population-frequency and clinically overlapping evidence reduced performance, whereas excluding AlphaMissense and ESM-derived features alone had minimal effect. Temporal ClinVar validation on newly observed pathogenic/benign variants achieved MCC = 0.7613, accuracy = 0.8798 and F1-score = 0.8750. The final model was applied to 90,643,830 hg38 missense variants to generate AnnotateMissense pathogenicity scores and binary prediction labels. Code and outputs are available at https://github.com/MuhammadMuneeb007/CAGI7_Annotate_All_Missense and https://doi.org/10.5281/zenodo.19981867.

• The paper introduces a comprehensive genome-wide annotation framework that integrates diverse genomic, transcript, and protein-level features to assess missense variant pathogenicity.
• It employs advanced machine learning models like XGBoost with rigorous cross-validation, achieving ROC-AUC of 0.995 and MCC of 0.941 to benchmark predictive performance.
• It demonstrates superior performance over established predictors by highlighting the critical role of integrated annotations and mitigating issues like circularity in variant effect prediction.

Genome-Wide Annotation and Benchmarking of Missense Pathogenicity: A Technical Analysis of AnnotateMissense

Workflow and Feature Integration

AnnotateMissense implements a comprehensive workflow for large-scale missense variant annotation, integrating multi-source evidence to predict missense pathogenicity. It leverages chromosome-divided missense variant files derived from dbNSFP v5.1, incorporating genomic, transcript, and protein-level descriptors across over 90 million variants. Annotation is performed using both RefSeq and Ensembl gene models via ANNOVAR, followed by augmentation with population frequency metrics (gnomAD, ExAC, 1000 Genomes), vertebrate conservation scores (PhyloP, GERP++, SiPhy, PhastCons), prior variant effect predictors (SIFT, PolyPhen-2, CADD, REVEL, etc.), protein LLM outputs (AlphaMissense, ESM), and engineered biological features (BLOSUM62, amino acid property changes, codon context, CpG status).

Feature engineering is subject to rigorous row-independent and fold-specific preprocessing. Filtering removes features with excessive missingness or constant values as well as fields that could enable direct target leakage from ClinVar labels. Categorical encoding is optimized based on cardinality. From an initial set of 334 candidate features, 207 are retained for modelling, balancing annotation comprehensiveness with data quality.

Benchmark Dataset Construction and Model Training

ClinVar-annotated missense variants serve as the gold-standard benchmark, with clinical significance harmonized into a binary target. Quality control excludes uncertain or conflicting annotations. Stratified five-fold cross-validation is utilized, with fold-specific preprocessing to guard against information leakage.

Supervised models include XGBoost (primary), Random Forest, FLAML AutoML, TabNet, and deep neural network architectures (PyTorch, TensorFlow). The primary evaluation metrics are Matthews Correlation Coefficient (MCC) and ROC-AUC, with binary encoding for variant classes. Class balancing and input standardization protocols are applied per fold. Models are robust to missing values, and only features common to both train and test partitions are retained.

Benchmarking and Ablation Analysis

Performance evaluation spans seven feature configurations: full benchmark (303 features), naive (41 features), location-oriented (56 features), and various circularity-controlled ablations. The full benchmark achieves the strongest results—XGBoost attains a mean test ROC-AUC of 0.99497 and mean test MCC of 0.94108. Restricted feature sets (naive/location-oriented/engineered) exhibit pronounced performance drops (MCC ≈ 0.5), underscoring the criticality of multi-layer integration.

Circularity-controlled ablations demonstrate that exclusion of prior predictors and population frequency features causes a marked reduction in MCC (to ≈ 0.72), while removing only AlphaMissense and ESM has negligible impact (MCC remains ≈0.94). This indicates the dominance of curated annotation-level predictors in recapitulating clinical ground truth, with protein-language-model features offering limited independent signal once broader annotation features are included.

Comparison to Established Pathogenicity Predictors

AnnotateMissense-trained models significantly outperform established tools (MetaRNN, BayesDel, PolyPhen-2, SIFT, REVEL, CADD, AlphaMissense, PROVEAN), with MCC and accuracy values substantially higher on ClinVar-labeled data. For instance, XGBoost achieves an MCC of 0.981 and accuracy of 0.991, compared to MCC of 0.883 and accuracy of 0.942 for MetaRNN, and MCC of 0.426 and accuracy of 0.670 for REVEL. Correlation analyses reflect high fidelity (Pearson r ≈ 0.98) with ClinVar category assignments in the full model.

Discordance analysis against comparators quantifies signal complementarity. AnnotateMissense captures overlapping but non-identical predictive information, providing higher resolution particularly on variants discordant with protein LLMs (ESM1v). On discordant cases relative to ESM1v, AnnotateMissense agrees with ClinVar ground truth in 68% of instances, illustrating both broad concordance and model-wise complementarity.

Temporal Validation and Biological Utility

Temporal ClinVar validation using newly observed pathogenic/benign variants yields MCC = 0.76 and accuracy = 0.88, demonstrating robust generalization to updated clinical annotations. Sensitivity and specificity remain high, and F1-score of 0.88 further reinforces reliable discrimination.

Biological utility is evaluated via prioritization of variants of uncertain significance (VUS). AnnotateMissense assigns higher pathogenicity scores to VUS within missense-intolerant genes (gnomAD mis_z > 3), consistent with selective constraint, despite gene identity not being explicitly used as a training feature. Statistical enrichment against gene constraint groups is significant ($p < 1 \times 10^{-50}$, Mann–Whitney $U$ test), corroborating that model predictions align with biologically meaningful gene-level intolerance.

Practical Implications and Theoretical Considerations

AnnotateMissense illustrates the necessity of annotation-level feature integration to approach clinical benchmarking standards in variant pathogenicity prediction. Reliance on engineered sequence-derived features alone (such as amino acid chemistry, codon bias, conservation scores) is insufficient for high-fidelity pathogenicity inference; interdisciplinary integration spanning population genomics, computational prediction, and evolving protein LLMs is essential.

Circularity-controlled ablations raise concern about overfitting to clinical database proxies within the training data. While removal of clinical predictors substantially impairs accuracy, the incremental value of protein LLMs is limited in the presence of expert-curated annotation features. This suggests a theoretical plateau in predictive signal for current missense effect models, with future progress likely dependent on novel, orthogonal evidence layers.

On the practical front, AnnotateMissense enables genome-wide inference for over 90 million variants, providing resources for triage, prioritization, and benchmarking. However, the workflow is intended for research prioritization and should not be interpreted as an independent clinical diagnostic system.

Speculative Outlook for Future Research

The results suggest saturation of predictive signal using current variant effect annotation paradigms. Future advances may require integration of orthogonal experimental readouts, improved modelling of epistatic interactions, and longitudinal clinical validation frameworks. Additional enrichment from protein-specific, cell-type-specific, or disease-context-specific constraints could further improve discriminatory signal beyond what is achievable via annotation alone.

Expansion to non-missense variants, regulatory regions, and incorporation of RNA/protein structure-aware models may also enhance coverage. The continued evolution of protein LLMs holds promise, but their marginal utility is limited unless coupled with broader annotation evidence.

Conclusion

AnnotateMissense presents a comprehensive, scalable framework for genome-wide missense pathogenicity prediction, achieving high concordance with clinical gold standards and outperforming established predictors. Rigorous benchmarking and ablation analyses highlight the necessity of integrated annotation evidence and expose limits of independent sequence-derived or protein language-model features. The system offers substantial value for research triage and variant prioritization, facilitating exploration and benchmarking at genome scale, while setting the stage for future methodological refinement and expansion in variant consequence prediction (2605.24520).