TCR-XAI Benchmark: Molecular XAI Evaluation

• TCR-XAI Benchmark is a quantitative evaluation framework that assesses AI explanations for TCR-pMHC interactions using residue-level experimental data.
• It employs specialized metrics such as ROC-AUC, Log-Odds, AOPC, and BRHR to measure the alignment of computational attributions with true binding residues.
• The benchmark enhances trust in deep learning models by enabling rigorous, structure-driven analysis of both post-hoc and explain-by-design explainability methods.

The TCR-XAI Benchmark is a quantitative evaluation framework for explainable AI (XAI) methods in the context of T cell receptor–peptide–MHC (TCR-pMHC) interaction modeling. Designed specifically for biological sequence modeling with deep learning, it enables rigorous assessment of model interpretability by integrating ground-truth structural data, specialized explainability metrics, and comparison protocols for both post-hoc and explain-by-design architectures. The benchmark has been employed to evaluate state-of-the-art models and explanation techniques in several recent studies.

1. Motivation and Overview

The motivation for the TCR-XAI Benchmark arises from the need to interpret complex transformer models employed in TCR-pMHC binding prediction. Traditional XAI methods have been designed for self-attention or encoder-only architectures and remain highly abstract and difficult to evaluate quantitatively in molecular biology contexts. TCR-XAI addresses this challenge by constructing an empirical, structure-driven evaluation set that enables fine-grained, residue-level assessment of model explanations. The central design incorporates a dataset of experimentally determined TCR-pMHC structures, permitting direct comparison of model-identified binding residues against physically established contacts (Li et al., 3 Jul 2025, Li et al., 5 Oct 2025).

2. Construction of the TCR-XAI Benchmark

The TCR-XAI benchmark consists of 274 experimentally determined TCR–pMHC protein structures collected from well-curated resources, such as STCRDab and TCR3d 2.0. Only samples with complete TCR α- and β-chain sequences, intact peptide epitopes, and high-confidence atomic coordinates are retained. Ground truth for binding is represented as inter-residue distances: for each complex, the minimum distances are computed between CDR3 residues of both chains and their corresponding peptide atoms. Short distances identify the critical contacts underlying molecular interaction and are used as an objective measure for evaluating the accuracy of model explanations. To accommodate minor misalignments between computational attention and true physical proximity, a one-residue positional tolerance is applied, and smoothing is performed using convolutional filtering (Li et al., 3 Jul 2025, Li et al., 5 Oct 2025).

Component	Source	Role in Evaluation
Structures	STCRDab, TCR3d 2.0	Empirical ground truth
Annotation	CDR3α, CDR3β, Peptide	Residue-level attribution
Metric	Inter-residue distances	Binding region identification

3. Metric Design and Quantitative Evaluation

The core interpretability metrics in TCR-XAI relate explanation outputs to experimentally validated binding regions. The following metrics are used to compare the predictive attribution of different methods:

• ROC-AUC: Explanatory scores thresholded against the binary ground truth (residue as binding or non-binding based on experimentally determined distances), generating Receiver Operating Characteristic curves for each chain (CDR3α, CDR3β, epitope).
• Log-Odds Score (LOdds): Measures the decrease in model confidence when the top-k high-importance residues (as given by the explanation) are perturbed. More negative values indicate stronger reliance on predicted contacts.
• Area Over the Perturbation Curve (AOPC): Aggregates the average drop in prediction confidence as additional important residues are removed.
• Binding Region Hit Rate (BRHR): Fraction of ground-truth binding residues captured within the top percentile of predicted importance scores.

The BRHR is mathematically defined as the fraction (top-n ranked explanation scores) that overlap with the set of true binding residues for each molecular interaction.

4. Evaluation of XAI Methods Using TCR-XAI

The benchmark has been used to compare post-hoc explanation techniques tailored to the TCR-pMHC context. Notably, Quantifying Cross-Attention Interaction (QCAI) extends Grad-CAM ideas to decoder cross-attention layers present in transformer models such as TULIP (Li et al., 3 Jul 2025). QCAI decomposes attention matrices into query and key contributions using gradients, ReLU activation, and matrix operations:

$$S(A_\ell) = \mathbb{E}_H(\operatorname{ReLU}\left(\frac{\partial L^c}{\partial A_\ell} \odot A_\ell\right)) + I$$

where $A_\ell$ is the cross-attention matrix at decoder layer $\ell$, $L^c$ is the class-specific loss, $\mathbb{E}_H$ denotes averaging over attention heads, and $I$ is the residual connection.

The QCAI method further calculates query-conditioned and key-conditioned importance, employing the Moore–Penrose pseudoinverse approximation to disentangle cross-stream influences. Recursive aggregation across layers yields residue-level importance vectors that can be directly mapped to ground-truth binding regions.

Models with explainable architectural components—such as TCR-EML (“Explainable Model Layers” for TCR-pMHC prediction)—use prototype (contact) layers informed by biochemical binding structure and fuse chain-peptide interactions explicitly. Their output contact maps are compared quantitatively using BRHR and ROC-AUC (with TCR-EML achieving up to 99.9% ROC-AUC and 71.4% mean BRHR on Top-100 epitopes) (Li et al., 5 Oct 2025).

Metric	QCAI (Cross-Attn)	TCR-EML (Contact Layer)
ROC-AUC	>0.60 for epitopes	Up to 0.999 Top-100
BRHR	Highest among post-hoc methods	~0.84–0.90
LOdds/AOPC	Strongest	Not reported for EML

A plausible implication is that TCR-EML’s explain-by-design approach yields both interpretability and superior predictive power, whereas QCAI excels in post-hoc explanation of transformers with cross-attention.

5. Comparison with Other XAI Evaluation Platforms

Unlike general-purpose XAI benchmarks (e.g., Focus score in image mosaics (Arias-Duart et al., 2021), hierarchical qualitative/quantitative scores in Compare-xAI (Belaid et al., 2022), or ground-truth pixel-based metrics in EXACT (Clark et al., 20 May 2024)), TCR-XAI is tailored for molecular biology and emphasizes residue-level evaluation within experimentally determined protein complexes. The use of actual molecular structures as ground truth marks a distinctive approach: while most platforms rely on simulated or synthetic datasets, TCR-XAI leverages high-fidelity empirical annotations to assess feature attribution in a biologically relevant domain.

6. Significance and Implications

The TCR-XAI Benchmark serves several pivotal roles in advancing XAI for biological sequence modeling:

• It provides an objective, quantitative standard for interpretation quality, enabling model developers to compare explanation strategies not only in terms of faithfulness but also mechanistic alignment with experimental data.
• The benchmark reveals systematic limitations of both post-hoc and black-box architectures. For instance, QCAI’s application demonstrates that existing methods designed for encoder-only attention mechanisms are suboptimal for cross-attention, demanding specialized techniques for biological transformers (Li et al., 3 Jul 2025).
• The empirical validation offered by TCR-XAI encourages the integration of architectural components that explicitly reflect known binding mechanisms—potentially influencing future “explain-by-design” deep learning models in immunoinformatics and structural biology (Li et al., 5 Oct 2025).

This suggests that rigorous benchmark frameworks such as TCR-XAI are prerequisite for trustworthy deployment of predictive models in high-stakes biomedical applications, where both prediction accuracy and interpretability are essential for scientific and clinical confidence.

7. Future Directions

Further expansion of the TCR-XAI benchmark is anticipated, potentially including larger datasets, additional annotation dimensions (e.g., solvent accessibility, mutation impact), and more complex biochemical contexts. Integration with other evaluation initiatives across omics and clinical domains may yield unified interpretability standards. The demonstrated utility of TCR-XAI in distinguishing mechanistically faithful models and explanations suggests ongoing adoption in both research and translational settings.