T-FIX: Expert Alignment Benchmark
- T-FIX is a benchmark assessing expert alignment of LLM free-form explanations by emphasizing domain-relevant criteria over mere plausibility.
- The benchmark spans seven datasets from cosmology, psychology, and medicine with multimodal data, totaling 700 curated examples.
- Its evaluation pipeline uses atomic claim extraction, relevancy filtering, and expert alignment scoring to determine explanation completeness based on trusted domain criteria.
T-FIX is a benchmark and evaluation pipeline for assessing whether a LLM’s free-form textual explanation is aligned with expert reasoning in knowledge-intensive settings. It formalizes expert alignment as whether an explanation emphasizes the criteria a domain expert would use when making the same prediction, rather than merely appearing plausible or reflecting the model’s internal computation. The benchmark spans seven datasets from seven tasks across cosmology, psychology, and medicine, with text, vision, and time-series modalities, and is intentionally capped at 700 examples total, with 100 per dataset, to keep evaluation affordable given the use of multiple LLM calls in the scoring pipeline (Havaldar et al., 6 Nov 2025).
1. Conceptual basis
T-FIX introduces expert alignment as a third dimension of explanation quality alongside plausibility and faithfulness. In this formulation, plausibility concerns whether an explanation seems coherent to a human observer, whereas faithfulness concerns whether the explanation reflects the model’s actual internal decision process. T-FIX argues that neither criterion is sufficient in expert-facing domains, because an explanation can be plausible and faithful while still relying on features that experts consider weak, irrelevant, or non-diagnostic. Its motivating contrast is clinical: an explanation for sepsis prediction may be coherent and even faithful, yet still be poorly aligned if it emphasizes cues such as glucose or patient demeanor instead of clinically central factors such as age, hypotension, or sepsis scoring criteria (Havaldar et al., 6 Nov 2025).
This reframing is significant because it makes explanation evaluation explicitly audience-aware. The intended users are not generic annotators but clinicians, astrophysicists, psychologists, and other specialists who expect explanations grounded in domain-validated principles. T-FIX therefore treats explanation quality as an epistemic question: whether the rationale is built from the kinds of evidence experts trust. The benchmark also positions itself relative to earlier work on expert-interpretable feature groups, including FIX, but addresses a different output regime: modern LLMs produce free-form natural-language explanations, not structured attributions, so expert alignment must be evaluated at the level of textual claims rather than feature vectors or saliency maps (Havaldar et al., 6 Nov 2025).
2. Benchmark composition
T-FIX covers seven tasks distributed across three broad areas and three modalities. Each dataset contributes 100 examples, often balanced across labels where possible, yielding the benchmark-wide total of 700 examples (Havaldar et al., 6 Nov 2025).
| Area | Dataset | Task |
|---|---|---|
| Cosmology | Mass Maps | Predict and from weak lensing maps |
| Cosmology | Supernova | Classify astrophysical objects from multivariate time-series light curves |
| Psychology | Politeness | Rate multilingual utterances on a 1–5 politeness scale |
| Psychology | Emotion | Classify text into 27 emotions or neutral |
| Medicine | Laparoscopic Cholecystectomy | Identify safe and unsafe incision regions in surgical frames |
| Medicine | Cardiac Arrest | Predict high risk of cardiac arrest within 5 minutes |
| Medicine | Sepsis | Predict high risk of developing sepsis within 12 hours |
The datasets are multimodal by design. Mass Maps uses images from CosmoGrid, and the examples were upsampled and color-mapped using expert-defined thresholds to make relevant structures more visible to multimodal LLMs. Supernova uses PLAsTiCC light curves rendered as plots, with examples sampled across train, validation, and test splits to mitigate class imbalance and target roughly 7–8 per class. Politeness uses multilingual text from the holistic politeness dataset, balanced across labels and languages including English, Spanish, Japanese, and Chinese. Emotion uses GoEmotions, restricted to texts longer than 20 characters. Cholecystectomy converts masks into a 9×16 grid, yielding indexed regions. Cardiac and Sepsis are both derived from MC-MED subsets, with task-specific definitions for positives and balanced evaluation sets of 100 samples each (Havaldar et al., 6 Nov 2025).
The composition matters because T-FIX is not a single-domain diagnostic instrument. It is designed to test whether expert alignment can be defined and measured consistently across heterogeneous reasoning settings, including regression, multiclass classification, binary prediction, and structured region selection (Havaldar et al., 6 Nov 2025).
3. Expert criteria and expert-interpretable features
T-FIX operationalizes “features interpretable to experts” as expert alignment criteria: domain-relevant concepts, cues, and heuristics that experts actually use for the task. Criteria development proceeds in two stages. First, OpenAI’s o3 model is prompted to perform a field-specific literature review using the task description, example input-output pairs, and instructions to generate criteria with citations from reputable sources. Second, a domain expert for each dataset reviews the resulting list, removes incorrect or irrelevant items, adds missing important items, and iterates until the list reflects peer-acceptable consensus. The paper notes that only one expert per domain was used, although the literature-seeding step was intended to reduce single-expert bias (Havaldar et al., 6 Nov 2025).
The resulting criteria are domain-specific and technically concrete. In Mass Maps they include lensing peak abundance, void size and frequency, filament thickness and sharpness, fine-scale clumpiness, connectivity of the cosmic web, and density contrast extremes. In Supernova they include contiguous non-zero flux, rise–decline rates, photometric amplitude, event duration, periodic light curves, and secondary maxima. In Politeness they include honorifics, gratitude expressions, apologies, indirect or modal requests, hedging, inclusive pronouns, greetings, compliments, softened disagreement, urgency language, and avoidance of profanity. In Emotion they include valence, arousal, explicit emotion words and emojis, expressive punctuation, threat or worry language, self-blame, praise, gratitude, affection, and relief indicators. In Cholecystectomy they include Calot’s triangle clearance, cystic plate exposure, visibility of only two tubular structures, position relative to the R4U line, safe distance from the common bile duct, and aberrant artery caution. In Cardiac they include ventricular tachyarrhythmias, ventricular ectopy or NSVT, bradycardia, ST-segment changes, prolonged QT interval, severe hyperkalemia signs, age, sex, underlying cardiac disease, and critical illness. In Sepsis they include elderly susceptibility, SIRS positivity, high qSOFA, elevated NEWS, elevated serum lactate, elevated shock index, sepsis-associated hypotension, SOFA increase, and early antibiotic or culture orders (Havaldar et al., 6 Nov 2025).
These criteria are the benchmark’s reference ontology. T-FIX does not ask whether an explanation contains arbitrary medically or scientifically sounding content; it asks whether each claim can be matched to one of these expert-endorsed categories, and how completely it captures that category’s meaning (Havaldar et al., 6 Nov 2025).
4. Evaluation pipeline and scoring
The T-FIX pipeline has three stages, implemented in the main experiments with GPT-4o as evaluator. First, Atomic Claim Extraction decomposes a free-form explanation into self-contained, indivisible claims that each express a single verifiable fact. Second, Relevancy Filtering removes claims that are unsupported by the input, speculative, irrelevant, mere repetitions of the answer, or otherwise not explanatory. A claim is retained only if it is clearly grounded in the input and directly contributes to explaining the prediction. The paper reports that, on average, 72% of claims extracted in Stage 1 pass this filter. Third, Alignment Scoring compares each retained claim to the expert criteria, assigns the most aligned criterion if any, and labels the claim as complete, partial, or none (Havaldar et al., 6 Nov 2025).
The label semantics are explicitly defined. None means the claim is unrelated to the expert category or misinterprets it. Partial means the claim refers to the category only incompletely, with missing details, vagueness, overgeneralization, or noise. Complete means the claim is specific, directly relevant, and fully captures the category’s meaning and intent. Filtered-out claims are automatically assigned none. The explanation-level score is then obtained by mapping complete to 1, partial to 0.5, and none to 0, and averaging across claims. In compact form, the scoring rule is: where according to the claim label (Havaldar et al., 6 Nov 2025).
T-FIX supplements the claim-average score with two dataset-level analyses. One is coverage via Shannon entropy, which measures how broadly a model’s explanations distribute themselves across the available expert criteria. High entropy indicates broad coverage; low entropy indicates repeated reliance on a narrow subset of criteria. The other is Pearson correlation between expert alignment scores and predictive performance, reported as accuracy or MSE depending on task. These additions are methodologically important because they distinguish local claim quality from global reasoning breadth and from raw task success (Havaldar et al., 6 Nov 2025).
5. Validation and empirical findings
Because the benchmark depends on an LLM-mediated evaluator, T-FIX validates each stage against human annotation and domain expert interviews. The annotation study covers 35 examples total, 5 per domain, with 295 extracted claims, 211 aligned claims, 6 annotators total, and 2 annotators per example. Annotators were PhD students in machine learning with experience evaluating LLM outputs. The rubric is three-way at every stage—A, B, C—mapped to 1.0, 0.5, and 0.0 respectively. The reported validation results are: Claim Extraction, , Accuracy 0.943, Cohen’s ; Relevancy Filtering, , Accuracy 0.871, Cohen’s ; and Expert Alignment, , Accuracy 0.923, Cohen’s 0. The authors characterize these as moderate-to-substantial agreement (Havaldar et al., 6 Nov 2025).
The generation-side benchmark evaluates GPT-4o, o1, Gemini-2.0-Flash, and Claude-3.5-Sonnet, each under four prompting strategies: Vanilla, Chain-of-Thought, Socratic Prompting, and Subquestion Decomposition. Across domains, the central empirical result is that current LLM explanations are only moderately expert-aligned. Representative scores include Claude-3.5-Sonnet vanilla (0.622) on Mass Maps, GPT-4o vanilla (0.726) on Supernova, o1 CoT (0.555) on Emotion, o1 CoT (0.387) on Cholecystectomy, GPT-4o CoT (0.472) on Cardiac, and o1 SubQ (0.574) on Sepsis. Cholecystectomy is generally the weakest domain overall, while Supernova and some medical text tasks produce materially higher scores (Havaldar et al., 6 Nov 2025).
Several benchmark-level conclusions follow. First, more sophisticated prompting does not reliably improve expert alignment: Chain-of-Thought, Socratic prompting, and Subquestion decomposition do not consistently outperform vanilla prompting. Second, expert alignment and predictive performance are only weakly coupled, so accurate predictions do not imply expert-grounded explanations. Third, broader coverage of expert criteria, measured by higher Shannon entropy, is associated with stronger T-FIX performance; the paper notes that lower-performing domains such as Cholecystectomy and Supernova exhibit lower entropy, whereas higher-performing domains such as Politeness and Sepsis show broader criterion coverage. Fourth, the qualitative case studies show that T-FIX rewards mentioning the right concepts in the right way, not merely producing plausible domain language. For example, in Sepsis, “Fever and high heart rate are potential signs of sepsis” is scored as complete under SIRS Positivity, whereas a claim about increased platelet count is weakly aligned because low platelets are the clinically relevant direction. In Supernova, a claim about a prominent peak followed by gradual decline aligns strongly with Rise–decline rates, while merely noting lack of periodicity is weak alignment under Periodic light curves (Havaldar et al., 6 Nov 2025).
6. Limitations, interpretation, and nomenclature
T-FIX is explicit about its limitations. The pipeline relies on hand-crafted GPT-4o prompts and therefore remains prompt-dependent. The main evaluator is GPT-4o, creating potential evaluator entanglement even though the framework is model-agnostic in principle. Manual validation covers only 35 examples. Criteria construction uses only one expert per domain. The claim-wise scoring procedure can underrepresent compositional reasoning, especially in medicine, where experts noted that clinically meaningful judgment may depend on combining multiple claims rather than mapping each claim to one criterion independently. Reproducibility is also constrained by the use of closed-source LLM APIs that change over time (Havaldar et al., 6 Nov 2025).
The evaluator-robustness analysis partially mitigates, but does not eliminate, these concerns. When the evaluator is swapped among GPT-4o, Gemini-2.0-Flash, and Qwen2.5-VL-7B-Instruct on one dataset per domain, rankings are largely consistent. On Mass Maps, all three evaluators rank Claude-3.5-Sonnet first and o1 second. On Cholecystectomy, GPT-4o and Qwen behave similarly, whereas Gemini is overly conservative and scores all models at zero. This suggests that the benchmark is largely robust to evaluator choice, but not fully immune (Havaldar et al., 6 Nov 2025).
A separate interpretive issue is nomenclature. T-FIX in this sense is an explanation-evaluation benchmark, and should be distinguished from similarly named systems in automated program repair. TFix+ is a self-configuring hybrid timeout bug-fixing framework for cloud systems (He et al., 2021), and TFix is also used as the name of a window-based program-repair baseline in security-vulnerability fixing comparisons (Berabi et al., 2024). The overlap in naming can obscure the fact that T-FIX, properly speaking, is concerned with expert alignment of explanations rather than code repair.
The benchmark’s forward-looking significance lies in this reframing. It suggests that explanation evaluation in expert domains should not stop at plausibility or faithfulness, and that instruction tuning or benchmark-specific training may be needed if LLMs are to produce explanations that domain experts regard as epistemically sound. The paper accordingly proposes extending T-FIX to more domains, using multiple experts per domain, conducting larger human studies, improving evaluator robustness, and studying how expert-aligned explanations affect expert trust and downstream decision-making in practice (Havaldar et al., 6 Nov 2025).