Beyond Topical Similarity: Contrastive Evidence Retrieval with Interpretable Attention Alignment in RAG

Abstract: Ensuring factuality and interpretability in RAG remains an open and urgent problem. We introduce Contrastive Evidence Rationale Attention (CERA), the first retrieval framework to employ subjectivity-based hard negative selection and inject an evidential inductive bias into contrastive learning through an auxiliary attention alignment loss. CERA fine-tunes a dense retriever using two training objectives: triplet-based contrastive learning and interpretable attention alignment, which supervises CLS-to-token attention using a part-of-speech-weighted masking distribution over human-annotated factual rationales as evidence signals. Experiments on a large corpus of clinical trial reports demonstrate that the subjectivity-based hard negative selection substantially improves retrieval effectiveness compared to both Contriever and hard negative selection baselines. Furthermore, rationale alignment improves faithfulness while maintaining competitive retrieval performance, supporting the hypothesis that attention can serve as a more faithful explanation of model behavior when guided by human rationales. Moving beyond topical similarity, CERA enables the retriever to identify the specific tokens that constitute supporting evidence, promoting more interpretable evidence selection in RAG systems.

• The paper introduces CERA, a retrieval framework that combines subjectivity-based hard negatives with rationale-supervised attention, raising Recall@5 from 0.4301 to 0.5608 and MRR from 0.3288 to 0.4296 over a hard-negative baseline.
• CERA’s POS-weighted attention alignment substantially improves explanation faithfulness, reducing sufficiency from 0.2073 to 0.0939 while modestly improving token-level rationale overlap, although stronger alignment can reduce early-ranking precision.
• The results suggest that evidence-aware retrieval can outperform topical similarity for clinical RAG, but broader validation is needed because CERA relies on expert rationales, a crude subjectivity detector, and evaluation within a single biomedical dataset.

Motivation and problem statement

Retrieval-augmented generation (RAG) systems depend on dense retrievers to surface passages that support factual inference, yet standard retrievers optimize topical similarity rather than evidential adequacy. The authors of "Beyond Topical Similarity: Contrastive Evidence Retrieval with Interpretable Attention Alignment in RAG" (2606.01482) argue that this distinction is consequential: a passage can be lexically and semantically close to a query while being subjective, contradictory, or factually unsupported, and conventional contrastive training treats all non-relevant documents as equally negative, conflating factual disagreement with irrelevance. A second concern is interpretability: most retrieval work optimizes what evidence is retrieved, relying on post-hoc explanation techniques, without intrinsic mechanisms that make the model's attention align with human-identified evidence.

The paper introduces Contrastive Evidence Rationale Attention (CERA), described by its authors as the first retrieval framework combining two mechanisms: subjectivity-based hard negative selection, which uses highly subjective non-evidential chunks as hard negatives in triplet training, and interpretable attention alignment, an auxiliary KL-divergence loss that supervises CLS-to-token attention distributions against POS-weighted human-annotated evidence rationales. Together these inject what the authors term an evidential inductive bias — a learning preference toward representations grounded in human-annotated evidence rather than distributional similarity alone.

Method

CERA fine-tunes Contriever (2606.01482) on the Evidence Inference 2.0 dataset (2606.01482), a benchmark of clinical trial reports where each instance pairs an intervention–comparator–outcome (ICO) query with expert-annotated evidence spans and an effect label. Documents are split into 3,393 files (80/20 train/test) and chunked using a heuristic based on annotation-length statistics; chunks overlapping gold evidence spans are positives, yielding 13,004 triplets over 1,916 queries.

The training objective has two components. First, a cosine-based triplet loss separates positive evidence chunks from negatives, where negatives are the top-$K=5$ most subjective non-overlapping chunks within the same document, scored via TextBlob subjectivity. This choice is motivated by an appendix corpus analysis using TXM: hierarchical clustering and hypergeometric specificity analysis show that evidential chunks are lexically distinct from subjective chunks — evidence contains statistical terminology, numerals, and comparative adverbs such as "significantly," whereas subjective negatives use more general, methodological vocabulary. Fine-tuning under this scheme also improves embedding geometry, increasing inter-class distance from 0.5901 to 0.8110 relative to self-supervised Contriever.

Second, an alignment loss minimizes KL divergence between softmax-normalized final-layer CLS-to-token attention and a weighted rationale distribution $\tilde{\mathbf{r}}$, where each token inside the gold span receives a POS-dependent weight (NOUN/PROPN/VERB = 1.0 down to PUNCT = 0.0). The full objective is $$\mathcal{L} = \mathcal{L}_{\text{triplet}} + \lambda \mathcal{L}_{\text{alignment}}$$, evaluated at $\lambda \in \{0.01, 0.05\}$. Training uses AdamW at $1\times10^{-6}$, batch size 8, margin 0.2, and 10 epochs.

Retrieval results

Evaluation is local: for each query, ranking is restricted to the chunk pool of the corresponding PMCID document, measuring fine-grained evidence identification. CERA substantially outperforms both base Contriever and a conventional hard-negative baseline, with gains concentrated in early ranks:

Metric	Contriever	HardNeg baseline	CERA	Δ vs. HardNeg
Recall@1	0.0214	0.1248	0.2016	+0.0768
Recall@5	0.1169	0.4301	0.5608	+0.1307
NDCG@10	0.1027	0.3717	0.4776	+0.1059
MAP@10	0.0632	0.2712	0.3745	+0.1033
MRR	0.1063	0.3288	0.4296	+0.1009

These are large absolute improvements, particularly given that the baseline already uses hard negative mining; the implication is that which negatives are selected matters more than hardness per se, and that subjectivity provides a more informative contrastive signal than lexical or embedding proximity. An ablation across batch sizes (4–32), learning rates, schedulers, and epochs confirms robustness: CERA$_{\text{alignment}}$ beats HardNeg$_{\text{base}}$ by roughly 5–7 absolute points in Recall@10 across all configurations, indicating the gain stems from the negative selection strategy rather than hyperparameter tuning.

A complementary factuality evaluation uses three LLM judges (GPT-5.4, Qwen3-Max, Mistral Large 3) scoring retrieved spans 0–3 for factual consistency and usefulness against gold spans. CERA improves jury scores at Rank 1 from 1.1881 (Contriever) to 1.9015, with consistent gains across all judges and ranks — notably, Mistral Large 3 roughly doubles its score at Rank 1 (0.9306 → 1.8544). This supports the claim that subjectivity-based negatives improve the factual quality of retrieved evidence, not merely overlap-based relevance.

Interpretability results

Explanation quality is assessed with ERASER-style metrics (2606.01482): plausibility (IOU-F1, Token-F1) and faithfulness (comprehensiveness, sufficiency). Rationale alignment yields clear faithfulness gains at modest plausibility cost:

Model	IOU-F1 ↑	Token-F1 ↑	Comprehensiveness ↑	Sufficiency ↓
Contriever-base	0.1393	0.5204	0.1729	0.4273
CERA	0.1514	0.5242	0.1056	0.2073
CERA$_{\text{alignment}}$ λ=0.01	0.1570	0.5286	0.1173	0.1250
CERA$_{\text{alignment}}$ λ=0.05	0.1701	0.5283	0.1113	0.0939

Sufficiency drops from 0.2073 to 0.0939 under stronger alignment regularization, meaning the attended tokens alone nearly reproduce the full model score — direct evidence that supervised attention becomes more faithful when guided by human rationales. This bears on the long-standing debate over whether attention is explanatory (2606.01482): rather than assuming attention is inherently interpretable, CERA makes it so through explicit supervision, consistent with prior rationale-alignment work in classification settings.

However, the trade-off is real. Increasing $\lambda$ from 0.01 to 0.05 degrades early-ranking metrics systematically (Recall@1 falls from 0.1898 to 0.1747; MRR from 0.4166 to 0.3934), while large-cutoff metrics remain stable. The authors interpret this as the alignment objective over-constraining the encoder's ability to learn sharp discriminative rankings, concluding that moderate alignment strength offers the best balance. Notably, even the strongest alignment setting still outperforms base Contriever by wide margins, so the trade-off operates within an overall improved regime.

Limitations and open questions

The paper concedes several constraints plainly. Evaluation is confined to Evidence Inference 2.0, leaving generalization to other domains and global (cross-document) retrieval untested. The alignment objective requires costly, potentially subjective expert rationales, limiting scalability. Although faithfulness improves, the authors acknowledge that attention weights may still not fully reflect model reasoning — a caveat that applies to any attention-based explanation. Only two values of $\tilde{\mathbf{r}}$0 are explored, multi-hop reasoning and long-range dependencies are not addressed, and TextBlob subjectivity is a crude proxy whose behavior outside clinical text is unknown. Finally, the authors state explicitly that CERA should not be used as a standalone clinical decision-making system without human oversight. Open questions include whether subjectivity-based negative selection transfers beyond biomedical corpora, and how alignment strength can be adapted automatically rather than fixed a priori.

Conclusion

CERA demonstrates that dense retrievers can be trained to distinguish evidential support from topical similarity through two complementary interventions: selecting hard negatives by subjectivity, and supervising attention with POS-weighted human rationales. The former produces large retrieval gains (Recall@10 up ~11 points over a hard-negative baseline), and the latter substantially improves explanation faithfulness (sufficiency reduced by more than half) at a small, tunable cost in ranking precision. The results support the position that retrieval objectives should encode evidential structure, and that attention can serve as a faithful explanation when explicitly aligned with human evidence — provided the supervision signal itself is available and reliable.