Anatomically Informed Attention Guidance

• Anatomically Informed Attention Guidance (AIAG) is a method that integrates explicit anatomical priors, such as expert annotations and segmentation masks, into neural network attention mechanisms.
• It leverages detailed signals like radiologist gaze, landmark data, and graph-based anatomical structures to constrain attention and improve model predictions.
• Applications span radiology, cardiology, neuroimaging, and pathology, demonstrating enhanced metrics like classification accuracy and AUC in diagnostic tasks.

Anatomically Informed Attention Guidance (AIAG) refers to a class of methodologies in computational imaging and biomedical machine learning that enforce neural network attention mechanisms to focus on anatomically or pathologically relevant regions. These frameworks integrate explicit anatomical priors—derived from expert annotations, domain knowledge, landmarks, masks, or auxiliary segmentation—directly into model architectures or training objectives, thereby constraining the regions where attention is allocated. AIAG is used to enhance interpretability, reduce spurious correlations, minimize the use of irrelevant background, and improve diagnostic accuracy across domains including radiology, pathology, neuroimaging, and ophthalmology.

1. Principles and Formal Definitions

AIAG operates by linking attention weights or distributions to anatomical information, either explicitly (via region annotations, gaze data, or spatial masks) or implicitly (through structure-informed graphs or feedback from adversarial modules). The main formalism involves:

• Attention Weight Alignment: Attention scores $a_i$ for pixels, patches, or tokens are optimized (using KL divergence, MSE, or cross-entropy) to match anatomical priors $p_i$ extracted from regions of interest.
• Region-constrained Attention: Models are trained to allocate a specified proportion of attention $s_\ell = \sum_{i \in \ell} a_i$ to pre-defined regions $\ell$, informed by anatomy or pathology segmentation.
• Explicit Anatomical Masking: Input or feature maps are masked (element-wise product) using computed heatmaps or binary segmentation maps: $X' = \hat{A} \odot X$, forcing downstream predictors to operate on relevant tissue only.

These constraints are typically incorporated into the loss function, either via direct penalization ($\mathcal{L}_{\textrm{attn}}$) or by guiding the structure of network modules (e.g., anatomical tokenization, cross-attention between regions).

2. Data Sources and Anatomical Knowledge Integration

AIAG requires reliable anatomical signals. Typical sources are:

• Eye-tracking and Radiologist Gaze: As used in the I-AI pipeline (Pham et al., 2023), gaze fixations synced to diagnostic transcripts are filtered to anatomical masks, yielding spatially continuous heatmaps $H_a(x,y)$ reflecting dwell time and location. This enables supervised alignment of attention maps.
• Expert Landmarks and Structural Masks: In frameworks such as ViACT (Thorley et al., 2 Nov 2025) and LungEvaty (Brandt et al., 25 Nov 2025), myocardium contours, lobe masks, and nodule boxes are extracted via preprocessing and used to define region indices or token sets.
• Domain-guided Graphs: TractGraphFormer (Chen et al., 11 Jul 2024) and TractGraphCNN (Chen et al., 2023) compute graph adjacencies based on anatomical proximity (white-matter fiber distances) or connectivity, constructing static adjacency matrices $A$ and EdgeConv neighborhoods.
• Semantic Guidance from Pathology Detectors: Semantics-Aware Attention Guidance (SAG) (Liu et al., 16 Apr 2024) extracts tissue masks ($m_i^{TG}$) and heuristic regions (entity clustering, $m_i^{HG}$) from whole-slide images, normalized to patch-level guidance signals.

This anatomical knowledge can be acquired from manual annotation, pretrained segmentation models (e.g., SAMed, UNETR++), or cell clustering and mask extraction algorithms.

3. Architectural Instantiations

AIAG is realized in diverse network architectures, each adapted for its domain:

• Vision-LLMs with Adapter Fusion: I-AI (Pham et al., 2023) incorporates anatomical heatmaps and textual prompts via a ViT-based adapter, fusing BiomedCLIP features with anatomical tokens and guiding both attention and classification. Heatmap decoders produce predicted $H_a$ and mask images for downstream prediction.
• Deforming Anatomical Token Transformers: ViACT (Thorley et al., 2 Nov 2025) operates solely on myocardium point-patch tokens; attention mechanisms never process non-cardiac background. Anatomically masked autoencoding ensures representation focus within the myocardium.
• Hybrid Graph CNN–Transformer Networks: TractGraphFormer (Chen et al., 11 Jul 2024) fuses feature representations from EdgeConv graph CNNs (encoding local anatomical geometry) and global self-attention Transformers. Anatomical connectivity biases are injected prior to attention calculation or via late fusion weights.
• Anatomically-Informed GANs: Attention-GAN (Emami et al., 2020) extracts spatial attention from discriminator activations and gates the generator input to focus on bone/air interfaces, indirectly encoding anatomy without segmentation maps.
• Cross-Attention within Anatomically Partitioned Volumes: AI-CNet3D (Kenia et al., 1 Oct 2025) divides 3D OCT data along clinically relevant axes and computes directional cross-attention between superior/inferior and ONH/macula sub-volumes. Channel Attention Representations (CAREs) are visualized and aligned to Grad-CAMs during fine-tuning.
• Semantic Guidance Losses in MIL/Transformer Pathology Models: SAG (Liu et al., 16 Apr 2024) regularizes slide-level attention weights to match tissue and heuristic masks via auxiliary MSE and tissue in/out loss components.

4. Training Objectives and Attention Loss Design

The core of AIAG is its loss construction. Key formulations include:

• KL Divergence for Patch-level Alignment: $L_\mathrm{pix} = D_{\mathrm{KL}}(p\|a)$ as in LungEvaty (Brandt et al., 25 Nov 2025), aligning attention to nodule masks.
• Region-Level Cross-Entropy: $L_\mathrm{reg} = -\sum_\ell y_\ell \log s_\ell$ on cumulative regional attention, operational in lobe-based lung cancer prediction (Brandt et al., 25 Nov 2025).
• Combined Heatmap Matching and Classification: $L = L_\mathrm{attn} + L_\mathrm{cls}$, with $L_\mathrm{attn}$ including logit L2, cross-entropy, and Dice loss—for attention mask accuracy in I-AI (Pham et al., 2023).
• Semantic Guidance for Patch Attention: SAG (Liu et al., 16 Apr 2024) uses $L_\mathrm{mse}$ for heuristic patch-wise guidance and $L_\mathrm{in/out}$ for tissue inclusion, combined as $$L_\mathrm{att} = \lambda_1 L_\mathrm{mse} + \lambda_2 L_\mathrm{in/out}$$.
• Unsupervised Consistency Losses: AI-CNet3D (Kenia et al., 1 Oct 2025) minimizes the MSE between CARE heatmaps and Grad-CAMs, $$L_\mathrm{unsup} = \|H_\mathrm{CARE} - H_\mathrm{GradCAM}\|^2$$, balancing with supervised BCE for overall anatomy-guided attention.

Hyperparameter tuning (e.g., weight $\lambda_\mathrm{AIAG}$, mask thresholds, blend coefficients) is performed via validation or cross-validation.

5. Quantitative and Qualitative Evaluations

AIAG frameworks consistently show quantitative improvements in interpretability and diagnostic accuracy. Representative findings include:

Model/Domain	AIAG Impact on Key Metric	Baseline	AIAG/Proposed
I-AI (CXR)	Classification Accuracy (%)	ResNet-CAM 71.6; Karargyris 75.1	76.9
LungEvaty (CT)	AUC for Y1 Lung Cancer Risk	Baseline 0.928	+AIAG 0.945
ViACT (Echo)	CA Classification Accuracy (%)	ViViT 78.08; MAE-ST 80.25	81.84
TractGraphFormer	Sex Prediction (ABCD/HCP)	SVM 73.5/90.7; CNN 85.1/93.8	85.5/94.8
SAG (Pathology)	Melanoma 4-class Accuracy (%)	ScAtNet 58.1	62.7
AI-CNet3D (OCT)	Topcon AUROC (%)	ViT 0.640; SE-ResNeXt 0.792	0.818

Qualitative heatmap visualizations indicate AIAG-guided models closely track radiologist gaze or pathology masks (I-AI, SAG), localize cardiac pathology (ViACT), and highlight anatomically critical structures (bone/air in GANs, superior/inferior retina in AI-CNet3D).

Ablation studies show that excluding non-anatomical regions (masking irrelevant pixels in I-AI) can preserve ≥80% accuracy with only 18% of the image, demonstrating the sufficiency of anatomy-driven attention. In pathology models, IoU between attention and tumor masks increases by 12% with SAG.

6. Limitations and Future Extensions

AIAG frameworks depend on high-quality anatomical priors (gaze data, segmentation masks, expert annotations). Inaccuracies in lobe segmentation or cell entity detection can misguide attention (Brandt et al., 25 Nov 2025, Liu et al., 16 Apr 2024). Pixel-level guidance generally improves short-term prediction (months to years in cancer risk), but has limited effect on long-term outcomes driven by diffuse changes.

Further directions include:

• End-to-end learning of guidance signals (differentiable cluster proposals).
• Extension to longitudinal and multimodal data (temporal attention steering).
• Validation across external cohorts and imaging platforms.
• Graphical models for explicit anatomical adjacency and hierarchical representation.

An explicit disadvantage is added training complexity and minor compute overhead, though inference remains unchanged.

7. Applications and Broader Significance

AIAG has wide applications in medical imaging tasks where interpretability and anatomical fidelity are paramount:

• Radiology: CXR, CT, and MRI diagnosis, particularly where radiologist gaze or pathology-localized prediction is critical.
• Cardiology: Echocardiographic analysis constrained to myocardium for pathological event detection.
• Neuroimaging: dMRI tractography analyses enforcing white-matter or gray-matter anatomical neighborhoods.
• Digital Pathology: Cancer diagnosis in WSIs using patch-level semantic and tissue guidance.
• Ophthalmology: Glaucoma classification on 3D OCT via cross-regional attention and anatomical consistency.
• Adversarial Modality Transfer: SynCT generation focused on anatomical outliers and defects.

The consistent gains in accuracy, interpretability, and the prevention of spurious attention indicate AIAG’s utility as a paradigm for integrating domain knowledge with neural attention. Its adaptability and modularity allow integration into transformer-based, CNN-based, and hybrid architectures, advancing anatomically rigorous computational imaging.