DinoAtten3D: Attention-Based Global Aggregation

• The paper presents DinoAtten3D, which adaptively aggregates DINOv2 feature embeddings via a trainable attention mechanism for accurate 3D MRI anomaly classification.
• It leverages a frozen Vision Transformer to extract per-slice features and a two-layer MLP to assign context-sensitive attention weights, balancing diagnostic precision with computational efficiency.
• Empirical evaluations demonstrate DinoAtten3D outperforms 3D CNN baselines by achieving up to 90% accuracy and superior AUC scores in detecting brain anomalies.

Attention-based Global Aggregation (DinoAtten3D) refers to a class of neural architectures that apply soft-attention mechanisms to adaptively weight and aggregate local or tokenized feature representations for the purpose of volumetric or set-level decision-making. In DinoAtten3D, this methodology is applied to the medical imaging context, specifically 3D brain MRI anomaly classification, by leveraging a frozen DINOv2 Vision Transformer (ViT) as a per-slice feature extractor and a trainable attention-based aggregation for global volumetric semantics (Rafsani et al., 15 Sep 2025). The approach is situated within a broader family of attention-based global aggregation strategies developed for a range of structured data—including 3D point clouds and graphs—in which the challenge is to capture long-range inter-element dependencies without incurring prohibitive computational costs (Deng et al., 2021, Li et al., 2024, Mostafa et al., 2020).

1. Core Architecture and Feature Flow

DinoAtten3D processes a 3D MRI volume by decomposing it into an ordered set of $N$ axial 2D slices $$\mathcal{S} = \{S_j \in \mathbb{R}^{C \times H \times W} \mid j=1, \ldots, N\}$$, where each slice is independently transformed into a $d$-dimensional embedding using the frozen DINOv2 ViT-S/14 backbone: $$f_{\rm Dino}\colon \mathbb{R}^{C \times H \times W} \longrightarrow \mathbb{R}^d$$, with $d=384$. The resulting slice embeddings $\mathbf{z}_j$ (for $j=1, \ldots, N$) are the basis for subsequent global aggregation.

Rather than relying on naive pooling operators (mean, max) or simple set-based MIL approaches, DinoAtten3D applies a learned, slice-level attention: a two-layer MLP computes an unnormalized relevance score $e_j$ for each slice embedding, which is transformed into an attention weight $\alpha_j$ via softmax normalization across all $N$ slices: $$\mathcal{S} = \{S_j \in \mathbb{R}^{C \times H \times W} \mid j=1, \ldots, N\}$$0 The attention-weighted sum produces a global, volume-level representation: $$\mathcal{S} = \{S_j \in \mathbb{R}^{C \times H \times W} \mid j=1, \ldots, N\}$$1 This vector feeds into a lightweight MLP head and a final linear classifier for anomaly prediction (Rafsani et al., 15 Sep 2025).

2. Attention-Based Global Aggregation: Methodological Foundations

The attention-based global aggregation approach instantiated in DinoAtten3D builds on general principles developed for 3D perception networks and graph models, where the key goal is efficient modeling of non-local dependencies and effective context pooling.

Key characteristics of attention-based global aggregation, as exemplified by DinoAtten3D, GA-Net (Deng et al., 2021), GAD (Li et al., 2024), and Permutohedral-GCN (Mostafa et al., 2020), include:

• Context-Sensitive Aggregation: Attention modules assign relevance weights to individual spatial or structural units (slices, points, or nodes), enabling adaptive pooling based on learned diagnostic or semantic importance rather than fixed spatial proximity or uniform treatment.
• Efficiency Considerations: Full non-local attention incurs $$\mathcal{S} = \{S_j \in \mathbb{R}^{C \times H \times W} \mid j=1, \ldots, N\}$$2 computational cost; DinoAtten3D sidesteps this by operating over a manageable number ($$\mathcal{S} = \{S_j \in \mathbb{R}^{C \times H \times W} \mid j=1, \ldots, N\}$$3) of slices and parametrizing attention as a shallow MLP. In large-scale point cloud settings, block-wise approximations (Deng et al., 2021) and fast filtering via permutohedral lattices (Mostafa et al., 2020) yield sub-quadratic or linear scaling.

Architecture	Data Domain	Attention Granularity	Global Feature Aggregation
DinoAtten3D (Rafsani et al., 15 Sep 2025)	3D brain MRI	2D slice-level	Soft attention over slice embeddings
GA-Net (Deng et al., 2021)	3D point cloud	Point-level (random cross)	RCAB + fusion with PIGA
GAD (Li et al., 2024)	3D point cloud	Point-level (positional)	Channel max-pool on transformer features
Permutohedral-GCN (Mostafa et al., 2020)	Graph	Node-level	Permutohedral Gaussian filter with global soft-attention

3. Composite Loss Function and Training Strategies

To address challenges of limited labelled data and severe class imbalance, DinoAtten3D integrates a composite loss:

• Supervised Contrastive Loss: Imposes similarity between final normalized embeddings of volumes with identical labels, while encouraging separation between distinct classes. For batch size $$\mathcal{S} = \{S_j \in \mathbb{R}^{C \times H \times W} \mid j=1, \ldots, N\}$$4 and temperature $$\mathcal{S} = \{S_j \in \mathbb{R}^{C \times H \times W} \mid j=1, \ldots, N\}$$5, similarity is $$\mathcal{S} = \{S_j \in \mathbb{R}^{C \times H \times W} \mid j=1, \ldots, N\}$$6, with loss: $$\mathcal{S} = \{S_j \in \mathbb{R}^{C \times H \times W} \mid j=1, \ldots, N\}$$7
• Class-Variance Regularization: Tightens intra-class clusters by penalizing within-class embedding spread: $$\mathcal{S} = \{S_j \in \mathbb{R}^{C \times H \times W} \mid j=1, \ldots, N\}$$8 Total loss is $$\mathcal{S} = \{S_j \in \mathbb{R}^{C \times H \times W} \mid j=1, \ldots, N\}$$9, where $d$0 (Rafsani et al., 15 Sep 2025).

All DINOv2 weights remain frozen. Only the attention and classification heads are trained, typically with Adam over 50–100 epochs, with early stopping on validation performance.

4. Empirical Performance and Benchmarking

DinoAtten3D demonstrates robust volumetric anomaly detection performance in data-constrained and imbalanced medical imaging settings. On the ADNI Alzheimer’s dataset (4,769 scans), pairwise discrimination achieves up to 87.8% accuracy (AUC = 0.865) for healthy control (HC) vs. Alzheimer's disease (AD). For HC vs. migraine in an institutional headache cohort, 90.0% accuracy and 0.992 AUC are reported. The model systematically outperforms 3D CNN baselines (e.g., 3D ResNet) and multiple-instance learning variants (SC-MIL) by 2–10% in both accuracy and AUC across tasks.

These results confirm that slice-level soft attention over pretrained DINOv2 embeddings is capable of effectively capturing both focal and distributed pathology in brain MRI, demonstrating particular advantages where inter-class distinctions are pronounced and annotation resources are scarce (Rafsani et al., 15 Sep 2025).

5. Relation to Broader Attention-Based Aggregation Paradigms

Attention-based global aggregation, as exemplified by DinoAtten3D, is co-evolving with similar strategies in other structured-data modalities:

• GA-Net (Deng et al., 2021): Defines both point-independent and point-dependent global attention modules for 3D point clouds, using Random Cross Attention Blocks (RCAB) to approximate full non-local dependencies with $d$1 computational cost, and per-point adaptive feature aggregation. Empirical evidence shows systematic improvements over RandLA-Net on datasets such as Semantic3D and S3DIS.
• GAD (Li et al., 2024): Combines a Contextual Position-enhanced Transformer for full-set attention with localized dual-domain KNN feature fusion, emphasizing the value of integrating global attention cues with local geometric/feature neighborhoods.
• Permutohedral-GCN (Mostafa et al., 2020): Develops a fully global attention layer for graphs based on a high-dimensional Gaussian kernel, implemented efficiently through permutohedral lattice filtering. The approach provides scalable, differentiable global aggregation in $d$2 time, outperforming GAT/GCN baselines—especially on graphs with low assortativity.

A common thematic element across these architectures is the trade-off between modeling expressivity (via truly global or near-global attention) and computational efficiency. Architectures like DinoAtten3D achieve tractability by limiting the global context to a natural subdivision (slices), with attention parametrized through a compact MLP; others (GA-Net, Permutohedral-GCN) employ structured low-rank or convolutional approximations.

6. Implementation Aspects and Public Resources

DinoAtten3D is implemented with standard neuroimaging preprocessing (N4 bias correction, skull stripping, non-linear registration to MNI-152, histogram-matching intensity normalization). Heavy 3D data augmentation is not required; randomization is limited to slice ordering during training. The full pipeline, pretrained weights, and scripts are available publicly at https://github.com/Rafsani/DinoAtten3D.git (Rafsani et al., 15 Sep 2025).

7. Context and Prospects

As the integration of foundation models and attention-based aggregation matures, future prospects include fine-tuning attention over temporally or hierarchically organized 3D data, scaling to larger contexts, and incorporating clinical metadata or spatial priors. A plausible implication is that the general framework established by DinoAtten3D (frozen pretrained patchwise feature extractors plus adaptive global attention) can enable high-performance transfer learning across a wide spectrum of volumetric and structured modality tasks where annotation remains a major bottleneck.