---
title: 'TomoGraphView: 3D Medical Classification'
url: https://www.emergentmind.com/topics/tomographview
type: topic
---

# TomoGraphView: 3D Medical Classification

Searching arXiv for the specified paper and closely related context.
TomoGraphView is a two-stage framework for 3D medical image classification that combines omnidirectional volume slicing with graph-based feature aggregation in order to preserve volumetric geometry while exploiting pretrained 2D vision models [2511.09605]. It is designed for CT and MRI volumes containing segmented lesions, and addresses a central limitation of slice-based decomposition methods: canonical planes such as axial, sagittal, and coronal may fail to capture the spatial extent of target structures when those structures are misaligned with standardized viewing planes, while conventional slice-wise aggregation can lose spatial coherence across slices [2511.09605]. In TomoGraphView, a 3D volume is first decomposed into semantically rich omnidirectional 2D slices, then the resulting slice embeddings are re-assembled by a spherical graph neural network prediction head. The method is evaluated on six oncology classification problems and is presented as a way to bridge pretrained 2D vision foundation models and volumetric medical imaging [2511.09605].

## 1. Conceptual basis and problem setting

3D medical image classification is described as challenging because volumetric data exhibit complex spatial relationships and long-range dependencies [2511.09605]. The same source identifies two additional constraints: training models from scratch suffers from low data regimes, and the absence of 3D large-scale multimodal datasets has limited the development of 3D medical imaging foundation models [2511.09605]. Recent work has nevertheless highlighted the utility of 2D vision foundation models, originally trained on natural images, as feature extractors for medical image analysis, but direct application of 2D models to 3D volumes through standard slice-based decomposition is characterized as suboptimal [2511.09605].

The central design of TomoGraphView is therefore to decouple feature extraction from volumetric aggregation. A frozen 2D encoder extracts representations from individual slices, while a graph-structured head restores structural context at the level of the volume [2511.09605]. This suggests a methodological compromise between purely 3D architectures, which may be constrained by data availability, and purely 2D slice pipelines, which may underrepresent three-dimensional geometry.

The framework assumes a CT or MRI volume $V \subset \mathbb{R}^3$ containing a segmented lesion. A tight bounding box is extracted around the lesion, and the volume is resampled to isotropic voxel size $1\,\mathrm{mm}$ before omnidirectional slice generation [2511.09605]. This preprocessing step is not incidental: the entire sampling procedure is defined geometrically relative to the lesion-centered volume.

## 2. Omnidirectional volume slicing

The first stage encloses the preprocessed volume in a sphere

$$
S = \{ x \in \mathbb{R}^3 : \|x\|_2 \le r \}
$$

whose center coincides with the center of $V$ and whose radius $r$ just covers the bounding box [2511.09605]. TomoGraphView then samples $N$ viewpoints on this sphere. Three fixed canonical points are defined as

$$
p_1 = (r,0,0), \quad p_2 = (0,r,0), \quad p_3 = (0,0,r),
$$

corresponding to the axial, coronal, and sagittal normals [2511.09605]. The remaining $N-3$ points are initialized uniformly on the sphere and optimized by minimizing the total electrostatic Thomson energy

$$
E = \sum_{i \ne j} \frac{1}{\|x_i - x_j\|^2},
$$

subject to $\|x_i\| = r$ for all $i$ [2511.09605]. In practice, each nonfixed point is moved along the gradient of $E$ and re-projected onto the sphere until convergence, yielding a near-uniform distribution of normals $\{p_i\}$ [2511.09605].

Each normal $p_i$ defines a slicing plane through the lesion. For each direction, the method aligns a plane whose normal vector is $p_i$ and scans along that direction to find the single cross-section that maximizes the lesion’s 2D area, or equivalently, intersects the lesion mask most fully [2511.09605]. Every viewpoint therefore yields one 2D slice $I_i$ of size $224 \times 224$ after cropping and resampling [2511.09605].

This construction is the defining departure from canonical slicing. Rather than restricting the representation to standardized anatomical planes, it samples the lesion from a near-uniform set of directions on a sphere. A plausible implication is that the representation becomes less sensitive to orientation mismatch between lesion morphology and canonical viewing planes, particularly in cases of anisotropic acquisition where the diagnostically salient cross-section may not align with axial, sagittal, or coronal directions.

## 3. Spherical graph construction and feature aggregation

After slice generation, TomoGraphView forms an undirected graph $G=(V,E,A)$ whose $N$ nodes correspond to the sampled slice planes [2511.09605]. Let $P=\{p_1,\dots,p_N\}$ be the set of unit-norm normals. A spherical Delaunay triangulation is performed on $P$, and two points $p_i,p_j$ are connected if they share an edge in the triangulation:

$$
E_{\text{local}} = \{(i,j) \mid p_i,p_j \text{ adjacent in triangulation}\}.
$$

This produces a sparse, near-regular mesh on the sphere [2511.09605].

To enable longer-range message passing, the method can also add all cross-connections $(i,j)$ for $i \ne j$ in complete-graph fashion, weighting each edge by the inverse of the hop distance in the local mesh [2511.09605]. If $d_G(v_i,v_j)$ is the shortest-path length between nodes $i$ and $j$ in the local mesh, then

$$
w_{ij} = \frac{1}{d_G(v_i,v_j)},
$$

and the weighted adjacency matrix $A$ is defined with $A_{ij}=w_{ij}$ for $i \ne j$ and zero diagonal [2511.09605].

Each node is attributed with a 384-dimensional feature vector $x_i$ obtained from a frozen DINOv2-ViT-small encoder:

$$
x_i = f_{\text{DINOv2}}(I_i) \in \mathbb{R}^{384}.
$$

The graph aggregation module is a message-passing GNN in the style of GraphSAGE [2511.09605]. With $h_i^{(0)}=x_i$, layer $k$ computes

$$
m_i^{(k)} = \bigoplus_{j \in \mathcal{N}(i)} \left(e_{ji} \cdot h_j^{(k-1)}\right),
$$

where $\mathcal{N}(i)$ are the neighbors of node $i$ and $\bigoplus$ is a permutation-invariant aggregator; TomoGraphView uses mean and max aggregations in parallel and concatenates their outputs [2511.09605]. Node states are updated as

$$
h_i^{(k)} = \mathrm{ReLU}\left(W^{(k)}[\,h_i^{(k-1)} \,\|\, m_i^{(k)}\,]\right).
$$

After $K$ layers, typically $K=2$, a global readout takes the dimension-wise mean over nodes,

$$
h_G = \frac{1}{N}\sum_{i=1}^N h_i^{(K)},
$$

and a linear classifier $\sigma$ produces the predicted logit $y_G=\sigma(h_G)$ for binary classification [2511.09605]. Training uses standard binary cross-entropy loss [2511.09605].

The graph component is not merely a classifier appended to slice embeddings. Its stated role is to preserve volumetric geometry during aggregation. This suggests that the spherical adjacency is intended to encode inter-view relationships intrinsic to the sampling procedure itself, rather than treating slices as an unordered set or as a simple sequence.

## 4. Integration of pretrained 2D vision models

TomoGraphView uses a DINOv2 vision transformer, specifically the small variant with 21 M parameters, pretrained on 142 M natural images via self-supervised learning [2511.09605]. Each $224 \times 224$ slice is passed through the frozen DINOv2 backbone, and the 384-dimensional $[\mathrm{CLS}]$ token embedding is used as the node feature $x_i$ [2511.09605].

No fine-tuning of DINOv2 is performed. Only the GNN head and a small MLP classification head, consisting of two layers with ReLU in between and approximately 100 k trainable parameters, are trained end-to-end on each task [2511.09605]. This sharply constrains the trainable portion of the system relative to full backbone optimization.

The architecture therefore relies on transfer from natural-image self-supervised pretraining while avoiding task-specific adaptation of the 2D backbone. This suggests that TomoGraphView treats DINOv2 as a generic slice encoder and places most domain adaptation burden on the graph-based prediction head. A plausible implication is that the framework is especially suited to settings where dataset size is insufficient for robust full-model fine-tuning.

## 5. Training protocol and evaluation setting

The framework is evaluated on six public oncological tasks: brain tumor grading using T1-c MRI with $n=501$, head–neck HPV status using CT with $n=545$, breast tumor grade using DCE-MRI with $n=192$, liver tumor subtype using CT with $n=176$, kidney tumor grade using CT with $n=210$, and soft-tissue sarcoma grade using T2FS MRI with $n=291$ [2511.09605]. Volumes are reoriented to RAS+, resampled to $1\,\mathrm{mm}$ isotropic resolution, and cropped around lesion masks [2511.09605].

Evaluation uses 5-fold stratified cross-validation with three folds for training, one for validation, and one for testing [2511.09605]. Optimization is performed with SGD using weight decay $=1\mathrm{e}{-3}$, momentum $0.9$, batch size $16$, and $300$ epochs [2511.09605]. The learning rate is linearly warmed up over $100$ epochs to a peak of $1\mathrm{e}{-3}$, then reduced by a factor of $0.95$ on any validation-AUROC stall of five epochs [2511.09605]. The best model by validation AUROC is evaluated on the held-out test fold, and no additional data augmentation is applied beyond cropping and resizing [2511.09605].

The primary metric is Area Under the Receiver-Operating Characteristic Curve (AUROC), with balanced accuracy, F1-score, and Matthews correlation coefficient (MCC) used in ablation tables [2511.09605]. The emphasis on AUROC is consistent with the binary classification formulation used throughout the reported experiments.

## 6. Quantitative performance and ablation results

Across all six tasks, TomoGraphView achieves an average test AUROC of $0.8372$ with $N=24$ viewpoints [2511.09605]. This is compared with $0.8154$ when omnidirectional slices are pooled by a simple MLP, and $0.7701$ for canonical slice-based MLP baselines [2511.09605]. In head–neck, kidney, and liver classification, the GNN head yields gains of $+3$ to $+5$ AUROC over the omnidirectional+MLP variant [2511.09605].

The method is also compared against five large-scale pretrained 3D backbones—Models Genesis, VOCO, VISTA3D, FMCIB, and SwinUNETR—using frozen embeddings plus a 100 k-parameter classifier [2511.09605]. Under this comparison, TomoGraphView surpasses the best 3D model, FMCIB, with average AUROC $0.8282$ versus $0.7170$ [2511.09605]. The report further states that even FMCIB fine-tuned for 200 epochs with large batch sizes does not match TomoGraphView’s average AUROC of $0.8328$ [2511.09605].

Three ablation axes are reported [2511.09605]. First, varying the number of omnidirectional views $N \in \{8,16,24\}$ shows steady improvement as $N$ increases, especially on datasets with high slice anisotropy, for example $z$-spacing $>3\,\mathrm{mm}$ [2511.09605]. Second, graph topology is varied among local Delaunay-only connectivity and complete graphs with uniform, inverse-hop, and inverse-hop$^2$ weighting. The complete graph with inverse-hop weighting yields the best mean AUROC of $0.8372$, while the sparse local mesh still attains $0.8305$ [2511.09605]. Third, alternative slice-aggregation baselines also benefit from omnidirectional slicing: LSTM AUROC rises from $0.7446 \rightarrow 0.7863$, and Medical Slice Transformer AUROC rises from $0.7746 \rightarrow 0.8198$, but neither matches the GNN head [2511.09605].

| Comparison | Reported average AUROC |
|---|---:|
| TomoGraphView, $N=24$ | 0.8372 |
| Omnidirectional slices + simple MLP | 0.8154 |
| Canonical slice-based MLP baseline | 0.7701 |
| Best 3D model (FMCIB) | 0.7170 |
| Local mesh only | 0.8305 |

These results isolate two contributions: the omnidirectional slicing strategy itself and the graph-based aggregation mechanism. The ablations indicate that both matter. Omnidirectional views improve multiple non-graph aggregators, while the GNN further improves performance over omnidirectional slices pooled without explicit geometry [2511.09605].

## 7. Limitations, clinical implications, and projected extensions

The framework currently requires a lesion segmentation mask or bounding box to center slice sampling, which introduces a dependency on pre-existing delineations [2511.09605]. The source notes, however, that in many clinical workflows bounding boxes suffice and could be auto-generated [2511.09605]. Another limitation is that slice interpolation across $N$ orientations adds modest preprocessing cost [2511.09605].

The reported clinical relevance is framed around improved AUROC on tumor grading and HPV status prediction, which is said to suggest potential to assist radiologists in risk stratification, especially when 3D models are too data-hungry to train from scratch [2511.09605]. This should not be read as a claim of clinical deployment; rather, it indicates the intended downstream role of the framework in oncology-oriented decision support.

Several future directions are explicitly identified. One is integration of a detection module to propose bounding boxes, potentially making the pipeline end-to-end [2511.09605]. Another is the possibility of leveraging multimodal large language models for in-context 3D localization [2511.09605]. The framework is also described as flexible with respect to stronger 2D encoders or 3D feature extractors, and as potentially extensible to multi-class problems or survival-time prediction [2511.09605].

A common misconception would be to interpret TomoGraphView as a native 3D foundation model. The reported formulation does not do so. Instead, it is a wrapper that couples omnidirectional 2D sampling to a spherical-mesh GNN and a frozen pretrained 2D encoder [2511.09605]. Its contribution lies in how 3D structure is sampled and reconstructed at the representation level, not in end-to-end volumetric pretraining.

Source: https://www.emergentmind.com/topics/tomographview