ViTaL-Net: Multimodal Ovarian Tumor Classification

• The paper introduces ViTaL-Net, a multimodal neural network that integrates ultrasound, tabular, and linguistic data for six-category ovarian tumor classification.
• It employs a Triplet Hierarchical Offset Attention Mechanism to fuse features at the latent level, outperforming single-modality and simple concatenation strategies.
• Empirical results show approximately 85% accuracy and high AUC across major tumor types, highlighting its clinical potential and limitations with underrepresented classes.

Searching arXiv for papers explicitly using or closely matching “ViTaL-Net” to ground the article. ViTaL-Net is a multimodal neural network introduced for multi-pathological ovarian tumor recognition under a clinical setting in which radiologists and gynecologists use ultrasound images, structured examination data, and ultrasound reports jointly rather than relying on a single source of evidence. In the associated work, ViTaL-Net is defined as the principal model and benchmark on the ViTaL dataset, which contains Visual, Tabular and Linguistic modality data of 496 patients across six pathological categories and supports multi-class ovarian tumor classification rather than only benign–malignant discrimination (Zhou et al., 6 Jul 2025).

1. Clinical scope and problem formulation

ViTaL-Net is designed for a setting in which ovarian tumors must be classified into six specific pathological categories: Mature cystic teratoma (M.C.T), Endometriotic cyst (E.C), Serous cystadenoma (S.C), Mucinous cystadenoma (M.C), Thecomatous fibroma (T.F), and High-grade serous carcinoma (H.G.S.C) (Zhou et al., 6 Jul 2025). The motivating claim is that it is insufficient to merely distinguish between benign and malignant ovarian tumors in clinical practice, because subtype-level distinctions affect treatment and prognosis.

The learning problem is posed over a dataset

$$\mathcal{D} = \{(\mathbf{V}_n^i, \mathbf{T}_n, \mathbf{L}_n), y_n\}_{n=1}^N,$$

where $\mathbf{V}_n^i$ denotes the $i$-th ultrasound image slice for patient $n$, $\mathbf{T}_n$ the tabular features, $\mathbf{L}_n$ the linguistic features, and $y_n \in \{1,\dots,C\}$ with $C=6$ classes (Zhou et al., 6 Jul 2025). Training minimizes average loss over the training set,

$$\min_{f \in \mathcal{F}} \frac{1}{N} \sum_{n=1}^{N} \mathcal{L}(f(D_n), y_n),$$

with $f$ instantiated as the ViTaL-Net pipeline of modality encoders, the fusion block, and a linear decoder (Zhou et al., 6 Jul 2025).

A central design premise is that single-modality systems are inadequate for this task. The paper explicitly frames prior ovarian-tumor AI systems as predominantly image-only, usually limited to coarse benign/malignant classification, and often based on simple decision-level fusion when more than one modality is used. ViTaL-Net is therefore positioned as a three-modality alternative in which interaction among modalities is handled at the feature level rather than deferred to post hoc combination of independent predictions (Zhou et al., 6 Jul 2025).

2. Data foundation: the ViTaL dataset

The ViTaL dataset contains data from 496 patients and is organized into three modality-specific subsets: visual data from 2216 two-dimensional ultrasound images, tabular data from medical examinations of 496 patients, and linguistic data from ultrasound reports of 496 patients (Zhou et al., 6 Jul 2025). Each image has only one tumor type.

The visual modality consists of ultrasound images whose original sizes vary from 302–1135 px width and 226–794 px height; during preprocessing, they are resized to 224 × 224 by random resize and crop, with augmentation by flips and rotations (Zhou et al., 6 Jul 2025). The tabular modality is a 10-dimensional patient-level feature vector containing Age, BMI, Abdominal pain (yes/no), Abdominal bloating (yes/no), CA-125, CEA, CA19-9, AFP, CA15-3, and Maximum tumor diameter (Zhou et al., 6 Jul 2025). The linguistic modality consists of ultrasound reports describing tumor size, location, echo patterns, and related findings (Zhou et al., 6 Jul 2025).

Tabular preprocessing is modality-aware. The paper states that min–max scaling is used for age, BMI, and maximum tumor diameter,

$\mathbf{V}_n^i$0

0–1 encoding is used for abdominal pain and abdominal bloating, Gaussian normalization is used for highly skewed markers such as CEA and AFP,

$\mathbf{V}_n^i$1

and robust scaling is used for extreme markers such as CA125, CA199, and CA153,

$\mathbf{V}_n^i$2

These preprocessing choices are presented as part of the model’s practical pipeline rather than as a separate methodological contribution (Zhou et al., 6 Jul 2025).

The dataset also contains an additional “Other” category in the raw data, but this category is excluded from the main experiments due to heterogeneity and small sample size (Zhou et al., 6 Jul 2025). This exclusion matters materially: the paper reports 70.39% ACC, 0.86 AUC when “Other” is included and 85.59% ACC, 0.95 AUC when it is excluded (Zhou et al., 6 Jul 2025).

3. Network architecture and modality-specific encoders

At a high level, ViTaL-Net has three stages: single-modality feature extraction, Triplet Hierarchical Offset Attention Mechanism (THOAM) fusion, and a linear decoder for multi-class classification (Zhou et al., 6 Jul 2025). Each modality is first encoded independently into a shared latent dimension $\mathbf{V}_n^i$3, after which cross-modal attention is applied.

The visual branch uses MobileNet as backbone. Given an image batch, the visual encoder produces

$\mathbf{V}_n^i$4

A Global Average Pooling (GAP) operation then removes the spatial dimensions:

$\mathbf{V}_n^i$5

yielding an image-level representation

$\mathbf{V}_n^i$6

This pooled representation is the visual input to THOAM (Zhou et al., 6 Jul 2025).

The tabular branch uses TabNet, producing

$\mathbf{V}_n^i$7

and the linguistic branch uses BERT, producing a sentence-level feature

$\mathbf{V}_n^i$8

The critical alignment condition is therefore

$\mathbf{V}_n^i$9

which permits cross-attention to operate between any pair of modalities using standard linear projections for queries, keys, and values (Zhou et al., 6 Jul 2025).

This suggests an "embedding alignment" interpretation of ViTaL-Net: the model is not performing raw early fusion, but rather learning in a space where ultrasound appearance, structured laboratory and symptom variables, and report text are all represented at the same dimensionality before interaction. The paper presents this as a prerequisite for the subsequent attention-based fusion, not as a standalone objective (Zhou et al., 6 Jul 2025).

4. Triplet Hierarchical Offset Attention Mechanism

The core of ViTaL-Net is the Triplet Hierarchical Offset Attention Mechanism (THOAM), described as a cross-attention-based fusion mechanism that uses pairwise interactions between three modalities, applies them in a hierarchical sequence, and introduces offsets in a conceptual sense rather than as a separate explicit learnable term (Zhou et al., 6 Jul 2025). The hierarchy is two-stage: first Visual ↔ Tabular, then (Visual + Tabular) ↔ Linguistic.

Level 1: visual–tabular cross-attention

Given $i$0 and $i$1, THOAM computes

$i$2

$i$3

$i$4

The attention weights are then

$i$5

and the fused representation is

$i$6

This stage is intended to let visual features attend selectively to clinically relevant tabular attributes such as tumor markers, symptoms, and tumor size (Zhou et al., 6 Jul 2025).

Level 2: fused visual–tabular representation with linguistic features

The second stage uses the intermediate fused feature $i$7 as queries and linguistic features as keys and values:

$i$8

$i$9

$n$0

Attention proceeds as

$n$1

$n$2

The final fused representation concatenates this attended output with the original visual feature:

$n$3

The paper notes a minor typo in one equation, but the intended operation is precisely this concatenation of the final attended representation with the original visual embedding (Zhou et al., 6 Jul 2025).

The terms triplet, hierarchical, and offset have specific meanings in the paper. Triplet refers to the three modalities V, T, and L. Hierarchical refers to the sequence V+T first, then VT+L. Offset refers to the way attention weights adjust the contribution of one modality relative to another, rather than to a standalone offset parameter (Zhou et al., 6 Jul 2025). A common misconception is therefore to treat THOAM as a special-purpose offset-attention variant with an extra learnable correction term; the paper does not formulate it that way.

5. Decoder, optimization, and empirical performance

The classifier at the top of ViTaL-Net is a linear decoder:

$n$4

where $n$5 is the $n$6-th element of the fused feature vector, $n$7 maps features to classes, and $n$8 is the bias (Zhou et al., 6 Jul 2025). Training uses standard multi-class cross-entropy,

$n$9

with no additional contrastive, triplet, or auxiliary objective reported for THOAM itself (Zhou et al., 6 Jul 2025).

The implementation is reported in PyTorch with MMPretrain, trained on an NVIDIA RTX 4080 Super. For the backbones used in ViTaL-Net, the optimizer is SGD with initial learning rate 1e-3, StepLR decay factor 0.3 at epochs 30, 60, 90, weight decay 1e-5, batch size 32, and 100 epochs (Zhou et al., 6 Jul 2025). The paper does not mention explicit class weighting, focal loss, or resampling despite class imbalance (Zhou et al., 6 Jul 2025).

Against unimodal image baselines such as DenseNet-121, ResNet-34, ResNet-50, SE-ResNet-50, ShuffleNetV1, MobileNetV2, ViT, and EfficientNetV2, the strongest image-only result is reported for MobileNet, with 74.54% overall accuracy and per-class AUC between 0.71 and 0.98 (Zhou et al., 6 Jul 2025). Multimodal comparators include CBAM, ITCM, and TFA-LT, with overall accuracies of 76.04%, 73.71%, and 76.72%, respectively (Zhou et al., 6 Jul 2025).

ViTaL-Net is reported at 83.08% overall accuracy in one results table and 85.59% in the modality-fusion ablation table; the paper summary characterizes its overall performance as about 85% (Zhou et al., 6 Jul 2025). It also emphasizes accuracies exceeding 90% on the two most common pathological types of ovarian tumor and reports per-class AUCs of 0.98 for T.F, 0.98 for E.C, 0.99 for M.C.T, 0.93 for S.C, 0.98 for H.G.S.C, and 0.82 for M.C (Zhou et al., 6 Jul 2025).

Sensitivity and specificity analyses show that performance is uneven across classes. For ViTaL-Net, M.C is the hardest class, with SEN 34.15% and SPE 96.22%, whereas E.C and M.C.T show SEN 93.02% and 91.41%, respectively (Zhou et al., 6 Jul 2025). This suggests that the model is highly specific across classes but still susceptible to under-detection in the more difficult or less separable categories.

6. Ablations, interpretation, and position in the literature

The clearest empirical case for ViTaL-Net comes from the modality and fusion ablations. Using only V, T, or L gives 74.54 / 0.87, 59.46 / 0.84, and 69.01 / 0.85 in ACC / AUC, respectively. Pairwise combinations improve performance: V + T yields 78.39 / 0.94, V + L yields 78.89 / 0.92, and T + L yields 80.40 / 0.92. The full V + T + L configuration achieves 85.59 / 0.95 (Zhou et al., 6 Jul 2025). The paper therefore treats full three-modality integration as essential rather than optional.

Fusion strategy matters comparably strongly. A simple concatenation baseline gives 75.88% ACC and 0.93 AUC, whereas attention fusion via THOAM gives 85.59% ACC and 0.95 AUC (Zhou et al., 6 Jul 2025). The implication drawn in the paper is that simple concatenation is insufficient for heterogeneous multimodal evidence, because it does not explicitly model inter-modality relevance and complementarity.

Qualitative analyses are consistent with the quantitative results. The paper reports that normalized confusion matrices show stronger diagonal mass for ViTaL-Net, ROC curves are closer to 1 for most pathologies, t-SNE embeddings are better separated than those of competing models, and CAM heatmaps indicate that multimodal fusion leads the network to focus more precisely on tumor regions in ultrasound images (Zhou et al., 6 Jul 2025). A plausible implication is that the non-visual modalities do not merely provide redundant patient context but alter the effective visual attention of the model.

Within the broader arXiv literature, the name ViTaL-Net is potentially ambiguous. The ovarian-tumor model described above is distinct from unrelated systems that use similar acronyms, including a vital data analysis platform using wearable sensor for ECG, acceleration, posture, fatigue, and relaxation analysis (Yamato, 2017), VITAL for latent reasoning in medical MLLMs (Li et al., 27 May 2026), ViTa for joint visual–tabular cardiac MRI representation learning (Zhang et al., 17 Apr 2025), ViTaL for visuo-tactile inference-time steering in robotics (Wu et al., 12 Jun 2026), VitaLLM for edge LLM acceleration (Lin et al., 30 Apr 2026, Lin et al., 1 May 2026), and MD-ViSCo as a unified model for multi-directional vital sign waveform conversion (Meyer et al., 10 Jun 2025). A common misconception is therefore to interpret “ViTaL-Net” as a settled term for a single architecture family across domains; the arXiv record instead shows multiple unrelated expansions and design goals.

The limitations reported for the ovarian-tumor ViTaL-Net are conventional but material: dataset size and imbalance, the exclusion of the heterogeneous “Other” class, and uncertainty about generalizability and clinical deployment because the data come from a single center and one scanner (Zhou et al., 6 Jul 2025). The stated future directions are to collect more data, especially for underrepresented classes, use stronger augmentation, and perform external validation across hospitals, scanners, and populations (Zhou et al., 6 Jul 2025). In that sense, ViTaL-Net is best understood not as a completed clinical system but as a reference multimodal architecture and benchmark for six-way ovarian tumor classification on aligned ultrasound, clinical-tabular, and report-text data.