Triplet Hierarchical Offset Attention (THOAM)

• The paper presents THOAM as a two-stage cross-attention fusion module that minimizes information loss by aligning visual, tabular, and linguistic clinical evidence.
• It employs a vision-anchored sequential fusion process where visual features first interact with tabular data before integrating linguistic cues for refined decision-making.
• Empirical results on the ViTaL dataset show substantial improvements in classification accuracy and AUC compared to simple concatenation and other fusion baselines.

Searching arXiv for the cited papers to ground the article and confirm metadata. Triplet Hierarchical Offset Attention Mechanism (THOAM) is the multimodal fusion core of ViTaL-Net, a network introduced for multi-pathology ovarian tumor recognition in the ViTaL benchmark (Zhou et al., 6 Jul 2025). In that setting, THOAM is designed to reduce the information loss associated with “simple and rigid fusion” of heterogeneous modalities by enhancing relevance and complementarity across three sources of clinical evidence: ultrasound images, structured examination indicators, and ultrasound reports (Zhou et al., 6 Jul 2025). The mechanism operates after single-modality encoding and before the final linear classifier, performing staged cross-attention over a visual feature, a tabular feature, and a linguistic feature. Although its name suggests a fully formalized triplet, hierarchical, and offset-based attention framework, the manuscript defines most concretely a two-stage multimodal cross-attention procedure anchored on vision; the “offset” component is named but not mathematically specified (Zhou et al., 6 Jul 2025).

1. Definition and task context

THOAM was proposed in the context of the ViTaL dataset, which contains Visual, Tabular and Linguistic modality data of 496 patients across six pathological categories (Zhou et al., 6 Jul 2025). The three subsets comprise visual data from 2216 two-dimensional ultrasound images, tabular data from medical examinations of 496 patients, and linguistic data from ultrasound reports of 496 patients (Zhou et al., 6 Jul 2025). The classification task uses six pathology classes: Mature cystic teratoma, Endometriotic cyst, Serous cystadenoma, Mucinous cystadenoma, Thecomatous fibroma, and High-grade serous carcinoma (Zhou et al., 6 Jul 2025).

The motivating claim is that ovarian tumor recognition should not rely on ultrasound images alone, nor on crude decision-level fusion, because clinically useful evidence is distributed across three different sources (Zhou et al., 6 Jul 2025). Ultrasound images encode morphology and local lesion appearance; tabular data encode symptoms, tumor markers, and size-related variables; ultrasound reports encode textual descriptions of location, shape, and lesion characteristics (Zhou et al., 6 Jul 2025). THOAM is introduced specifically to minimize the loss incurred during feature fusion of this heterogeneous information and to strengthen cross-modal relevance and complementarity (Zhou et al., 6 Jul 2025).

This framing also reflects a broader criticism of multimodal fusion by direct concatenation. The ViTaL work states that when multimodal information is fused too rigidly, such as through decision-level fusion or direct concatenation, “significant loss” can occur because the model lacks an explicit mechanism to align which clinical variables or textual cues are relevant to which visual findings (Zhou et al., 6 Jul 2025). A plausible implication is that THOAM should be understood primarily as an alignment-oriented fusion module rather than as an independent feature extractor.

2. Architectural placement in ViTaL-Net

ViTaL-Net is described as three stages: single-modality feature extraction, fusion of features from different modalities, and final multi-pathology classification (Zhou et al., 6 Jul 2025). THOAM occupies the second stage. It receives modality-specific embeddings from three distinct backbones: MobileNet for ultrasound images, TabNet for tabular clinical data, and BERT for ultrasound reports (Zhou et al., 6 Jul 2025).

The feature outputs are denoted

$(F_V, F_T, F_L).$

The visual feature map is originally spatial,

$F_V \in \mathbb{R}^{N \times C \times H \times W},$

where $N$ is batch size, $C$ the channel dimension, and $H,W$ spatial dimensions (Zhou et al., 6 Jul 2025). Before multimodal attention, the image feature is reduced by global average pooling: $(F'_V, F_T, F_L) \in \mathbb{R}^{N \times C}.$ THOAM then performs sequential fusion: first visual with tabular, yielding an intermediate representation $F_1$; then $F_1$ interacts with the linguistic feature to produce the final multimodal output (Zhou et al., 6 Jul 2025). The output is sent to a linear decoder for classification into one of six ovarian tumor categories (Zhou et al., 6 Jul 2025).

The module’s role is therefore not auxiliary. It is the central fusion engine of ViTaL-Net. Its stated inputs are the pooled visual feature $F'_V$, the tabular feature $F_T$, and the linguistic feature $F_V \in \mathbb{R}^{N \times C \times H \times W},$0; its outputs are an intermediate visual-tabular fused feature $F_V \in \mathbb{R}^{N \times C \times H \times W},$1 and a final fused representation $F_V \in \mathbb{R}^{N \times C \times H \times W},$2 (Zhou et al., 6 Jul 2025). The manuscript also states: $F_V \in \mathbb{R}^{N \times C \times H \times W},$3 Here “Concact” evidently means concatenate (Zhou et al., 6 Jul 2025).

A notable architectural property is that vision remains the anchor throughout the sequential fusion process. The first query comes from visual features, and the second query comes from the visual-tabular fusion. This design choice is explicitly aligned with the downstream task, where the primary observation is the ultrasound image and the tabular and textual modalities provide auxiliary context (Zhou et al., 6 Jul 2025).

3. Mechanism: triplet, hierarchical, attention, and “offset”

The “triplet” aspect of THOAM refers to the use of three modalities: visual, tabular, and linguistic (Zhou et al., 6 Jul 2025). The “hierarchical” aspect is embodied by a two-stage fusion order rather than by a deep formal hierarchy, multi-scale feature pyramid, or layer-wise hierarchy. The sequence is:

1. visual $F_V \in \mathbb{R}^{N \times C \times H \times W},$4 tabular fusion
2. fused visual-tabular $F_V \in \mathbb{R}^{N \times C \times H \times W},$5 linguistic fusion

The manuscript explicitly notes that hierarchy should therefore be understood as staged multimodal fusion, not as local/global or coarse/fine hierarchy in a formal sense (Zhou et al., 6 Jul 2025).

The attention itself is standard scaled dot-product cross-attention. In the first stage, the visual feature is the query source and the tabular feature provides keys and values: $F_V \in \mathbb{R}^{N \times C \times H \times W},$6 The surrounding text indicates that $F_V \in \mathbb{R}^{N \times C \times H \times W},$7 here should likely be interpreted as the pooled feature $F_V \in \mathbb{R}^{N \times C \times H \times W},$8, since attention starts after global average pooling; the notation is inconsistent in the manuscript (Zhou et al., 6 Jul 2025). The raw attention score is computed as

$F_V \in \mathbb{R}^{N \times C \times H \times W},$9

followed by

$N$0

The second stage repeats the same pattern, now using the intermediate fused representation as query and the linguistic feature as key/value: $N$1 Then

$N$2

followed by a linear projection and concatenation with the original visual feature (Zhou et al., 6 Jul 2025).

The manuscript’s most significant conceptual ambiguity concerns “offset.” It repeatedly uses the phrase “offset attention” and says it “enhances the ability to focus on specific regions or elements within each modality,” but it does not provide a mathematical definition of an offset term, an additive bias, a learned displacement, a residual difference such as $N$3, or a distinct parameter $N$4 (Zhou et al., 6 Jul 2025). The safest interpretation given the text is that “offset” refers informally to refined attention-based adjustment or alignment of modality features during staged fusion rather than to a rigorously defined offset operator (Zhou et al., 6 Jul 2025).

This limited formalization distinguishes THOAM from architectures in which hierarchy or offsets are mathematically primary. A later work on hierarchical self-attention, for example, derives a block-constrained hierarchy-aware attention law over tree-structured signals, but explicitly does not define a triplet mechanism or explicit learned offsets (Amizadeh et al., 18 Sep 2025). Conversely, a heterogeneous graph triplet attention network defines explicit triadic attention, but not hierarchical or offset attention in the strict sense (Tanvir et al., 2023). THOAM occupies a different position: its technical substance is staged triplet multimodal cross-attention centered on visual features (Zhou et al., 6 Jul 2025).

4. Modality encoders and representation preparation

The representations entering THOAM are produced by three modality-specific encoders. For ultrasound images, inputs are 2D scans resized or cropped to $N$5, with horizontal flips, vertical flips, and random rotations used for augmentation (Zhou et al., 6 Jul 2025). Feature extraction uses MobileNet, selected because it was the best image-only baseline among the tested models and because it is lightweight (Zhou et al., 6 Jul 2025). The resulting spatial feature map is

$N$6

which is reduced by global average pooling: $N$7 After pooling, the visual representation becomes

$N$8

For tabular clinical data, THOAM uses features generated by TabNet (Zhou et al., 6 Jul 2025). The structured input is 10-dimensional and consists of age, BMI, abdominal pain, abdominal bloating, five tumor markers, and maximum tumor diameter (Zhou et al., 6 Jul 2025). The reported preprocessing includes min-max normalization for age, BMI, and maximum diameter; binary 0/1 encoding for abdominal pain and abdominal bloating; Gaussian normalization for CEA and AFP,

$N$9

and robust scaling for markers with extreme outliers such as CA125,

$C$0

The manuscript’s printed robust-scaling equation is malformed, but these are the intended normalizations described in the text (Zhou et al., 6 Jul 2025).

For linguistic data, the model uses BERT as a pretrained text encoder for ultrasound reports (Zhou et al., 6 Jul 2025). The paper states that BERT is used “to encode the text in the text modality, thereby obtaining its feature map,” but it does not specify the BERT variant, tokenizer, sequence length, whether fine-tuning is end-to-end, or how the sequence output is pooled into $C$1 (Zhou et al., 6 Jul 2025). Similarly, the exact shared feature dimension $C$2 is not reported (Zhou et al., 6 Jul 2025).

These omissions are consequential. The conceptual logic of THOAM is recoverable from the manuscript, but exact architectural replication is not possible from the provided text alone because no feature dimension $C$3, attention dimension $C$4, number of heads, number of THOAM layers, dropout rate, exact TabNet configuration, or BERT variant is specified (Zhou et al., 6 Jul 2025).

5. Operational interpretation and learning objective

The staged fusion process can be restated directly from the textual description and formulas in the ViTaL paper (Zhou et al., 6 Jul 2025). First, each modality is encoded: $C$5 Then visual features query tabular features: $C$6

$C$7

Next, the fused visual-tabular representation queries linguistic features: $C$8

$C$9

Finally, the second-stage fusion result is concatenated with the original visual feature: $H,W$0 and passed to a linear decoder: $H,W$1 The manuscript also describes the decoder as a standard affine classifier: $H,W$2

Operationally, THOAM is therefore closer to pairwise staged cross-attention with residual preservation of the original visual signal than to a dedicated three-way multilinear interaction operator (Zhou et al., 6 Jul 2025). Alignment occurs because one modality queries another; relevance enhancement occurs because the attention weights select the most useful tabular or textual components relative to the current query; complementarity is modeled because nonvisual modalities successively refine the visual representation rather than replacing it; and the final concatenation with $H,W$3 preserves original visual context (Zhou et al., 6 Jul 2025).

The training objective in ViTaL-Net is classification only. The multimodal dataset is written as

$H,W$4

where $H,W$5 is the $H,W$6-th visual slice for patient $H,W$7, $H,W$8 is the tabular data, $H,W$9 is the linguistic data, and $(F'_V, F_T, F_L) \in \mathbb{R}^{N \times C}.$0 is the class label (Zhou et al., 6 Jul 2025). The empirical risk minimization objective is described generally as minimizing average loss over the dataset, with standard cross-entropy used for classification: $(F'_V, F_T, F_L) \in \mathbb{R}^{N \times C}.$1 or equivalently

$(F'_V, F_T, F_L) \in \mathbb{R}^{N \times C}.$2

The paper does not introduce any THOAM-specific auxiliary alignment loss, contrastive loss, consistency loss, reconstruction loss, or multimodal fusion regularizer (Zhou et al., 6 Jul 2025).

6. Empirical behavior, evidence, and limitations

The clearest empirical evidence for THOAM concerns fusion quality. On ViTaL, the all-modality comparison between simple concatenation and attention fusion reports:

Fusion setting	ACC	AUC
Concatenation	75.88%	0.93
Attention Fusion / THOAM	85.59%	0.95

The paper explicitly attributes the gain to better handling of large representation differences among modalities and to reducing information loss during fusion (Zhou et al., 6 Jul 2025).

Against external fusion baselines, THOAM is also reported as the best-performing method in the main comparison table:

Method	ACC
CBAM	76.04%
ITCM	73.71%
TFA-LT	76.72%
THOAM	83.08%

The manuscript contains a discrepancy between this $(F'_V, F_T, F_L) \in \mathbb{R}^{N \times C}.$3 ACC figure and the $(F'_V, F_T, F_L) \in \mathbb{R}^{N \times C}.$4 ACC figure reported in the all-modality ablation table, and it does not explain the difference (Zhou et al., 6 Jul 2025). The safest conclusion is limited: in all reported comparisons, THOAM is the strongest fusion method, but the exact evaluation correspondence between tables is not clarified (Zhou et al., 6 Jul 2025).

The modality ablation study supports the claim of complementarity:

Modality configuration	ACC	AUC
Visual only	74.54%	0.87
Tabular only	59.46%	0.84
Linguistic only	69.01%	0.85
Visual + Tabular	78.39%	—
Visual + Linguistic	78.89%	—
Tabular + Linguistic	80.40%	—
Visual + Tabular + Linguistic	85.59%	0.95

These results indicate that tabular and linguistic inputs contribute meaningfully despite weaker unimodal performance, because they supply information not visible in the image alone (Zhou et al., 6 Jul 2025). The paper explicitly gives high-grade serous carcinoma as an example: it is difficult to separate at the image level, but abdominal pain, bloating, and tumor marker patterns in the tabular modality provide useful extra evidence, while the text report reinforces lesion location, shape, and descriptive context (Zhou et al., 6 Jul 2025).

The qualitative analyses are consistent with this interpretation. The ViTaL paper states that the confusion matrix for ViTaL-Net has the strongest main diagonal, ROC curves are closer to 1, and t-SNE clusters are more separated (Zhou et al., 6 Jul 2025). It also reports that incorporating tabular and textual modalities helps the visual branch generate heatmaps more focused on tumor regions (Zhou et al., 6 Jul 2025). This suggests that THOAM does not merely improve late classification but may also improve the discriminativeness of visually grounded representations.

The limitations are equally clear. Some pathological categories have few samples, especially high-grade serous carcinoma, which may cause overfitting (Zhou et al., 6 Jul 2025). The “Other” category was excluded from the main task because its heterogeneity caused a sharp performance drop:

• including “Other”: $(F'_V, F_T, F_L) \in \mathbb{R}^{N \times C}.$5 ACC, $(F'_V, F_T, F_L) \in \mathbb{R}^{N \times C}.$6 AUC
• excluding “Other”: $(F'_V, F_T, F_L) \in \mathbb{R}^{N \times C}.$7 ACC, $(F'_V, F_T, F_L) \in \mathbb{R}^{N \times C}.$8 AUC (Zhou et al., 6 Jul 2025)

The paper does not discuss missing-modality handling, and nothing in THOAM is described as supporting absent modalities; all experiments assume complete image-tabular-text triplets (Zhou et al., 6 Jul 2025). This suggests that the mechanism is designed for complete multimodal settings rather than for robust partial-observation inference.

7. Relation to adjacent attention paradigms

THOAM shares terminology with several attention traditions but is not reducible to any of them. Its “triplet” aspect does not denote three-way tensorized attention over entities in the sense used in heterogeneous graph triplet modeling. In HeTriNet, for example, triplet attention is explicitly defined over a central node and a neighboring pair through a triadic score

$(F'_V, F_T, F_L) \in \mathbb{R}^{N \times C}.$9

which models the importance of a pair for an entity (Tanvir et al., 2023). THOAM does not implement such a triplet scoring function. Instead, it realizes triplet interaction by chaining two pairwise cross-attention stages across three modalities (Zhou et al., 6 Jul 2025).

Its “hierarchical” aspect is likewise narrower than in mathematically explicit hierarchy-aware attention frameworks. Hierarchical Self-Attention introduces a rooted-tree representation of nested signals, a recursive interaction energy, a block-constrained attention matrix, and an exact dynamic-programming algorithm, but it explicitly does not provide an explicit triplet interaction tensor or learned geometric offsets (Amizadeh et al., 18 Sep 2025). THOAM shares only the high-level intuition that hierarchy can be expressed through staged aggregation, not the formal machinery of recursive hierarchical normalization (Zhou et al., 6 Jul 2025, Amizadeh et al., 18 Sep 2025).

The closest precedent for the term “triplet attention” in computer vision is the convolutional Triplet Attention module, which uses a three-branch structure to capture cross-dimension interaction among $F_1$0, $F_1$1, and $F_1$2 by tensor permutation, Z-pooling, shallow convolutional gating, and averaging (Misra et al., 2020). That module is neither hierarchical nor offset-based in the strict sense (Misra et al., 2020). THOAM differs fundamentally in operating across heterogeneous modalities rather than across tensor axes within one visual feature map (Zhou et al., 6 Jul 2025).

The principal misconception to avoid is therefore terminological inflation. THOAM is not a formally specified offset-attention framework, not a generalized tree-structured hierarchical attention law, and not a triadic attention kernel in the graph-attention sense. The technically grounded description supported by the ViTaL manuscript is more precise: THOAM is a staged three-modality cross-attention fusion mechanism, centered on visual features, followed by concatenation with the original visual representation before linear classification (Zhou et al., 6 Jul 2025).

This characterization also explains its practical significance. In ovarian tumor recognition, where some categories are difficult to distinguish visually and useful diagnostic evidence is distributed across morphology, symptoms, biomarkers, and textual descriptions, rigid concatenation can obscure cross-modal relevance (Zhou et al., 6 Jul 2025). THOAM addresses that problem by reweighting tabular and textual information relative to a vision-anchored query sequence, thereby improving six-class ovarian tumor recognition on the ViTaL benchmark (Zhou et al., 6 Jul 2025).