Modality-Aware Similarity Thresholds

• Modality-aware similarity thresholds are adaptive decision mechanisms that adjust similarity evaluation based on sensor modality, representation, and retrieval context.
• They leverage techniques such as rank-based neighbor selection, modality-specific score standardization, and soft gating to overcome modality gaps in multimodal tasks.
• These mechanisms improve system performance by calibrating similarity scores without relying on rigid global cutoffs, enabling context-sensitive decision boundaries.

Searching arXiv for the papers on arXiv to ground citations. Search "VI-MMRec Similarity-Aware Training Cost-free Virtual User-Item Interactions for Multimodal Recommendation" Modality-aware similarity thresholds are criteria that determine when similarity evidence should be accepted, attenuated, standardized, or routed differently as a function of modality, modality pair, or retrieval context. In the supplied literature, this topic does not reduce to a single hard cutoff of the form $\mathcal{S}>\tau$. Instead, modality awareness appears through several recurrent mechanisms: rank-based neighbor selection, modality-specific score standardization, pre-softmax scaling, soft interpolation masks, pairwise correlation coefficients, weak-combination mining, and confidence-triggered routing. A broad survey of modality-aware feature matching supports the general conclusion that similarity thresholds are not globally transferable across modalities because score meaning depends on sensing domain, representation, and post-verification stage (Liu et al., 30 Jul 2025).

1. Problem settings that make modality-aware thresholds necessary

Several supplied studies motivate modality-aware thresholding by showing that similarity scores are not intrinsically comparable across modalities. In mixed text-image retrieval, the central failure mode is the modality gap: even when a vision-LLM embeds text and images into a shared space, a text query often gives higher cosine scores to text candidates than to image candidates regardless of true relevance, so a single global ranking rule or raw cosine threshold becomes unreliable (Yamashita et al., 27 Nov 2025). In multimodal sentiment analysis, weak modality correlations arise from voice discrepancy, content discrepancy, alignment discrepancy, and clarity discrepancy, so uniform treatment of cross-modal interactions can be suboptimal (Li et al., 2024). In semantic location prediction from social-media text-image pairs, the stated difficulties are noise and modality heterogeneity, which make fixed cross-modal interaction strength undesirable (Zhang et al., 2024).

These settings differ, but they share a structural property: the same numerical similarity value can have different operational meaning depending on whether it is measured within a modality, across modalities, or after fusion. This suggests that “threshold” should be interpreted broadly. In some systems it means a literal cutoff; in others it means a modality-conditioned ranking rule, a calibration transform, or a soft gate whose effect is equivalent to thresholding without binary rejection.

2. Rank-based selection and score calibration

One common replacement for explicit score thresholds is rank-based selection. In VI-MMRec, virtual user-item interactions are generated from modality-specific cosine similarities computed on frozen pretrained features, but the paper does not use a rule of the form $\mathcal{S}_{i,j}>\tau_m$. Instead, for each observed interaction and each modality $m$, it keeps the top-$k$ most similar items,

$$j \in \mathcal{I}_{u,i}^m \quad \text{s.t.} \quad \mathcal{S}_{i,j}^m \in \text{top-}k(\mathcal{S}_{i,p}^m \mid p\in\mathcal{I},\, p\neq i),$$

with $k \in \{5,10,20\}$. The paper states that the method is modality-aware in similarity computation but not modality-aware in thresholding in the strict sense, because filtering is controlled by one shared top-$k$ hyperparameter. It further reports recommended $k=10$ for most datasets and recommended $k=20$ for TikTok, interpreting the latter as a consequence of having more modalities (Xu et al., 9 Dec 2025).

A different solution is modality-specific score standardization. In mixed text-image retrieval, similarity is calibrated by estimating per-modality positive-pair statistics,

$$\mu_m = \frac{1}{|\mathcal{P}_m|}\sum_{(q,d_m^+) \in \mathcal{P}_m}\cos(f(q), f(d_m^+)),$$

$\mathcal{S}_{i,j}>\tau_m$0

and then transforming raw cosine scores by

$\mathcal{S}_{i,j}>\tau_m$1

Here the “threshold-like” role is moved from raw cosine space into standardized-score space. The supplied analysis explicitly notes that the paper frames this primarily as ranking normalization, but that a shared global threshold becomes more plausible only after modality-specific standardization (Yamashita et al., 27 Nov 2025).

Taken together, these two designs show a basic distinction. Rank-based selection avoids assuming comparable score scales, whereas standardization attempts to construct such a common scale explicitly. The former is local and ordinal; the latter is global and distributional.

3. Soft thresholds: scaling, gating, and correlation weighting

A second recurring pattern is the use of soft mechanisms that behave like thresholds without binary selection. In TCL-MAP, the Modality-Aware Prompting module aligns prompt tokens with video and audio through similarity matrices

$\mathcal{S}_{i,j}>\tau_m$2

The supplied description states that $\mathcal{S}_{i,j}>\tau_m$3 and $\mathcal{S}_{i,j}>\tau_m$4 are called “threshold hyper-parameters,” but they do not implement binary thresholding, top-$\mathcal{S}_{i,j}>\tau_m$5 selection, or masking. Their practical role is pre-softmax scaling, followed by

$\mathcal{S}_{i,j}>\tau_m$6

Accordingly, the threshold analogue is continuous concentration control rather than hard acceptance or rejection (Zhou et al., 2023).

SG-MFT uses a different soft mechanism. It computes coarse-grained modality-wise similarity $\mathcal{S}_{i,j}>\tau_m$7, normalizes it into $\mathcal{S}_{i,j}>\tau_m$8, and then interpolates between cross-attention and self-attention:

$\mathcal{S}_{i,j}>\tau_m$9

The supplied analysis explicitly states that no explicit similarity threshold is defined; instead, $m$0 behaves as a soft gate. High similarity leans toward cross-attention, low similarity toward self-attention. Fine-grained interaction is handled analogously through the similarity-aware matrix $m$1 in the feed-forward block (Zhang et al., 2024).

CorMulT introduces yet another soft-threshold substitute through pairwise modality correlation coefficients,

$m$2

which are used to scale cross-modal transformer outputs, for example

$m$3

The supplied discussion is explicit that there is no hard threshold on correlation. Instead, weakly correlated modality pairs are attenuated and strongly correlated pairs are amplified. This is threshold-like only in the sense of continuous gating (Li et al., 2024).

These systems converge on a common design principle: rather than deciding whether similarity exceeds a universal cutoff, they shape downstream computation by varying how sharply, how strongly, or in which pathway similarity is allowed to act.

4. Dataset-level modality relevance and weak-combination selection

Modality-aware thresholding can also be expressed at the level of dataset statistics or modality-combination difficulty, not only per pair or per token. VI-MMRec supplements top-$m$4 retrieval with a statistically informed weight allocation mechanism based on overlap with real interactions. For modality $m$5,

$m$6

and the modality weight is

$m$7

The supplied description stresses that these weights do not replace top-$m$8 selection; they act as a soft confidence mechanism after candidate generation. This is modality-aware calibration in the form of post-selection weighting, not direct thresholding (Xu et al., 9 Dec 2025).

MMANet operates at a different level. In Margin-aware Distillation, it computes pairwise relation matrices with cosine similarity,

$m$9

constructs a relation-gap vector $k$0, measures teacher uncertainty by entropy,

$k$1

and weights the distillation loss by $k$2. The paper interprets higher classification uncertainty as greater proximity to decision boundaries. In Modality-aware Regularization, weak modality combinations are mined by comparing predicted class-count distributions to the complete-modality baseline with

$k$3

averaging over warm-up epochs, and selecting

$k$4

This is not a numeric threshold, but it is a hard modality-aware selection rule induced by ranking (Wei et al., 2023).

A plausible implication is that modality-aware thresholding often emerges when systems must decide not only whether two items are similar, but also which modality combinations are systematically weak, noisy, or unreliable. In that setting, ranking and adaptive masking can replace scalar cutoffs.

5. Explicit thresholds and decision boundaries

Among the supplied studies, the clearest direct evidence for modality-dependent decision thresholds comes from human preference annotation. In controlled text-versus-audio evaluation of identical semantic content, raters used larger average rating differences when committing to a winner in text than in audio: text $k$5, 95\% CI $k$6, $k$7; audio $k$8, 95\% CI $k$9, $$j \in \mathcal{I}_{u,i}^m \quad \text{s.t.} \quad \mathcal{S}_{i,j}^m \in \text{top-}k(\mathcal{S}_{i,p}^m \mid p\in\mathcal{I},\, p\neq i),$$0; Mann–Whitney $$j \in \mathcal{I}_{u,i}^m \quad \text{s.t.} \quad \mathcal{S}_{i,j}^m \in \text{top-}k(\mathcal{S}_{i,p}^m \mid p\in\mathcal{I},\, p\neq i),$$1, $$j \in \mathcal{I}_{u,i}^m \quad \text{s.t.} \quad \mathcal{S}_{i,j}^m \in \text{top-}k(\mathcal{S}_{i,p}^m \mid p\in\mathcal{I},\, p\neq i),$$2, $$j \in \mathcal{I}_{u,i}^m \quad \text{s.t.} \quad \mathcal{S}_{i,j}^m \in \text{top-}k(\mathcal{S}_{i,p}^m \mid p\in\mathcal{I},\, p\neq i),$$3. The paper states that audio raters exhibit narrower decision thresholds, and explicitly recommends calibrating thresholds per modality or retaining the continuous signal rather than using a fixed binarization threshold learned from text data (Broukhim et al., 26 Feb 2026).

This result is conceptually distinct from engineering thresholds in multimodal architectures. In MTNet for RGBT tracking, the explicit thresholds are not modality-similarity cutoffs but confidence thresholds in a template update strategy: confidence greater than $$j \in \mathcal{I}_{u,i}^m \quad \text{s.t.} \quad \mathcal{S}_{i,j}^m \in \text{top-}k(\mathcal{S}_{i,p}^m \mid p\in\mathcal{I},\, p\neq i),$$4 for $$j \in \mathcal{I}_{u,i}^m \quad \text{s.t.} \quad \mathcal{S}_{i,j}^m \in \text{top-}k(\mathcal{S}_{i,p}^m \mid p\in\mathcal{I},\, p\neq i),$$5 consecutive frames defines steady state; confidence between $$j \in \mathcal{I}_{u,i}^m \quad \text{s.t.} \quad \mathcal{S}_{i,j}^m \in \text{top-}k(\mathcal{S}_{i,p}^m \mid p\in\mathcal{I},\, p\neq i),$$6 and $$j \in \mathcal{I}_{u,i}^m \quad \text{s.t.} \quad \mathcal{S}_{i,j}^m \in \text{top-}k(\mathcal{S}_{i,p}^m \mid p\in\mathcal{I},\, p\neq i),$$7 defines transient steady state; confidence lower than $$j \in \mathcal{I}_{u,i}^m \quad \text{s.t.} \quad \mathcal{S}_{i,j}^m \in \text{top-}k(\mathcal{S}_{i,p}^m \mid p\in\mathcal{I},\, p\neq i),$$8 and accumulated up to $$j \in \mathcal{I}_{u,i}^m \quad \text{s.t.} \quad \mathcal{S}_{i,j}^m \in \text{top-}k(\mathcal{S}_{i,p}^m \mid p\in\mathcal{I},\, p\neq i),$$9 times defines unstable state. The paper’s main modality-aware similarity mechanism remains soft spatial gating through $k \in \{5,10,20\}$0 and $k \in \{5,10,20\}$1, not hard thresholding of RGB-versus-thermal similarity (Hou et al., 24 Aug 2025). HMAR shows the same distinction in medical retrieval: it introduces a routing gate

$k \in \{5,10,20\}$2

with $k \in \{5,10,20\}$3, together with top-$k \in \{5,10,20\}$4 global candidate selection and top-$k \in \{5,10,20\}$5 local refinement, but the supplied analysis is explicit that these are retrieval-mode controls rather than true modality-specific similarity thresholds (Yuan, 17 Mar 2026).

The literature therefore supports a sharp distinction between decision thresholds and similarity thresholds. Human studies can expose genuine modality-dependent decision boundaries. By contrast, many machine-learning systems use thresholds primarily to regulate confidence, routing, or update state, while keeping similarity handling itself soft or rank-based.

6. Misconceptions, limitations, and boundary conditions

A common misconception is that modality-aware similarity thresholds must be explicit cutoffs of the form $k \in \{5,10,20\}$6. The supplied papers repeatedly show otherwise. VI-MMRec uses top-$k \in \{5,10,20\}$7 retrieval and overlap-based weighting; TCL-MAP uses pre-softmax scaling; SG-MFT uses soft interpolation masks; CorMulT uses pairwise correlation coefficients; MMANet uses entropy weighting and argmax-based weak-combination mining. In all of these cases, thresholding is better understood as modality-aware control of similarity influence than as literal hard rejection.

Another misconception is that a shared embedding space automatically yields comparable scores. The modality-gap results in mixed text-image retrieval directly contradict that assumption: text and image candidates can occupy systematically different score ranges even under the same pretrained vision-LLM, and raw cosine thresholds can therefore be meaningless across modalities (Yamashita et al., 27 Nov 2025). The controlled preference study reaches a parallel conclusion on the human side: text and audio judgments can be similarly reliable in aggregate while still exhibiting near-chance cross-modality agreement at a zero threshold and substantially different decisive margins (Broukhim et al., 26 Feb 2026).

A further limitation is evidential rather than conceptual. Not every paper title supplied here yields usable technical content for this topic. For the item labeled “A Similarity Inference Metric for RGB-Infrared Cross-Modality Person Re-identification,” the supplied document is stated to be an IJCAI formatting template example containing no method, experiments, equations, datasets, or thresholds, so no claims about similarity inference or RGB–IR threshold design can be extracted from that source as given (Jia et al., 2020).

The supplied literature therefore supports a restrained encyclopedic conclusion. Modality-aware similarity thresholds are not a single algorithmic object. They are a family of modality-conditioned decision mechanisms that include top-$k \in \{5,10,20\}$8 selection, z-score standardization, pre-softmax scaling, soft gating, pairwise correlation weighting, weak-combination mining, and, in some settings, explicit human or system-level thresholds. The unifying principle is that similarity values acquire operational meaning only relative to modality, modality pair, and downstream verification context.