Dual Bank Sinkhorn Normalization in Retrieval

• Dual Bank Sinkhorn Normalization (DBSN) is a probability-balancing framework that extends Sinkhorn Normalization to mitigate hubness using dual query and target banks.
• It refines target-side compensation by narrowing the distributional gap and enforcing a doubly stochastic coupling that balances both query and target marginals.
• Empirical results show DBSN improves retrieval accuracy and reduces skewness in high-dimensional embedding spaces across various cross-modal tasks.

Dual Bank Sinkhorn Normalization (DBSN) is a probability-balancing framework for hubness reduction in cross-modal retrieval. It was introduced in "Hubness Reduction with Dual Bank Sinkhorn Normalization for Cross-Modal Retrieval" (Pan et al., 4 Aug 2025) as an extension of Sinkhorn Normalization (SN) to the query-agnostic setting in which the true test query set is unavailable at inference time. The method addresses the claim that balancing target probabilities alone is inadequate for robust hubness mitigation, and instead estimates hubness through a dual-bank construction containing both a query bank and a corresponding target bank. In the paper’s formulation, DBSN refines target-side compensation by narrowing the distributional gap between the proxy bank and the retrieval targets, while preserving the entropy-regularized optimal-transport view that underlies SN (Pan et al., 4 Aug 2025).

1. Hubness in cross-modal retrieval

In high-dimensional embedding spaces, hubness refers to a small subset of target items appearing as nearest neighbors of disproportionately many queries, while other targets are rarely retrieved (Pan et al., 4 Aug 2025). In cross-modal retrieval, this phenomenon degrades the precision of similarity measurements even when the underlying embedding model measures pairwise similarity accurately.

The paper quantifies hubness through the $k$-occurrence distribution over targets. Let $S \in \mathbb{R}^{m \times n}$ be the similarity matrix between $m$ queries $Q=\{q_i\}$ and $n$ targets $T=\{t_j\}$. For each target $t_j$, the $k$-occurrence is defined as

$KOcc_k(j)=\#\{i: t_j \in kNN(q_i)\}.$

A common hubness measure is the skewness of the $KOcc_k$ distribution:

$S \in \mathbb{R}^{m \times n}$0

where $S \in \mathbb{R}^{m \times n}$1 and $S \in \mathbb{R}^{m \times n}$2 are the mean and standard deviation of $S \in \mathbb{R}^{m \times n}$3. High skewness indicates severe hubness (Pan et al., 4 Aug 2025).

This formalization is central because DBSN is not defined as a generic post-processing heuristic. It is introduced as a mechanism for modifying retrieval probabilities so that the target distribution is less dominated by hubs, and, in the SN extension, so that both query and target marginals are balanced (Pan et al., 4 Aug 2025).

2. From Inverted Softmax to joint probability balancing

A major antecedent in the paper is Inverted Softmax (IS). For similarities $S \in \mathbb{R}^{m \times n}$4 and temperature $S \in \mathbb{R}^{m \times n}$5, IS defines

$S \in \mathbb{R}^{m \times n}$6

The paper interprets IS as an entropy-regularized one-sided normalization that balances the target marginal:

$S \in \mathbb{R}^{m \times n}$7

with $S \in \mathbb{R}^{m \times n}$8 and

$S \in \mathbb{R}^{m \times n}$9

Under this view, IS injects a target-specific compensation

$m$0

which flattens the retrieval probability across targets (Pan et al., 4 Aug 2025).

The paper’s central limitation claim is that IS only enforces uniformity on the target side, leaving query marginals unconstrained. This can bias hubness estimation when query distributions vary. The stated goal is therefore to achieve a doubly stochastic coupling $m$1 whose rows and columns sum to uniform marginals, so that both query- and target-side probabilities are balanced and asymmetric neighbor effects are reduced on both dimensions (Pan et al., 4 Aug 2025).

This argument is also the paper’s main corrective to a common misconception in retrieval post-processing: that target-side calibration alone is sufficient. The method explicitly contends that balancing target probabilities alone is inadequate (Pan et al., 4 Aug 2025).

3. Sinkhorn Normalization as the immediate precursor

Sinkhorn Normalization is the paper’s doubly stochastic alternative to IS. It begins from the cost matrix $m$2, or equivalently from the Gibbs kernel $m$3, and solves the entropy-regularized optimal-transport problem

$m$4

where

$m$5

with uniform marginals $m$6 and $m$7 (Pan et al., 4 Aug 2025).

Introducing dual vectors $m$8 and $m$9, the solution has the form

$Q=\{q_i\}$0

The Sinkhorn updates are

$Q=\{q_i\}$1

$Q=\{q_i\}$2

According to the paper, SN yields the unique doubly stochastic projection of $Q=\{q_i\}$3, enforcing uniform marginals on both modalities and thereby minimizing asymmetric neighbor bias (Pan et al., 4 Aug 2025).

In the retrieval setting, SN is therefore a balancing operator over the similarity-induced Gibbs kernel rather than merely a reweighting rule for nearest-neighbor scores. Its significance in the paper is theoretical as much as empirical: it provides a principled transition from one-sided target normalization to a doubly constrained coupling.

4. Dual-bank construction and the DBSN mechanism

The motivation for DBSN arises in the query-agnostic setting. At inference, the true test query set $Q=\{q_i\}$4 is unavailable, so a proxy query bank $Q=\{q_i\}$5—for example, training or validation queries—is used instead. The paper states that if $Q=\{q_i\}$6 in distribution, estimated hubness from SN on $Q=\{q_i\}$7 is biased, which degrades performance (Pan et al., 4 Aug 2025).

DBSN introduces an auxiliary target bank to mitigate this gap. The construction is:

• a query bank $Q=\{q_i\}$8, typically all training or validation queries;
• a target bank $Q=\{q_i\}$9, consisting of auxiliary targets such as a training gallery;
• an extended target set $n$0 with similarity blocks

$n$1

The method computes a coupling

$n$2

by solving

$n$3

with

$n$4

From the final column-scaling vectors $n$5, DBSN extracts hubness scalars

$n$6

and adjusts the original similarities by

$n$7

The paper’s theoretical justification is Proposition 2, described as a triangle inequality for EMD. It states that extending $n$8 to $n$9 reduces divergence,

$T=\{t_j\}$0

which yields more accurate hubness estimates (Pan et al., 4 Aug 2025).

The core distinction between SN and DBSN is therefore not the use of Sinkhorn scaling itself, but the bank design used to estimate the target-side compensation term when the true query distribution is unknown. DBSN is the paper’s answer to the claim that a query bank alone produces suboptimal results because of a significant distributional gap between the query bank and targets (Pan et al., 4 Aug 2025).

5. Iterative algorithm and implementation details

The paper presents an iterative Sinkhorn algorithm specialized to the dual-bank setting. It initializes

$T=\{t_j\}$1

and then, for $T=\{t_j\}$2, updates

$T=\{t_j\}$3

$T=\{t_j\}$4

The final coupling is

$T=\{t_j\}$5

and the hubness estimate is

$T=\{t_j\}$6

Final retrieval ranks are obtained by sorting each row of the adjusted $T=\{t_j\}$7 in descending order (Pan et al., 4 Aug 2025).

The reported hyperparameters are specific. SN uses $T=\{t_j\}$8, while IS uses $T=\{t_j\}$9. The number of iterations is typically 10–20 for convergence. Bank sizes $t_j$0 and $t_j$1 are set equal to training set sizes, such as approximately $t_j$2 for Flickr30k, in order to cover the target distribution. For image–text, video–text, and audio–text embeddings, the paper reports CLIP ViT-B/32 as the backbone; the audio results additionally include CLAP zero-shot evaluation in the reported benchmarks (Pan et al., 4 Aug 2025).

These details place DBSN as a post-hoc normalization layer over pretrained embedding spaces rather than a replacement for the embedding backbone itself. The method modifies score calibration and retrieval ordering after similarity computation.

6. Empirical performance, scope, and relation to matrix scaling

The evaluation spans image–text retrieval on Flickr30k and MS-COCO; video–text retrieval on MSR-VTT, Didemo, ActivityNet, VATEX, and MSVD; and audio–text retrieval on AudioCaps and Clotho. The metrics are Recall@K for $t_j$3, together with Mean Rank and Median Rank (Pan et al., 4 Aug 2025).

The paper reports that SN improves over baseline and IS by an average of $t_j$4 Recall@1 across tasks. Representative results include Flickr30k in CLIP zero-shot, where Recall@1 increases from $t_j$5 to $t_j$6, and MSR-VTT, where it increases from $t_j$7 to $t_j$8. In audio–text retrieval, AudioCaps improves from $t_j$9 to $k$0 with SN, and Clotho from $k$1 to $k$2. In the query-agnostic setting, DBSN is reported to close the gap to oracle SN; on zero-shot CLIP for Flickr30k with a training bank, the result moves from SN Recall@1 $k$3 to DBSN $k$4, compared with oracle $k$5 (Pan et al., 4 Aug 2025).

The paper also reports consistent skewness reduction, with Flickr30k skewness decreasing from $k$6 to $k$7 with SN, and states that this reduction correlates with accuracy (Pan et al., 4 Aug 2025). Within the paper’s own framework, this supports the argument that hubness mitigation is not merely a geometric regularization objective but one that is directly connected to retrieval effectiveness.

A broader theoretical connection appears in "From Local Updates to Global Balance: A Framework for Distributed Matrix Scaling" (Aletti et al., 3 Jun 2025). That paper studies matrix scaling processes derived from the classical Sinkhorn algorithm and describes local row and column normalization updates that converge to a doubly stochastic matrix under arbitrary update order, provided each row and column index is selected infinitely often. In that exposition, dual banks correspond to row-bank and column-bank diagonal factors, and the current scaled matrix is written as $k$8 (Aletti et al., 3 Jun 2025). This is not the same retrieval formulation as DBSN in cross-modal retrieval, but it situates the dual-bank language within a larger Sinkhorn and matrix-scaling tradition. A plausible implication is that DBSN’s retrieval-specific use of dual banks inherits conceptual support from the broader theory of doubly stochastic scaling, even though its concrete construction with query and target banks is specific to cross-modal hubness reduction (Aletti et al., 3 Jun 2025).

Across these formulations, the stable point of comparison is the doubly stochastic objective: in the retrieval paper, it is a coupling over similarities; in the distributed scaling paper, it is a scaled matrix over local updates. DBSN’s distinctive contribution is to use that balancing principle to estimate hubness under query uncertainty by augmenting the target side with an auxiliary bank (Pan et al., 4 Aug 2025).