--- title: Dynamic Inverted Softmax (DIS) url: https://www.emergentmind.com/topics/dynamic-inverted-softmax-dis type: topic --- # Dynamic Inverted Softmax (DIS) Dynamic Inverted Softmax (DIS) encompasses a set of adaptive normalization techniques designed to improve both retrieval accuracy and computational efficiency in high-dimensional embedding spaces. DIS is prominent in two central application domains: Robust cross-modal retrieval with hubness mitigation as formalized in Querybank Normalisation (QB-Norm), and large-scale learning/inference acceleration via adaptive sampled softmax algorithms such as the MIDX-Sampler. Both methods employ dynamic, query-dependent mechanisms to suppress hubs selectively or to construct efficient proposal distributions for importance sampling, yielding robust performance under distributional shift and extreme data cardinality. ## 1. Mathematical Formulation and Derivation The DIS mechanism was introduced in two distinct but conceptually aligned modalities: embedding space normalisation for cross-modal retrieval [2112.12777] and adaptive sampled softmax for extreme classification and large-scale neural models [2501.08563]. ### DIS in Cross-Modal Retrieval Given encoders $f_q : m_q \to \mathbb{R}^C$ and $f_g : m_g \to \mathbb{R}^C$ mapping queries and gallery items into a joint $C$-dimensional space, the base similarity is \[ s_q(j) = \mathrm{sim}(f_q(q), f_g(g_j)) \] A querybank $B = \{b_i\}_{i=1}^N$ of $N$ probes is constructed. Each gallery item $g_j$ receives a probe vector $p_j(i) = \mathrm{sim}(f_q(b_i), f_g(g_j))$. The static Inverted Softmax (IS) normalises gallery similarities as \[ \eta_q^{\mathrm{IS}}(j) = \frac{\exp(\beta s_q(j))}{\sum_{i=1}^N \exp(\beta p_j(i))} \] DIS introduces a data-driven activation set $\mathcal{A}$ comprising gallery items receiving top-$k$ votes by the querybank probes. For a given query, DIS reweights only if the top raw-similarity hit $j^* = \arg\max_j s_q(j)$ is itself a hub ($j^* \in \mathcal{A}$): \[ \eta_q^{\mathrm{DIS}}(j) = \begin{cases} \frac{\exp(\beta s_q(j))}{\sum_{i=1}^N \exp(\beta p_j(i))} & j^* \in \mathcal{A} \\ s_q(j) & \text{otherwise} \end{cases} \] ### DIS in Adaptive Sampled Softmax (MIDX-Sampler) For large-class softmax, let queries be $z \in \mathbb{R}^D$, classes $q_i \in \mathbb{R}^D$, and scores $o_i = z^\top q_i$. The ideal softmax is \[ p(i|z) = \frac{\exp(o_i)}{\sum_{j=1}^N \exp(o_j)} \] To enable efficient sampling, DIS decomposes $p(i|z)$ via product or residual quantization. With $K$ codewords per block and $B$ blocks, each class is mapped to $(k_1, k_2)$, and $\Omega_{k_1,k_2}$ contains class indices sharing codeword assignments. The dynamic (DIS) proposal distribution replaces the residual softmax by a uniform: \[ Q_{midx}(i|z) = P^1_z(k_1(i)) \cdot P^2_z(k_2(i)|k_1(i)) \cdot \frac{1}{|\Omega_{k_1,k_2}|} \] Sampling and importance weighting are done sublinearly in $N$. ## 2. Comparison of DIS to Static Methods DIS techniques provide targeted hub penalisation and adaptivity not present in their static analogues: - **Static IS** divides all similarities by a hubness denominator for every gallery, regardless of actual hub status, often resulting in over-penalisation for poor or mismatched querybanks [2112.12777]. - **DIS (QB-Norm)** applies softmax only when a query's top result is a flaggable hub; otherwise, raw scoring is retained, increasing robustness to querybank/galleries with domain mismatch or low overlap. - **Static proposals in sampled softmax** (uniform, unigram) exhibit large KL and gradient biases, poor convergence, and slow optimisation [2501.08563]. DIS via MIDX-Sampler achieves codeword-structured adaptivity, reducing bias and preserving computational efficiency. ## 3. Algorithms and Implementation ### QB-Norm DIS Inference Pipeline Precompute probe similarities and the activation set $\mathcal{A}$. At inference: ```python # Pseudocode for DIS in QB-Norm [2112.12777] for query in queries: s = [sim(f_q(query), f_g(g_j)) for g_j in gallery] j_star = argmax_j s[j] if j_star in activation_set: eta = [exp(beta * s[j]) / denom[j] for j in gallery] else: eta = s ranking = argsort_descending(eta) ``` ### MIDX-Sampler DIS Workflow Construct product/residual quantization codebooks. For each query, sample negatives per codeword distribution and evaluate the dynamic proposal $Q_{midx}(i|z)$: ```python # Pseudocode sketch for MIDX-Sampler DIS [2501.08563] for t in range(M): k1 = sample(P1_z) k2 = sample(P2_z_given_k1) i = uniform(Ω_k1_k2) Q = P1_z[k1] * P2_z[k2|k1] / len(Ω_k1_k2) record(i, Q) # Use corrected logits and self-normalized importance sampling for training ``` ## 4. Hyperparameters and Selection Criteria Key DIS parameters determine both performance and generalisation properties: | Hyperparameter | Role in QB-Norm DIS[2112.12777] | Role in MIDX-Sampler DIS[2501.08563] | |-----------------------|---------------------------------------|--------------------------------------| | Inverse temperature ($\beta$) | Controls peakiness; optimal around 20 (or $1/1.99$ for CLIP models) | Implicit in codeword inner products capacity | | Querybank size ($N$) | $5000$–$10,000$ probes; saturates at $N \sim 10,000$ | Codebook size $K$ (balancing distortion and speed) | | Top-$k$ selection | $k=1$ gives maximal robustness | Not applicable; uniform sampling at codeword level | | Similarity measure | Cosine similarity in $\mathbb{R}^C$ | Inner product in $\mathbb{R}^D$ | Selection is typically done via held-out validation; increasing $N$ or $K$ reduces hubness and quantization distortion but incurs additional memory. ## 5. Integration in End-to-End Systems ### QB-Norm Pipeline - **Training:** Encoders $f_q$, $f_g$ learned via standard ranking/contrastive loss. - **Indexing:** Gallery embeddings and querybank fixed; probe similarities and activation set precomputed. - **Inference:** Queries scored and ranked via DIS; no retraining or parameter updates required. ### Sampled Softmax for Large-Scale Learning - **Index construction:** Product/residual quantization codebooks computed offline. - **Proposal decomposition:** DIS enables sublinear sampling and softmax approximation, reducing both bias and compute. - **Training/inference:** Gradient estimation and prediction performed on dynamically sampled negatives. ## 6. Empirical Performance and Robustness ### Retrieval tasks [2112.12777]: - **Video-text retrieval (MSR-VTT):** R@1 improves from $29.6 \rightarrow 33.3$; CLIP2Video improves R@1 $45.6 \rightarrow 47.2$. - **Recall increases:** Across six video-text datasets, all recall metrics rise by $1$–$5$ points. - **Image/text/audio tasks:** CLIP zero-shot on MSCOCO R@1 improves $37.8 \rightarrow 41.4$; text-audio retrieval rises $23.1 \rightarrow 23.9$. - **Hubness reduction:** Example: MSR-VTT $k$-occurrence skewness drops $0.94 \rightarrow 0.51$. ### Robustness to Distributional Shift: - DIS preserves baseline performance under far-domain querybanks, unlike static IS or CSLS which collapse [2112.12777]. - Ablations show smooth degradation away from $\beta \sim 20$, querybank size saturation, and maximum robustness at $k=1$. ### Large-Scale Learning (MIDX-Sampler) [2501.08563]: - **Sampling cost:** $O(KD + K^2 + M)$, sublinear in number of classes. - **Bias/convergence:** KL-divergence and gradient bias provably bounded; smaller residual distortion $\|\tilde o\|_\infty$ leads to superior generalisation and faster convergence. - **Empirical speedup:** MIDX-Sampler matches full softmax in perplexity, ranking, and recall with $5\times$–$10\times$ sampling and $10$–$100\times$ memory improvements over kernel methods. ## 7. Context, Significance, and Current Challenges Dynamic Inverted Softmax provides modular, data-driven normalisation strategies that bypass limitations of static global transformations. In retrieval, DIS robustly demotes hubs only where necessary, preserving performance under query/resource limitations and domain mismatch. In sampled softmax, DIS enables scalable, theoretically justified learning for extreme class cardinality. These mechanisms require no retraining or backpropagation through the normaliser, making them practical augmentations for existing neural models. A plausible implication is that future extensions may leverage learnable codeword assignment and additional adaptivity in sampling or hub detection to further reduce bias and accelerate convergence. Current limitations include memory footprint for large querybanks/codebooks and sensitivity to quantization or probe selection. Nevertheless, the empirical consistency and theoretical foundation make DIS central to both retrieval and extreme learning architectures [2112.12777, 2501.08563]. Source: https://www.emergentmind.com/topics/dynamic-inverted-softmax-dis