Hystar: Hypernetwork-driven Style-adaptive Retrieval via Dynamic SVD Modulation

Abstract: Query-based image retrieval (QBIR) requires retrieving relevant images given diverse and often stylistically heterogeneous queries, such as sketches, artworks, or low-resolution previews. While large-scale vision--language representation models (VLRMs) like CLIP offer strong zero-shot retrieval performance, they struggle with distribution shifts caused by unseen query styles. In this paper, we propose the Hypernetwork-driven Style-adaptive Retrieval (Hystar), a lightweight framework that dynamically adapts model weights to each query's style. Hystar employs a hypernetwork to generate singular-value perturbations ($ΔS$) for attention layers, enabling flexible per-input adaptation, while static singular-value offsets on MLP layers ensure cross-style stability. To better handle semantic confusions across styles, we design StyleNCE as part of Hystar, an optimal-transport-weighted contrastive loss that emphasizes hard cross-style negatives. Extensive experiments on multi-style retrieval and cross-style classification benchmarks demonstrate that Hystar consistently outperforms strong baselines, achieving state-of-the-art performance while being parameter-efficient and stable across styles.

• The paper introduces a novel hypernetwork-driven dynamic SVD modulation that adapts attention layers for per-query style variation.
• It combines dynamic spectral scaling in attention layers with static singular value biases in MLPs to achieve robust performance across diverse and out-of-distribution styles.
• Using the StyleNCE contrastive loss with optimal transport weighting, Hystar demonstrates state-of-the-art results on benchmarks like DSR and DomainNet.

Hypernetwork-driven Style-adaptive Retrieval via Dynamic SVD Modulation: An Analysis

Introduction and Motivation

"Hystar: Hypernetwork-driven Style-adaptive Retrieval via Dynamic SVD Modulation" (2605.10009) proposes a novel framework for query-based image retrieval (QBIR) under heterogeneous and often out-of-distribution visual styles. Traditional QBIR pipelines and large-scale VLRMs such as CLIP deliver strong zero-shot retrieval but degrade under distributional shifts caused by novel query styles—sketches, paintings, low-res images, or stylized representations—because static parameter-efficient fine-tuning (PEFT) and cluster-based prompt learning approaches lack the flexibility for per-query adaptation.

Hystar addresses these limitations with a dynamic-static hybrid modulation framework. The core advancement is a hypernetwork that generates query-specific singular value perturbations on attention layers via SVD modulation, enabling per-query adaptability. Simultaneously, global static singular value biases are learned for MLP layers to maintain cross-style robustness. The authors further introduce StyleNCE, an optimal-transport-weighted contrastive objective that prioritizes hard negatives, enhancing alignment of style-diverse query-image pairs.

Technical Framework

Dynamic-static Hybrid SVD Modulation

Hystar's approach relies on the spectral analysis of weight matrices within the vision backbone (e.g., CLIP, BLIP). Instead of predicting dense low-rank updates (as in LoRA), SVD decomposition is applied to pretrained weights $W_0 = U \Sigma V^T$. Updates are then applied only to the diagonal singular value components:

$W = U (\Sigma + \Delta \Sigma) V^T$

where $$\Delta \Sigma = \Delta \Sigma_{\text{dyn}} + \Delta \Sigma_{\text{stat}}$$ is decomposed into:

• Dynamic $\Delta \Sigma_{\text{dyn}}$: Generated by a hypernetwork conditioned on a style embedding $z$ (extracted via DINOv2 from query input), this vector applies per-query, style-specific spectral scaling, implemented only in attention layers for high expressivity.
• Static $\Delta \Sigma_{\text{stat}}$: A global learnable bias, applied throughout training, but shared across all inputs—specifically acting on MLP layers to provide consistent, stable adaptation.

This design maintains the original semantic subspace (generated by $U$ and $V$) and only rescales directions therein. Notably, hypernetwork output dimensionality scales with $\min(d_1, d_2)$ per attention layer, ensuring efficiency relative to dense parameter updates.

StyleNCE Contrastive Loss

Standard contrastive objectives (e.g., InfoNCE) are agnostic to the semantic "difficulty" of negatives, which is suboptimal when easy negatives overwhelm the loss and do not reflect hard cross-style ambiguities. StyleNCE incorporates optimal transport (OT) to assign higher importance to hard negatives within each batch:

$$\mathcal{L}_{\text{StyleNCE}} = -\frac{1}{N} \sum_{i=1}^N \log \frac{\exp(\text{sim}(q_i, p_i)/T)}{\exp(\text{sim}(q_i,p_i)/T) + \gamma \sum_{j\neq i} w_{ij} \exp(\text{sim}(q_i, p_j)/T)}$$

where $W = U (\Sigma + \Delta \Sigma) V^T$0 are obtained by solving an OT problem (using the Sinkhorn algorithm), based on batch-wise similarity cost matrices.

OT-based weighting ensures the loss surface prioritizes hard negative separation, leading to more robust style-independent feature learning.

Training and Inference

Hystar is integrated with pretrained backbones (e.g., CLIP, BLIP, ALBEF), using DINOv2 to extract per-query style embeddings. During both training and inference, the query's representation is adaptively encoded via dynamic modulation. This design ensures consistency and eliminates mode collapse to seen styles.

Empirical Results

Hystar's claims are anchored in strong empirical performance on multi-style QBIR and cross-domain classification tasks. Notable findings include:

• DSR Benchmark (Fine-grained Multi-style Retrieval):
  • Hystar-BLIP achieves Top-1 retrieval of 75.6% (Art), 91.0% (Sketch), and 98.8% (Low-Res), improving absolute accuracy by 21–25% over BLIP* for challenging style shifts.
  • Consistently outperforms CLIP/BLIP, LoRA, VPT, (IA)3, AdaptFormer, SSF, FreestyleRet, and recent multimodal baselines.
• DomainNet (Zero-shot Category Retrieval):
  • Demonstrates SOTA Top-1 accuracy across the most abstract styles (e.g., +3.5% vs. FreestyleRet on Surrealist Abstract Art).
  • Maintains robustness under severe domain shifts, where static and style-clustered baselines fail.
• Zero-shot Classification:
  • Superior generalization for both retrieval and classification, e.g., 71.2% Top-1 on Sketch (DomainNet), outperforming CLIP and other PEFT methods.
• Qualitative Feature Analysis:
  • t-SNE projections indicate Hystar achieves more compact, well-separated cross-style clusters, reflecting improved semantic structure and reduced style entanglement.

Ablation and Analysis

Detailed ablation studies validate the necessity of each component:

• Combining static and dynamic SVD modulation yields complementary gains: the static component stabilizes cross-style performance, while dynamic adaptation ensures flexibility for unseen and fine-grained styles.
• StyleNCE outperforms InfoNCE, triplet loss, and their hard-negative variants, confirming the efficacy of OT-based reweighting.
• Middle-layer dynamic injection provides the best balance in parameter usage and generalization (early/late layer injection is less effective).
• Hystar’s gain is not due to the choice of style extractor: DINOv2, VGG, and even CLIP-based features all improve over static baselines, but Hystar's main benefit is architectural (adaptive modulation).

In terms of computational cost, Hystar achieves favorable trade-offs, introducing modest parameter and latency overhead compared to static PEFT schemes while remaining far more efficient than full fine-tuning approaches.

Implications and Future Directions

Hystar demonstrates that dynamic, hypernetwork-driven modulation—especially when expressed in the spectral domain—enables robust query-based retrieval under open-ended style shifts, without reliance on exhaustive style annotations or per-style modules. The SVD-centric approach constrains adaptation within pretrained geometry, avoiding forgetting and catastrophic overfitting to transient style attributes.

The optimal-transport-based negative mining of StyleNCE provides a principled path for extending contrastive learning under severe modality or style shifts, with potential applicability beyond QBIR to general multimodal representation learning and open-world recognition.

Future work may explore:

• Extension to complex cross-modal retrieval (video, audio-visual, or text-art hybrid modalities).
• Active style-disentanglement via continuous modulation rather than discrete style clustering, supporting highly granular or mixed-style queries.
• Dynamic adaptation in the textual encoder or joint query/image pipeline for bidirectional retrieval.

Conclusion

Hystar represents a technically rigorous and empirically validated advance in parameter-efficient adaptation of large vision-LLMs for style-diverse retrieval contexts. Through dynamic SVD modulation, hypernetwork conditioning, and an OT-weighted contrastive objective, it achieves robust, state-of-the-art performance under heterogeneous and out-of-distribution style scenarios—outperforming prior static, cluster-based, and dense parameterization methods, while remaining computationally efficient and theoretically well-motivated. These contributions further establish the utility of dynamic, lightweight adaptation for scalable deployment of VLRMs and related architectures in real-world, style-rich environments.