Determine the causes of TopKSigLIP’s reduced linear-probe performance

Determine whether TopKSigLIP’s comparatively weaker image-encoder performance under linear probing is caused by its use of a lightweight ConvNeXt-Tiny vision encoder and a smaller training batch size than competing mammography vision–language models.

Background

TopKSigLIP performs strongly in zero-shot evaluation, but the performance gap relative to competing models narrows under linear probing because the baseline models benefit more from supervised fine-tuning. The paper notes that TopKSigLIP uses a 28.6-million-parameter ConvNeXt-Tiny encoder and batch size 32, whereas several competing systems use substantially larger vision backbones and batches exceeding 100.

The authors explicitly conjecture that these architectural and optimization differences explain the observed limitation. Establishing whether larger vision encoders and training batches improve the image representation would clarify whether the limitation is intrinsic to the TopK-Patch approach or primarily a consequence of the proof-of-concept configuration.

References

We conjecture this is due to two reasons: as a proof-of-concept, TopKSigLIP deliberately employs the lightweight ConvNeXt-Tiny encoder (28.6M parameters), which is significantly smaller than competing backbones such as DaViT (87.9M) and ViT-B (86M); and it is trained on a single GPU with batch size 32, versus batch sizes exceeding 100 in several baselines.

— Solving the Needle-in-a-Haystack Problem in Mammography Vision-Language Model with Differentiable Subset Sampling  (2609.03085 - Jeon et al., 2 Sep 2026) in Section 4.4, Results, subsection “Accuracy,” paragraph “Linear probing”