Resolve the reliability gap in few-shot hyperparameter search on Gemma-4

Establish a reliable hyperparameter-selection procedure for soft prompting on Gemma-4-12B-it that allows performance on an internal holdout drawn from the ten-shot support set to predict performance on unseen test images.

Background

On Qwen3-VL, the paper reports that a short holdout-based sweep can distinguish soft-prompt configurations in a manner consistent with test performance. On the architecturally unrelated Gemma-4-12B-it backbone, however, large holdout improvements sometimes reverse on the test split: configurations selected by the search lose to the untuned default on several datasets.

The unresolved issue is therefore not whether soft prompting can produce gains on Gemma-4, but how to select prompt position, length, initialization, learning rate, accumulation, and training budget reliably under the extremely limited ten-shot supervision. A solution would make the method more dependable across backbones and reduce the risk of selecting configurations that overfit the internal holdout.

References

We report this as an open reliability gap in the search procedure itself, not as evidence against the underlying method: the same $10$-shot budget that lets \S\ref{sec:ablations}'s Qwen3-VL search reliably rank configurations does not, on this second backbone, carry enough signal to always rank them correctly.

— Your Model Already Knows Don't Teach It, Learn to Ask It: Soft Prompting for Few-Shot Adaptation of Vision-Language Models  (2609.11310 - Gare et al., 10 Sep 2026) in Appendix, Section “Gemma-4 Generalisation: Search Methodology and Full Results,” final paragraph