Set Pivot Learning for VFM Adaptation

• Set Pivot Learning (SPL) is a formulation of generalized segmentation that replaces traditional source-target domains with a Knowledge Set and Application Set for adaptive, dynamic prompt tuning.
• It employs a frozen Vision Foundation Model along with lightweight modules like the Dynamic Class-aware Prompter and Prompt-guided Feature Focuser to refine visual features for segmentation.
• Empirical results demonstrate that SPL significantly outperforms classical domain generalization methods in both synthetic-to-real and real-to-real segmentation scenarios.

Set Pivot Learning (SPL) is a formulation of generalized segmentation introduced in the era of Vision Foundation Models (VFMs) that replaces the classical source-domain/target-domain language of domain generalization with the paired notions of a Knowledge Set (KS) and an Application Set (AS), while treating a pretrained VFM as the pivot for adaptation (Li et al., 3 Aug 2025). In this formulation, the central problem is no longer rigid domain transfer under a strict target-invisibility assumption, but adaptive refinement of already broad visual representations for downstream segmentation scenarios. The formulation was introduced “for the first time” as a VFM-centric alternative to conventional domain generalization, with two stated attributes—dynamic adaptation and VFM-centric tuning—and was instantiated through a Dynamic Prompt Fine-Tuning pipeline for generalized semantic segmentation (Li et al., 3 Aug 2025).

1. Historical placement and motivation

Set Pivot Learning was proposed against the background of classical domain generalization (DG) for semantic segmentation, where one assumes a labeled source domain and a different target domain that is completely unavailable during training (Li et al., 3 Aug 2025). In that conventional setup, DG methods typically use standard backbones such as ResNet-50/101, VGG, or ShuffleNet pretrained on ImageNet, and seek domain-invariant features through feature alignment, normalization or whitening, style transfer, adversarial augmentation, meta-learning, contrastive learning, or feature disentanglement (Li et al., 3 Aug 2025).

The motivation for SPL is the claim that VFMs such as CLIP, DINOv2, SAM, EVA02, and MAE destabilize the classical DG assumption. Because such models are pretrained on broad web-scale data, the invisibility of the target domain during pretraining is “practically impossible to ensure,” and a frozen VFM backbone can already outperform many DG methods built on classical backbones in cross-domain segmentation (Li et al., 3 Aug 2025). The resulting argument is that the main research problem shifts from transferring features from a cleanly separated source to an unseen target toward refining and refocusing already strong generic representations for specific downstream conditions.

Within this framing, SPL is presented as a response to several limitations of conventional DG: rigid domain transfer, static alignment, poor adaptation to evolving real-world conditions, and a mismatch between assumptions tailored to ImageNet-pretrained backbones and the reality of VFM pretraining (Li et al., 3 Aug 2025). This suggests a broader conceptual move from domain-bound generalization to set-based adaptation around a pretrained representational anchor.

2. Formal definition and problem setting

The defining move in Set Pivot Learning is the replacement of “source domain” and “target domain” by two sets (Li et al., 3 Aug 2025):

• The Knowledge Set $\mathcal{S}^K$ is all labeled data available for training,

$\mathcal{S}^K = \{ (x_i, y_i) \}_{i=1}^{N},$

without the requirement that it come from a single domain or a set of well-separated domains.

• The Application Set $\mathcal{S}^A$ is the deployment or evaluation data,

$\mathcal{S}^A = \{ (x_j, y_j) \}_{j=1}^{M}.$

The stated point of this redefinition is that KS can be a union of heterogeneous datasets—synthetic, real, multiple countries, or multiple weather types—while AS need not be assumed invisible to the VFM during pretraining (Li et al., 3 Aug 2025). Training still uses only KS, but the backbone may already encode AS-like information from pretraining.

Given a pretrained VFM and a labeled KS, SPL aims to learn a segmentation model

$\varphi_\theta : x \mapsto \hat{y},$

where the trainable parameters $\theta$ are concentrated in lightweight adaptation modules on top of a frozen VFM (Li et al., 3 Aug 2025). The paper identifies these modules as the prompter, feature focuser, and task head.

The “pivot” in Set Pivot Learning is explicitly the VFM backbone itself. It is treated as a shared representational reference connecting the heterogeneous KS and the diverse AS, so SPL does not attempt direct KS–AS alignment in raw pixel space or conventional feature space. Instead, it relies on the VFM as a broad prior and adapts around it using prompts and feature focusing (Li et al., 3 Aug 2025).

A notable negative definition is equally important: the paper states that SPL is primarily a problem formulation and a design philosophy, not a new explicit loss. The segmentation objective is inherited from the downstream architecture, while SPL is realized through the architecture and training protocol built around the frozen VFM (Li et al., 3 Aug 2025).

3. Core attributes and conceptual commitments

The paper assigns two central attributes to Set Pivot Learning (Li et al., 3 Aug 2025).

The first is dynamic adaptation. Classical DG is described as producing a static domain-invariant feature space, whereas SPL uses mechanisms that vary with the input and downstream task. In the proposed instantiation, a Dynamic Class-aware Prompter produces image-specific class prompts and similarity weights, and a Prompt-guided Feature Focuser uses those prompts to reweight and refine VFM features layer by layer. The intended consequence is that different weather, lighting, or synthetic-versus-real conditions induce different prompt sets and therefore different feature emphases (Li et al., 3 Aug 2025).

The second is VFM-centric tuning. The VFM backbone is frozen throughout training, and only lightweight modules are trained: the Dynamic Class-aware Prompter, the Prompt-guided Feature Focuser, and the Mask2Former segmentation head (Li et al., 3 Aug 2025). The rationale given in the source is that this preserves cross-domain robustness, avoids overfitting to KS, and reduces the computational and memory costs associated with full VFM fine-tuning.

The paper further contrasts SPL with three neighboring paradigms. Relative to classical DG, SPL does not prioritize domain-invariant alignment. Relative to standard fine-tuning, it avoids updating a large fraction of VFM parameters and thereby avoids the risk of “destroying cross-domain robustness.” Relative to standard transfer learning, it abandons the “source domain $\rightarrow$ target domain” framing in favor of KS/AS and emphasizes refinement rather than transfer in the narrow sense (Li et al., 3 Aug 2025).

This suggests that SPL is less a replacement for segmentation losses than a reframing of what counts as the task in the presence of broad pretrained visual priors.

4. Architectural instantiation: Dynamic Prompt Fine-Tuning

The proposed implementation of Set Pivot Learning is Dynamic Prompt Fine-Tuning, a four-part pipeline consisting of a VFM backbone, a Dynamic Class-aware Prompter (DCP), a Prompt-guided Feature Focuser (PFF), and a Mask2Former segmentation head (Li et al., 3 Aug 2025).

VFM backbone

The paper experiments with MAE, CLIP, SAM, EVA02, and DINOv2, using mainly ViT-L backbones, except for SAM, which uses ViT-H (Li et al., 3 Aug 2025). If the VFM layers are denoted by $L_1,\ldots,L_N$, then the intermediate visual features are

$F_i = L_i(x), \quad i = 1,\ldots,N.$

These features are then modulated by PFF before being passed to the segmentation decoder (Li et al., 3 Aug 2025).

Dynamic Class-aware Prompter

DCP operates per input image. It begins from an initial category library containing the 1000 ImageNet classes, then enriches that library using GPT or BLIP-based VQA on sampled KS images, yielding a fixed final category library $\mathcal{C}$ used during training and inference (Li et al., 3 Aug 2025).

For an input image $\mathcal{S}^K = \{ (x_i, y_i) \}_{i=1}^{N},$0, DCP uses a CLIP model $\mathcal{S}^K = \{ (x_i, y_i) \}_{i=1}^{N},$1 to compute similarity scores $\mathcal{S}^K = \{ (x_i, y_i) \}_{i=1}^{N},$2 between the image and each class prompt in $\mathcal{S}^K = \{ (x_i, y_i) \}_{i=1}^{N},$3,

$\mathcal{S}^K = \{ (x_i, y_i) \}_{i=1}^{N},$4

After softmax normalization, a filter threshold $\mathcal{S}^K = \{ (x_i, y_i) \}_{i=1}^{N},$5 retains classes with $\mathcal{S}^K = \{ (x_i, y_i) \}_{i=1}^{N},$6, producing class prompts and associated similarity scores. Hierarchical agglomerative clustering with threshold $\mathcal{S}^K = \{ (x_i, y_i) \}_{i=1}^{N},$7 then removes redundancy among semantically close prompts, selecting the most central class in each cluster, and iteratively adjusting thresholds until the number of prompts is below a maximum $\mathcal{S}^K = \{ (x_i, y_i) \}_{i=1}^{N},$8, given as an example of 30 (Li et al., 3 Aug 2025).

The result is an image-specific, reduced prompt set together with prompt relevance scores. The paper characterizes this as a semantic mining and filtering mechanism.

Prompt-guided Feature Focuser

PFF is the module that injects prompt information into the frozen VFM. It has three parts: similarity-based prompt fusion, text-to-text self-attention, and image-to-text cross-attention (Li et al., 3 Aug 2025).

Selected class prompts are encoded by the CLIP text encoder into embeddings $\mathcal{S}^K = \{ (x_i, y_i) \}_{i=1}^{N},$9. Similarity scores are logarithmically normalized, expanded to the prompt embedding dimension, and fused by an MLP: $\mathcal{S}^A$0 For each VFM layer $\mathcal{S}^A$1, PFF introduces learnable tokens

$\mathcal{S}^A$2

concatenates them with the fused prompts,

$\mathcal{S}^A$3

and applies multi-head self-attention: $\mathcal{S}^A$4

The refined prompt tokens then interact with image features $\mathcal{S}^A$5 via cross-attention,

$\mathcal{S}^A$6

followed by an MLP and a residual connection: $\mathcal{S}^A$7 These $\mathcal{S}^A$8 features are then consumed by Mask2Former for semantic segmentation (Li et al., 3 Aug 2025).

The article’s central architectural claim is therefore precise: dynamic prompts determine what textual concepts are relevant in a scene, and PFF uses those concepts to modulate frozen VFM features without updating the backbone itself.

5. Task formulation, optimization, and empirical results

The testbed for Set Pivot Learning is generalized street scene semantic segmentation (Li et al., 3 Aug 2025). Training uses KS data such as GTA5, SYNTHIA, GTA5 + SYNTHIA, or Cityscapes, while evaluation uses AS datasets such as Cityscapes, BDD-100K, Mapillary, and ACDC. The evaluation metric is mIoU on validation splits.

The optimization protocol is deliberately light-touch with respect to the backbone. The VFM remains frozen, while the trainable components are the PFF modules, the Mask2Former decoder, and the MLP fusion component in DCP; the paper explicitly states AdamW with learning rate $\mathcal{S}^A$9, weight decay 0.05, 40K iterations, and input crops of size $\mathcal{S}^A = \{ (x_j, y_j) \}_{j=1}^{M}.$0 (Li et al., 3 Aug 2025). The segmentation objective is taken directly from Mask2Former rather than redefined within SPL.

On synthetic $\mathcal{S}^A = \{ (x_j, y_j) \}_{j=1}^{M}.$1 real generalized segmentation, the reported results show that frozen VFMs already substantially exceed ResNet-based DG baselines, and that Dynamic Prompt Fine-Tuning yields further gains (Li et al., 3 Aug 2025). With a frozen EVA02 backbone trained on GTA5 and evaluated on Cityscapes, BDD-100K, and Mapillary, the reported average mIoU values are 56.2 for the VFM backbone only, 60.8 for VPT, 63.6 for Rein, and 64.7 for the SPL-based method. With frozen DINOv2, the corresponding averages are 61.1 for the backbone only, 64.3 for Rein, and 65.8 for the SPL-based method (Li et al., 3 Aug 2025).

On real $\mathcal{S}^A = \{ (x_j, y_j) \}_{j=1}^{M}.$2 real generalization with Cityscapes as KS and ACDC, BDD-100K, and Mapillary as AS, the DINOv2-based SPL method is also reported to improve over Rein: 80.1, 74.1, 71.3, and 56.4 mIoU on ACDC fog, rain, snow, and night, respectively, together with 65.2 on BDD-100K and 75.1 on Mapillary (Li et al., 3 Aug 2025). In a multi-KS setup using GTA5 + SYNTHIA, the reported average mIoU rises from 62.7 for Rein to 64.0 for the SPL-based method, described as a +1.3% gain (Li et al., 3 Aug 2025).

The ablation studies are central to the method’s interpretation. The paper reports that near-optimal mIoU is reached with 1/4 of GTA5, which it interprets as evidence that strong VFMs plus dynamic prompt tuning reduce dependence on large labeled KS sizes (Li et al., 3 Aug 2025). It also reports that DCP is robust across reasonable ranges of $\mathcal{S}^A = \{ (x_j, y_j) \}_{j=1}^{M}.$3, $\mathcal{S}^A = \{ (x_j, y_j) \}_{j=1}^{M}.$4, and the maximum number of output classes $\mathcal{S}^A = \{ (x_j, y_j) \}_{j=1}^{M}.$5, with the best result around $\mathcal{S}^A = \{ (x_j, y_j) \}_{j=1}^{M}.$6, including an example of 67.9 mIoU on Cityscapes (Li et al., 3 Aug 2025). For PFF, self-attention alone or cross-attention alone improves over the baseline, but the full combination performs best. For the learnable token length $\mathcal{S}^A = \{ (x_j, y_j) \}_{j=1}^{M}.$7, the best average mIoU is reported at 75, with the range from 50 to 100 described as very similar (Li et al., 3 Aug 2025).

These results support the paper’s stated thesis that the gains are not merely from using a stronger frozen encoder, but from a VFM-centric adaptation mechanism that dynamically selects prompts and uses them to refocus feature processing.

6. Scope, limitations, and relation to other “SPL” terms

The paper explicitly motivates Set Pivot Learning as better aligned with practical deployment settings in which strong VFMs are already available, training data is heterogeneous, environments change over time, and strict guarantees of target-domain invisibility during pretraining are unrealistic (Li et al., 3 Aug 2025). It further suggests extensions to object detection, panoptic segmentation, video understanding, other VFM backbones, and possibly online or continual adaptation (Li et al., 3 Aug 2025).

The paper also identifies limitations. DCP depends on CLIP-based semantics, so prompt selection may be suboptimal if CLIP’s semantic space does not align with the application domain. The method does not perform explicit unlabeled adaptation on AS, remaining a KS-supervised approach. Finally, adding DCP, PFF, and Mask2Former increases architectural complexity relative to a simple fine-tuned head, even though the VFM is frozen (Li et al., 3 Aug 2025).

A recurring source of confusion is that the acronym SPL is already established elsewhere with different meanings. In the older machine-learning literature, SPL commonly denotes Self-Paced Learning, an easy-to-hard training regime based on sample weights and age parameters, with a substantial theory around latent non-convex objectives and majorization–minimization (Meng et al., 2015); (Liu et al., 2018). That meaning is also used in QSAR modeling, where SPL-Logsum combines Self-Paced Learning with Logsum-penalized logistic regression for descriptor selection and classification (Xia et al., 2018). In yet another later usage, SPL denotes Swap-guided Preference Learning for personalized RLHF, where fictitious swap annotators and a swap-guided regularizer are used to mitigate posterior collapse in variational preference learning (Kim et al., 13 Mar 2026).

Set Pivot Learning is therefore a distinct, later usage of the acronym, specific to VFM-based generalized segmentation. Its defining contribution is not an easy-to-hard optimization rule or a preference-learning regularizer, but a reframing of generalized segmentation around a frozen VFM as a pivot between a heterogeneous Knowledge Set and a deployment-oriented Application Set, together with a dynamic prompt-based adaptation mechanism built on top of that pivot (Li et al., 3 Aug 2025).