Concept-Constrained Prompt Learning for Few-Shot CLIP Adaptation

Abstract: Few-shot prompt learning is an effective strategy for adapting CLIP to downstream tasks, but class-only prompt optimization can overfit base-class supervision and weaken transfer to unseen classes. We propose Concept-Constrained Prompt Learning (CCPL), a lightweight regularization framework that anchors learnable class prompts to frozen concept-level text prototypes without updating CLIP encoders. CCPL learns a set of shared context tokens, instantiates class prompts by appending class names, and constructs frozen concept prototypes from a class-level concept bank. During training, a text-space cosine consistency objective aligns learnable class-prompt embeddings with frozen concept prototypes; concept dropout provides additional regularization against over-reliance on fixed concept lists. At inference, CCPL optionally fuses class-prompt logits with concept-prototype logits using a controllable ensemble weight alpha. Our default configuration uses text-space concept regularization lambda = 0.5, concept dropout p = 0.3 and weak concept-guided fusion (alpha = 0.1), with no KL-based prediction consistency term. Experiments under identical automatically-generated fallback splits show that CCPL improves the base-to-new harmonic mean on DTD (+0.6) and EuroSAT (+2.9) compared with CoOp, while remaining near-neutral on OxfordPets (-0.1). Ablations indicate that text-space concept regularization is consistently beneficial, while the best concept-guided inference strength is dataset- and protocol-sensitive. These results suggest concept constraints are most effective when concept prototypes align naturally with dataset semantics, and identify fine-grained categories as a current boundary condition. The code is released at: https://github.com/richael-sang/concept-constrained-prompt-learning.

• The paper introduces CCPL, a method that regularizes prompt learning using frozen concept prototypes to prevent overfitting in few-shot CLIP adaptation.
• It employs a cosine consistency objective and concept dropout to anchor learned prompt embeddings to semantically rich text prototypes without updating CLIP parameters.
• Experimental results demonstrate improved new-class accuracy and harmonic mean on datasets like EuroSAT, highlighting a trade-off between base and new class performance.

Concept-Constrained Prompt Learning for Few-Shot CLIP Adaptation: A Technical Overview

Motivation and Problem Formulation

The adaptation of CLIP-style vision-LLMs under few-shot regimes poses a fundamental challenge: optimizing for task specialization while preserving the rich open-vocabulary structure originally learned from massive, diverse image-text pairs. Text-based prompt learning, exemplified by CoOp, leverages context token optimization without updating encoders, providing parameter efficiency and strong base-class accuracy. However, such methods risk overfitting to semantically sparse class names, degrading transfer to novel categories. The paper introduces Concept-Constrained Prompt Learning (CCPL), which regularizes learned class prompts using frozen concept-level text prototypes, constructed from a curated concept bank per dataset class.

Figure 1: CCPL schematic illustrating regularization of learned class-prompt embeddings toward frozen semantic prototypes in text space.

CCPL anchors prompt optimization to semantic directions aligned with concept-level textual representations, aiming to restrict the drift of prompt embeddings while maintaining the CLIP encoder's semantic interpretability. The approach keeps all CLIP parameters frozen, introducing only shared context tokens as trainable parameters.

Approach: Concept Anchoring and Regularization

CCPL's method comprises several distinct steps:

• Concept Prototype Construction: For each class $c$, a set of concept phrases is extracted from a hand-crafted bank and encoded via the frozen CLIP text encoder. Their normalized embeddings are averaged to form the class-specific semantic anchor.
• Prompt Learning: The prompt for each class is composed of shared learnable context tokens appended to the class name. Prompt embeddings are generated by passing these composites through the frozen text encoder.
• Cosine Consistency Objective: During training, a text-space cosine loss penalizes deviation between the learned prompt and the frozen concept prototype for base classes:

$$\mathcal{L}_{\text{text}} = \frac{1}{|\mathcal{C}_{\text{base}}|} \sum_{c \in \mathcal{C}_{\text{base}}} (1 - \cos(\mathbf{t}_c^{\text{cls}}(\theta),\, \mathbf{t}_c^{\text{con}}))$$

• Concept Dropout: To prevent over-reliance on any single phrase, concept dropout randomly masks concept phrases per class during training.
• Inference Fusion: At inference, class-prompt logits and concept-prototype logits are combined via a convex weight $\alpha$, allowing tunable reliance on concept-driven predictions.

Experimental Evaluation and Results

The evaluation protocol uses 4-shot adaptation and a base-to-new split on three datasets: DTD (textures), EuroSAT (satellite scenes), and OxfordPets (fine-grained breeds). All models employ CLIP ViT-B/16 with identical fallback protocol splits and frozen encoders. Key results are summarized as follows:

Figure 2: Harmonic mean improvement for DTD and EuroSAT, with EuroSAT's gains driven by increased new-class accuracy.

• EuroSAT: CCPL raises new-class accuracy (56.3% to 60.6%) and the harmonic mean (66.0 to 68.9), mainly from enhanced transferability via semantic alignment.
• DTD: Moderate harmonic mean gain ($+$0.6), with stable new-class accuracy and marginal increase in base-class performance.
• OxfordPets: Neutral effect; harmonic mean change is $-$0.1, demonstrating boundary conditions for concept constraints in fine-grained settings.

Seed robustness checks confirm directionally consistent gains in DTD across distinct initialization seeds.

Figure 3: Seed robustness on DTD; CCPL-default consistently exceeds CoOp across tested seeds.

Ablation and Parameter Sensitivity

The ablation study clarifies component contributions:

• Text-Space Regularization: Removing the cosine consistency term reduces the harmonic mean, underscoring its role in improving new-class generalization.
• Inference Fusion Weight ($\alpha$): Higher $\alpha$ increases new-class accuracy and overall harmonic mean in EuroSAT, but at the expense of base-class performance. $\alpha$ thus enables explicit control over the base-new accuracy trade-off.

  Figure 4: EuroSAT ablation showing harmonic mean for variants and alpha-induced base-new trade-off.

Further analysis confirms harmonic mean maximization at $\alpha=0.4$ for EuroSAT, with the corresponding reduction in base-class performance.

Figure 5: Alpha sensitivity on EuroSAT; large $\alpha$ improves new-class accuracy while decreasing base-class metrics.

Analysis of Concept Quality and Dataset Dependence

The effectiveness of CCPL is tightly linked to the quality and specificity of the concept prototypes:

• Scene and texture datasets (EuroSAT, DTD) benefit from concept phrases capturing salient, class-discriminative attributes.
• Fine-grained tasks (OxfordPets) show limited gains, as template-based phrases lack breed-specific information.

A representative concept phrase table illustrates the semantic richness for EuroSAT versus the generic nature for OxfordPets.

Implications, Limitations, and Future Directions

The practical implications are twofold:

1. CCPL requires hand-crafted or dataset-specific concept banks to achieve meaningful regularization. Its utility tracks the semantic fidelity of available concept descriptors.
2. The method delivers efficient few-shot adaptation via lightweight regularization, without encoder updates, and offers interpretable control over trade-offs through inference fusion.

Limitations include the reliance on manually constructed concept banks, restricted dataset coverage, protocol differences with published baselines, and unmeasured alignment dynamics. Prospective directions comprise automatic concept generation via LLMs, learnable concept weighting, broader dataset validation, and official split corroboration.

Conclusion

Concept-Constrained Prompt Learning (CCPL) introduces a principled, parameter-efficient regularization mechanism for prompt learning in CLIP, leveraging frozen concept prototypes to mitigate base-class overfitting and enhance semantic transfer to unseen classes. Significant gains manifest in scene and texture recognition tasks, with explicit identification of boundary conditions in fine-grained settings. CCPL contributes a practical framework for the principled adaptation of vision-LLMs under few-shot supervision, foregrounding the crucial role of semantically anchored regularization.