Prompt-Driven Instance Normalization (PIN)

• PIN is a feature normalization method that aligns visual features with semantic prompts for zero-shot domain adaptation without using target images.
• It leverages CLIP’s latent multimodal space by optimizing channel-wise statistics to achieve competitive performance in tasks like segmentation and object detection.
• The approach offers efficiency and prompt flexibility, while also presenting challenges such as prompt dependency and limitations imposed by CLIP’s pretraining distribution.

Prompt-driven Instance Normalization (PIN) is a normalization and feature adaptation technique that enables zero-shot domain adaptation in deep neural networks by aligning input feature statistics to semantic targets specified via natural language prompts. Unlike conventional style transfer or domain adaptation methods, PIN requires no images from the target domain. Instead, it leverages the latent multimodal space of pretrained vision-LLMs (notably CLIP) to steer source-domain features toward the semantics of a descriptive prompt, enabling practical adaptation in settings where target data is unavailable or resource constraints preclude heavyweight adaptation protocols.

1. Definition and Conceptual Overview

Prompt-driven Instance Normalization (PIN) is an affine normalization mechanism formulated for zero-shot domain adaptation scenarios in vision. PIN adapts source image features to a target domain described solely by a language prompt, without requirement for target images at any stage. The main technical novelty lies in optimizing the channel-wise mean and standard deviation parameters of instance normalization so that, after normalization, the CLIP-image embedding of the normalized features is as close as possible (under cosine similarity) to the CLIP-text embedding of the prompt describing the target domain. This strategy facilitates semantic alignment across domains using only semantic cues rather than visual samples (Fahes et al., 2022, &&&1&&&).

2. Mathematical Formulation

Let $f \in \mathbb{R}^{h \times w \times c}$ denote the low-level visual features extracted from a frozen backbone (typically the initial layers of CLIP's visual encoder) for a source image. Let $T_\mathrm{Target}$ denote the prompt (e.g., "driving in snow" or "a road scene on a rainy night"), and let $\mathrm{txt}(T_\mathrm{Target})$ denote its CLIP text embedding.

PIN is defined as

$$\mathrm{PIN}(f, \mu, \sigma) = \sigma \left( \frac{f - \mu(f)}{\sigma(f)} \right) + \mu$$

where:

• $\mu(f), \sigma(f)$ are the per-channel mean and standard deviation across spatial dimensions of $f$.
• $\mu, \sigma$ are optimizable statistics—not originating from data, but learned to steer $f$ toward the target semantic.

The semantic alignment objective is

$$L_{\mu, \sigma}\left(\bar{f}_{s \rightarrow t}, \mathrm{TrgEmb}\right) = 1 - \frac{\bar{f}_{s \rightarrow t} \cdot \mathrm{TrgEmb}}{\|\bar{f}_{s \rightarrow t}\| \, \|\mathrm{TrgEmb}\|}$$

where:

• $\bar{f}_{s \rightarrow t}$ is the CLIP image encoder embedding of the normalized feature $\mathrm{PIN}(f_s, \mu, \sigma)$,
• $\mathrm{TrgEmb}$ is the CLIP text encoder embedding of the prompt $T_\mathrm{Target}$.

The minimization of this loss results in style parameters ($\mu, \sigma$) that re-statisticize the source feature such that its latent representation in CLIP space approaches the intended target domain, as defined by the prompt.

3. Methodological Workflow and Integration

PIN is central to the teacher-student pipelines in prompt-driven zero-shot domain adaptation frameworks, including PØDA (Fahes et al., 2022) and Prmpt2Adpt (Farrukh et al., 20 Jun 2025). The adaptation process follows:

1. Feature Extraction: A small cache of low-level source features ($f_s$) is obtained using a frozen vision-language backbone (e.g., CLIP).
2. Prompt Embedding: The domain description is encoded as a text embedding using the CLIP text encoder.
3. Style Mining via PIN: For each source feature, PIN parameters ($\mu, \sigma$) are optimized—initialized from source statistics, updated via gradient descent on $L_{\mu, \sigma}$.
4. Augmentation: The resulting PIN-normalized features constitute "prompt-steered" source features semantically aligned with the prompt, serving as synthetic target-like features.
5. Fine-Tuning Task Head: These features, paired with original source annotations, are used to fine-tune only the task-specific (e.g., detection or segmentation) head, keeping the backbone frozen.
6. Self-Training: For frameworks with a student model (e.g., Prmpt2Adpt), the adapted teacher generates pseudo-labels for the student, enabling efficient and lightweight adaptation.

Resource constraints are respected by retaining only a handful of source images in memory and forgoing any target data at any stage.

4. Experimental Evaluation and Empirical Impact

PIN yields significant gains in domain adaptation benchmarks, even in the absence of target images:

• For semantic segmentation (ACDC Night: Cityscapes $\rightarrow$ ACDC-Night), PIN achieves mean IoU of 25.03% compared to 18.31% for source-only and 21.38% for CLIPstyler (an image-space stylization baseline).
• It consistently matches or outperforms one-shot domain adaptation (SM-PPM) despite requiring zero target images.
• For object detection (MDS-A dataset in Prmpt2Adpt (Farrukh et al., 20 Jun 2025)), PIN enables competitive performance at up to 7$\times$ faster adaptation and 5$\times$ faster inference than prior methods (PODA, ULDA), while using as few as five source images.
• Qualitative validation demonstrates that PIN-guided features are more semantically congruent with prompt intent in CLIP space, yielding superior pseudo-labels for downstream adaptation.

Ablation studies confirm that PIN applied to early (low-level) features maximizes adaptation efficacy; increasing the number of "mined" prompt-driven styles improves performance stability.

5. Theoretical Connections and Distinctions

PIN draws direct inspiration from Adaptive Instance Normalization (AdaIN), which stylizes content images through the affine transfer of statistics from reference images. Distinctively, PIN does not require any real or synthetic target images, as the "style" is determined by optimizing statistics to satisfy a multimodal (vision-language) semantic similarity criterion.

Where ILM~Norm (Jia et al., 2019) meta-learns normalization parameters as functions of instance statistics via an auto-encoder, PIN employs direct semantic guidance by grounding instance normalization in explicit prompt targets within the CLIP latent space. This affords adaptation by intention, not by implicit correlation to instance statistics.

PIN is most naturally positioned within the spectrum of prompt-driven zero-shot domain adaptation techniques, exploiting CLIP's universal latent space to perform feature-space augmentation that preserves structure while imparting contextual adaptation cues derived from the prompt.

6. Practical Advantages and Limitations

The principal strengths of PIN are:

• Zero-shot adaptation: Effective with only source data and natural language prompts—no target images or distribution statistics required.
• Computational efficiency: Feature-level transformation avoids the overhead and artifacts associated with image-level style transfer (e.g., CLIPstyler), while being over 200$\times$ faster in stylization step.
• Generalization and versatility: Demonstrated across semantic segmentation, detection, and classification.
• Prompt flexibility: Robust to prompt rewording, provided semantic relevance is maintained.

Limitations include:

• Prompt semantic dependence: Effectiveness relies on the supplied prompt's domain relevance; unrelated prompts can degrade adaptation.
• Latent space limitations: Coverage is bound by CLIP's pretraining distribution—performance may deteriorate for out-of-distribution prompts/domains.
• Optimization drift: Without careful initialization, statistic optimization might diverge if the CLIP semantic manifold is not adequately covered.

7. Extensions, Generalizations, and Related Methods

The PIN approach generalizes to a variety of tasks (segmentation, detection, classification) and is broadly agnostic to the backbone architecture, as long as a CLIP-compatible feature extractor and text encoder are available. Its design is particularly suitable for on-the-fly adaptation in memory- and compute-bounded environments (e.g., drones, robotics).

PIN is distinguished from internal-statistic meta-normalizations (ILM~Norm (Jia et al., 2019)) by its explicit, externally driven semantics, while being markedly more tractable and invertible than generative domain translation or pixel-space style transfer. The feature-space perspective and direct semantic guidance point toward future research in cross-modal normalization using universal latent spaces, further decoupling domain alignment from both visual and statistical correlation constraints.

---

Aspect	PIN	Related Methods
Domain signal	Language prompt (external semantics)	AdaIN: reference image; ILM~Norm: statistics
Requires target images	No	AdaIN: Yes; ILM~Norm: No
Level of adaptation	Feature-level (affine normalization)	AdaIN: feature; CLIPstyler: pixel
Optimization criterion	Semantic proximity in CLIP space	AdaIN: statistical matching; ILM~Norm: meta-learning statistics
Applications	Zero-shot DA, segmentation, detection, classification	Varies

PIN represents an instantiation of prompt-steered, zero-shot adaptation strategies, establishing a computationally efficient and semantically defined pathway for domain generalization in vision tasks (Fahes et al., 2022, Farrukh et al., 20 Jun 2025).