Prompt-driven Normalization (PDNorm)

• Prompt-driven Normalization is a feature-level method that optimizes instance normalization parameters using textual prompts to align source features with a target domain style.
• It employs gradient descent on CLIP-derived alignment loss to adjust affine style transformations, preserving content semantics while adapting appearance.
• Empirical studies show that PDNorm robustly improves performance in tasks like segmentation, detection, and classification without requiring target image data.

Prompt-driven Normalization (PDNorm), specifically instantiated as Prompt-driven Instance Normalization (PIN), is a feature-level normalization strategy in vision-LLMs that enables zero-shot domain adaptation using only a natural language description of the target domain. By leveraging pretrained contrastive vision-LLMs such as CLIP, PIN optimizes affine style transformations of intermediate source features to approximate the imagined appearance of a prompt-specified target domain, while explicitly preserving content semantics. This approach introduces a paradigm in which domain adaptation is feasible without requiring any access to target images or statistical priors beyond a textual prompt (Fahes et al., 2022).

1. Objective and Conceptual Framework

Prompt-driven Normalization addresses the task of "Prompt-driven Zero-shot Domain Adaptation." The central goal is to adapt a source-trained model $M$ (trained on image-label pairs $(x, y)$ from a source domain) to perform well on a target domain, which is not observed during training but is described by a single prompt $t$ (e.g., "driving at night"). The methodology centers on transporting intermediate feature maps $f \in \mathbb{R}^{C\times H\times W}$ from their native source style toward the style that would be elicited by the prompt, while leaving pixel-wise semantics and label supervision intact.

This process is operationalized through a two-stage pipeline:

• Style Mining: For each source feature, find per-channel scale ($\mu_t$) and bias ($\sigma_t$) that, when applied, align the CLIP image embedding of the stylized feature with the CLIP text embedding of the prompt.
• Adaptation: At each training step, randomly select one of these mined $(\mu_t, \sigma_t)$ pairs, stylize a source feature, and continue to train the model’s task head on these augmented features using ground-truth labels (Fahes et al., 2022).

2. Mathematical Formulation of PIN

PIN is rooted in the family of normalization methods, generalizing the template of Instance Normalization (IN) or Adaptive Instance Normalization (AdaIN), but replaces the style statistics with prompt-driven, optimizable variables. For an intermediate feature $f\in\mathbb{R}^{C\times H\times W}$:

• Source statistics are computed as follows:
  • $\mu(f)_c = \frac{1}{HW}\sum_{h,w} f_{c,h,w}$
  • $$\sigma(f)_c = \sqrt{\frac{1}{HW}\sum_{h,w}(f_{c,h,w}-\mu(f)_c)^2}$$
• PIN transforms features as:

  $(x, y)$0

where $(x, y)$1 and $(x, y)$2 are per-channel vectors (learned style parameters) driven by the prompt.

In this notation, $(x, y)$3 and $(x, y)$4 are affine transformations parameterized by the prompt $(x, y)$5.

3. Prompt-Driven Style Optimization

Unlike AdaIN, PIN avoids mapping the prompt directly to style parameters via an auxiliary MLP. Instead, for each source feature, it optimizes $(x, y)$6 and $(x, y)$7 directly to minimize a CLIP-derived prompt-alignment loss:

• For each source feature, initialize $(x, y)$8 and $(x, y)$9 to preserve semantic fidelity.
• For $t$0 steps (typically $t$1), perform gradient descent on:

  $t$2

  where $t$3 and $t$4. This objective aligns the pooled image feature of the stylized source feature $t$5 with the text embedding of the prompt.

No auxiliary MLP or additional regularization is required; the initialization from source statistics acts as a strong prior for content preservation.

4. Losses, Adaptation Pipeline, and Integration

The adaptation workflow can be summarized in two stages:

• Style Mining Stage: For each source feature, only $t$6 is minimized to populate a pool $t$7 of $t$8 pairs. No task (e.g., segmentation) labels are used in this stage.
• Adaptation Stage: During model fine-tuning, features are randomly stylized using $t$9. The task head is trained exclusively with the standard task objective $f \in \mathbb{R}^{C\times H\times W}$0, with no further image-text alignment loss. PIN is inserted at a single low-level layer (e.g., after Layer1 in a DeepLabv3+ model with a CLIP-ResNet backbone). The backbone remains frozen to maximize zero-shot generalization; only the task head is updated. At inference, PIN is bypassed or replaced by source statistics, and predictions are made with the adapted task head (Fahes et al., 2022).

Empirical findings indicate hyperparameters such as $f \in \mathbb{R}^{C\times H\times W}$1 mining steps, batch size $f \in \mathbb{R}^{C\times H\times W}$2, learning rate $f \in \mathbb{R}^{C\times H\times W}$3 for style optimization, and full utilization of the source set for style mining provide robust adaptation behavior. Training with the entire backbone frozen, rather than early-layer fine-tuning, enhances generalization to the prompt-imagined target style.

5. Empirical Performance and Observations

Extensive experiments on semantic segmentation, object detection, and classification tasks demonstrate that PIN-based prompt-driven augmentation substantially surpasses CLIP-based style transfer baselines. Notably, in semantic segmentation on Cityscapes→ACDC-Night adaptation, performance improves rapidly with increasing mining steps up to approximately $f \in \mathbb{R}^{C\times H\times W}$4–$f \in \mathbb{R}^{C\times H\times W}$5, after which plateau or mild decline is observed due to potential over-stylization. Even small style pools (e.g., $f \in \mathbb{R}^{C\times H\times W}$6) yield benefits though with higher variance; utilizing the full pool of source images yields maximal stability.

Prompt wording is robust to semantically equivalent expressions; arbitrary or irrelevant prompts can degrade adaptation quality. At inference, omitting or trivializing PIN incurs no penalty, as the adapted head maintains target domain robustness even without explicit style steering (Fahes et al., 2022).

6. Implementation Pipeline and Pseudocode

The procedural workflow for PIN can be summarized as follows:

1. Source Training: Train the task head on unmodified source features.
2. Style Mining: For each source image, extract features, initialize style parameters to source statistics, and apply $f \in \mathbb{R}^{C\times H\times W}$7 steps of gradient descent on $f \in \mathbb{R}^{C\times H\times W}$8 to mine $f \in \mathbb{R}^{C\times H\times W}$9 that best align with the prompt’s text embedding.
3. Adaptation: Across several epochs, sample a random source image and a style pair $\mu_t$0 from the mined pool, stylize the feature, and train the task head (with frozen backbone) to minimize task loss.
4. Inference: Process target images with the adapted task head, bypassing or trivializing PIN.

This yields a plug-and-play "feature styler" that, conditioned purely on CLIP text embeddings, generates diverse and semantically-faithful variants of the source domain, supporting zero-shot adaptation without target data (Fahes et al., 2022).

7. Context and Significance

Prompt-driven Normalization via PIN advances unsupervised domain adaptation by exploiting vision-language pretraining and parametric feature steering in the absence of any target data. Unlike conventional domain adaptation which requires at least a few (or many) target samples, PIN demonstrates that natural language prompts sufficed to drive meaningful and robust appearance transformations at the intermediate representation level, transmitted through the lens of CLIP’s dual encodings. This approach permits adaptation not only for segmentation but also object detection and classification, with empirical results showing improvements even over one-shot unsupervised domain adaptation. A plausible implication is that CLIP-style models, coupled with prompt-driven feature normalization, enable a new research axis where flexible prompt engineering combined with modular normalization can deliver robust adaptation without direct supervision from the target domain (Fahes et al., 2022).