Prototypical Progressive Alignment and Reweighting (PPAR)

• PPAR is a framework for single-source domain generalization in semantic segmentation that leverages CLIP-based text prototypes to establish semantically invariant class anchors.
• It employs progressive alignment where shallow features align to visual-text prototypes and deep features align to original text prototypes for robust domain transfer.
• The method introduces pixel-wise reweighting to suppress unreliable source pixels, thereby reducing negative transfer under severe domain shifts.

Searching arXiv for PPAR and closely related alignment papers. Prototypical Progressive Alignment and Reweighting (PPAR) is a framework for single-source domain generalization semantic segmentation in which a segmentation model is trained on a labeled source domain only and is expected to generalize to unseen target domains at test time. Its core design combines CLIP-based text prototype generation, progressive prototype alignment, and prototype-guided reweighting. The method is motivated by the claim that class-wise prototypes can function as relatively stable, semantically consistent cues, but that naive source-batch centroids, coarse alignment strategies, and equal treatment of all source features are inadequate under severe shifts in style, color, texture, and dataset-specific appearance. PPAR therefore replaces source-centric prototypes with CLIP-derived text prototypes, aligns shallow and deep representations differently, and explicitly downweights unreliable source pixels to reduce negative transfer (Zhang et al., 16 Jul 2025).

1. Problem setting and motivating limitations

PPAR addresses generalizable semantic segmentation, particularly urban-scene segmentation under strong domain shift. The training regime is single-source domain generalization rather than unsupervised domain adaptation: the method uses labeled source data only during training, and no target data are available until evaluation on unseen domains. Within this setting, prototype-based methods are attractive because a class prototype is intended to capture a class-level centroid that is more stable than individual samples and can therefore act as a domain-invariant semantic anchor (Zhang et al., 16 Jul 2025).

The paper identifies three concrete shortcomings in prior prototype-based segmentation. First, many methods rely on a naive prototype computed as the source-batch class centroid. In the paper’s formulation, with labeled source images and masks $\{x_s, y_s\}$, the feature extractor $F(\cdot)$ produces masked class features

$${f^n}^{(h,w)} = F(x_s)^{(h,w)} \,\mathbb{1}(y_s^{(h,w,n)} == 1),$$

and the naive class prototype is the source centroid

$$p^n = \frac{\sum_{x_s\in X_s}\sum_h\sum_w {f^n}^{(h,w)}}{\sum_{x_s\in X_s}\sum_h\sum_w \mathbb{1}(y_s^{(h,w,n)}==1)}.$$

PPAR treats this prototype as source-biased, vulnerable to source outliers, and prone to overfitting rather than approximating an “overall-domain” centroid.

Second, the paper argues that prior alignment strategies are too coarse. Single-layer prototypical alignment (SPA) aligns only one layer, while multi-layer prototypical alignment (MPA) aligns shallow and deep features in the same way even though those layers encode different kinds of information. This, according to the paper, can suppress useful low-level visual cues in shallow layers.

Third, existing methods usually treat all source features equally, even though some samples or pixels are harder to transfer and some are unrelated or harmful. PPAR is explicitly designed to suppress negative transfer from source outliers or weakly transferable source pixels.

A common misconception is to read PPAR as a target-adaptive method in the style of pseudo-label-based adaptation. It is not. A second misconception is to interpret “progressive” as an explicit time-scheduled curriculum over samples. In PPAR, “progressive” is primarily depth-aware and easy-to-difficult by representation level, not a thresholded sample-mining schedule (Zhang et al., 16 Jul 2025).

2. CLIP-derived prototype system: OTP and VTP

PPAR’s prototype construction is centered on the CLIP text encoder $F_t(\cdot)$, which the paper treats as a source of generalized class representations. The method defines two fixed class prototypes per semantic class $n$: the Original Text Prototype (OTP) and the Visual Text Prototype (VTP).

Prototype	Definition	Intended role
OTP	$P_{to}^n = F_t(T_o^n)$	Pure category semantics
VTP	$P_{tv}^n = F_t(T_v^n)$	Category semantics plus representative visual factors

For OTP, $T_o^n$ is simply the class name string, such as “car,” “person,” or “building,” and the prototype is

$P_{to}^n = F_t(T_o^n).$

OTP is therefore the CLIP text embedding of the class name and serves as the pure category-level prototype (Zhang et al., 16 Jul 2025).

VTP augments the class name with representative source-domain visual factors before encoding. The paper uses two such factors: color and local texture. The representative color is the mode of class pixels,

$F(\cdot)$0

and the representative local texture is obtained from Local Binary Pattern values,

$F(\cdot)$1

These are concatenated with the class name,

$F(\cdot)$2

and the VTP is then

$F(\cdot)$3

The paper states that OTP and VTP are fixed CLIP-derived class prototypes, one per class, with no learnable update mechanism. From the implementation discussion, the segmentation feature dimension is remapped to 512 to match the dimension of the CLIP text feature, so the prototypes are effectively 512-dimensional. This suggests that PPAR’s prototype design is not merely a regularizer attached to source features; it is a replacement of source-derived centroids with text-conditioned anchors that are meant to approximate a broader cross-domain semantic prior (Zhang et al., 16 Jul 2025).

3. Progressive alignment across representation depth

The method’s main alignment mechanism is the prototypical progressive alignment (PPA). Its premise is that shallow and deep features should not be aligned identically because shallow features preserve more style and basic visual information, whereas deep features preserve more category semantics. PPAR therefore aligns shallow features to VTPs and deep features to OTPs.

The progressive alignment loss is

$F(\cdot)$4

where $F(\cdot)$5 is shallow alignment and $F(\cdot)$6 is deep alignment. For each layer, class centroids are still computed from segmentation features using the prototype formula of Eq. (2), yielding shallow centroids $F(\cdot)$7 and deep centroids $F(\cdot)$8. The shallow centroid is aligned to the VTP via KL divergence,

$F(\cdot)$9

and the deep centroid is aligned to the OTP,

$${f^n}^{(h,w)} = F(x_s)^{(h,w)} \,\mathbb{1}(y_s^{(h,w,n)} == 1),$$0

The crucial point is that “easy-to-difficult” here is conceptual rather than a time-varying stage schedule. Aligning shallow features to VTP is considered easier because VTP still contains representative source-like visual factors and thus lies closer to source features. Aligning deep features to OTP is considered harder because OTP strips away style and functions as a more domain-agnostic semantic anchor. The curriculum is therefore: preserve and align representative visual information at shallow layers through VTP, then align deeper semantic representations to pure semantic prototypes (Zhang et al., 16 Jul 2025).

The paper also validates where shallow alignment should occur in a ResNet backbone. In ResNet, Stage 1–4 correspond to $${f^n}^{(h,w)} = F(x_s)^{(h,w)} \,\mathbb{1}(y_s^{(h,w,n)} == 1),$$1. Deep alignment is fixed at Stage 4, while shallow alignment is tested at Stages 1–3; Stage 1 performs best. Multi-stage shallow alignment is also tested, and single-stage shallow alignment at Stage 1 is reported as superior.

4. Prototype-guided reweighting and the training objective

The second major component is prototypical reweighting (PR), which explicitly estimates source-feature reliability at the pixel level. This is a direct response to the paper’s claim that some source pixels are unrelated, harmful, or difficult to transfer, and should contribute less to the supervised loss.

Given the last-layer feature $${f^n}^{(h,w)} = F(x_s)^{(h,w)} \,\mathbb{1}(y_s^{(h,w,n)} == 1),$$2, the method compares it to OTPs through a softmax over dot products: $${f^n}^{(h,w)} = F(x_s)^{(h,w)} \,\mathbb{1}(y_s^{(h,w,n)} == 1),$$3 This yields a class-prototype similarity distribution for each pixel. The uncertainty of that distribution is measured by entropy,

$${f^n}^{(h,w)} = F(x_s)^{(h,w)} \,\mathbb{1}(y_s^{(h,w,n)} == 1),$$4

Low entropy indicates strong association with one OTP and is interpreted as high reliability; high entropy indicates weak correlation with generalized class semantics.

The uncertainty is then normalized and exponentiated into a positive weight,

$${f^n}^{(h,w)} = F(x_s)^{(h,w)} \,\mathbb{1}(y_s^{(h,w,n)} == 1),$$5

so that higher uncertainty produces lower weight. These weights are inserted into a reweighted cross-entropy segmentation loss: $${f^n}^{(h,w)} = F(x_s)^{(h,w)} \,\mathbb{1}(y_s^{(h,w,n)} == 1),$$6

The complete objective is

$${f^n}^{(h,w)} = F(x_s)^{(h,w)} \,\mathbb{1}(y_s^{(h,w,n)} == 1),$$7

with

$${f^n}^{(h,w)} = F(x_s)^{(h,w)} \,\mathbb{1}(y_s^{(h,w,n)} == 1),$$8

Substituting the alignment terms gives

$${f^n}^{(h,w)} = F(x_s)^{(h,w)} \,\mathbb{1}(y_s^{(h,w,n)} == 1),$$9

Another common misconception is that PPAR’s reweighting is class-level or target-level. It is not. The paper defines it as an explicit pixel-wise weighting mechanism on source supervision, driven by OTP-feature similarity, and interprets it as a way to reduce negative transfer from outliers and poorly transferable pixels (Zhang et al., 16 Jul 2025).

5. Architecture, theory, and empirical profile

Architecturally, PPAR uses a CNN backbone and an ASPP classifier head. The main backbones are ResNet-50, MobileNet-V2, and ShuffleNet-V2, all initialized from ImageNet pretraining. The final-layer feature dimension is projected to 512. The CLIP text encoder is used only to generate OTP and VTP, and the prototype modules are used only during training; at inference, the trained segmentation network is run directly without CLIP encoding (Zhang et al., 16 Jul 2025).

Training uses SGD with initial learning rate $$p^n = \frac{\sum_{x_s\in X_s}\sum_h\sum_w {f^n}^{(h,w)}}{\sum_{x_s\in X_s}\sum_h\sum_w \mathbb{1}(y_s^{(h,w,n)}==1)}.$$0, momentum $$p^n = \frac{\sum_{x_s\in X_s}\sum_h\sum_w {f^n}^{(h,w)}}{\sum_{x_s\in X_s}\sum_h\sum_w \mathbb{1}(y_s^{(h,w,n)}==1)}.$$1, weight decay $$p^n = \frac{\sum_{x_s\in X_s}\sum_h\sum_w {f^n}^{(h,w)}}{\sum_{x_s\in X_s}\sum_h\sum_w \mathbb{1}(y_s^{(h,w,n)}==1)}.$$2, batch size $$p^n = \frac{\sum_{x_s\in X_s}\sum_h\sum_w {f^n}^{(h,w)}}{\sum_{x_s\in X_s}\sum_h\sum_w \mathbb{1}(y_s^{(h,w,n)}==1)}.$$3, crop size $$p^n = \frac{\sum_{x_s\in X_s}\sum_h\sum_w {f^n}^{(h,w)}}{\sum_{x_s\in X_s}\sum_h\sum_w \mathbb{1}(y_s^{(h,w,n)}==1)}.$$4, and maximum iterations 100,000, with early stopping at 60,000 iterations. The implementation is in PyTorch. The evaluation metric is mIoU. The datasets are GTA5, SYNTHIA, Cityscapes, BDD100K, Mapillary, and IDD.

The theoretical analysis is framed through domain generalization theory based on Ben-David et al. and Albuquerque et al. The single source is augmented into multiple source-like domains $$p^n = \frac{\sum_{x_s\in X_s}\sum_h\sum_w {f^n}^{(h,w)}}{\sum_{x_s\in X_s}\sum_h\sum_w \mathbb{1}(y_s^{(h,w,n)}==1)}.$$5, and the unseen target distribution is assumed to lie in the convex hull of source distributions: $$p^n = \frac{\sum_{x_s\in X_s}\sum_h\sum_w {f^n}^{(h,w)}}{\sum_{x_s\in X_s}\sum_h\sum_w \mathbb{1}(y_s^{(h,w,n)}==1)}.$$6 The target risk is then bounded by weighted source risk terms, inter-source divergence $$p^n = \frac{\sum_{x_s\in X_s}\sum_h\sum_w {f^n}^{(h,w)}}{\sum_{x_s\in X_s}\sum_h\sum_w \mathbb{1}(y_s^{(h,w,n)}==1)}.$$7, and a residual term involving the approximation of the unseen target by the source convex hull. The paper’s interpretation is that prototypical reweighting lowers the effective source risk by reducing the impact of outliers or unrelated pixels, while progressive alignment lowers inter-source divergence by aligning augmented source representations toward text prototypes that approximate a pseudo overall-domain representation.

Empirically, the paper reports 43.1\% average mIoU with ResNet-50, outperforming SAN-SAW (38.5), SIL (40.4), PinMem (41.0), SHADE (42.5), and ISW (35.1). On lightweight backbones, PPAR reaches 37.4\% average mIoU on ShuffleNet-V2 and 38.2\% on MobileNet-V2. In multi-source DG, it obtains 42.3\% average mIoU for $$p^n = \frac{\sum_{x_s\in X_s}\sum_h\sum_w {f^n}^{(h,w)}}{\sum_{x_s\in X_s}\sum_h\sum_w \mathbb{1}(y_s^{(h,w,n)}==1)}.$$8 and 56.1\% for $$p^n = \frac{\sum_{x_s\in X_s}\sum_h\sum_w {f^n}^{(h,w)}}{\sum_{x_s\in X_s}\sum_h\sum_w \mathbb{1}(y_s^{(h,w,n)}==1)}.$$9 (Zhang et al., 16 Jul 2025).

The ablations isolate the value of both alignment and reweighting. On $F_t(\cdot)$0 with ResNet-50, the baseline is 35.1 average mIoU. Using only shallow progressive alignment with local texture yields 38.8, using only color yields 38.3, using only reweighting yields 38.7, combining the two VTP components yields 39.5, and combining all components yields 40.1. In a second ablation, SPA-NP and MPA-NP both score 36.2, SPA-OTP scores 37.7, MPA-OTP 37.9, SPA-VTP 37.1, MPA-VTP 37.5, while PPA (Ours) reaches 39.5. The paper also reports qualitative improvements in confusion matrices and t-SNE plots, with reduced class confusion and tighter clustering for classes such as road, sky, rider, and bus (Zhang et al., 16 Jul 2025).

6. Relation to earlier and adjacent alignment methods

PPAR is most directly related to prior work on prototype-based alignment, but its formulation differs in three specific ways: it uses CLIP-derived fixed text prototypes rather than source centroids, it assigns different prototype roles to shallow and deep layers, and it uses those same prototypes to derive an explicit pixel-wise reweighting signal.

A useful precursor is the Progressive Feature Alignment Network (PFAN), which was proposed for closed-set unsupervised domain adaptation rather than domain generalization. PFAN combines an Easy-to-Hard Transfer Strategy (EHTS), Adaptive Prototype Alignment (APA), and temperature-scaled source classification, thereby introducing progressive target inclusion and prototype-based class alignment under pseudo-label noise. However, its reweighting is largely implicit through binary target selection weights $F_t(\cdot)$1, a curriculum threshold $F_t(\cdot)$2, and similarity-adaptive prototype update weights $F_t(\cdot)$3, rather than the explicit pixel-wise reliability weighting used in PPAR (Chen et al., 2018).

Two later papers clarify the specificity of PPAR’s design by contrast. ProtoConNet is relevant to the “prototypical alignment” aspect because it includes a Prototypical Alignment module for open-set few-shot classification, but it does not present a literal progressive alignment and reweighting formulation; its closest weighting mechanism is attention-like fusion of contextual prototypes, not reliability-aware prototype reweighting (Shi et al., 16 Jul 2025). OneEncoder is highly relevant to the “progressive alignment” phrase because it incrementally aligns new modalities into a frozen shared space, yet it contains neither class prototypes nor prototype-guided reweighting in the PPAR sense (Faye et al., 2024).

This suggests that PPAR occupies a distinct position: it is neither a pseudo-label-driven UDA method nor a generic progressive alignment framework. It is specifically a CLIP-assisted domain generalization method for semantic segmentation in which text prototypes are treated as generalized class anchors, alignment is stratified by representation depth, and reliability estimation is built directly into the supervised segmentation loss. Its stated limitations follow from the same design choices: the method assumes that CLIP text embeddings are meaningful approximations to generalized class centroids, that representative source color and texture can be summarized by mode statistics, and that the task involves unseen domains with shared classes, not unseen classes (Zhang et al., 16 Jul 2025).