---
title: Prototypical Progressive Alignment and Reweighting (PPAR)
url: https://www.emergentmind.com/topics/prototypical-progressive-alignment-and-reweighting-ppar
type: topic
---

# Prototypical Progressive Alignment and Reweighting (PPAR)

Searching arXiv for PPAR and closely related alignment papers.
Prototypical Progressive Alignment and Reweighting (PPAR) is a framework for **single-source domain generalization semantic segmentation** in which a segmentation model is trained on a labeled source domain only and is expected to generalize to **unseen target domains** at test time. Its core design combines **CLIP-based text prototype generation**, **progressive prototype alignment**, and **prototype-guided reweighting**. The method is motivated by the claim that class-wise prototypes can function as relatively stable, semantically consistent cues, but that naive source-batch centroids, coarse alignment strategies, and equal treatment of all source features are inadequate under severe shifts in **style, color, texture, and dataset-specific appearance**. PPAR therefore replaces source-centric prototypes with CLIP-derived text prototypes, aligns shallow and deep representations differently, and explicitly downweights unreliable source pixels to reduce **negative transfer** [2507.11955].

## 1. Problem setting and motivating limitations

PPAR addresses **generalizable semantic segmentation**, particularly urban-scene segmentation under strong domain shift. The training regime is **single-source domain generalization** rather than unsupervised domain adaptation: the method uses labeled source data only during training, and no target data are available until evaluation on unseen domains. Within this setting, prototype-based methods are attractive because a class prototype is intended to capture a **class-level centroid** that is more stable than individual samples and can therefore act as a **domain-invariant semantic anchor** [2507.11955].

The paper identifies three concrete shortcomings in prior prototype-based segmentation. First, many methods rely on a **naive prototype** computed as the source-batch class centroid. In the paper’s formulation, with labeled source images and masks \(\{x_s, y_s\}\), the feature extractor \(F(\cdot)\) produces masked class features
\[
{f^n}^{(h,w)} = F(x_s)^{(h,w)} \,\mathbb{1}(y_s^{(h,w,n)} == 1),
\]
and the naive class prototype is the source centroid
\[
p^n = \frac{\sum_{x_s\in X_s}\sum_h\sum_w {f^n}^{(h,w)}}{\sum_{x_s\in X_s}\sum_h\sum_w \mathbb{1}(y_s^{(h,w,n)}==1)}.
\]
PPAR treats this prototype as **source-biased**, vulnerable to source outliers, and prone to overfitting rather than approximating an “overall-domain” centroid.

Second, the paper argues that prior alignment strategies are too coarse. **Single-layer prototypical alignment (SPA)** aligns only one layer, while **multi-layer prototypical alignment (MPA)** aligns shallow and deep features in the same way even though those layers encode different kinds of information. This, according to the paper, can suppress useful low-level visual cues in shallow layers.

Third, existing methods usually **treat all source features equally**, even though some samples or pixels are harder to transfer and some are unrelated or harmful. PPAR is explicitly designed to suppress negative transfer from **source outliers or weakly transferable source pixels**.

A common misconception is to read PPAR as a target-adaptive method in the style of pseudo-label-based adaptation. It is not. A second misconception is to interpret “progressive” as an explicit time-scheduled curriculum over samples. In PPAR, “progressive” is primarily **depth-aware and easy-to-difficult by representation level**, not a thresholded sample-mining schedule [2507.11955].

## 2. CLIP-derived prototype system: OTP and VTP

PPAR’s prototype construction is centered on the CLIP text encoder \(F_t(\cdot)\), which the paper treats as a source of **generalized class representations**. The method defines two fixed class prototypes per semantic class \(n\): the **Original Text Prototype (OTP)** and the **Visual Text Prototype (VTP)**.

| Prototype | Definition | Intended role |
|---|---|---|
| OTP | \(P_{to}^n = F_t(T_o^n)\) | Pure category semantics |
| VTP | \(P_{tv}^n = F_t(T_v^n)\) | Category semantics plus representative visual factors |

For OTP, \(T_o^n\) is simply the class name string, such as “car,” “person,” or “building,” and the prototype is
\[
P_{to}^n = F_t(T_o^n).
\]
OTP is therefore the CLIP text embedding of the class name and serves as the pure category-level prototype [2507.11955].

VTP augments the class name with representative source-domain visual factors before encoding. The paper uses two such factors: **color** and **local texture**. The representative color is the mode of class pixels,
\[
C^n = Mo\!\left(I(x_s \,\mathbb{1}(y_s^{n} == 1))\right),
\]
and the representative local texture is obtained from Local Binary Pattern values,
\[
LT^n = Mo\!\left(LBP(x_s \,\mathbb{1}(y_s^{n} == 1))\right).
\]
These are concatenated with the class name,
\[
T_v^n = \phi\!\left(T_o^n,\ \text{``with color ''},\ C^n,\ \text{`` with local texture ''},\ LT^n\right),
\]
and the VTP is then
\[
P_{tv}^n = F_t(T_v^n).
\]

The paper states that OTP and VTP are **fixed CLIP-derived class prototypes**, one per class, with no learnable update mechanism. From the implementation discussion, the segmentation feature dimension is remapped to **512** to match the dimension of the CLIP text feature, so the prototypes are effectively 512-dimensional. This suggests that PPAR’s prototype design is not merely a regularizer attached to source features; it is a replacement of source-derived centroids with text-conditioned anchors that are meant to approximate a broader cross-domain semantic prior [2507.11955].

## 3. Progressive alignment across representation depth

The method’s main alignment mechanism is the **prototypical progressive alignment (PPA)**. Its premise is that shallow and deep features should not be aligned identically because shallow features preserve more style and basic visual information, whereas deep features preserve more category semantics. PPAR therefore aligns **shallow features to VTPs** and **deep features to OTPs**.

The progressive alignment loss is
\[
\mathcal{L}_{pa} = \mathcal{L}_{as} + \mathcal{L}_{ad},
\]
where \(\mathcal{L}_{as}\) is shallow alignment and \(\mathcal{L}_{ad}\) is deep alignment. For each layer, class centroids are still computed from segmentation features using the prototype formula of Eq. (2), yielding shallow centroids \(P_s^n\) and deep centroids \(P_d^n\). The shallow centroid is aligned to the VTP via KL divergence,
\[
\mathcal{L}_{as} = \sum_{n=0}^{N} P_s^n \log \frac{P_s^n}{P_{tv}^n},
\]
and the deep centroid is aligned to the OTP,
\[
\mathcal{L}_{ad} = \sum_{n=0}^{N} P_d^n \log \frac{P_d^n}{P_{to}^n}.
\]

The crucial point is that “easy-to-difficult” here is **conceptual rather than a time-varying stage schedule**. Aligning shallow features to VTP is considered easier because VTP still contains representative source-like visual factors and thus lies closer to source features. Aligning deep features to OTP is considered harder because OTP strips away style and functions as a more domain-agnostic semantic anchor. The curriculum is therefore: preserve and align representative visual information at shallow layers through VTP, then align deeper semantic representations to pure semantic prototypes [2507.11955].

The paper also validates where shallow alignment should occur in a ResNet backbone. In ResNet, Stage 1–4 correspond to \(\{\text{conv2\_x}, \text{conv3\_x}, \text{conv4\_x}, \text{conv5\_x}\}\). Deep alignment is fixed at Stage 4, while shallow alignment is tested at Stages 1–3; **Stage 1 performs best**. Multi-stage shallow alignment is also tested, and single-stage shallow alignment at Stage 1 is reported as superior.

## 4. Prototype-guided reweighting and the training objective

The second major component is **prototypical reweighting (PR)**, which explicitly estimates source-feature reliability at the **pixel level**. This is a direct response to the paper’s claim that some source pixels are unrelated, harmful, or difficult to transfer, and should contribute less to the supervised loss.

Given the last-layer feature \(f_l^{(h,w)}\), the method compares it to OTPs through a softmax over dot products:
\[
{S^n}^{(h,w)} = \frac{\exp(P_{to}^n \cdot f_l^{(h,w)})}{\sum_{i=1}^{N}\exp(P_{to}^i \cdot f_l^{(h,w)})}.
\]
This yields a class-prototype similarity distribution for each pixel. The uncertainty of that distribution is measured by entropy,
\[
U^{(h,w)} = -\sum_{n=1}^{N} S^n^{(h,w)} \log S^n^{(h,w)}.
\]
Low entropy indicates strong association with one OTP and is interpreted as high reliability; high entropy indicates weak correlation with generalized class semantics.

The uncertainty is then normalized and exponentiated into a positive weight,
\[
R^{(h,w)} = \exp\!\left(-\frac{U^{(h,w)}-U_{\min}}{U_{\max}-U_{\min}}\right),
\]
so that higher uncertainty produces lower weight. These weights are inserted into a reweighted cross-entropy segmentation loss:
\[
\mathcal{L}_{rs} = -\sum_{h,w} R^{(h,w)} \cdot \sum_{n\in N} y^{(h,w,n)} \log M^{(h,w,n)}.
\]

The complete objective is
\[
\mathcal{L}_{all} = \mathcal{L}_{rs} + \alpha_{pa}\mathcal{L}_{pa},
\]
with
\[
\alpha_{pa} = 0.001.
\]
Substituting the alignment terms gives
\[
\mathcal{L}_{all} =
-\sum_{h,w} R^{(h,w)} \sum_{n\in N} y^{(h,w,n)}\log M^{(h,w,n)}
+ \alpha_{pa}\left(
\sum_{n=0}^{N} P_s^n \log \frac{P_s^n}{P_{tv}^n}
+
\sum_{n=0}^{N} P_d^n \log \frac{P_d^n}{P_{to}^n}
\right).
\]

Another common misconception is that PPAR’s reweighting is class-level or target-level. It is not. The paper defines it as an **explicit pixel-wise weighting mechanism on source supervision**, driven by OTP-feature similarity, and interprets it as a way to reduce negative transfer from outliers and poorly transferable pixels [2507.11955].

## 5. Architecture, theory, and empirical profile

Architecturally, PPAR uses a CNN backbone and an ASPP classifier head. The main backbones are **ResNet-50**, **MobileNet-V2**, and **ShuffleNet-V2**, all initialized from ImageNet pretraining. The final-layer feature dimension is projected to **512**. The CLIP text encoder is used only to generate OTP and VTP, and the prototype modules are **used only during training**; at inference, the trained segmentation network is run directly without CLIP encoding [2507.11955].

Training uses **SGD** with initial learning rate \(0.01\), momentum \(0.9\), weight decay \(0.0025\), batch size \(4\), crop size \(768\times768\), and maximum iterations **100,000**, with early stopping at **60,000** iterations. The implementation is in **PyTorch**. The evaluation metric is **mIoU**. The datasets are **GTA5, SYNTHIA, Cityscapes, BDD100K, Mapillary, and IDD**.

The theoretical analysis is framed through domain generalization theory based on Ben-David et al. and Albuquerque et al. The single source is augmented into multiple source-like domains \(D_s^i \sim P_s^i\), and the unseen target distribution is assumed to lie in the convex hull of source distributions:
\[
P_{ut} = \sum_{i=1}^{N_s}\alpha_i P_s^i,\qquad \sum_{i=1}^{N_s}\alpha_i = 1.
\]
The target risk is then bounded by weighted source risk terms, inter-source divergence \(\delta\), and a residual term involving the approximation of the unseen target by the source convex hull. The paper’s interpretation is that **prototypical reweighting** lowers the effective source risk by reducing the impact of outliers or unrelated pixels, while **progressive alignment** lowers inter-source divergence by aligning augmented source representations toward text prototypes that approximate a pseudo overall-domain representation.

Empirically, the paper reports **43.1\% average mIoU** with **ResNet-50**, outperforming SAN-SAW (**38.5**), SIL (**40.4**), PinMem (**41.0**), SHADE (**42.5**), and ISW (**35.1**). On lightweight backbones, PPAR reaches **37.4\%** average mIoU on **ShuffleNet-V2** and **38.2\%** on **MobileNet-V2**. In multi-source DG, it obtains **42.3\%** average mIoU for \(G+S\rightarrow\{C,B,M\}\) and **56.1\%** for \(G+S+I\rightarrow\{C,B,M\}\) [2507.11955].

The ablations isolate the value of both alignment and reweighting. On \(G\rightarrow\{C,B,M,S\}\) with ResNet-50, the baseline is **35.1** average mIoU. Using only shallow progressive alignment with local texture yields **38.8**, using only color yields **38.3**, using only reweighting yields **38.7**, combining the two VTP components yields **39.5**, and combining all components yields **40.1**. In a second ablation, **SPA-NP** and **MPA-NP** both score **36.2**, **SPA-OTP** scores **37.7**, **MPA-OTP** **37.9**, **SPA-VTP** **37.1**, **MPA-VTP** **37.5**, while **PPA (Ours)** reaches **39.5**. The paper also reports qualitative improvements in confusion matrices and t-SNE plots, with reduced class confusion and tighter clustering for classes such as road, sky, rider, and bus [2507.11955].

## 6. Relation to earlier and adjacent alignment methods

PPAR is most directly related to prior work on prototype-based alignment, but its formulation differs in three specific ways: it uses **CLIP-derived fixed text prototypes** rather than source centroids, it assigns **different prototype roles to shallow and deep layers**, and it uses those same prototypes to derive an **explicit pixel-wise reweighting signal**.

A useful precursor is the Progressive Feature Alignment Network (PFAN), which was proposed for **closed-set unsupervised domain adaptation** rather than domain generalization. PFAN combines an **Easy-to-Hard Transfer Strategy (EHTS)**, **Adaptive Prototype Alignment (APA)**, and temperature-scaled source classification, thereby introducing **progressive target inclusion** and **prototype-based class alignment** under pseudo-label noise. However, its reweighting is largely **implicit** through binary target selection weights \(w_j\), a curriculum threshold \(\tau(m)\), and similarity-adaptive prototype update weights \(\rho_t^2\), rather than the explicit pixel-wise reliability weighting used in PPAR [1811.08585].

Two later papers clarify the specificity of PPAR’s design by contrast. **ProtoConNet** is relevant to the “prototypical alignment” aspect because it includes a Prototypical Alignment module for open-set few-shot classification, but it does not present a literal **progressive alignment and reweighting** formulation; its closest weighting mechanism is attention-like fusion of contextual prototypes, not reliability-aware prototype reweighting [2507.11845]. **OneEncoder** is highly relevant to the “progressive alignment” phrase because it incrementally aligns new modalities into a frozen shared space, yet it contains neither class prototypes nor prototype-guided reweighting in the PPAR sense [2409.11059].

This suggests that PPAR occupies a distinct position: it is neither a pseudo-label-driven UDA method nor a generic progressive alignment framework. It is specifically a **CLIP-assisted domain generalization method for semantic segmentation** in which text prototypes are treated as generalized class anchors, alignment is stratified by representation depth, and reliability estimation is built directly into the supervised segmentation loss. Its stated limitations follow from the same design choices: the method assumes that CLIP text embeddings are meaningful approximations to generalized class centroids, that representative source color and texture can be summarized by mode statistics, and that the task involves unseen domains with **shared classes**, not unseen classes [2507.11955].

Source: https://www.emergentmind.com/topics/prototypical-progressive-alignment-and-reweighting-ppar