Pixel-level Semantic Sampling Alignment

• The paper introduces PSA by enforcing pixel-wise alignment with class-balanced sampling and contrastive losses to mitigate class imbalance and granularity gaps.
• It demonstrates significant mIoU improvements in remote sensing and open-vocabulary segmentation by targeting minority classes and refining pixel-text correspondence.
• PSA leverages modular loss functions with backbones like ResNet-50 and CLIP, proving robust across diverse multimodal and fine-grained segmentation tasks.

Pixel-level Semantic sampling Alignment (PSA) is a class of strategies for pixel-wise semantic feature alignment in dense prediction tasks, primarily semantic segmentation. PSA techniques address limitations of global or region-based alignment by enforcing pixel-level correspondence between modalities or between image and text descriptions. Recent instantiations have been motivated by challenges endemic to remote sensing with missing modalities (Wang et al., 24 Jan 2026) and open-vocabulary semantic segmentation with weak text supervision (Liu et al., 2024). PSA integrates class-balanced sampling with fine-grained, contrastive or alignment-based objective functions, explicitly focusing optimization on challenging category instances and mitigating class imbalance or granularity gaps.

1. Motivations and Fundamental Challenges

Pixel-level Semantic sampling Alignment strategies were developed to overcome two central challenges in semantic segmentation:

• Class Imbalance in Multimodal Settings: In remote sensing, majority-class labels (e.g., water, forest) dominate pixel distributions, overwhelming global alignment losses. Minority-classes (e.g., buildings, vehicles) are underrepresented, leading to poor cross-modal alignment and degraded Intersection-over-Union (IoU), especially under missing-modality conditions at inference time (Wang et al., 24 Jan 2026).
• Alignment Granularity Gap in Open-Vocabulary Segmentation: Methods based on global or region-level image-text alignment suffer from misalignment at the pixel-level, especially when dense pixelwise labels are not available. This results in inferior zero-shot segmentation performance, as coarse-aligned models oversmooth or fragment fine-grained semantics (Liu et al., 2024).

PSA directly addresses these challenges by enforcing class-balanced, pixel-level alignment via explicit sampling and loss construction, enabling robust minority-class recognition and precise cross-domain correspondence.

2. Mathematical Formulation and Loss Construction

Two prominent PSA instantiations have been proposed: cross-modality pixel alignment for remote sensing (Wang et al., 24 Jan 2026) and cross-modal pixel-text alignment for open-vocabulary segmentation (Liu et al., 2024). Both involve two core steps: pixel sampling and pixelwise contrastive objective.

Cross-Modality Alignment (STARS)

Given two modalities (e.g., optical and SAR), a batch ground-truth label $Y$, and a defined class set $C_\mathrm{valid}$, the procedure is as follows:

1. Class-Balanced Sampling: For each class $c\in C_\mathrm{valid}$, $N$ pixel coordinates are sampled (with replacement if necessary) to construct the index set $I$ such that $|I| = N \cdot |C_\mathrm{valid}|$.
2. Feature Extraction and Normalization: For $i\in I$, extract corresponding feature vectors $f_1^i$, $f_2^i$ from two modalities and apply L2 normalization.
3. Similarity Computation: Cosine similarity $$\operatorname{sim}(u, v) = \frac{u \cdot v}{\|u\|_2 \|v\|_2}$$ is computed for each pair among sampled pixels.
4. Alignment Loss:

$$L_{psc} = -\sum_{i \in I} \log \left( \frac{ \sum_{j \in P(i)} \exp( \operatorname{sim}(\tilde f_1^i, \tilde f_2^j)/\tau ) }{ \sum_{k \in I} \exp( \operatorname{sim}(\tilde f_1^i, \tilde f_2^k)/\tau ) } \right),$$

where $\tau$ is a temperature hyperparameter and $P(i)$ are positive (same-class) indices.

Cross-Modal (Pixel-Text) Alignment (MGCA)

For each image-text pair:

• Pixel-Text Similarity: $S_{ij}^p = V_i^p \cdot T_j$, where $V_i^p$ is the $p$-th pixel embedding and $T_j$ the text embedding.
• Pixel-level Positives/Negatives:
  • Positives: Semi-hardest pixels within the pseudo-object mask (top $k_{pix}$ fraction by $S_{ii}^p$).
  • Negatives: For each unpaired text, the pixel with maximal $S_{ij}^p$.
• Symmetric Contrastive Loss:

$$L^{pix} = -\frac{1}{B} \sum_{i=1}^B \left[ \log \frac{\exp(S_{ii}^+/\tau)}{\exp(S_{ii}^+/\tau) + \sum_{j\neq i} \exp(S_{ij}^-/\tau)} + \log \frac{\exp(S_{ii}^+/\tau)}{\exp(S_{ii}^+/\tau) + \sum_{j\neq i} \exp(S_{ji}^-/\tau)} \right].$$

All objectives use temperature scaling, with semi-hard sampling targeting informative pixel pairs (Liu et al., 2024).

3. Class-Balanced Pixel Sampling

A signature component of PSA is the class-balanced pixel sampling scheme designed to equalize representation of all semantic classes in the loss calculation, thereby preventing dominance by majority classes and ensuring minority-class features drive the alignment process.

For each training batch (Wang et al., 24 Jan 2026):

1. Identify Valid Classes: $C_\mathrm{valid} = \mathrm{unique}(Y)$, possibly excluding “background”/“ignore.”
2. Index Collection and Sampling: For each $c\in C_\mathrm{valid}$, collect all pixel positions $I_c$ with $Y_b(x,y) = c$.
3. Sampling per Class: If $|I_c| \geq N$, sample $N$ unique indices without replacement. Otherwise, sample with replacement.
4. Index Set Assembly: Concatenate to yield $I = \cup_c S_c$, with uniform per-class sample counts.

This process ensures balanced class contributions to the loss and stabilizes training, especially when minority classes contain few labeled pixels. Empirical ablations demonstrate pronounced impact on both overall and per-class performance (Wang et al., 24 Jan 2026).

4. Practical Implementation Details

Key architectural and training configurations for PSA include:

• Feature Backbone: STARS employs shared/specific ResNet-50 encoders up to stage-4 (Wang et al., 24 Jan 2026). MGCA leverages a frozen CLIP ViT-B/16 visual encoder and CLIP text transformer, with a lightweight gated-conv decoder (Liu et al., 2024).
• Hyperparameters:
  • Number of samples per class ($N$): 32 (optimal; performance degrades for $N < 32$ or $N > 32$).
  • Temperature ($\tau$): STARS uses 0.2, with sensitivity tested in [0.1, 0.5].
  • Loss weight ($\alpha$): 0.5 found optimal for balancing pixel-alignment influence.
  • Batch size: 8 for remote sensing; 1024 for text-supervised segmentation.
• Optimization Strategy: Adam optimizer with weight decay and cosine learning-rate schedules. Gradient clipping is applied for stability.

The PSA loss is modular and can be combined with other objectives, such as bidirectional neural contrastive similarity loss (NCS) in STARS (Wang et al., 24 Jan 2026) or object- and region-level contrastive losses in MGCA (Liu et al., 2024).

5. Empirical Evaluation and Ablations

Across both multimodal remote sensing and open-vocabulary segmentation, PSA demonstrates substantial improvements:

• STARS (Remote Sensing, Missing-Modality): Incorporation of PSA yields an mIoU gain of +9.34% on EarthMiss and +4.77% on WHU-OPT-SAR datasets relative to the baseline SAR-only model. Performance peaks at $N=32$ samples per class and $\alpha = 0.5$ loss weight; deviation leads to reduced accuracy. Combined with bidirectional NCS loss, PSA delivers maximum gains in both overall and minority-class IoU (Wang et al., 24 Jan 2026).

Variant	EarthMiss ΔmIoU	WHU-OPT-SAR ΔmIoU
Baseline-SAR	–	–
+NCS ($L_{ncs}$)	+7.26	+3.40
+NCS + PSA ($L_{psc}$)	+9.34	+4.77
Full STARS (w/ trans)	+9.60	+5.95

• MGCA (Text-Supervised Segmentation): PSA enables direct optimization at pixel granularity by mining semi-hard positives and hardest negatives, bridging the region-to-pixel alignment gap. Empirical results indicate the approach yields state-of-the-art zero-shot segmentation on CC3M, confirmed by comprehensive comparisons and qualitative analysis (Liu et al., 2024).

6. Relation to Broader Research and Applications

PSA represents a methodological advance in both the remote sensing and open-vocabulary segmentation domains, serving two broad roles:

• In remote sensing, PSA augments multimodal fusion strategies to handle missing modalities, addressing feature collapse and underrepresented object categories (Wang et al., 24 Jan 2026).
• In vision–language pretraining, PSA fills the granularity gap between region/object and pixel representations, enabling dense alignment critical for fine-grained downstream tasks (Liu et al., 2024).

A plausible implication is that PSA can generalize to any scenario with label imbalance and multi-level correspondence structure, including medical imaging, autonomous driving, and large-scale video–text alignment.

7. Limitations and Practical Considerations

Although PSA mitigates class imbalance and granularity gaps, its effectiveness depends on several design choices:

• The sampling quota $N$ and loss weight $\alpha$ require empirical tuning for best results.
• While symmetrized formulations exist (anchoring both modalities), empirical results suggest one-way alignment suffices when paired with complementary objectives (Wang et al., 24 Jan 2026).
• For extremely rare classes or extremely weak supervision, sampling with replacement may introduce variance; however, empirical results demonstrate improved minority class recognition under realistic conditions.

Overall, Pixel-level Semantic sampling Alignment (PSA) constitutes a robust, theoretically grounded, and empirically validated approach for class-balanced, pixel-precise semantic alignment across modalities and tasks (Wang et al., 24 Jan 2026, Liu et al., 2024).