DEARLi: Decoupled Panoptic Segmentation

• DEARLi is a semi-supervised panoptic segmentation framework that decouples recognition and localization to handle few-label, many-class settings.
• It leverages a frozen ConvNeXt-CLIP backbone for semantic recognition and SAM for class-agnostic localization, enhancing pseudo-labeling quality.
• The three-stage pipeline (CLIP pretraining, SAM decoder warm-up, and Mean Teacher training) achieves higher PQ and mIoU in low-label regimes.

DEARLi, short for Decoupled Enhancement of Recognition and Localization for Semi-supervised Panoptic Segmentation, is a semi-supervised panoptic segmentation framework that treats recognition and localization as distinct subproblems and improves each with a different foundation model (Martinović et al., 14 Jul 2025). It is designed for settings with a small labeled set,

$$\mathcal{X}^l=\left\{\left(x_i^l, y^l_i\right)\right\}_{i=1}^{N_l},$$

and a much larger unlabeled set,

$\mathcal{X}^u=\left\{x_i^u\right\}_{i=1}^{N_u},$

with $N_l \ll N_u$. The method is motivated by large-taxonomy, low-label regimes such as ADE20K, where many classes are rare and standard panoptic models generalize poorly from few labeled examples. Its central design choice is selective use of foundation models: CLIP is used only for recognition, and SAM is used only for localization (Martinović et al., 14 Jul 2025).

1. Problem setting and formal basis

DEARLi studies semi-supervised panoptic segmentation, where the objective is to exploit both labeled and unlabeled images to improve segmentation quality (Martinović et al., 14 Jul 2025). The paper frames this as especially difficult in large-taxonomy, low-label regimes, because recognition and localization degrade for different reasons: recognition suffers from label scarcity and long-tail class frequency, while localization suffers from insufficient supervision for precise mask boundaries.

The model is built on a mask transformer backbone, specifically Mask2Former-style set prediction (Martinović et al., 14 Jul 2025). The mask transformer predicts:

• mask-wide class logits $\mathbf{P} \in \mathbb{R}^{N \times (C+1)}$,
• pixel-level localization maps $$\boldsymbol{\sigma} \in \mathbb{R}^{N \times H \times W}$$,

where the extra $(C+1)$-th class is “no object.” Training uses bipartite matching $\mathcal M$ and the compound loss

$$\mathcal{L} = \sum_i^N \mathcal{L}_{cls}(\mathbf{P}_i, y^{GT}_{\mathcal{M}(i)}) + \hspace{-1.5em} \sum_{ y_{\mathcal{M}(i)}^{GT} \neq C + 1}{ \hspace{-1.5em} \mathcal{L}_{loc}(\boldsymbol{\sigma}_i, \boldsymbol{\sigma}^{GT}_{\mathcal{M}(i)})}.$$

Here $\mathcal{L}_{cls}$ is cross-entropy over segments, and $\mathcal{L}_{loc}$ is the usual dice plus binary cross-entropy localization loss (Martinović et al., 14 Jul 2025).

This formulation makes the decoupling explicit. Classification errors are associated primarily with $\mathcal{X}^u=\left\{x_i^u\right\}_{i=1}^{N_u},$0, whereas boundary and mask-quality errors are associated with $\mathcal{X}^u=\left\{x_i^u\right\}_{i=1}^{N_u},$1. A plausible implication is that the framework can insert external priors into each term without forcing a single foundation model to solve the entire panoptic problem.

2. Decoupling recognition and localization

The defining idea of DEARLi is that foundation models are most useful when their strengths are isolated and targeted (Martinović et al., 14 Jul 2025). The paper argues that CLIP contributes large-scale semantic priors and zero-shot class knowledge, whereas SAM contributes strong class-agnostic region proposal and boundary quality. Rather than merging them into a single dense prediction pipeline, DEARLi assigns each model a restricted role.

For recognition, the method uses CLIP to improve class selection for mask candidates. For localization, it uses SAM pseudo-labels to warm up the decoder in a class-agnostic fashion (Martinović et al., 14 Jul 2025). The decoupling is therefore not only conceptual but architectural: recognition enhancement enters through classification and pseudo-label re-ranking, while localization enhancement enters through decoder pretraining on masks.

This separation also addresses a common misconception about the framework. DEARLi does not use CLIP to predict pixels directly, and it does not use SAM to solve category labeling (Martinović et al., 14 Jul 2025). The paper explicitly notes that a naive strategy of classifying SAM masks with CLIP yields only 8.8 PQ on ADE20K, attributing this to a granularity mismatch between SAM’s regions and the dataset taxonomy (Martinović et al., 14 Jul 2025). That negative result is integral to the method’s design rationale.

3. Recognition enhancement with CLIP

To improve recognition, DEARLi freezes a ConvNeXt-CLIP backbone pretrained on LAION-2B (Martinović et al., 14 Jul 2025). According to the paper, freezing the backbone has two immediate effects: it preserves zero-shot alignment, and it reduces memory and computation because gradients through the backbone are unnecessary.

The final classifier is tied to text embeddings from the CLIP text encoder, producing the M2F-Lang variant (Martinović et al., 14 Jul 2025). For a mask candidate, dense CLIP features $\mathcal{X}^u=\left\{x_i^u\right\}_{i=1}^{N_u},$2 are pooled with the predicted mask using

$\mathcal{X}^u=\left\{x_i^u\right\}_{i=1}^{N_u},$3

The resulting mask-level visual embedding is compared with CLIP text embeddings $\mathcal{X}^u=\left\{x_i^u\right\}_{i=1}^{N_u},$4 to obtain zero-shot class probabilities (Martinović et al., 14 Jul 2025).

For unlabeled data, the teacher first predicts mask candidates and mask-transformer class probabilities $\mathcal{X}^u=\left\{x_i^u\right\}_{i=1}^{N_u},$5. CLIP zero-shot probabilities $\mathcal{X}^u=\left\{x_i^u\right\}_{i=1}^{N_u},$6 are then computed for the same masks, and the final posterior is a weighted geometric mean

$\mathcal{X}^u=\left\{x_i^u\right\}_{i=1}^{N_u},$7

with $\mathcal{X}^u=\left\{x_i^u\right\}_{i=1}^{N_u},$8 in all experiments (Martinović et al., 14 Jul 2025). These ensembled probabilities are converted into hard pseudo-labels through standard panoptic inference and used in the unlabeled consistency loss.

The paper reports that CLIP posterior ensembling matters: using $\mathcal{X}^u=\left\{x_i^u\right\}_{i=1}^{N_u},$9 alone improves over decoder-only teacher predictions, but the combined ensemble $N_l \ll N_u$0 is best (Martinović et al., 14 Jul 2025). It also reports that ensembling is more valuable when used for pseudo-label generation than only at inference time. This supports the paper’s claim that CLIP is most effective as a mask-level re-ranking mechanism, not as a dense segmenter.

4. Localization enhancement with SAM

DEARLi addresses localization by introducing decoder warm-up with SAM, abbreviated DeWa in the paper’s pipeline description (Martinović et al., 14 Jul 2025). SAM generates class-agnostic pseudo-label masks $N_l \ll N_u$1, and the decoder is first trained on both labeled and unlabeled images using these masks while optimizing only the localization term

$N_l \ll N_u$2

The warm-up stage is explicitly class-agnostic (Martinović et al., 14 Jul 2025). Its purpose is to teach the decoder objectness and boundary alignment before classification learning. The paper contrasts this with the failed alternative of directly classifying SAM masks, reinforcing the view that SAM should be used only for what it is best at: region proposal and boundary quality.

Ablation results indicate that SAM warm-up matters (Martinović et al., 14 Jul 2025). Random decoder initialization is weaker than SAM warm-up, and supervised decoder pretraining can even hurt, which the paper interprets as suggesting overfitting. The reported gain from SAM-based class-agnostic warm-up is about 1.4–2.7 PQ points depending on split, with larger benefits in the hardest low-label regimes (Martinović et al., 14 Jul 2025).

This stage is central to the “localization” half of the framework’s title. It does not attempt to inject semantics into SAM outputs; rather, it uses SAM as a source of class-agnostic spatial supervision. A plausible implication is that DEARLi’s performance depends not on SAM’s compatibility with the target taxonomy, but on the transferability of its boundary and objectness priors.

5. Three-stage pipeline and semi-supervised training

DEARLi uses a three-stage pipeline (Martinović et al., 14 Jul 2025).

Stage	Function	Key specification
1	CLIP pretraining	Frozen OpenCLIP ConvNeXt-B backbone pretrained on LAION-2B
2	Decoder warm-up with SAM (DeWa)	Class-agnostic pseudo-labels from SAM; optimize only $N_l \ll N_u$3
3	Semi-supervised training with Mean Teacher	Supervised loss on labeled data and consistency loss on unlabeled data

In the final stage, the student is trained on labeled data with supervised loss and on unlabeled data with consistency loss (Martinović et al., 14 Jul 2025). The unlabeled prediction is produced on a strongly augmented image,

$N_l \ll N_u$4

while the pseudo-label is produced by the teacher on a weakly augmented image,

$N_l \ll N_u$5

Teacher parameters are updated by exponential moving average,

$N_l \ll N_u$6

with $N_l \ll N_u$7 in practice (Martinović et al., 14 Jul 2025).

The pseudo-label quality in this Mean Teacher setup is improved by the CLIP–M2F posterior ensemble described above. The paper notes that inference-time ensembling can also be applied, though it helps less than using the ensemble during pseudo-label generation (Martinović et al., 14 Jul 2025). This suggests that DEARLi’s main gains arise during training-time supervision refinement rather than only from test-time aggregation.

6. Empirical behavior, efficiency, and evaluation

The headline ADE20K result reported for the most difficult regime uses only 158 labeled images (Martinović et al., 14 Jul 2025). Under that setting, the paper reports:

• DEAR: 27.7 PQ
• DEARLi: 29.9 PQ
• DEARLi semantic inference: 38.9 mIoU

The same work states that DEAR and DEARLi outperform prior semi-supervised semantic segmentation methods on ADE20K, with DEAR at 36.5 mIoU and DEARLi at 38.9 mIoU, despite being trained only for the panoptic objective (Martinović et al., 14 Jul 2025). On additional benchmarks, the paper reports 38.8 PQ on COCO-Panoptic with 232 labeled images, and 54.6 mIoU on COCO-Objects at 1/512 (Martinović et al., 14 Jul 2025).

The paper emphasizes that the method is particularly effective in few-label, many-class settings (Martinović et al., 14 Jul 2025). It reports that the largest gains occur on underrepresented classes, which is consistent with the claim that CLIP’s semantics compensate for label scarcity. It also reports that ConvNeXt-L improves over ConvNeXt-B, and that larger frozen backbones can approach gains comparable to doubling model capacity (Martinović et al., 14 Jul 2025).

A notable practical claim is computational efficiency. The paper states that DEAR achieves these gains with 8× less GPU memory than competing semi-supervised semantic segmentation approaches (Martinović et al., 14 Jul 2025). The stated reasons are architectural: the backbone is frozen, the ConvNeXt CLIP features are reused, and no expensive fine-tuning through the backbone is needed. It further notes that backbone fine-tuning roughly doubles memory and can hurt low-label performance by causing vision-language misalignment, and that experiments with frozen backbones are runnable on a single A100 40GB GPU (Martinović et al., 14 Jul 2025).

7. Scope, interpretation, and related acronym usage

DEARLi’s contribution is specific to semi-supervised panoptic segmentation and should not be conflated with other methods that use similar acronyms. The acronym DEAR also appears in unrelated work on Disentangled Environment and Agent Representations for visual reinforcement learning (Pore et al., 2024) and Depth-Enhanced Action Recognition (Rahmaniboldaji et al., 2024). DEARLi is distinct from both; its subject matter is panoptic segmentation under label scarcity, and its design centers on decoupling recognition from localization (Martinović et al., 14 Jul 2025).

Within its own domain, the most important conceptual point is that the method does not advocate indiscriminate insertion of foundation models into the segmentation stack. Instead, it argues for selective deployment: CLIP for semantics, SAM for boundaries (Martinović et al., 14 Jul 2025). This suggests a broader methodological lesson for semi-supervised dense prediction: when foundation models encode different priors, the effective strategy may be to map those priors onto the subproblems they solve best, rather than to force a single fused model to handle recognition, localization, and pseudo-labeling uniformly.

In that sense, DEARLi can be understood as a framework for task-aligned decomposition in semi-supervised panoptic segmentation. Its reported behavior in ADE20K and COCO, especially in large-taxonomy low-label settings, indicates that the separation of recognition and localization is not merely interpretive but operationally consequential (Martinović et al., 14 Jul 2025).