---
title: 'DEARLi: Decoupled Panoptic Segmentation'
url: https://www.emergentmind.com/topics/dearli
type: topic
---

# DEARLi: Decoupled Panoptic Segmentation

DEARLi, short for **Decoupled Enhancement of Recognition and Localization for Semi-supervised Panoptic Segmentation**, is a semi-supervised panoptic segmentation framework that treats **recognition** and **localization** as distinct subproblems and improves each with a different foundation model [2507.10118]. It is designed for settings with a small labeled set,
\[
\mathcal{X}^l=\left\{\left(x_i^l, y^l_i\right)\right\}_{i=1}^{N_l},
\]
and a much larger unlabeled set,
\[
\mathcal{X}^u=\left\{x_i^u\right\}_{i=1}^{N_u},
\]
with \(N_l \ll N_u\). The method is motivated by large-taxonomy, low-label regimes such as ADE20K, where many classes are rare and standard panoptic models generalize poorly from few labeled examples. Its central design choice is selective use of foundation models: **CLIP** is used only for recognition, and **SAM** is used only for localization [2507.10118].

## 1. Problem setting and formal basis

DEARLi studies **semi-supervised panoptic segmentation**, where the objective is to exploit both labeled and unlabeled images to improve segmentation quality [2507.10118]. The paper frames this as especially difficult in **large-taxonomy, low-label regimes**, because recognition and localization degrade for different reasons: recognition suffers from label scarcity and long-tail class frequency, while localization suffers from insufficient supervision for precise mask boundaries.

The model is built on a **mask transformer** backbone, specifically **Mask2Former-style set prediction** [2507.10118]. The mask transformer predicts:

- mask-wide class logits \(\mathbf{P} \in \mathbb{R}^{N \times (C+1)}\),
- pixel-level localization maps \(\boldsymbol{\sigma} \in \mathbb{R}^{N \times H \times W}\),

where the extra \((C+1)\)-th class is “no object.” Training uses bipartite matching \(\mathcal M\) and the compound loss
\[
\mathcal{L} = \sum_i^N \mathcal{L}_{cls}(\mathbf{P}_i, y^{GT}_{\mathcal{M}(i)}) + \hspace{-1.5em} \sum_{ y_{\mathcal{M}(i)}^{GT} \neq C + 1}{ \hspace{-1.5em} \mathcal{L}_{loc}(\boldsymbol{\sigma}_i, \boldsymbol{\sigma}^{GT}_{\mathcal{M}(i)})}.
\]
Here \(\mathcal{L}_{cls}\) is cross-entropy over segments, and \(\mathcal{L}_{loc}\) is the usual dice plus binary cross-entropy localization loss [2507.10118].

This formulation makes the decoupling explicit. Classification errors are associated primarily with \(\mathcal{L}_{cls}\), whereas boundary and mask-quality errors are associated with \(\mathcal{L}_{loc}\). A plausible implication is that the framework can insert external priors into each term without forcing a single foundation model to solve the entire panoptic problem.

## 2. Decoupling recognition and localization

The defining idea of DEARLi is that **foundation models are most useful when their strengths are isolated and targeted** [2507.10118]. The paper argues that **CLIP** contributes large-scale semantic priors and zero-shot class knowledge, whereas **SAM** contributes strong class-agnostic region proposal and boundary quality. Rather than merging them into a single dense prediction pipeline, DEARLi assigns each model a restricted role.

For **recognition**, the method uses CLIP to improve class selection for mask candidates. For **localization**, it uses SAM pseudo-labels to warm up the decoder in a class-agnostic fashion [2507.10118]. The decoupling is therefore not only conceptual but architectural: recognition enhancement enters through classification and pseudo-label re-ranking, while localization enhancement enters through decoder pretraining on masks.

This separation also addresses a common misconception about the framework. DEARLi does **not** use CLIP to predict pixels directly, and it does **not** use SAM to solve category labeling [2507.10118]. The paper explicitly notes that a naive strategy of classifying SAM masks with CLIP yields only **8.8 PQ on ADE20K**, attributing this to a granularity mismatch between SAM’s regions and the dataset taxonomy [2507.10118]. That negative result is integral to the method’s design rationale.

## 3. Recognition enhancement with CLIP

To improve recognition, DEARLi freezes a **ConvNeXt-CLIP** backbone pretrained on **LAION-2B** [2507.10118]. According to the paper, freezing the backbone has two immediate effects: it preserves zero-shot alignment, and it reduces memory and computation because gradients through the backbone are unnecessary.

The final classifier is tied to text embeddings from the CLIP text encoder, producing the **M2F-Lang** variant [2507.10118]. For a mask candidate, dense CLIP features \(\mathbf{F}\in \mathbb{R}^{H' \times W' \times D}\) are pooled with the predicted mask using
\[
\mathbf{e}_{v_i}=\mathcal{MP}\left(\mathbf{F}, \mathbf{M}_i\right) = \frac{\sum_{r,c}^{HW}\mathbf{F}[r,c,:]\cdot \mathbf{M}_i[r,c]}{\sum_{r,c}^{HW} \mathbf{M}_i[{r,c}]}.
\]
The resulting mask-level visual embedding is compared with CLIP text embeddings \(\{\mathbf{e}_{t_j}\}_{j=1}^C\) to obtain zero-shot class probabilities [2507.10118].

For unlabeled data, the teacher first predicts mask candidates and mask-transformer class probabilities \(\mathbf{P}_{MT}\). CLIP zero-shot probabilities \(\mathbf{P}_{CLIP}\) are then computed for the same masks, and the final posterior is a **weighted geometric mean**
\[
\mathbf{P}_{\mathrm{ENS}} = \left(\mathbf{P}_{\mathrm{MT}}\right)^\alpha \odot \left(\mathbf{P}_\mathrm{CLIP}\right)^{1-\alpha},
\]
with \(\alpha=0.6\) in all experiments [2507.10118]. These ensembled probabilities are converted into hard pseudo-labels through standard panoptic inference and used in the unlabeled consistency loss.

The paper reports that **CLIP posterior ensembling matters**: using \(\mathbf{P}_{CLIP}\) alone improves over decoder-only teacher predictions, but the combined ensemble \(\mathbf{P}_{ENS}\) is best [2507.10118]. It also reports that ensembling is more valuable when used for pseudo-label generation than only at inference time. This supports the paper’s claim that CLIP is most effective as a **mask-level re-ranking mechanism**, not as a dense segmenter.

## 4. Localization enhancement with SAM

DEARLi addresses localization by introducing **decoder warm-up with SAM**, abbreviated **DeWa** in the paper’s pipeline description [2507.10118]. SAM generates class-agnostic pseudo-label masks \(\{\mathbf{M}_i\}\), and the decoder is first trained on both labeled and unlabeled images using these masks while optimizing only the localization term
\[
\mathcal{L}_{loc}.
\]

The warm-up stage is explicitly **class-agnostic** [2507.10118]. Its purpose is to teach the decoder objectness and boundary alignment before classification learning. The paper contrasts this with the failed alternative of directly classifying SAM masks, reinforcing the view that SAM should be used only for what it is best at: region proposal and boundary quality.

Ablation results indicate that **SAM warm-up matters** [2507.10118]. Random decoder initialization is weaker than SAM warm-up, and supervised decoder pretraining can even hurt, which the paper interprets as suggesting overfitting. The reported gain from SAM-based class-agnostic warm-up is about **1.4–2.7 PQ points depending on split**, with larger benefits in the hardest low-label regimes [2507.10118].

This stage is central to the “localization” half of the framework’s title. It does not attempt to inject semantics into SAM outputs; rather, it uses SAM as a source of class-agnostic spatial supervision. A plausible implication is that DEARLi’s performance depends not on SAM’s compatibility with the target taxonomy, but on the transferability of its boundary and objectness priors.

## 5. Three-stage pipeline and semi-supervised training

DEARLi uses a **three-stage pipeline** [2507.10118].

| Stage | Function | Key specification |
|---|---|---|
| 1 | CLIP pretraining | Frozen OpenCLIP ConvNeXt-B backbone pretrained on LAION-2B |
| 2 | Decoder warm-up with SAM (DeWa) | Class-agnostic pseudo-labels from SAM; optimize only \(\mathcal{L}_{loc}\) |
| 3 | Semi-supervised training with Mean Teacher | Supervised loss on labeled data and consistency loss on unlabeled data |

In the final stage, the student is trained on labeled data with supervised loss and on unlabeled data with consistency loss [2507.10118]. The unlabeled prediction is produced on a strongly augmented image,
\[
o_i^u=h_{\theta_{stud} \left(\mathcal{S}(x_i^u)\right),
\]
while the pseudo-label is produced by the teacher on a weakly augmented image,
\[
\hat{y}^u_i=h_{\theta_{teach} \left(\mathcal{W}(x_i^u)\right).
\]
Teacher parameters are updated by exponential moving average,
\[
\theta_{teach}^{t} = \gamma \theta_{teach}^{t-1} + \left(1-\gamma\right) \theta_{stud}^{t-1},
\]
with \(\gamma=0.999\) in practice [2507.10118].

The pseudo-label quality in this Mean Teacher setup is improved by the CLIP–M2F posterior ensemble described above. The paper notes that inference-time ensembling can also be applied, though it helps less than using the ensemble during pseudo-label generation [2507.10118]. This suggests that DEARLi’s main gains arise during training-time supervision refinement rather than only from test-time aggregation.

## 6. Empirical behavior, efficiency, and evaluation

The headline ADE20K result reported for the most difficult regime uses only **158 labeled images** [2507.10118]. Under that setting, the paper reports:

- **DEAR:** **27.7 PQ**
- **DEARLi:** **29.9 PQ**
- **DEARLi semantic inference:** **38.9 mIoU**

The same work states that DEAR and DEARLi outperform prior semi-supervised semantic segmentation methods on ADE20K, with **DEAR** at **36.5 mIoU** and **DEARLi** at **38.9 mIoU**, despite being trained only for the **panoptic objective** [2507.10118]. On additional benchmarks, the paper reports **38.8 PQ** on **COCO-Panoptic** with **232 labeled images**, and **54.6 mIoU** on **COCO-Objects** at **1/512** [2507.10118].

The paper emphasizes that the method is particularly effective in **few-label, many-class** settings [2507.10118]. It reports that the largest gains occur on **underrepresented classes**, which is consistent with the claim that CLIP’s semantics compensate for label scarcity. It also reports that **ConvNeXt-L** improves over **ConvNeXt-B**, and that larger frozen backbones can approach gains comparable to doubling model capacity [2507.10118].

A notable practical claim is computational efficiency. The paper states that DEAR achieves these gains with **8× less GPU memory** than competing semi-supervised semantic segmentation approaches [2507.10118]. The stated reasons are architectural: the backbone is **frozen**, the **ConvNeXt CLIP features are reused**, and no expensive fine-tuning through the backbone is needed. It further notes that backbone fine-tuning roughly doubles memory and can hurt low-label performance by causing vision-language misalignment, and that experiments with frozen backbones are runnable on a **single A100 40GB GPU** [2507.10118].

## 7. Scope, interpretation, and related acronym usage

DEARLi’s contribution is specific to **semi-supervised panoptic segmentation** and should not be conflated with other methods that use similar acronyms. The acronym **DEAR** also appears in unrelated work on **Disentangled Environment and Agent Representations** for visual reinforcement learning [2407.00633] and **Depth-Enhanced Action Recognition** [2408.15679]. DEARLi is distinct from both; its subject matter is panoptic segmentation under label scarcity, and its design centers on decoupling **recognition** from **localization** [2507.10118].

Within its own domain, the most important conceptual point is that the method does **not** advocate indiscriminate insertion of foundation models into the segmentation stack. Instead, it argues for selective deployment: CLIP for semantics, SAM for boundaries [2507.10118]. This suggests a broader methodological lesson for semi-supervised dense prediction: when foundation models encode different priors, the effective strategy may be to map those priors onto the subproblems they solve best, rather than to force a single fused model to handle recognition, localization, and pseudo-labeling uniformly.

In that sense, DEARLi can be understood as a framework for **task-aligned decomposition** in semi-supervised panoptic segmentation. Its reported behavior in ADE20K and COCO, especially in large-taxonomy low-label settings, indicates that the separation of recognition and localization is not merely interpretive but operationally consequential [2507.10118].

Source: https://www.emergentmind.com/topics/dearli