Unsupervised Collaborative Domain Adaptation for Driving Scene Parsing

Abstract: Reliable driving scene parsing is a fundamental capability for autonomous vehicles operating in open and dynamic driving environments. However, adapting perception models to new deployment domains remains challenging because pixel-level annotations are expensive to obtain, while source-domain data are often inaccessible due to privacy, security, or ownership constraints. Existing source-free unsupervised domain adaptation methods typically rely on a single pre-trained source model, which makes the adapted perception system vulnerable to source-specific biases and limits its robustness under diverse road layouts, illumination conditions, weather patterns, and traffic conditions. This article presents an unsupervised collaborative domain adaptation (UCDA) framework for driving scene parsing in a source-free setting, which transfers complementary knowledge from multiple pre-trained source models to a unified target model without accessing any original source samples. To compare predictions from independently trained models, UCDA constructs a class-level prototype memory bank and estimates cross-model prediction reliability through prototype similarity, reducing the effect of inconsistent confidence scales across source models. Based on the resulting complementary supervision, UCDA adopts a two-stage transfer strategy: multiple source models are first refined on unlabeled target-domain driving data through collaborative optimization with positive and negative consistency constraints, and their validated expertise is then distilled into a single deployable target model. Comprehensive evaluations on public driving-scene datasets and real-world data collected from an autonomous vehicle platform demonstrate that UCDA effectively consolidates complementary multi-source knowledge, improving target-domain scene parsing reliability and generalization across diverse driving environments.

• The paper introduces unsupervised collaborative domain adaptation (UCDA) which combines knowledge from multiple pre-trained source models to adapt a single deployable target model without accessing any source data, addressing domain-shift issues in driving scene parsing.
• UCDA achieves significant improvements in cross-domain performance, such as a +5.1 mIoU gain on BDD100K 16-class tasks, demonstrating its effectiveness in handling diverse and challenging target datasets.
• The method employs a prototype-aware reliability estimation for scoring predictions reliably and synthesizing complementary labels ensuring positive and negative constraints on pixel-level predictions.

Problem and motivation

Driving scene parsing models degrade under domain shift—unseen cities, weather, illumination, and sensor configurations—and pixel-level annotation for every deployment environment is impractical. Source-free unsupervised domain adaptation (SFUDA) addresses the privacy constraint by adapting pre-trained source models using only unlabeled target data, but existing SFUDA methods rely on a single pre-trained source model. The authors identify two consequences of this single-model design: adapted systems inherit source-specific biases (e.g., overfitting to a particular city layout or weather distribution), and long-tail or safety-critical categories under-represented in the source domain transfer poorly.

Extending to multiple pre-trained source models introduces its own difficulties. Independently trained models produce confidence scores on incompatible scales, so softmax probability or entropy are unreliable cross-model selection criteria. Moreover, when all source models share failure modes on rare conditions, high-confidence-only pseudo-labeling ignores informative structure in ambiguous regions and can reinforce shared errors—a negative transfer dynamic.

The paper proposes unsupervised collaborative domain adaptation (UCDA), which transfers complementary knowledge from $K$ pre-trained source models into one deployable target model without accessing any source samples (2606.01818).

Method

UCDA operates in two stages: collaborative refinement of the source models, followed by multi-model distillation into a unified target model.

Prototype-aware reliability estimation. For each source model, class-level prototypes $\boldsymbol{p}_e^k$ are computed as logit centroids over pixels assigned to class $e$ by an initial argmax pseudo-label map; all prototypes form a per-model prototype matrix, aggregated across models into a memory bank. Pixel-wise reliability is then measured by cosine similarity between the pixel's logit vector and the candidate class prototype:

$$\boldsymbol{R}_c^k(\boldsymbol{x}) = \frac{(\boldsymbol{L}_t^k(\boldsymbol{x}))^\top \boldsymbol{p}_c^k}{\lVert\boldsymbol{L}_t^k(\boldsymbol{x})\rVert_2 \lVert\boldsymbol{p}_c^k\rVert_2}$$

Because cosine similarity is invariant to logit magnitude, it sidesteps calibration mismatches across models—an assumption supported empirically by outperforming entropy- and softmax-based selection (see below).

Complementary supervision synthesis. Positive pseudo-labels (maximum softmax probability) and negative pseudo-labels (minimum probability) are extracted from each source model. Per pixel, the positive prediction is taken from the model with maximum reliability for that label ($u = \arg\max_k R_p^k$), while the negative cue comes from the most reliable unlikely-category signal ($v = \arg\min_k R_n^k$). Both discrete labels and soft logits are retained as supervision.

Decoupled collaborative optimization. Pixels are partitioned by a confidence threshold $\tau$ on the reliability-selected positive logits. High-confidence regions receive masked cross-entropy against synthesized pseudo-labels. Low-confidence regions receive two consistency losses: a positive consistency loss aligning each source model with the integrated soft targets, and a negative consistency loss suppressing classes flagged as unlikely. The negative term is motivated explicitly: since each pixel has many negative candidates but one positive class, negative supervision adds constraints where hard labels would be unreliable.

Knowledge infusion. Refined source models serve as frozen teachers; the student target model is trained with the same decoupled objective, yielding a single model whose inference cost equals that of a standard segmenter.

Experimental results

The evaluation spans synthetic-to-real (GTA5, SYNTHIA, Synscapes) and real-to-real (Mapillary Vistas, Cityscapes → Cityscapes, BDD100K, ACDC) adaptation, using both DeepLab-V2/ResNet-101 and SegFormer-style MiT-B5 architectures.

Key quantitative findings:

Benchmark / setting	Best prior SFUDA	UCDA (T)	Gain
Cityscapes 19-class, ResNet-101	SND, 55.6 mIoU	56.3	+0.7
Cityscapes 16-class, MiT-B5	ATP, 66.6 mIoU	67.3 (72.2 mIoU*)	+0.7
BDD100K 16-class, ResNet-101	SND, 47.8 mIoU	52.9	+5.1
ACDC 19-class, MiT-B5	ATP, 65.6 mIoU	69.3	+3.7

Under ResNet-101 on both Cityscapes protocols, UCDA surpasses not only source-free methods but also source-dependent methods such as CaCo, ARAS, and PBAL despite never touching source data. The largest gains appear on the more heterogeneous target domains: BDD100K (+5.1 mIoU over SND) and ACDC (+3.7 over ATP), supporting the claim that multi-source collaboration pays off most where target diversity exposes single-source bias. On ACDC, UCDA improves safety-relevant categories including road, sidewalk, bus, and train.

Several specific results deserve emphasis:

• Pseudo-label quality: on the Cityscapes 16-class task, prototype-cosine selection reaches 53.7% mIoU versus 51.7% (entropy), 51.8% (softmax), 49.9% (majority vote), and 52.8% (probability averaging). This isolates the reliability mechanism as the source of improvement rather than mere ensembling.
• Ablations: removing either consistency loss degrades performance—for example, dropping the negative consistency loss reduces knowledge-infusion mIoU from 54.6% to 51.6% in the two-source configuration, confirming that negative cues contribute beyond standard self-training.
• Source scaling: performance increases monotonically with the number of sources, from 37.2% mIoU (single SYNTHIA model) to 66.1% (five mixed synthetic/real sources), indicating robustness to heterogeneous source configurations.
• Generalization: adapted only on Cityscapes, the UCDA model achieves the best average mIoU (48.5%) across Cityscapes, BDD100K, and ACDC, versus 41.8% for IAPC—the strongest cross-target result reported.
• Deployment: on data collected from a NIO ET9 platform (33 sensors, eight-megapixel cameras), UCDA attains 55.8% mIoU versus 52.6% for SND at identical parameters (43.9M), FLOPs, FPS (~51), and latency (~19.5 ms); the accuracy gain carries no inference overhead because all methods share the same target architecture.

Limitations and open questions

The authors concede several constraints. Effectiveness depends on the quality and diversity of available source models—weak or heavily biased teachers provide little complementary value. Prototype-based reliability estimation can still be fooled: an incorrect prediction that aligns strongly with an incorrect class prototype may be selected, propagating errors into collaborative optimization. This manifests concretely as reduced IoU on some dominant categories (e.g., road on the 19-class Cityscapes protocol relative to the best individual model), reflecting residual difficulty in preserving all source strengths during collaboration. Training-time cost grows linearly with the number of source models, even though inference cost does not. An open question left by the paper is whether reliability estimation can be made robust to systematically misaligned prototypes without target annotations, and whether training cost can be amortized for larger source pools.

Conclusion

This paper extends SFUDA for driving scene parsing from single-source to multi-source settings via three mechanisms: prototype-similarity-based cross-model reliability estimation, decoupled positive/negative consistency optimization of uncertain regions, and two-stage refine-then-distill knowledge transfer. The empirical case is strongest where single-source methods fail most—heterogeneous targets like BDD100K and adverse-condition ACDC—and includes validation on proprietary vehicle-platform data at unchanged deployment cost. The framework's applicability to other privacy-constrained dense-prediction domains, and its behavior when source models share systematic biases rather than complementary ones, remain open questions.