PAN-Unified: Unified Vision & Remote Sensing

• PAN-Unified is a framework that integrates tasks like panoptic segmentation, depth-aware parsing, pansharpening, panorama generation, and action recognition into one end-to-end model.
• It employs shared representations with dynamic convolution and transformer-based attention to fuse multimodal data and ensure robust domain generalization.
• Demonstrated benefits include improved performance metrics, reduced parameter counts, and real-time inference, facilitating robust deployment across varied sensors and environments.

PAN-Unified is an umbrella term for unified architectures and methodologies deployed across diverse computer vision and remote sensing domains, focusing on instance-level scene understanding, multi-modal fusion, and domain generalization. These frameworks prioritize single-network, end-to-end learning paradigms over task-specific multi-branch systems and consistently emphasize shared representations, efficient computation, and unified inference protocols. PAN-Unified architectures now underpin panoptic segmentation, depth-aware scene parsing, pansharpening, panorama generation, and action recognition, each with category-specific technical innovations.

1. Unified Frameworks and Problem Formulation

PAN-Unified frameworks address tasks that traditionally required separate models or branches, by consolidating multiple outputs or modalities into a single, jointly optimized system. In panoptic segmentation and depth-aware scene parsing, PAN-Unified approaches simultaneously predict semantic segmentation (“stuff”), object instances (“things”), and auxiliary quantities such as depth (Gao et al., 2022, Xiong et al., 2019, Li et al., 2020, Zhang et al., 2022, Li et al., 2023). For remote sensing pansharpening, PAN-Unified refers to architectural and data-level strategies that achieve robust fusion of multispectral and panchromatic imagery across diverse sensors (Zhang et al., 20 Jul 2025, Cui et al., 25 Oct 2025). In generation tasks, PAN-Unified enables a single model to produce either full panoramas from text prompts or to complete partial panoramic views via a shared pipeline (Feng et al., 7 Dec 2025). In action recognition, PAN architectures aggregate motion and appearance cues with minimal branch separation (Zhang et al., 2020).

Formally, the PAN-Unified principle treats disparate prediction tasks—semantic class assignment, instance ID, depth regression, image fusion, or panoramic synthesis—as outputs of a shared model, often parameterized by transformer, dynamic convolution, or multi-head attention mechanisms.

2. Core Architectural Strategies

PAN-Unified designs routinely deploy backbone sharing and dynamic head construction. For DPS, PAN-Unified leverages instance-specific dynamic convolution kernels to predict both masks and depth maps, generating per-instance kernels for both tasks and exploiting high-level object context in depth prediction (Gao et al., 2022). UPSNet implements a parameter-free panoptic head that fuses semantic and instance logits for pixelwise classification with on-the-fly channel expansion to accommodate variable instance counts (Xiong et al., 2019). In domain-adaptive panoptic segmentation, UniDAformer replaces dual networks with a DETR-style transformer, a unified mask-classifier head, and hierarchical mask calibration to rectify pseudo-labels at region, superpixel, and pixel scales (Zhang et al., 2022). PanopticPartFormer++ models things, stuff, and part queries jointly within a decoupled transformer decoder architecture, introducing global-part masked cross-attention for unified part-whole predictions (Li et al., 2023).

For pansharpening and multi-sensor fusion, PAN-Unified architectures such as PanTiny adopt single-encoder, lightweight CNN+Transformer hybrids, trained on multiple datasets with universal loss functions and minimal architectural cruft (Zhang et al., 20 Jul 2025). JoPano for panorama generation utilizes a single frozen DiT backbone with joint-face adapters, allowing both text-to-panorama and view-to-panorama synthesis in a unified conditional framework (Feng et al., 7 Dec 2025).

3. Unified Inference, Training Protocols, and Losses

PAN-Unified frameworks eliminate the train-test divide by deploying identical propagation mechanisms and fusion strategies at both stages. In panoptic segmentation, end-to-end learning of dense instance affinity ensures fully differentiable training and inference without post-processing heuristics; the panoptic map arises from straightforward channelwise argmax (Li et al., 2020). UPSNet’s panoptic head enables backpropagation of the panoptic loss to all bottom modules, and supports variable instance counts with no need for heuristic merging at inference (Xiong et al., 2019). UniDAformer, through momentum teacher-student self-training coupled with hierarchical mask calibration, achieves robust domain adaptation, sharing computation across semantic and instance segmentation (Zhang et al., 2022).

In pansharpening, PanTiny demonstrates that all-in-one training across multiple sensors is more effective than brute-force scaling of model size, balancing spectral, perceptual, and focal errors via a universal composite loss (Charbonnier, SSIM, Focal regression) (Zhang et al., 20 Jul 2025). UniPAN proposes distribution transformation by quantile-based mapping of input pixels to a common statistical domain, enabling train-once, deploy-forever generalizability without parameter adjustment at inference (Cui et al., 25 Oct 2025).

For panorama synthesis, JoPano’s unified loss is a mean-squared error between predicted and ground-truth diffusion velocities, applicable regardless of T2P or V2P task toggle (Feng et al., 7 Dec 2025). In action recognition, the PAN framework integrates persistence-of-appearance and multi-timescale pooling with standard cross-entropy, eschewing auxiliary motion-specific branches (Zhang et al., 2020).

4. Technical Innovations and Comparative Results

PAN-Unified frameworks have spurred advances across multiple axes:

• Instance-level dynamic depth estimation: Adaptive Kernel Fusion and instance-wise normalization deliver ∼2% DPQ improvements in DPS benchmarks, with RMSE for monocular depth matching or exceeding panoptic-aware SOTA (Gao et al., 2022).
• Unified domain adaptation: UniDAformer achieves 33.0 mPQ on SYNTHIA→Cityscapes vs. 27.9 for two-branch baselines, halving parameter count and accelerating inference to 5.2 fps (Zhang et al., 2022).
• Panoptic-part segmentation: PanopticPartFormer++ improvements of up to +2% PartPQ and +3% PWQ on Cityscapes PPS, +5% PartPQ on Pascal PPS, state-of-the-art scores with transformer-based global-part cross-attention (Li et al., 2023).
• Pansharpening: PanTiny’s “multi-in-one” training boosts full-resolution QNR from ~0.79 to ~0.89 and matches or surpasses much heavier architectures, while requiring ≤82K parameters (Zhang et al., 20 Jul 2025). UniPAN’s distribution transformation increases mean QNR by up to +0.08, and generalizes across six sensors for twelve model backbones (Cui et al., 25 Oct 2025).
• Panorama generation: JoPano sets new records in both T2P and V2P with FID, CLIP-FID, IS, and CLIP-Score improvements over PanFusion, SMGD, and Diffusion360, and achieves seam-consistency scores (Seam-SSIM, Seam-Sobel) nearly matching ground-truth after Poisson blending (Feng et al., 7 Dec 2025).
• Action recognition: PAN achieves 8196 fps (PA module) vs. FlowNet2.0’s 25 fps and matches or outperforms flow-dependent SOTA at one to two orders lower FLOPs (Zhang et al., 2020).

Model/Method	Domain	Key Metric(s)	Unified Training vs. Separated	Notable Innovations
PAN-Unified DPS	Scene parsing	DPQ (Cityscapes)	+2.1% absolute DPQ gain	Instance-wise kernel; AKF; FSF
UPSNet	Panoptic seg.	PQ (COCO/Citysc.)	+0.8 PQ over heuristic merge	Param-free head; variable inst. count
PanTiny (Big)	Pansharpening	QNR (full-res)	QNR ↑ 0.08–0.10 (all-in-one)	Multi-dataset; universal loss
UniPAN	Pan-sharpening	QNR (cross-sensor)	+0.01–0.08 QNR majority	Quantile-based dist. unification
JoPano	Panorama gen.	Seam-SSIM, FID	SOTA on T2P/V2P benchmarks	DiT+adapter; Poisson blending
UniDAformer	DA panoptic	mPQ/SQ/RQ	+5–10 mPQ over baseline	One-branch; HMC

5. Practical Deployment and Generalization

A consistent theme of PAN-Unified is practical, robust deployment across variable data sources, sensor types, and domain shifts. In remote sensing, UniPAN’s quantile-based transformation supports plug-and-play deployment for unseen satellites—after a brief estimation of pixel quantiles, any trained backbone can generalize without retraining (Cui et al., 25 Oct 2025). PanTiny’s minimal footprint and all-in-one model deployment simplify edge applications, matching state-of-the-art at orders-of-magnitude lower cost (Zhang et al., 20 Jul 2025). In segmentation, unification collapses dual networks and reduces parameter demands and inference time (e.g., UniDAformer 78M params/5.2 fps vs. CVRN’s 185M/0.36 fps) (Zhang et al., 2022). PAN architectures for action recognition eliminate dependence on dense optical flow, enabling real-time inference (Zhang et al., 2020).

6. Limitations and Future Research Directions

While PAN-Unified frameworks substantially advance efficiency and performance, several limitations remain:

• Scaling to extremely high-resolution (e.g., true-4K panoramas) may require more powerful backbones and extensive pretraining (JoPano) (Feng et al., 7 Dec 2025).
• Some normalization strategies may perturb original data statistics more than necessary; e.g., Gaussian targets in UniPAN may degrade some metrics slightly vs. uniform targets (Cui et al., 25 Oct 2025).
• Unified models may observe small performance drops in highly specialized (sensor-specific) tasks, though absolute performance and generalization remain superior when averaging across domains (Zhang et al., 20 Jul 2025).
• Multi-task unification occasionally calls for more sophisticated loss balancing, especially when secondary outputs are critical (e.g., depth vs. semantics in DPS) (Gao et al., 2022).

This suggests that future work in PAN-Unified architectures will continue to refine backbone, attention, and fusion modules for increased scalability, fine-grained cross-domain adaptation, and even broader multimodal unification—including dynamic video, temporal coherence, and generative data augmentation (Feng et al., 7 Dec 2025, Li et al., 2023).

7. Conceptual Impact and Research Trends

The emergence of PAN-Unified frameworks marks a trend away from brute-force model scaling and task-specific customization, in favor of data- and architecture-level unification. This trend encompasses:

• End-to-end differentiable training for multi-output segmentation (Li et al., 2020, Xiong et al., 2019);
• Parameter-free heads and instance-adaptive kernels (Gao et al., 2022);
• Data-space unification for robust model generalization (Cui et al., 25 Oct 2025, Zhang et al., 20 Jul 2025);
• Adapter-based cross-task generative modeling (Feng et al., 7 Dec 2025);
• Real-time, explainable action recognition via boundary-based cues (Zhang et al., 2020);
• Hierarchical, query-based transformer decoders bridging instance, stuff, and part predictions (Li et al., 2023, Zhang et al., 2022).

A plausible implication is that unified paradigms will directly inform the next generation of efficient multi-task, multi-domain vision models—progressing toward universal scene parsing, fusion, and generation at scale.