MTPano: Panoramic Multi-Task Model

• The paper introduces a label-free training pipeline that transforms unlabeled panoramas into pseudo-label supervision for semantic segmentation, depth, and surface normal estimation.
• It employs a dual-branch architecture that separates rotation-invariant and rotation-variant tasks, using geometric cues and a latitude-adaptive token mixer to address ERP distortions.
• Benchmark results on Structured3D and Stanford2D3D demonstrate significant performance gains over specialist and multi-task baselines, validating its innovative approach.

MTPano is a multi-task panoramic foundation model for dense scene understanding in equirectangular panoramas. It is designed for semantic segmentation, depth estimation, and surface normal estimation under a label-free training pipeline that projects unlabeled panoramas into perspective patches, generates pseudo-labels with off-the-shelf perspective foundation models, and reprojects those predictions back as patch-wise supervision. Its central architectural claim is that panoramic multi-task learning should distinguish between rotation-invariant tasks such as depth and segmentation and rotation-variant tasks such as surface normals, because naive feature sharing across these task types introduces negative transfer in spherical coordinates (Zhang et al., 5 Feb 2026).

1. Problem formulation and scope

MTPano addresses panoramic dense prediction in the equirectangular projection (ERP) setting, where a single panorama covers the full $360^\circ$ field of view but also introduces non-uniform geometric distortion. The paper identifies two obstacles. The first is the scarcity of high-resolution, multi-task panoramic annotations. The second is that directly adapting perspective foundation models to panoramas is ineffective because ERP distorts local neighborhoods and because different dense tasks obey different coordinate conventions.

The model’s main tasks are semantic segmentation, depth estimation, and surface normal estimation. During training it also uses auxiliary dense tasks: image gradient estimation, edge distance field (EDF), and metric point map estimation. This task inventory is important because the paper does not cast MTPano as a single-task specialist. Instead, it presents the model as a panoramic multi-task foundation model whose purpose is to learn shared semantics and geometry while controlling task interference.

A recurring misconception is that panoramic multi-task learning is simply perspective multi-task learning applied to a wider image. MTPano rejects that premise. Its argument is that ERP distortion and task-specific geometric behavior change the optimization problem itself: segmentation and depth can be treated as rotation-invariant in the paper’s taxonomy, whereas normals are explicitly rotation-variant and must remain tied to absolute spherical orientation (Zhang et al., 5 Feb 2026).

2. Label-free supervision pipeline

The training pipeline begins with an unlabeled panorama $I_{pano}$. For each panorama, the method samples $N=32$ random perspective crops with FoV in $[80^\circ,120^\circ]$, yaw in $[0^\circ,360^\circ]$, and pitch in $[-90^\circ,90^\circ]$. These perspective patches are then processed by pre-trained perspective models: InternImage-H for semantic segmentation, and MoGe-2 for depth and surface normals.

The key design choice is that pseudo-labels are generated after converting panoramas into perspective patches. The paper describes these pseudo-labels as “domain-gap-free” because the teacher models operate in their native perspective domain rather than directly on ERP imagery. The patch predictions are then reprojected back to spherical coordinates and used as patch-wise supervision. The method explicitly avoids stitching overlapping pseudo-labels into a full panoramic target, because the paper reports that stitching introduces artifacts from scale inconsistencies and overlap noise. Losses are instead computed only on valid pixels in the reprojected patches.

MTPano is trained on 140,884 unlabeled panoramas drawn from Structured3D (20,041), Sun360 (34,260), Matterport3D (10,359), and DiT360 synthetic panoramas (76,224). Ground-truth labels are discarded even when available. This is what “label-free” means in the paper: the model does not require manual panoramic labels, but it is still trained with supervision in the form of pseudo-labels and auxiliary targets.

The pipeline is meant to transform the abundance of unlabeled panoramas into dense panoramic supervision without direct panoramic annotation. A plausible implication is that MTPano treats perspective foundation models as supervision sources rather than as final inference models, thereby shifting the adaptation burden from direct ERP finetuning to pseudo-label transfer (Zhang et al., 5 Feb 2026).

3. Task taxonomy and spherical geometry

A defining aspect of MTPano is its separation of dense tasks into rotation-invariant and rotation-variant groups. The paper denotes the invariant group as $\mathcal{F}_{inv}$ and gives semantic segmentation and depth as examples. The variant group is denoted $\mathcal{F}_{var}$ and surface normals are the canonical example. The motivation is geometric: some outputs should remain stable under changes of observer orientation, while others should transform with the camera or spherical frame.

For the rotation-variant branch, the paper defines the spherical ray direction at latitude $\varphi$ and longitude $\theta$ as

$I_{pano}$0

This direction vector is combined with absolute spherical position and injected only into the variant branch. The corresponding claim is that normals require absolute orientation cues, while segmentation and depth should not be forced to encode that dependency.

The same geometric logic appears in the auxiliary metric point map. Using normalized image coordinates $I_{pano}$1, the paper defines

$I_{pano}$2

$I_{pano}$3

and then converts depth to a dense 3D point map by

$I_{pano}$4

The point map is used as an auxiliary signal for the variant stream and provides explicit 3D coordinate structure. In this formulation, MTPano does not rely on a generic notion of “geometry awareness”; it uses ray direction, absolute spherical position, and point-map supervision as concrete geometry carriers (Zhang et al., 5 Feb 2026).

4. Panorama-Dual-BridgeNet

The model uses a DINOv3-Large backbone and a DPT dense prediction head, but its central architectural component is the Panorama-Dual-BridgeNet (PD-BridgeNet). PD-BridgeNet contains three named parts: geometric-aware feature disentanglement, an ERP Token Mixer, and a Gradient-Truncated Bridge mechanism.

The architecture splits processing into an Invariant Stream and a Variant Stream. In the variant stream, spherical geometry is injected by a FiLM-style modulation. If $I_{pano}$5 denotes the shared backbone features, the modulated variant feature is

$I_{pano}$6

Here $I_{pano}$7 and $I_{pano}$8 are predicted from the concatenation of ray direction and positional embeddings of $I_{pano}$9. This branch-specific conditioning is the mechanism by which MTPano disentangles orientation-sensitive from orientation-insensitive features.

To address ERP distortion, the paper introduces a latitude-adaptive token mixer:

$N=32$0

with

$N=32$1

Near the equator, the mixer favors a standard $N=32$2 kernel. Near the poles, it increases the contribution of a wider $N=32$3 kernel to compensate for horizontal stretching in ERP.

Cross-task interaction is handled by a Bridge Feature Extractor inspired by BridgeNet. Lightweight auxiliary heads produce intermediate task-specific features, these are flattened into tokens, backbone features serve as queries, and concatenated task-specific features serve as keys and values in cross-attention. The crucial optimization constraint is gradient truncation: cross-stream features participate in the forward pass, but a detach operation prevents gradients from one stream from corrupting the other. The paper’s claim is not that cross-task interaction is eliminated, but that incompatible backward coupling is blocked while beneficial forward information sharing remains.

Two additional stabilizers are reported. Injection layers are initialized with Zero-Convolution so that they start as identity mappings, and training uses progressive warmup, first freezing the backbone and training task heads, then unfreezing for end-to-end optimization (Zhang et al., 5 Feb 2026).

5. Auxiliary tasks and objective design

The auxiliary tasks are not peripheral additions. The paper states that they “fertilize the cross-task learning process.” Image gradients are computed with Sobel filters:

$N=32$4

EDF is produced from a high-thresholded edge mask with $N=32$5 using the Jump Flooding Algorithm to compute Euclidean distance to the nearest edge. Metric point maps are formed from depth and spherical rays via $N=32$6.

These auxiliary targets are assigned asymmetrically. The invariant stream receives image gradient and EDF cues, which the paper describes as dense boundary and texture priors. The variant stream receives metric point maps, which provide explicit 3D coordinate priors. This asymmetric auxiliary design is consistent with the paper’s main thesis that panoramic multi-task learning should respect task geometry rather than assume uniform sharing.

The total objective is

$N=32$7

The reported loss weights are $N=32$8, with auxiliary task weight $N=32$9. Geometry regularization from SPGen is added specifically for depth and normals to recover high-frequency detail.

Training uses input resolution $[80^\circ,120^\circ]$0, Adam, learning rate $[80^\circ,120^\circ]$1, weight decay $[80^\circ,120^\circ]$2, 100,000 iterations, 8 NVIDIA A100 GPUs, batch size 4 per GPU, and progressive warmup for 1000 steps. These details indicate that MTPano is positioned not as a lightweight adaptation recipe, but as a panoramic foundation-model training framework with substantial pretraining-scale optimization (Zhang et al., 5 Feb 2026).

6. Benchmarks, performance, and ablations

MTPano is evaluated on Structured3D and Stanford2D3D. The reported metrics are mIoU for semantic segmentation; AbsRel, RMSE, and $[80^\circ,120^\circ]$3 for depth; and mean and median angular error plus percentages within $[80^\circ,120^\circ]$4, $[80^\circ,120^\circ]$5, and $[80^\circ,120^\circ]$6 for normals.

On Structured3D, the reported results are: semantic segmentation mIoU 75.66; depth AbsRel 0.0248, RMSE 0.0968, $[80^\circ,120^\circ]$7, $[80^\circ,120^\circ]$8, $[80^\circ,120^\circ]$9; and normals mean 3.850, median 0.0100, $[0^\circ,360^\circ]$0, $[0^\circ,360^\circ]$1, $[0^\circ,360^\circ]$2. The paper states that these results exceed specialist baselines including SFSS-MMSI for segmentation, DAP for depth, and PanoNormal for normal estimation, and also improve substantially over panoramic multi-task baselines InvPT, BridgeNet, and TaskPrompter.

On Stanford2D3D, the reported results are: semantic segmentation mIoU 69.47; depth AbsRel 0.0675, RMSE 0.4317, $[0^\circ,360^\circ]$3, $[0^\circ,360^\circ]$4, $[0^\circ,360^\circ]$5; and normals mean 9.706, median 0.9314, $[0^\circ,360^\circ]$6, $[0^\circ,360^\circ]$7, $[0^\circ,360^\circ]$8. The paper reports that depth is slightly worse in AbsRel than DA$[0^\circ,360^\circ]$9 at 0.0616, but still much stronger than all compared multi-task baselines.

The ablations are organized around the full design. On Structured3D with DINOv3-Small, a single-task learning baseline reports 64.21 mIoU, 0.1989 RMSE, and 5.9120 mean normal error. The full MTPano model reports 66.64 mIoU, 0.1432 RMSE, and 5.4740 mean normal error, with $[-90^\circ,90^\circ]$0. Additional ablations show that pseudo-label-based multi-task learning already outperforms STL, that removing geometric modulation hurts performance, that gradient truncation improves optimization by reducing harmful cross-branch interference, and that auxiliary heads improve all tasks by enriching bridge interactions. The paper also reports qualitative evidence that cross-task learning can denoise pseudo-label artifacts, including projection seams and depth or normal discontinuities (Zhang et al., 5 Feb 2026).

7. Interpretation, adjacent work, and limitations

MTPano’s main significance lies in combining three ideas that are usually separated: pseudo-label transfer from perspective foundation models, geometry-aware multi-task disentanglement for ERP panoramas, and constrained cross-task interaction through gradient truncation. The paper presents this combination as a way to make panoramic multi-task learning viable without manual panoramic labels.

A second misconception addressed by the model is that “label-free” means unsupervised learning in the strict sense. MTPano is not unsupervised. Its supervision is derived from perspective pseudo-labels, auxiliary geometric targets, and geometry regularization; what it removes is the need for manual panoramic multi-task annotation.

The paper also makes MTPano distinct from adjacent panorama-native work. A related but different direction is PanoWorld, which studies pano-native multimodal reasoning over ERP panoramas for semantic anchoring, spherical localization, reference-frame transformation, and depth-aware 3D spatial reasoning, using a Qwen3.5-VL backbone with Spherical Spatial Cross-Attention rather than a dense prediction architecture (Wang et al., 13 May 2026). This suggests that panoramic modeling is splitting into at least two strands: dense geometric-semantic prediction, represented by MTPano, and panorama-native multimodal spatial reasoning, represented by PanoWorld.

The limitations stated or implied in the MTPano paper are concrete. Performance depends on the quality of perspective pseudo-labels. The model inherits output characteristics of the teacher models and requires benchmark-specific finetuning to align prediction spaces. It also adds architectural complexity through dual branches, auxiliary tasks, and multiple projection steps. These limitations do not negate the central result, but they delimit the method’s scope: MTPano is a panoramic foundation model built around label transfer and geometry-aware multi-task design, not a generic replacement for all panoramic perception systems (Zhang et al., 5 Feb 2026).