MGNet: Mask Generation for Autonomous Driving

• MGNet is a Mask Generation Network that jointly performs panoptic segmentation and self-supervised monocular depth estimation using a shared encoder–decoder architecture.
• It employs a bottom-up mask generation approach by grouping semantic predictions, instance-center heatmaps, and offset fields, eliminating the need for proposal-based methods.
• The design leverages panoptic outputs to inform geometric depth scaling, achieving competitive performance in real-time autonomous driving benchmarks.

Searching arXiv for the relevant MGNet papers and acronym disambiguation. MGNet most commonly denotes the model introduced in “MGNet: Monocular Geometric Scene Understanding for Autonomous Driving,” where it is described as a Mask Generation Network for jointly solving panoptic segmentation and self-supervised monocular depth estimation in one efficient architecture (Schön et al., 2022). In this formulation, monocular geometric scene understanding is the combination of scene parsing at panoptic granularity and dense geometric inference from monocular video. The model is designed for low latency and, at deployment, produces dense 3D point clouds with instance aware semantic labels from single high-resolution camera images. Its “mask generation” mechanism is bottom-up: masks are generated from semantic prediction, instance-center heatmaps, and pixel-wise offset fields, rather than from proposal-based crop-and-mask heads (Schön et al., 2022).

1. Definition and nomenclature

In the autonomous-driving literature, MGNet refers to a multi-task framework whose two core tasks are panoptic segmentation and self-supervised monocular depth estimation (Schön et al., 2022). Panoptic segmentation provides “what is where,” including both “stuff” and “thing” classes, while monocular depth estimation provides “how far away” every visible point is. The joint output can be lifted into a dense 3D point cloud with semantic and instance-aware labels.

The acronym is overloaded across arXiv. “MGNet” also names the Multi-Glimpse Network, a recurrent downsampled-attention classifier rather than a mask predictor (Tan et al., 2021); MgNet, a multigrid-inspired image-classification architecture rather than a segmentation model (He et al., 2021); and the Multi-Stage Goal-Driven Network for pedestrian trajectory prediction (Wu et al., 2024). A later weakly supervised camouflaged object detector also uses the name MGNet, but there it denotes a Mask-guided Network rather than the autonomous-driving model discussed here (Li et al., 25 May 2026). In this encyclopedia sense, Mask Generation Network (MGNet) refers specifically to the monocular geometric scene-understanding framework of Schoen and collaborators (Schön et al., 2022).

arXiv id	Expansion of MGNet	Domain
(Schön et al., 2022)	Mask Generation Network	Monocular geometric scene understanding
(Tan et al., 2021)	Multi-Glimpse Network	Image classification
(He et al., 2021)	MgNet	Multigrid-inspired classification
(Wu et al., 2024)	Multi-Stage Goal-Driven Network	Trajectory prediction
(Li et al., 25 May 2026)	Mask-guided Network	Camouflaged object detection

2. Overall architecture

MGNet is an encoder–decoder multi-task network with one shared encoder and three task-specific decoders: one for semantic segmentation, one for instance segmentation through center and offset prediction, and one for depth estimation (Schön et al., 2022). The encoder extracts multiscale feature maps up to output stride 32. The paper evaluates MobileNetV3, MNASNet100, EfficientNetLite0, and ResNet18, and ultimately uses ResNet18 as the main choice because it gives the best balance of PQ, RMSE, and FPS.

On top of the deepest backbone feature map, MGNet adds a Global Context Module (GCM) based on global average pooling. This enlarges the effective receptive field and injects image-level context into both scene parsing and depth estimation. Each decoder then fuses multiscale features using Attention Refinement Modules (ARMs) on the last two backbone feature maps and Feature Fusion Modules (FFMs) to combine higher-level contextual features with low-level spatial detail through skip connections. The authors explicitly avoid BiSeNet’s bilateral spatial/context path because it increased latency without improving performance (Schön et al., 2022).

The task heads remain lightweight. Each head uses a $3 \times 3$ convolution followed by a $1 \times 1$ convolution to map decoder features to semantic logits, instance-center heatmaps, offset vectors, or depth logits. The depth head output is passed through a sigmoid, following Monodepth2 practice. This organization makes the model a genuinely shared multi-task system rather than a late fusion of independent panoptic and depth networks (Schön et al., 2022).

3. Bottom-up mask generation and panoptic decoding

The mask-generation component follows the bottom-up representation of Panoptic-DeepLab (Schön et al., 2022). Instead of proposal generation and RoI mask heads, each object instance is represented by three dense outputs: a semantic segmentation map, an instance center heatmap, and a pixel-wise 2D offset field that points from each foreground “thing” pixel to the center of its instance. In this sense, MGNet generates masks indirectly by grouping pixels around predicted centers.

For semantic segmentation, the model predicts a class distribution at every pixel. The semantic loss is a weighted bootstrapped cross-entropy,

$\mathcal{L}_{\text{seg}} = -\frac{1}{K}\sum_{i=1}^{N}\omega_i \cdot \mathds{1}\!\left[p_{i,y_i}<t_K\right]\cdot \log p_{i,y_i},$

with $\omega_i = 3$ for pixels belonging to instances smaller than $64\times64$ and $\omega_i = 1$ otherwise. The instance-center branch predicts a 2D Gaussian heatmap with fixed $\sigma = 8$ pixels for each object center and is trained with an MSE term. The offset branch predicts $\mathbf{o}_i=(\Delta x_i,\Delta y_i)$ for each “thing” pixel and is trained with an $L_1$ regression loss (Schön et al., 2022).

The full panoptic objective is

$$\mathcal{L}_{\text{pan}} = \mathcal{L}_{\text{seg}} + 200 \cdot \mathcal{L}_{\text{mse}} + 0.01 \cdot \mathcal{L}_{L_1}.$$

At inference time, semantic argmax yields class labels, keypoint-based non-maximum suppression with max-pooling kernel size 7 and confidence threshold 0.3 extracts center candidates, and only pixels predicted as “thing” classes participate in instance grouping. Each “thing” pixel is shifted by its predicted offset vector and assigned to the closest detected center keypoint. All pixels assigned to the same center form one instance mask, and the instance class is determined by majority voting over semantic labels inside that mask (Schön et al., 2022).

This design is the defining “mask generation” feature of MGNet. It is fully convolutional, avoids proposal generation and RoI refinement, and is therefore aligned with the model’s real-time objective. A plausible implication is that MGNet belongs more naturally to the bottom-up panoptic lineage than to Mask R-CNN-style mask-head architectures.

4. Self-supervised depth estimation and geometric coupling

The depth branch is based on Monodepth2 and an improved Dense Geometrical Constraints (DGC) module (Schön et al., 2022). During training, MGNet uses monocular video triplets $1 \times 1$0. The network predicts a depth map for $1 \times 1$1, while a separate ResNet18-based pose network predicts 6 DoF relative camera motion between adjacent frames. The pose network is used only during training.

Depth supervision is obtained by view synthesis. Neighboring frames are warped into the target viewpoint, producing $1 \times 1$2 and $1 \times 1$3, and photometric consistency is enforced by

$1 \times 1$4

with $1 \times 1$5. MGNet adopts the minimum reprojection error strategy from Monodepth2 and includes the unwarped source images in the candidate set, implementing auto-masking. A valid-projection mask $1 \times 1$6 ignores invalid geometric projections. Edge-aware smoothness regularization is applied to mean-normalized inverse depth, and the final depth loss is

$1 \times 1$7

There is no explicit supervised depth loss (Schön et al., 2022).

A distinctive feature is the coupling between segmentation and geometry in scale recovery. Monocular self-supervised depth is only known up to scale, so MGNet uses its own panoptic output to identify road pixels as ground points in an improved DGC module. For each ground point $1 \times 1$8, camera height is estimated as

$1 \times 1$9

the median estimated height is $\mathcal{L}_{\text{seg}} = -\frac{1}{K}\sum_{i=1}^{N}\omega_i \cdot \mathds{1}\!\left[p_{i,y_i}<t_K\right]\cdot \log p_{i,y_i},$0, and absolute depth is recovered by

$\mathcal{L}_{\text{seg}} = -\frac{1}{K}\sum_{i=1}^{N}\omega_i \cdot \mathds{1}\!\left[p_{i,y_i}<t_K\right]\cdot \log p_{i,y_i},$1

This is one of the clearest explicit cross-task interactions in the model: panoptic road prediction improves depth scaling by reducing false ground-point classifications such as car roofs (Schön et al., 2022).

5. Optimization, data, and inference behavior

MGNet addresses multi-task loss balancing with homoscedastic uncertainty weighting layered on top of the fixed coefficients already used in the single-task formulations (Schön et al., 2022). With learned variables $\mathcal{L}_{\text{seg}} = -\frac{1}{K}\sum_{i=1}^{N}\omega_i \cdot \mathds{1}\!\left[p_{i,y_i}<t_K\right]\cdot \log p_{i,y_i},$2, the total loss is

$\mathcal{L}_{\text{seg}} = -\frac{1}{K}\sum_{i=1}^{N}\omega_i \cdot \mathds{1}\!\left[p_{i,y_i}<t_K\right]\cdot \log p_{i,y_i},$3

The rationale is that fixed coefficients encode prior task knowledge, while learned uncertainties adapt the balance during optimization (Schön et al., 2022).

The model is evaluated on Cityscapes and KITTI. Cityscapes provides 5,000 fine-annotated images with panoptic labels, 19 evaluation classes, and video sequences; MGNet trains on 20 classes by adding ego-car. KITTI Eigen split has no panoptic labels, so the authors use their best Cityscapes MGNet to generate pseudo panoptic labels on KITTI and then train jointly on those (Schön et al., 2022).

At test time, MGNet takes a single RGB image. The pose network and photometric machinery are removed. The network outputs semantic segmentation logits, an instance center heatmap, an instance offset field, and a relative depth map, which is scaled to absolute depth. These are decoded into a panoptic segmentation map and then into a 3D point cloud with semantic/instance labels. For 3D output, sky and ego-car are excluded because they do not correspond to useful valid scene geometry (Schön et al., 2022).

6. Empirical performance and ablation structure

On Cityscapes validation, the final MGNet reports PQ = 55.7, RMSE = 8.3, and 30 FPS at full resolution $\mathcal{L}_{\text{seg}} = -\frac{1}{K}\sum_{i=1}^{N}\omega_i \cdot \mathds{1}\!\left[p_{i,y_i}<t_K\right]\cdot \log p_{i,y_i},$4 (Schön et al., 2022). Runtime is measured over 500 forward passes, including GPU data loading and post-processing, with TensorRT optimization unless otherwise stated. The corresponding latency is 44.4 ms, and on KITTI $\mathcal{L}_{\text{seg}} = -\frac{1}{K}\sum_{i=1}^{N}\omega_i \cdot \mathds{1}\!\left[p_{i,y_i}<t_K\right]\cdot \log p_{i,y_i},$5, the model reaches 82 FPS (Schön et al., 2022). Training uses 4 NVIDIA RTX 2080 Ti GPUs.

On KITTI Eigen split, MGNet reports Abs Rel = 0.095, RMSE = 3.761, $\mathcal{L}_{\text{seg}} = -\frac{1}{K}\sum_{i=1}^{N}\omega_i \cdot \mathds{1}\!\left[p_{i,y_i}<t_K\right]\cdot \log p_{i,y_i},$6, $\mathcal{L}_{\text{seg}} = -\frac{1}{K}\sum_{i=1}^{N}\omega_i \cdot \mathds{1}\!\left[p_{i,y_i}<t_K\right]\cdot \log p_{i,y_i},$7, and $\mathcal{L}_{\text{seg}} = -\frac{1}{K}\sum_{i=1}^{N}\omega_i \cdot \mathds{1}\!\left[p_{i,y_i}<t_K\right]\cdot \log p_{i,y_i},$8 (Schön et al., 2022). The paper emphasizes competitiveness with real-time capable methods rather than maximal single-task accuracy. Compared with a lightweight Panoptic-DeepLab variant using MobileNetV3, MGNet is slightly better in PQ (55.7 vs 55.4) and much faster (44.4 ms vs 63 ms) while also solving depth estimation (Schön et al., 2022).

The ablations make the task-coupling story explicit. For backbone choice on Cityscapes, ResNet18 gives PQ 53.3, RMSE 8.8, FPS 30.0, compared with PQ 48.6, RMSE 9.4, FPS 29.8 for MobileNetV3, PQ 50.8, RMSE 9.0, FPS 28.3 for MNASNet100, and PQ 52.8, RMSE 8.6, FPS 27.7 for EfficientNetLite0 (Schön et al., 2022). For optimization strategy, the single-task baseline yields PQ 54.1, RMSE 9.3, the naive multi-task baseline yields PQ 53.3, RMSE 8.8, adding uncertainty weighting yields PQ 54.2, RMSE 8.6, and adding video sequence training yields the final PQ 55.7, RMSE 8.3 (Schön et al., 2022). This shows that multi-task learning is beneficial, but only if loss balancing is handled carefully.

7. Relation to other mask-oriented work and major limitations

MGNet’s mask-generation mechanism is specific to bottom-up panoptic grouping. It should not be conflated with several other mask uses in the literature. MaskPlus improves the mask generation branch inside Mask R-CNN for instance segmentation, focusing on contextual fusion, deconvolutional pyramid refinement, boundary refinement, quasi-multitask learning, and biased training (Xu et al., 2019). “Mask Embedding in conditional GAN for Guided Synthesis of High Resolution Images” uses masks as conditioning inputs for high-resolution face synthesis rather than generating masks (Ren et al., 2019). Mask-ControlNet introduces an additional mask prompt derived from SAM for reference-image-guided diffusion generation, again using masks as conditioning rather than as predicted panoptic outputs (Huang et al., 2024). “Mask Based Unsupervised Content Transfer” learns a soft spatial mask for localized content insertion, while MaskUNet masks diffusion-model parameters rather than spatial outputs (Mokady et al., 2019, Wang et al., 6 May 2025). These comparisons delimit the meaning of MGNet’s “mask generation”: it is the generation of panoptic instance masks from dense center-offset representations, embedded in a geometry-aware autonomous-driving system.

The main limitations are those of its design goals. MGNet prioritizes real-time efficiency over maximum single-task accuracy, so its PQ is behind large high-accuracy panoptic methods and its depth quality is behind the strongest specialized self-supervised depth models (Schön et al., 2022). Its depth branch inherits the standard assumptions of photometric monocular training: mostly static scenes, camera motion, limited occlusions, and scale ambiguity unless additional constraints are added. Absolute scale recovery depends on ground geometry and known camera height, making it dataset- and camera-dependent. On KITTI, panoptic supervision is based on pseudo labels, which may cap segmentation quality. Finally, except for the road-mask usage in DGC, most segmentation–depth interaction remains implicit (Schön et al., 2022).

Taken together, MGNet is best understood as a bottom-up panoptic mask generator fused with a self-supervised monocular depth estimator under a shared low-latency encoder–decoder design (Schön et al., 2022). Its distinctive contribution is not merely that it predicts masks and depth at once, but that it turns panoptic road masks into a geometric prior for absolute depth scaling, thereby linking semantic scene parsing and monocular geometry within a single real-time framework.