---
title: 'MGNet: Mask Generation for Autonomous Driving'
url: https://www.emergentmind.com/topics/mask-generation-network-mgnet
type: topic
---

# MGNet: Mask Generation for Autonomous Driving

Searching arXiv for the relevant MGNet papers and acronym disambiguation.
MGNet most commonly denotes the model introduced in “MGNet: Monocular Geometric Scene Understanding for Autonomous Driving,” where it is described as a **Mask Generation Network** for jointly solving **panoptic segmentation** and **self-supervised monocular depth estimation** in one efficient architecture [2206.13199]. In this formulation, monocular geometric scene understanding is the combination of scene parsing at panoptic granularity and dense geometric inference from monocular video. The model is designed for low latency and, at deployment, produces dense 3D point clouds with instance aware semantic labels from single high-resolution camera images. Its “mask generation” mechanism is bottom-up: masks are generated from semantic prediction, instance-center heatmaps, and pixel-wise offset fields, rather than from proposal-based crop-and-mask heads [2206.13199].

## 1. Definition and nomenclature

In the autonomous-driving literature, MGNet refers to a multi-task framework whose two core tasks are panoptic segmentation and self-supervised monocular depth estimation [2206.13199]. Panoptic segmentation provides “what is where,” including both “stuff” and “thing” classes, while monocular depth estimation provides “how far away” every visible point is. The joint output can be lifted into a dense 3D point cloud with semantic and instance-aware labels.

The acronym is overloaded across arXiv. “MGNet” also names the **Multi-Glimpse Network**, a recurrent downsampled-attention classifier rather than a mask predictor [2111.02018]; **MgNet**, a multigrid-inspired image-classification architecture rather than a segmentation model [2112.07441]; and the **Multi-Stage Goal-Driven Network** for pedestrian trajectory prediction [2406.18050]. A later weakly supervised camouflaged object detector also uses the name MGNet, but there it denotes a **Mask-guided Network** rather than the autonomous-driving model discussed here [2605.25385]. In this encyclopedia sense, **Mask Generation Network (MGNet)** refers specifically to the monocular geometric scene-understanding framework of Schoen and collaborators [2206.13199].

| arXiv id | Expansion of MGNet | Domain |
|---|---|---|
| [2206.13199] | Mask Generation Network | Monocular geometric scene understanding |
| [2111.02018] | Multi-Glimpse Network | Image classification |
| [2112.07441] | MgNet | Multigrid-inspired classification |
| [2406.18050] | Multi-Stage Goal-Driven Network | Trajectory prediction |
| [2605.25385] | Mask-guided Network | Camouflaged object detection |

## 2. Overall architecture

MGNet is an **encoder–decoder multi-task network** with **one shared encoder** and **three task-specific decoders**: one for semantic segmentation, one for instance segmentation through center and offset prediction, and one for depth estimation [2206.13199]. The encoder extracts multiscale feature maps up to output stride 32. The paper evaluates **MobileNetV3, MNASNet100, EfficientNetLite0, and ResNet18**, and ultimately uses **ResNet18** as the main choice because it gives the best balance of PQ, RMSE, and FPS.

On top of the deepest backbone feature map, MGNet adds a **Global Context Module (GCM)** based on global average pooling. This enlarges the effective receptive field and injects image-level context into both scene parsing and depth estimation. Each decoder then fuses multiscale features using **Attention Refinement Modules (ARMs)** on the last two backbone feature maps and **Feature Fusion Modules (FFMs)** to combine higher-level contextual features with low-level spatial detail through skip connections. The authors explicitly avoid BiSeNet’s bilateral spatial/context path because it increased latency without improving performance [2206.13199].

The task heads remain lightweight. Each head uses a \(3 \times 3\) convolution followed by a \(1 \times 1\) convolution to map decoder features to semantic logits, instance-center heatmaps, offset vectors, or depth logits. The depth head output is passed through a **sigmoid**, following Monodepth2 practice. This organization makes the model a genuinely shared multi-task system rather than a late fusion of independent panoptic and depth networks [2206.13199].

## 3. Bottom-up mask generation and panoptic decoding

The mask-generation component follows the bottom-up representation of **Panoptic-DeepLab** [2206.13199]. Instead of proposal generation and RoI mask heads, each object instance is represented by three dense outputs: a semantic segmentation map, an instance center heatmap, and a pixel-wise 2D offset field that points from each foreground “thing” pixel to the center of its instance. In this sense, MGNet generates masks indirectly by grouping pixels around predicted centers.

For semantic segmentation, the model predicts a class distribution at every pixel. The semantic loss is a weighted bootstrapped cross-entropy,
\[
\mathcal{L}_{\text{seg}} = -\frac{1}{K}\sum_{i=1}^{N}\omega_i \cdot \mathds{1}\!\left[p_{i,y_i}<t_K\right]\cdot \log p_{i,y_i},
\]
with \(\omega_i = 3\) for pixels belonging to instances smaller than \(64\times64\) and \(\omega_i = 1\) otherwise. The instance-center branch predicts a **2D Gaussian heatmap** with fixed \(\sigma = 8\) pixels for each object center and is trained with an MSE term. The offset branch predicts \(\mathbf{o}_i=(\Delta x_i,\Delta y_i)\) for each “thing” pixel and is trained with an \(L_1\) regression loss [2206.13199].

The full panoptic objective is
\[
\mathcal{L}_{\text{pan}} = \mathcal{L}_{\text{seg}} + 200 \cdot \mathcal{L}_{\text{mse}} + 0.01 \cdot \mathcal{L}_{L_1}.
\]
At inference time, semantic argmax yields class labels, **keypoint-based non-maximum suppression** with max-pooling kernel size 7 and confidence threshold 0.3 extracts center candidates, and only pixels predicted as “thing” classes participate in instance grouping. Each “thing” pixel is shifted by its predicted offset vector and assigned to the closest detected center keypoint. All pixels assigned to the same center form one instance mask, and the instance class is determined by majority voting over semantic labels inside that mask [2206.13199].

This design is the defining “mask generation” feature of MGNet. It is fully convolutional, avoids proposal generation and RoI refinement, and is therefore aligned with the model’s real-time objective. A plausible implication is that MGNet belongs more naturally to the bottom-up panoptic lineage than to Mask R-CNN-style mask-head architectures.

## 4. Self-supervised depth estimation and geometric coupling

The depth branch is based on **Monodepth2** and an improved **Dense Geometrical Constraints (DGC)** module [2206.13199]. During training, MGNet uses monocular video triplets \(I_{t-1}, I_t, I_{t+1}\). The network predicts a depth map for \(I_t\), while a separate **ResNet18-based pose network** predicts 6 DoF relative camera motion between adjacent frames. The pose network is used only during training.

Depth supervision is obtained by view synthesis. Neighboring frames are warped into the target viewpoint, producing \(\hat{I}_{t-1\rightarrow t}\) and \(\hat{I}_{t+1\rightarrow t}\), and photometric consistency is enforced by
\[
\mathcal{L}_{\text{phot}}(I_t, \hat{I}_t) =
\alpha \cdot \frac{1-\mathrm{SSIM}(I_t,\hat{I}_t)}{2}
+ (1-\alpha)\cdot |I_t-\hat{I}_t|,
\]
with \(\alpha = 0.85\). MGNet adopts the **minimum reprojection error** strategy from Monodepth2 and includes the unwarped source images in the candidate set, implementing **auto-masking**. A valid-projection mask \(M_v\) ignores invalid geometric projections. Edge-aware smoothness regularization is applied to mean-normalized inverse depth, and the final depth loss is
\[
\mathcal{L}_{\text{depth}} = \mathcal{L}_{\text{phot}} + 0.001 \cdot \mathcal{L}_{\text{smooth}}.
\]
There is no explicit supervised depth loss [2206.13199].

A distinctive feature is the coupling between segmentation and geometry in scale recovery. Monocular self-supervised depth is only known up to scale, so MGNet uses its own panoptic output to identify **road pixels** as ground points in an improved DGC module. For each ground point \(p_i=(x,y,z)^\top\), camera height is estimated as
\[
h(p_i)=N(p_i)^\top p_i,
\]
the median estimated height is \(\hat{h}_m\), and absolute depth is recovered by
\[
d_{\text{abs}} = \frac{h}{\hat{h}_m} \, d_{\text{rel}}.
\]
This is one of the clearest explicit cross-task interactions in the model: panoptic road prediction improves depth scaling by reducing false ground-point classifications such as car roofs [2206.13199].

## 5. Optimization, data, and inference behavior

MGNet addresses multi-task loss balancing with **homoscedastic uncertainty weighting** layered on top of the fixed coefficients already used in the single-task formulations [2206.13199]. With learned variables \(s_i=\log(\sigma_i^2)\), the total loss is
\[
\begin{aligned}
\mathcal{L} &= \exp(-s_0)\cdot \mathcal{L}_{\text{seg}}
+ 0.5\cdot \exp(-s_1)\cdot 200 \cdot \mathcal{L}_{\text{mse}} \\
&\quad + 0.5\cdot \exp(-s_2)\cdot 0.01 \cdot \mathcal{L}_{L_1}
+ 0.5\cdot \exp(-s_3)\cdot \mathcal{L}_{\text{phot}} \\
&\quad + 0.5\cdot \exp(-s_4)\cdot 0.001 \cdot \mathcal{L}_{\text{smooth}}
+ 0.5 \cdot \sum_i s_i .
\end{aligned}
\]
The rationale is that fixed coefficients encode prior task knowledge, while learned uncertainties adapt the balance during optimization [2206.13199].

The model is evaluated on **Cityscapes** and **KITTI**. Cityscapes provides 5,000 fine-annotated images with panoptic labels, 19 evaluation classes, and video sequences; MGNet trains on **20 classes** by adding **ego-car**. KITTI Eigen split has no panoptic labels, so the authors use their best Cityscapes MGNet to generate **pseudo panoptic labels** on KITTI and then train jointly on those [2206.13199].

At test time, MGNet takes a **single RGB image**. The pose network and photometric machinery are removed. The network outputs semantic segmentation logits, an instance center heatmap, an instance offset field, and a relative depth map, which is scaled to absolute depth. These are decoded into a panoptic segmentation map and then into a **3D point cloud with semantic/instance labels**. For 3D output, **sky** and **ego-car** are excluded because they do not correspond to useful valid scene geometry [2206.13199].

## 6. Empirical performance and ablation structure

On **Cityscapes** validation, the final MGNet reports **PQ = 55.7**, **RMSE = 8.3**, and **30 FPS** at full resolution \(1024 \times 2048\) [2206.13199]. Runtime is measured over 500 forward passes, including GPU data loading and post-processing, with TensorRT optimization unless otherwise stated. The corresponding latency is **44.4 ms**, and on **KITTI** \(384 \times 1280\), the model reaches **82 FPS** [2206.13199]. Training uses **4 NVIDIA RTX 2080 Ti GPUs**.

On **KITTI Eigen split**, MGNet reports **Abs Rel = 0.095**, **RMSE = 3.761**, \(\delta<1.25 = 0.902\), \(\delta<1.25^2 = 0.979\), and \(\delta<1.25^3 = 0.992\) [2206.13199]. The paper emphasizes competitiveness with real-time capable methods rather than maximal single-task accuracy. Compared with a lightweight Panoptic-DeepLab variant using MobileNetV3, MGNet is slightly better in PQ (**55.7 vs 55.4**) and much faster (**44.4 ms vs 63 ms**) while also solving depth estimation [2206.13199].

The ablations make the task-coupling story explicit. For backbone choice on Cityscapes, **ResNet18** gives **PQ 53.3, RMSE 8.8, FPS 30.0**, compared with **PQ 48.6, RMSE 9.4, FPS 29.8** for MobileNetV3, **PQ 50.8, RMSE 9.0, FPS 28.3** for MNASNet100, and **PQ 52.8, RMSE 8.6, FPS 27.7** for EfficientNetLite0 [2206.13199]. For optimization strategy, the **single-task baseline** yields **PQ 54.1, RMSE 9.3**, the **naive multi-task baseline** yields **PQ 53.3, RMSE 8.8**, adding **uncertainty weighting** yields **PQ 54.2, RMSE 8.6**, and adding **video sequence training** yields the final **PQ 55.7, RMSE 8.3** [2206.13199]. This shows that multi-task learning is beneficial, but only if loss balancing is handled carefully.

## 7. Relation to other mask-oriented work and major limitations

MGNet’s mask-generation mechanism is specific to bottom-up panoptic grouping. It should not be conflated with several other mask uses in the literature. **MaskPlus** improves the mask generation branch inside Mask R-CNN for instance segmentation, focusing on contextual fusion, deconvolutional pyramid refinement, boundary refinement, quasi-multitask learning, and biased training [1907.06713]. **“Mask Embedding in conditional GAN for Guided Synthesis of High Resolution Images”** uses masks as conditioning inputs for high-resolution face synthesis rather than generating masks [1907.01710]. **Mask-ControlNet** introduces an additional mask prompt derived from SAM for reference-image-guided diffusion generation, again using masks as conditioning rather than as predicted panoptic outputs [2404.05331]. **“Mask Based Unsupervised Content Transfer”** learns a soft spatial mask for localized content insertion, while **MaskUNet** masks diffusion-model parameters rather than spatial outputs [1906.06558; 2505.03097]. These comparisons delimit the meaning of MGNet’s “mask generation”: it is the generation of panoptic instance masks from dense center-offset representations, embedded in a geometry-aware autonomous-driving system.

The main limitations are those of its design goals. MGNet prioritizes **real-time efficiency** over maximum single-task accuracy, so its PQ is behind large high-accuracy panoptic methods and its depth quality is behind the strongest specialized self-supervised depth models [2206.13199]. Its depth branch inherits the standard assumptions of photometric monocular training: mostly static scenes, camera motion, limited occlusions, and scale ambiguity unless additional constraints are added. Absolute scale recovery depends on **ground geometry** and **known camera height**, making it dataset- and camera-dependent. On KITTI, panoptic supervision is based on **pseudo labels**, which may cap segmentation quality. Finally, except for the road-mask usage in DGC, most segmentation–depth interaction remains implicit [2206.13199].

Taken together, MGNet is best understood as a **bottom-up panoptic mask generator fused with a self-supervised monocular depth estimator** under a shared low-latency encoder–decoder design [2206.13199]. Its distinctive contribution is not merely that it predicts masks and depth at once, but that it turns panoptic road masks into a geometric prior for absolute depth scaling, thereby linking semantic scene parsing and monocular geometry within a single real-time framework.

Source: https://www.emergentmind.com/topics/mask-generation-network-mgnet