MODNet: Real-Time Trimap-Free Portrait Matting via Objective Decomposition (2011.11961v4)

Abstract: Existing portrait matting methods either require auxiliary inputs that are costly to obtain or involve multiple stages that are computationally expensive, making them less suitable for real-time applications. In this work, we present a light-weight matting objective decomposition network (MODNet) for portrait matting in real-time with a single input image. The key idea behind our efficient design is by optimizing a series of sub-objectives simultaneously via explicit constraints. In addition, MODNet includes two novel techniques for improving model efficiency and robustness. First, an Efficient Atrous Spatial Pyramid Pooling (e-ASPP) module is introduced to fuse multi-scale features for semantic estimation. Second, a self-supervised sub-objectives consistency (SOC) strategy is proposed to adapt MODNet to real-world data to address the domain shift problem common to trimap-free methods. MODNet is easy to be trained in an end-to-end manner. It is much faster than contemporaneous methods and runs at 67 frames per second on a 1080Ti GPU. Experiments show that MODNet outperforms prior trimap-free methods by a large margin on both Adobe Matting Dataset and a carefully designed photographic portrait matting (PPM-100) benchmark proposed by us. Further, MODNet achieves remarkable results on daily photos and videos. Our code and models are available at https://github.com/ZHKKKe/MODNet, and the PPM-100 benchmark is released at https://github.com/ZHKKKe/PPM.

• The paper introduces MODNet, a trimap-free approach that decomposes portrait matting into semantic estimation, detail prediction, and fusion branches for robust real-time performance.
• It employs efficient atrous spatial pyramid pooling and self-supervised constraints to reduce computation and mitigate domain shift, achieving 67 fps on standard GPUs.
• The model outperforms prior methods on the Adobe Matting and PPM-100 benchmarks, and its open-source release supports a wide range of practical applications.

Analysis of MODNet: Real-Time Trimap-Free Portrait Matting via Objective Decomposition

The paper presents MODNet, a model designed for efficient and effective portrait matting without relying on auxiliary inputs such as trimaps. Traditional matting methods, which often require such additional inputs or involve complex multi-staged processing, present limitations for real-time applications. In contrast, MODNet introduces an innovative methodology by addressing matting through the simultaneous optimization of sub-objectives using explicit constraints.

Key Contributions and Techniques

MODNet's architecture is built around three branches: semantic estimation, detail prediction, and semantic-detail fusion. This decomposition of the matting process allows the model to handle portrait matting efficiently with a single RGB image input. The model's architecture leverages MobileNetV2 as its backbone, chosen for its lightweight and efficient design suitable for real-time applications.

Two novel techniques underpin the MODNet's efficiency:

1. Efficient Atrous Spatial Pyramid Pooling (e-ASPP): This module effectively fuses multi-scale features in a computationally efficient manner. By altering the standard ASPP structure, the e-ASPP significantly reduces computational overhead while maintaining performance.
2. Self-supervised Sub-objectives Consistency (SOC): Addressing the domain shift problem common in trimap-free methods, SOC adapts the model to real-world data without requiring annotated training data. It imposes self-supervised constraints among sub-objective predictions, enhancing generalization.

Performance and Evaluation

MODNet has demonstrated significant performance improvements over existing trimap-free matting methods. It operates at 67 frames per second on a GTX 1080Ti GPU, which underscores its suitability for real-time applications. The paper reports that MODNet surpasses previous methods on both the Adobe Matting Dataset and a newly proposed benchmark, PPM-100, which provides a diverse set of test images to challenge matting models more comprehensively than previous synthetic benchmarks.

The model's robustness extends to daily photos and videos, with its code and models being made publicly available. This open-source approach allows for broader validation and integration into various applications.

Implications and Future Directions

Practically, MODNet holds potential for real-time applications like camera previews or video conferencing where computational resources and latency are critically constrained. Theoretically, the decomposition of a complex objective into simpler sub-objectives for simultaneous optimization might inspire similar approaches in other domains of AI.

Future research could investigate the incorporation of temporal information to handle videos with strong motion blurs, a limitation mentioned for MODNet. Additionally, further developments could explore the model's adaptability to other domains where trimap-free methods might be beneficial.

In summary, MODNet presents a meaningful advancement in trimap-free portrait matting, effectively balancing performance, efficiency, and applicability in real-world use cases. The insights from this research could influence continued innovation in both specific applications of matting and broader AI research methodologies.