PP-HumanSeg: Connectivity-Aware Portrait Segmentation with a Large-Scale Teleconferencing Video Dataset (2112.07146v1)

Abstract: As the COVID-19 pandemic rampages across the world, the demands of video conferencing surge. To this end, real-time portrait segmentation becomes a popular feature to replace backgrounds of conferencing participants. While feature-rich datasets, models and algorithms have been offered for segmentation that extract body postures from life scenes, portrait segmentation has yet not been well covered in a video conferencing context. To facilitate the progress in this field, we introduce an open-source solution named PP-HumanSeg. This work is the first to construct a large-scale video portrait dataset that contains 291 videos from 23 conference scenes with 14K fine-labeled frames and extensions to multi-camera teleconferencing. Furthermore, we propose a novel Semantic Connectivity-aware Learning (SCL) for semantic segmentation, which introduces a semantic connectivity-aware loss to improve the quality of segmentation results from the perspective of connectivity. And we propose an ultra-lightweight model with SCL for practical portrait segmentation, which achieves the best trade-off between IoU and the speed of inference. Extensive evaluations on our dataset demonstrate the superiority of SCL and our model. The source code is available at https://github.com/PaddlePaddle/PaddleSeg.

• The paper introduces PP-HumanSeg with a large-scale teleconferencing video dataset and a novel Semantic Connectivity-aware Learning framework.
• The paper presents ConnectNet, an ultra-lightweight encoder-decoder model optimized for low latency and high segmentation accuracy using depth-wise separable convolutions.
• The paper demonstrates significant performance improvements, achieving high mIoU and real-time operation on resource-limited systems for teleconferencing.

PP-HumanSeg: Connectivity-Aware Portrait Segmentation for Teleconferencing

The paper "PP-HumanSeg: Connectivity-Aware Portrait Segmentation with a Large-Scale Teleconferencing Video Dataset" addresses the demand for effective portrait segmentation in the context of video conferencing, a need that has become pronounced with the widespread move towards remote communication. As video conferencing becomes a staple, the ability to segment and replace backgrounds in real time is crucial, particularly when participants join from diverse environments with non-uniform lighting and backgrounds.

Introduction to PP-HumanSeg

The authors introduce an open-source solution, PP-HumanSeg, marked by three main contributions:

1. A large-scale dataset specifically geared towards video conferencing scenarios.
2. A novel Semantic Connectivity-aware Learning (SCL) framework.
3. An ultra-lightweight model termed ConnectNet.

These contributions aim to enhance the performance of portrait segmentation under practical resource constraints typical in teleconferencing setups.

Dataset Construction

A significant aspect of this work is the creation of a comprehensive dataset comprising 291 videos across 23 teleconferencing scenarios, including 14,117 densely annotated frames. This dataset includes real-world variability, such as participants wearing masks, varying postures, and diverse actions, capturing authentic teleconferencing environments. This richness in data addresses deficiencies in previous datasets, which either lack quality or contain redundant synthesized images unsuitable for this domain.

Semantic Connectivity-aware Learning (SCL)

The authors propose SCL to improve segmentation by incorporating a novel Semantic Connectivity (SC) loss. This loss function evaluates the connectivity between predicted and actual segmentation results, emphasizing completeness and consistency over traditional pixel-level classification methods. The SC loss functions as a regularizer, complementing traditional segmentation losses to enhance the model's ability to maintain the integrity of segmented person instances. This is especially beneficial for maintaining coherent segmentation when dealing with non-rigid objects like human figures in dynamic video conferencing environments.

ConnectNet: An Ultra-lightweight Solution

ConnectNet, the ultra-lightweight model proposed in this research, employs an encoder-decoder architecture optimized for low latency and high performance. It integrates depth-wise separable convolution and skip connections to ensure efficient computation and accurate segmentation in resource-limited settings. Remarkably, ConnectNet achieves a balance between inference speed and segmentation accuracy, outperforming other state-of-the-art models in terms of the trade-off between Mean Intersection over Union (mIoU) and processing time.

Experimental Validation and Impact

The empirical results highlight the superiority of the proposed approach. Extensive evaluations underline that the SCL framework and ConnectNet model improve segmentation accuracy, achieving a high mIoU while operating efficiently on limited computational resources. The lightweight nature and robust performance of ConnectNet position it as a viable solution for real-time applications in video conferencing scenarios.

Implications and Future Directions

The research introduces foundational elements for further exploration in AI-assisted video teleconferencing. By addressing the unique challenges posed by this new norm, the paper sets a benchmark for future datasets and segmentation methods. Subsequent research could expand the dataset to include more diverse environments or further optimize segmentation approaches for even faster real-time performance. The integration of additional contextual data, like audio cues or more advanced machine learning techniques, could also enhance segmentation robustness in various conditions.

In conclusion, the PP-HumanSeg framework represents an important advancement in video conferencing technology, providing an open-source tool that combines practical applicability with cutting-edge research in semantic segmentation. This work is poised to significantly influence future developments in AI-driven communication technologies. The dataset and source code availability encourage ongoing collaboration and innovation in this vital field.