Towards Light-weight and Real-time Line Segment Detection (2106.00186v3)

Abstract: Previous deep learning-based line segment detection (LSD) suffers from the immense model size and high computational cost for line prediction. This constrains them from real-time inference on computationally restricted environments. In this paper, we propose a real-time and light-weight line segment detector for resource-constrained environments named Mobile LSD (M-LSD). We design an extremely efficient LSD architecture by minimizing the backbone network and removing the typical multi-module process for line prediction found in previous methods. To maintain competitive performance with a light-weight network, we present novel training schemes: Segments of Line segment (SoL) augmentation, matching and geometric loss. SoL augmentation splits a line segment into multiple subparts, which are used to provide auxiliary line data during the training process. Moreover, the matching and geometric loss allow a model to capture additional geometric cues. Compared with TP-LSD-Lite, previously the best real-time LSD method, our model (M-LSD-tiny) achieves competitive performance with 2.5% of model size and an increase of 130.5% in inference speed on GPU. Furthermore, our model runs at 56.8 FPS and 48.6 FPS on the latest Android and iPhone mobile devices, respectively. To the best of our knowledge, this is the first real-time deep LSD available on mobile devices. Our code is available.

• The paper introduces Mobile LSD (M-LSD), a novel light-weight and real-time line segment detector designed for efficient deployment on resource-constrained devices like mobile phones.
• M-LSD achieves significant architectural efficiency with a minimized backbone and simplified processing, enabling inference speeds of up to 200.8 FPS on GPUs and outperforming existing slender models.
• The research incorporates innovative training schemes including Segments of Line (SoL) augmentation and geometric loss functions to enhance performance and leverage auxiliary geometric data for its lightweight architecture.

Light-weight and Real-time Line Segment Detection with M-LSD

This paper introduces Mobile LSD (M-LSD), a real-time and light-weight line segment detector specifically designed for resource-constrained environments. The paper provides detailed insights into the architectural decisions that enable significant improvements in efficiency and performance metrics, paving the way for practical applications in mobile settings.

Architectural Efficiency for Real-time Application

The researchers addressed limitations typically encountered in previous deep learning-based line segment detection methods, including substantial model size and computational cost. M-LSD achieves a reduction in complexity without compromising competitive performance by minimizing the network backbone and eliminating the multi-module processing traditionally used for line prediction. This architectural simplification is pivotal, allowing inference speeds of up to 200.8 FPS on GPUs, outperforming existing slender models such as TP-LSD-Lite by a notable margin.

Novel Training Strategies

M-LSD incorporates innovative training schemes designed to augment the performance of its lightweight architecture. Key contributions include Segments of Line segment (SoL) augmentation and matching and geometric loss functions.

1. SoL Augmentation: This technique enhances the training dataset by dividing line segments into multiple subparts, enabling the model to assimilate more auxiliary geometric data. This strategy significantly aids in overcoming challenges related to line segment lengths exceeding receptive field sizes or closely situated centers of distinct line segments.
2. Matching and Geometric Loss: The introduction of these loss functions allows the model to leverage additional geometric cues. Matching loss optimizes the alignment of predicted line segments with ground truth by considering geometric relationships, while geometric loss aids in refining feature map training by incorporating junction and line segmentation, and length and degree regression.

Performance and Deployment Metrics

M-LSD's efficacy is demonstrated through extensive evaluations on the Wireframe and YorkUrban datasets, where it achieves competitive performance benchmarks across several metrics (e.g., F$^H$, sAP$^{10}$, LAP). Impressively, with only 2.5% of the model size compared to TP-LSD-Lite, M-LSD-tiny delivers significant speed enhancements.

Moreover, the paper highlights M-LSD's operational feasibility in real-world environments by successfully deploying it on Android and iPhone devices, showcasing resource-efficient inference speeds without extensive memory requirements. This marks a notable milestone, with M-LSD variants being the first reported real-time LSD suitable for mobile applications.

Implications and Future Prospects

The implications of this research extend to real-world applications in mobile and embedded vision systems, facilitating fast and efficient line segment detection. This enables practical deployment scenarios such as real-time image processing on mobile platforms for tasks including pose estimation, 3D reconstruction, and image rectification.

Further exploration into expanding the utility of the training schemes with existing methods suggests potential collaboration strategies across various line detection models. The flexibility and scalability of M-LSD positions it as a candidate for integration into broader computer vision tasks that require efficient resource management and rapid inference capabilities.

In conclusion, M-LSD's development marks a significant advancement in the field of real-time line segment detection, offering an optimized solution that balances model size and computational demands with performance excellence. Future developments may explore further refinement of its training strategies or broader applications beyond mobile device implementations.