Interacted Planes Reveal 3D Line Mapping

Abstract: 3D line mapping from multi-view RGB images provides a compact and structured visual representation of scenes. We study the problem from a physical and topological perspective: a 3D line most naturally emerges as the edge of a finite 3D planar patch. We present LiP-Map, a line-plane joint optimization framework that explicitly models learnable line and planar primitives. This coupling enables accurate and detailed 3D line mapping while maintaining strong efficiency (typically completing a reconstruction in 3 to 5 minutes per scene). LiP-Map pioneers the integration of planar topology into 3D line mapping, not by imposing pairwise coplanarity constraints but by explicitly constructing interactions between plane and line primitives, thus offering a principled route toward structured reconstruction in man-made environments. On more than 100 scenes from ScanNetV2, ScanNet++, Hypersim, 7Scenes, and Tanks&Temple, LiP-Map improves both accuracy and completeness over state-of-the-art methods. Beyond line mapping quality, LiP-Map significantly advances line-assisted visual localization, establishing strong performance on 7Scenes. Our code is released at https://github.com/calmke/LiPMAP for reproducible research.

• The paper introduces LiP-Map, a novel joint optimization framework that models 3D line segments as boundaries of planar patches to boost mapping accuracy and completeness.
• It employs robust geometric criteria, integrating angular consistency and orthogonal endpoint distance, and validates performance on datasets like ScanNetV2 and Tanks and Temples.
• The framework enhances visual localization by fusing 2D–3D line correspondences with point-based pipelines, achieving reliable pose accuracy and efficient scene representation.

Interacted Planes Reveal 3D Line Mapping: An Authoritative Summary

Introduction and Motivation

The presented work, "Interacted Planes Reveal 3D Line Mapping" (2602.01296), introduces LiP-Map, a novel line–plane joint optimization framework for 3D line mapping from multi-view RGB images. Addressing the deficiencies of canonical point-based and matching-based line reconstruction pipelines, this work posits that 3D line segments in man-made environments are best modeled as structural boundaries—edges of optimized planar patches—rather than isolated primitives triangulated from unreliable multi-view correspondences.

LiP-Map builds upon the emerging paradigm of structured scene reconstruction that leverages learnable geometric primitives, especially planes and lines, offering compact, symbolic, and topologically rich spatial representations. The framework explicitly models geometric and topological interactions between rectangular planar primitives and associated line segments, leading to significant improvements in reconstruction accuracy, completeness, and efficiency with runtime typically around 3–5 minutes per scene.

Line–Plane Joint Modeling and Optimization

The framework begins with posed multi-view images, where each input is supplemented by depth and normal maps provided by pretrained geometric foundation models (such as Metric3Dv2 and Omnidata) and dense 2D line segment detections from arbitrary detectors (e.g., DeepLSD, LSD, HAWPv3, ScaleLSD).

Each scene is parameterized by a set of learnable 3D planes, each with center, quaternion-based rotation (for orientation), and directional radii for flexible rectangular extents. Edges of these plane rectangles are used as candidate 3D lines. This representation enables natural alignment of reconstructed surfaces and their physical boundaries.

Figure 1: Overview of the line–plane joint optimization framework leveraging synergies between planes and lines in 3D.

2D line segments are robustly associated with candidate 3D plane edges based on two geometric criteria: angular consistency and orthogonal endpoint distance in the projected image space. Each pixel in a detected 2D line's 1-pixel region emits a ray and casts for the nearest intersected plane and its projected edges, thus providing robust assignments even for noisy or non-perfect initial plane arrangements.

Losses for joint optimization comprise plane rendering losses supervised by depth/normal maps, 2D alignment losses for endpoints and orthogonal deviations, and a group loss to encourage spatial coherence among 3D edges assigned to the same 2D line track. After convergence, strict geometric thresholds are used to select and finalize the reconstructed 3D line map, thus increasing reliability and filtering spurious lines.

Figure 2: Assignment process of a 2D line segment to the best matching 3D plane edge based on angle and distance criteria.

Experimental Evaluation and Quantitative Results

LiP-Map is evaluated on over 100 scenes from ScanNetV2, ScanNet++, Hypersim, 7Scenes, and Tanks and Temples. Extensive quantitative comparisons are made with prevailing methods: LIMAP (with/without depth), CLMAP, and other baselines. Benchmarks use both line- and junction-level metrics (accuracy, completeness, precision, recall, F-score) and line-track metrics (length recall, inlier percentage).

LiP-Map consistently yields substantially lower completeness error and higher recall/F1 scores, often reconstructing an order of magnitude more valid 3D lines for dense indoor scenes (e.g., 2,900–3,500 lines vs. ~200–600 for LIMAP/CLMAP on ScanNetV2). Length recall metrics also demonstrate robust coverage, and LiP-Map maintains high precision even as completeness is increased.

Ablation studies reveal that the hybrid loss design (endpoint, orthogonal, and group) and joint angle/distance assignment both contribute critically to optimal performance. The framework is robust to hyperparameter choices (plane splitting thresholds, initialization counts), view sparsity, and input variations.

Figure 3: Comparison of 3D line mapping results across methods and datasets. LiP-Map result is shown in the bottom row, capturing more structural line features.

Figure 4: Qualitative comparison on ScanNetV2, illustrating LiP-Map's superior coverage of structural boundaries.

Line-Assisted Visual Localization

LiP-Map demonstrates practical utility in visual localization tasks, where 2D–3D line correspondences are fused with classic point-based pipelines (e.g., COLMAP, HLoc, Pl2Map). Integration of LiP-Map's robust line primitives substantially improves pose accuracy and robustness, especially in textureless or repetitive regions where points alone are inadequate.

On the challenging 7Scenes benchmark, line-assisted pose estimation exhibits reduced median translation/rotation errors and heightened pose accuracy across scenes, sometimes outperforming prior point–line joint methods and learning-based relocalizers, verifying the meaningful geometric consistency achieved by line–plane optimization.

Figure 5: Line-assisted visual localization showing improved pose consistency by leveraging LiP-Map line correspondences.

Generalization and Scalability

While indoor, planar-dominated environments benefit most from this approach, experiments on non-planar and generic scenes (e.g., DTU, Tanks and Temples, BlenderMVS) demonstrate that LiP-Map, with dense plane initialization and strict assignment thresholds, can approximate curved and irregular structures via the union of small planar patches. However, reconstruction fidelity of highly non-planar objects is inherently limited; spurious planes are filtered out in the final mapping stage.

The framework scales efficiently with the number of views and input planes, exhibiting graceful runtime scaling and stable convergence. The line merging strategy (local/global DBSCAN clustering and PCA alignment) further consolidates redundant or fragmented lines, improving symbolic map compactness and overall quality.

Figure 6: Structure reconstruction and 3D line maps on scenes with curved or non-planar geometry.

Theoretical and Practical Implications

LiP-Map’s success exemplifies a shift toward hybrid geometric learning—using parameterized differentiable primitives and explicit geometric reasoning rather than implicit or brute-force neural field representations. It demonstrates that symbolic, structured scene models are compatible with the scale and realism of modern multi-view datasets, offering strong numerical results for scene abstraction, localization, and spatial intelligence.

This approach opens avenues for efficient vectorized representations, topological reasoning, architectural modeling, real-time mapping, and spatial AI—especially for robotics and AR. The principled coupling of planes and lines sets a foundation for future research in compact scene understanding, structured primitives, and hybrid optimization.

Limitations and Future Directions

Plane-based line representations are most suitable for man-made, architectural environments; generalization to highly curved or non-planar objects remains restricted. Redundant (duplicate) lines may arise from noise in 2D detection, and aggressive pruning or merging strategies are needed. Further refinements could include more expressive primitive sets (curves, higher-order patches), probabilistic topological reasoning, and integration with learned scene semantics.

Potential future developments involve scale-invariant hybrid models, self-supervised spatial priors, and fusion with neural radiance field methods for photometric consistency. Efficient structured reconstruction may spur advances in downstream tasks: map compression, navigation, object recognition, and semantic SLAM.

Conclusion

LiP-Map delivers a robust, efficient, and principled framework for 3D line mapping in structured environments, significantly outperforming previous methods in geometric accuracy, completeness, and topological fidelity. Its joint plane–line optimization and direct exploitation of physical scene structure set a new standard for symbolic 3D spatial representations, with broad implications for computer vision, robotics, and spatial AI. The released code and thorough benchmarking enable reproducible research and future extension.