Unified and Efficient Point-Line Local Features

Abstract: Multi-view computer vision pipelines typically rely on accurate sparse keypoints and robust descriptors. While incorporating line features has shown clear benefits for matching and pose estimation, existing point-line approaches remain inefficient: they detect points and lines separately, use increasingly heavy networks, and depend on CPU-bound heuristics that hinder real-time performance. We introduce a Unified Efficient Points and Lines (UPAL) feature extractor that jointly extracts keypoints, line segments, and feature descriptors within a single lightweight architecture. A shared backbone provides common representations that feed different branches for point and line features. Line segments are recovered through an accelerated post-processing stage, an enhanced and highly efficient variant of the LSD algorithm. UPAL matches or exceeds state-ofthe-art performance in both point and line applications while significantly reducing computational cost, achieving, for instance, a 4x speedup and 10x smaller memory footprint over the ALIKED + DeepLSD pipeline. Code is publicly available at https://github.com/francois141/upal.

• The paper introduces UPAL, a 0.78M-parameter network that jointly extracts keypoints, 128-dimensional descriptors, and line segments using shared features and teacher-based distillation.
• The paper shows that GPU-accelerated line processing cuts post-processing latency from 233 ms to 11 ms, while the full pipeline reaches 70 ms—4× faster than an ALIKED plus DeepLSD system.
• The paper demonstrates strong results across homography estimation, pose estimation, localization, and reconstruction, including 75.0% overall points-and-lines accuracy on 7Scenes, while identifying line matching and descriptor quality as remaining limitations.

Motivation and problem statement

Multi-view geometric pipelines—feature matching, visual localization, SLAM, and 3D reconstruction—increasingly combine point and line features, since lines are prevalent in man-made environments and complement points in low-texture scenes. Existing point-line systems, however, are computationally unattractive: they run two separate extractors (e.g., ALIKED for points plus DeepLSD for lines), rely on increasingly large backbones, and depend on CPU-bound heuristics such as the LSD line segment detector, whose cost grows quadratically with image size. The paper introduces UPAL (Unified Efficient Points and Lines), a single lightweight network that jointly predicts keypoints, line segments, and descriptors in one forward pass, together with a GPU-accelerated variant of LSD for line extraction.

Architecture

UPAL adopts the ALIKED-n16 encoder as its shared backbone: four convolutional blocks with deformable convolutions in the last two stages, whose outputs are upsampled and concatenated into a bottleneck feature map. Three lightweight heads operate on this bottleneck:

• Keypoint branch: a Score Map Head followed by Differentiable Keypoint Detection (DKD) with NMS and softargmax subpixel refinement.
• Descriptor branch: the Sparse Deformable Descriptor Head (SDDH) from ALIKED, producing 128-dimensional descriptors via learned deformable sampling.
• Line branch: only three convolutional layers predicting a line distance field; the angle field is computed directly from image gradients on the GPU rather than predicted by the network.

The design rests on an empirical observation that keypoint score maps and line distance fields are strongly correlated—keypoints frequently lie at line endpoints and intersections—justifying a shared representation. The full model has only 0.78M parameters, roughly 10× fewer than DeepLSD's U-Net backbone alone.

Distillation-based training

Rather than training from scratch, UPAL distills the best-performing component of each prior method. After evaluating detectors and descriptors in isolation, the authors select SuperPoint and DaD as keypoint teachers (their score maps are fused element-wise via maximum), ALIKED-n32 as the descriptor teacher, and DeepLSD as the line teacher. Training uses a weighted binary cross-entropy loss for keypoints ($\lambda = 200$), a normalized L1 loss restricted to pixels within radius $r=5$ of ground-truth lines for the distance field, and an L1 descriptor loss over the top 1000 keypoints. Training is efficient: roughly one hour for the line branch and twelve hours for the remaining components on four TITAN GPUs.

A notable finding is that joint training improves the point features themselves: on MegaDepth relative pose estimation, joint point-line training yields +3.0 to +3.2 AUC over a points-only variant of the same architecture, indicating that line supervision regularizes the backbone toward more geometrically consistent keypoints.

Accelerated LSD post-processing

The paper contributes three modifications to the LSD pipeline inherited from DeepLSD. First, removing the learned angle-field head both simplifies the network and improves accuracy—the ablation shows that DeepLSD without angle prediction achieves lower localization error (1.341 vs. 1.432) and higher repeatability than vanilla DeepLSD, because the predicted angle field degrades under challenging illumination such as RDNIM's day-night pairs. Second, gradient computation and image preprocessing are moved to the GPU, leaving only line growing on CPU. Third, seed points are subsampled with stride 2 and filtered to the top 20% of lowest distance-field values, cutting latency by roughly 3× with no measurable performance loss. The combined effect reduces post-processing latency from 233 ms (DeepLSD) to 11 ms while slightly improving repeatability (28.9 vs. 27.2).

Point evaluation

On HPatches homography estimation, UPAL attains the best results among lightweight methods (39.2 / 67.9 / 77.7 AUC at 1/3/5 px) at 62 ms latency, outperforming SuperPoint, XFeat, Wireframe, and PLNet, and approaching heavyweight baselines such as DeDoDe v2. On MegaDepth it ranks second among all methods (58.2 AUC@5°) behind DaD+DeDoDe v2, at comparable latency. On ScanNet indoor pose estimation, UPAL again ranks near the top of the lightweight group (15.6 AUC@5°). The authors attribute the strong indoor performance to supervising keypoints with corner-oriented teachers (SuperPoint, DaD) rather than ALIKED's own detector, which underperforms indoors, and to joint line supervision encouraging stable keypoints along structures.

Line evaluation

Despite its small line branch (~780k parameters), UPAL achieves the highest repeatability across HPatches and RDNIM among all compared methods—including LSD, ELSED, SOLD2, M-LSD, DeepLSD, Linea, ScaleLSD, Wireframe, and PLNet—and often ranks second in localization error and homography estimation. On RDNIM it surpasses its own teacher DeepLSD (repeatability 13.3 vs. 10.1 at 1 px), which the authors attribute to discarding the angle field. This supports the claim that generic line detection is a low-level task solvable with very few parameters, in contrast to capacity-heavy approaches like ScaleLSD.

Downstream tasks

For visual localization on the Stairs scene of 7Scenes using hloc and LIMAP, UPAL achieves the best result in both categories: 49.1% pose accuracy at 5 cm / 5° with points alone, and 54.6% with points and lines, exceeding SuperPoint+DeepLSD (49.6%) and Wireframe (53.8%). Across all seven 7Scenes scenes, UPAL obtains the best overall points+lines accuracy (75.0%), though PLNet and SuperPoint+DeepLSD remain stronger on individual scenes such as Heads and Office where long structural lines dominate. On ETH3D multi-view triangulation, UPAL exceeds SuperPoint, ALIKED, and Wireframe in both accuracy and completeness, but falls behind DISK, which the authors report transparently.

Efficiency

The headline efficiency claim is a 4× speedup and 10× smaller memory footprint relative to the ALIKED + DeepLSD pipeline: 70 ms versus 286 ms GPU latency on an RTX 2080 Ti, with 0.78M versus 9.2M parameters. Compared to prior joint extractors, UPAL has an order of magnitude fewer parameters than Wireframe (5.8M) and PLNet (7.5M) and 37% lower GPU latency. On a low-end GTX 1050 Ti, UPAL runs in 186 ms—about 18× faster than DaD + DeDoDe v2 + ScaleLSD (3474 ms)—demonstrating that heavyweight modern detectors degrade disproportionately on constrained hardware, whereas UPAL remains viable for embedded deployment.

Limitations and open questions

The paper concedes several limitations. Line matching relies solely on endpoint descriptors fed through mutual nearest neighbors and Sinkhorn assignment, without a dedicated line descriptor or learned matcher; qualitative results show that viewpoint-varying HPatches pairs matched reliably by points are almost entirely mismatched as lines, because LSD segment endpoints shift under perspective change. The authors leave integration with a lightweight jointly learned point-line matcher to future work. Additionally, the ETH3D triangulation gap behind DISK indicates that distilled descriptors do not uniformly match the strongest specialized models, and the visual localization advantage over PLNet and SuperPoint+DeepLSD is scene-dependent rather than universal. Whether the distillation ceiling can be raised—for example by adding stronger teachers without sacrificing the 0.78M parameter budget—remains open.

Conclusion

UPAL demonstrates that state-of-the-art point and line feature extraction can be unified in a single 0.78M-parameter network trained by distilling complementary teachers, with a GPU-accelerated LSD post-processing stage reducing line extraction latency by more than an order of magnitude. It matches or exceeds specialized baselines across homography estimation, relative pose, line repeatability, localization, and reconstruction benchmarks while delivering a 4× end-to-end speedup over the strongest modular pipeline, making joint point-line perception practical on embedded hardware.