DeepDetect: Unified Dense Keypoint Detector

• DeepDetect is a learned keypoint detector that integrates classical geometric features with semantic edge cues via deep learning, ensuring high density and repeatability.
• It employs multi-detector fusion by combining outputs from seven keypoint and two edge detectors to generate rich, binary supervision masks capturing diverse image structures.
• Its ESPNet-based architecture and rigorous training protocol yield superior performance across metrics, demonstrating robustness under degradation and reduced background noise.

DeepDetect is a learned, all-in-one dense keypoint detector that unifies the geometric strengths of classical keypoint detectors with the semantic awareness afforded by edge detectors via deep learning. Designed to address the limitations of both traditional and learning-based methods, DeepDetect produces extremely dense, repeatable, and semantically focused keypoints, demonstrating robustness under photometric degradation and providing superior performance across standard keypoint detection benchmarks (Tareen et al., 20 Oct 2025).

1. Background and Motivations

Keypoint detection underpins many computer vision tasks, including image registration, structure-from-motion, 3D reconstruction, visual odometry, and SLAM. Classical detectors such as SIFT, SURF, ORB, BRISK, FAST, and AGAST are highly robust to covariant transformations (scale, rotation, affine blur) but typically yield sparse keypoints, fail in low-contrast or degraded conditions, and require manual threshold tuning. Moreover, these methods lack semantic priors, treating all corners and blobs equally, which can result in background clutter and omission of visually important structures.

Learning-based detectors (e.g., SuperPoint, R2D2, LF-Net, D2-Net) improve keypoint repeatability but generally remain sparse and do not incorporate explicit multi-cue or semantic supervision, leading to suboptimal performance in cluttered or low-visibility images. These limitations motivate the design of DeepDetect: a detector with dense semantic focus, robustness to severe degradations, and unified supervision from both classical and edge-based cues.

2. Ground-Truth Mask Generation via Multi-Detector Fusion

DeepDetect employs a fusion strategy to generate rich binary supervision masks for training. Given an input image $I \in \mathbb{R}^{H \times W \times 3}$, outputs from seven classical keypoint detectors—SIFT, ORB, BRISK, FAST, AGAST, Harris, and Shi-Tomasi—are each converted to binary masks $M_d(I) \in \{0,1\}^{H \times W}$. Outputs from two edge detectors (Canny, Sobel) similarly yield $M_e(I)$. The final supervision mask is computed as a pixel-wise logical OR:

$$M(I) = \bigvee_{d \in D} M_d(I) \;\vee\; \bigvee_{e \in E} M_e(I)$$

where $D$ and $E$ are the sets of keypoint and edge detectors, respectively. This unified mask supervises the network to focus on a wide array of structural and textural cues, including corners, blobs, ridges, and edges, without any manual annotation and across a range of degradation settings.

3. ESPNet-Based Model Architecture

The DeepDetect detector uses the ESPNet architecture as its backbone for efficient pixel-wise keypoint probability prediction. ESPNet features an encoder–decoder design, with the encoder consisting of an initial $3\times3$ convolution (16 channels), followed by a stack of ESP modules:

• Each ESP module applies a $1\times1$ pointwise convolution, splits features into $k$ branches, and processes each branch through a $3\times3$ dilated convolution (dilation rates $\{1,2,\ldots,k\}$).
• Outputs are concatenated and fused with a residual connection.
• Encoder down-samples spatially via strided ESP modules to $1/32$ resolution.

The decoder employs hierarchical up-sampling (nearest-neighbor or learnable deconvolution), with skip connections to the encoder, and outputs a single-channel logits map $z \in \mathbb{R}^{H \times W}$. The total model size is approximately $1.82$ MB.

4. Training Protocol and Loss Optimization

DeepDetect is trained on a dataset comprising 41,000 images from MS-COCO and NewTsukuba, with 25% undergoing synthetic brightness and contrast degradation. The dataset is split into 33,000 training and 8,000 validation images, all at $480 \times 480$ resolution. The network is optimized using the Adam algorithm ($\beta_1=0.9$, $\beta_2=0.999$), an initial learning rate of $10^{-3}$, cosine-annealing schedule, batch size of 64, and 100 epochs.

The loss function is pixel-wise binary cross-entropy (BCE):

$$L(y, z) = -\frac{1}{N} \sum_{i=1}^{N} \left[ y_i \log \sigma(z_i) + (1 - y_i)\log(1 - \sigma(z_i)) \right]$$

where $y = M(I) \in \{0, 1\}^N$ is the supervision mask, $z$ are logits, $N = H \cdot W$, and $\sigma$ is the sigmoid. No focal loss or explicit regularization is used; the model checkpoint with minimum validation loss is selected.

5. Keypoint Extraction and Inference Workflow

During inference, the network outputs per-pixel logits $z_p$, converted to probabilities $P_p = \sigma(z_p)$. A fixed threshold $\tau$ (default 0.5) yields a binary mask $\hat{M}(I)_p$:

$$\hat{M}(I)_p = \begin{cases} 1 & \text{if } P_p \geq \tau \ 0 & \text{otherwise} \end{cases}$$

Non-maximum suppression (e.g., $3 \times 3$ window) is applied to the probability map to extract local maxima as discrete keypoints, each assigned a confidence score $P_p$. This method ensures density and semantic concentration of keypoints, as demonstrated by both qualitative and quantitative analyses.

6. Quantitative Evaluation and Comparative Performance

DeepDetect's performance is evaluated on the Oxford Affine Covariant Regions dataset across several key metrics:

Detector	Avg. Keypoint Density $\bar{\rho}$	Avg. Repeatability $\bar{R}$	Total Correct Matches	F-KP Ratio
DeepDetect	0.5143	0.9582	$\approx$59,003	0.7093
SIFT (def)	$\approx$0.0090	$\approx$0.75	$\sim$6,000	-
SIFT (low)	$\approx$0.1993	$\approx$0.58	$\sim$16,000	-

Metrics definitions:

• Keypoint Density $\rho$: $N/(H \cdot W)$, where $N$ is the number of detected keypoints.
• Repeatability $R$: $N_{A \cap B} / \min(N_A, N_B)$ for keypoints matched under homography.
• F-KP Ratio: $N_F / N_T$, measuring proportion of keypoints in foreground regions.
• Correct Matches: Number of descriptor matches within 1 pixel of ground-truth correspondence.

DeepDetect significantly outperforms both classical and prior learning-based detectors in terms of density, repeatability, and match count, with robustness to photometric degradation and minimal background noise.

7. Qualitative Properties, Limitations, and Future Prospects

DeepDetect produces very dense keypoints concentrated on semantically salient structures (e.g., road signs, building edges, textured facades), and maintains robustness under adverse conditions (low-light, fog, smoke) without parameter adaptation. Background regions exhibit low keypoint noise (high F-KP).

Notable limitations include potential overproduction of keypoints in textureless areas due to lighting artifacts, increased false correspondences in highly repetitive patterns, and dependence on a fixed threshold $\tau$ that governs the density-precision trade-off. Application-specific settings may benefit from adaptive thresholding or learnable non-maximum suppression.

Proposed future directions include integrating a learned descriptor head for end-to-end matching, enhancing thresholding/NMS strategies, and extending robustness to dynamic scenes (Tareen et al., 20 Oct 2025).