Sparse Deformable Multi-Scale Detector

• SDMSD is a detector architecture that integrates multi-scale feature extraction, dense proposal generation, and deformable transformer attention for robust object detection.
• It combines classical anchor-based methods with transformer sparsification to enhance recall for small, densely packed objects while optimizing efficiency.
• The framework is validated on industrial tasks like X-ray NDT, achieving high precision and real-time performance through adaptive non-maximum suppression and set-based training.

The Sparse Deformable Multi-Scale Detector (SDMSD) is a detector architecture that integrates dense multi-scale proposal generation, non-maximum suppression-based sparsification, and deformable transformer attention mechanisms for end-to-end object detection. SDMSD unifies classical anchor-based detection concepts with the computational and convergence advantages of recent deformable and sparse transformer approaches. It achieves high recall for small, densely packed objects and is optimized for high-resolution and real-time industrial inspection tasks, notably X-ray non-destructive testing (NDT) (Liu et al., 20 Jul 2025, Zhu et al., 2020, Roh et al., 2021).

1. Architectural Overview

SDMSD is composed of several intertwined stages:

• Backbone and Multi-Scale Feature Pyramid: A convolutional backbone (e.g., ResNet-50) extracts feature maps at multiple resolutions (C₂–C₅). An FPN merges these channels to form a hierarchy $\{X^1, ..., X^L\}$ of $L$ multi-scale feature maps, capturing both small (high-resolution) and large (low-resolution) structures.
• Dense Proposal Generation: On each $X^l$ feature map, a 3×3 convolutional head predicts $K$ anchor-based bounding box offsets $\{\delta x, \delta y, \delta w, \delta h\}$ and a confidence score $c$ per spatial position, yielding an initial dense set $B_\text{dense}$ of candidate regions.
• Non-Maximum Suppression (NMS) Sparsification: Standard IoU-based NMS prunes highly overlapping, lower-confidence boxes from $B_\text{dense}$, producing a much smaller set $S_\text{sparse}$ of high-confidence, non-overlapping proposals.
• Transformer Encoder–Decoder with Deformable Attention: $S_\text{sparse}$ defines the initial set of object queries for a multi-layer transformer. Each decoder layer applies multi-scale deformable cross-attention, wherein each query dynamically attends to a small, learned set of offsets around its reference point at each feature scale, rather than densely attending to the entire feature map.
• Set-Based Hungarian Training: Final box/class outputs are matched one-to-one with ground truth using Hungarian assignment, training with a weighted sum of classification, $\ell_1$, and generalized IoU losses.

2. Mathematical Formulation and Workflow

Dense Proposal Generation

For each location $(i, j)$ in map $X^l$ of size $H^l \times W^l$, $K$ anchors are predicted:

$$b' = (x_a + \delta x,\, y_a + \delta y,\, w_a \cdot \exp(\delta w),\, h_a \cdot \exp(\delta h)),$$

where $(x_a, y_a, w_a, h_a)$ is the anchor parameterization, and $(\delta x, \delta y, \delta w, \delta h)$ are network outputs.

All candidate boxes form $B_\text{dense} = \{ b_n \}_{n=1}^{n^D}$, where $n^D \approx \sum_l H^l W^l K$, each with confidence $c_n \in [0,1]$.

NMS Sparsification

NMS selects the subset

$$S_\text{sparse} = \text{NMS}(B_\text{dense}, \tau_{\text{IoU}}) = \{ b_i \in B_\text{dense}\ :\ \forall b_j \in B_\text{dense}, c_j > c_i\ \Rightarrow\ \text{IoU}(b_i, b_j) < \tau_{\text{IoU}} \},$$

typically with $\tau_\text{IoU}=0.5$. Pseudocode involves sorting by confidence and greedily discarding boxes with IoU above threshold.

Deformable Transformer Decoding

Decoder layers refine the proposals in $S_\text{sparse}$ by attending to up to $K$ sampled points per feature level $L$, per attention head, using bilinear interpolation at learned offsets around per-query reference points.

The attention computation for a query $f_q$ and normalized reference $p̂_q$:

$$\mathrm{Attn}(f_q, p̂_q, \{x^l\}) = \sum_{m=1}^M W_m \left[ \sum_{\ell=1}^L \sum_{k=1}^K A_{m\ell qk} \cdot W_m' x^l(\phi_\ell(p̂_q) + \Delta_{m\ell qk}) \right]$$

where $\phi_\ell(p̂_q)$ maps normalized reference into map $l$, and $A_{m\ell qk}$ are learned attention weights.

3. Computational Complexity and Efficiency

SDMSD’s key efficiency derives from two levels of sparsification:

1. Proposal Sparsification: Reduces the number of queries passed to the transformer from tens of thousands (dense proposals) to a few hundred via NMS, minimizing the workload in the expensive transformer decoder.
2. Deformable Attention: For each decoder query, attends only to $K \ll (HW)$ points at each scale, yielding sub-linear complexity in feature map size:
   • Standard attention: $O((HW)^2)$
   • Deformable attention: $O(\#\text{queries} \cdot \text{heads} \cdot K)$

This structure ensures near-real-time throughput, even on high-resolution inputs.

4. Empirical Performance and Ablation Evidence

On the GDXray+ defect detection benchmark, SDMSD achieves:

Metric	Overall	Small (<$32^2$)	Medium ($32^2\leq$area$<96^2$)	Large ($\geq96^2$)
Precision	92.62%	90.94%	93.34%	94.70%
Recall	98.99%	98.57%	98.61%	100.00%
F1-score	95.57%	--	--	--

Replacement of the dense proposal+NMS with direct deformable DETR queries lowers F1 by approximately 10 points, highlighting the critical role of dense-to-sparse proposal generation in hard small-object settings (Liu et al., 20 Jul 2025).

In ablation, increasing the number of sampling points per query from $K=4$ to $K=8$ yields significant gains, with diminishing returns beyond $K=8$. Disabling cross-level fusion (multi-scale attention) reduces recall for small objects by ~3 F1 points. Adjusting the NMS $\tau_\text{IoU}$ from 0.5 to 0.7 marginally raises recall but decreases precision due to more overlapping false positives; $\tau_{IoU}=0.5$ provides the best trade-off.

5. Connections to Deformable DETR and Sparse DETR

The SDMSD concept is instantiated both in the InsightX Agent (Liu et al., 20 Jul 2025) and as the core architectural advancement in Deformable DETR (Zhu et al., 2020) and further extended in Sparse DETR (Roh et al., 2021). These lines of work share:

• Multi-scale feature integration with dynamic deformable attention.
• Encoder/decoder sparsification mechanisms: Deformable DETR restricts attention per-query via learned offsets. Sparse DETR introduces a learnable scoring network to select which encoder tokens participate in attention, reducing encoder computational cost by up to 82% and boosting throughput by over 40% while retaining or improving AP in COCO tests.

A summary table contextualizing the computational trade-offs:

Model	Encoder FLOPs Reduction	FPS Improvement	COCO AP ($\rho$=10%)
Deformable DETR+	baseline	baseline	46.0
Sparse DETR ($\rho=10\%$)	–41%	+39%	45.3
Sparse DETR ($\rho=30\%$)	–32%	+28%	46.0

Auxiliary detection losses on sparsified encoder tokens, and DAM-based scoring for token selection, further improve training stability and performance (Roh et al., 2021).

6. Application in Large Multimodal and Agentic Frameworks

In InsightX Agent (Liu et al., 20 Jul 2025), SDMSD serves as the core defect proposal engine within a broader LMM-based agentic framework for X-ray NDT analysis. Dense proposals from SDMSD are validated and refined by an Evidence-Grounded Reflection (EGR) tool, implementing a chain-of-thought quality assurance loop. Integration of SDMSD with interpretability and self-assessment modules addresses practical demands for operator trust and diagnostic transparency in critical inspection domains.

7. Limitations and Future Directions

While SDMSD delivers marked efficiency and accuracy gains, performance in settings with extreme object density, heavy occlusion, or challenging domain transfer remains an area of ongoing research. Effective setting of sparsification (NMS thresholds, number of sampling points, sparsity ratio in Sparse DETR) is task-dependent, and hyper-parameter choice significantly affects the precision–recall trade-off. Further improvements may be realized via adaptive or learned proposal generation, differentiable NMS, and contextual feature integration in downstream decision frameworks.

---

SDMSD constitutes a robust detector paradigm for dense and small-object detection, combining classical proposals with deformable and sparsity-aware transformer processing. Its theoretical efficiency and empirical effectiveness are substantiated on both industrial (e.g. X-ray NDT) and generic benchmarks, and it provides a template for future developments in scalable visual reasoning systems (Liu et al., 20 Jul 2025, Zhu et al., 2020, Roh et al., 2021).