Lightweight Downsample Projector for Dense Prediction

• LDP is a neural architecture search framework that uses a multi-scale backbone with block-level diversification to optimize dense prediction networks.
• It utilizes Assisted Tabu Search with memory-based mutations and proxy evaluation to efficiently explore the architecture space, reducing search time to about 4.3 GPU days.
• LDP achieves state-of-the-art performance across tasks such as monocular depth estimation, semantic segmentation, and image super-resolution while significantly reducing model parameters.

The Lightweight Downsample Projector (LDP) refers to a neural architecture search (NAS) framework designed for identifying compact, high-performing dense prediction networks, with a focus on tasks including monocular depth estimation, semantic segmentation, and image super-resolution. The LDP framework leverages a pre-defined multi-scale backbone and a novel Assisted Tabu Search (ATS) to efficiently explore and optimize architectures per task and parameter budget, achieving significant reductions in model size and computational demands while maintaining or exceeding state-of-the-art prediction accuracy (Huynh et al., 2022).

1. Architectural Foundations of LDP

LDP is grounded on a pre-defined generic backbone that is densely structured as a multi-scale pyramid, incorporating blocks such as encoder, decoder, refinement, and both downsampling and upsampling modules. Each block itself comprises several layers, with the layer type and structure selected from a diverse pool of candidate operations. Options include standard 2D convolutions, depthwise convolutions, inverted bottlenecks, micro-blocks, and varying kernel sizes, squeeze–excitation ratios, and skip connection types.

A distinguishing feature of the LDP framework is the allowance for "layer diversification" (Editor's term): rather than replicating an optimal cell throughout the network, LDP enables each block to be composed of different operation types and configurations, leading to architectures tailored for dense prediction tasks. The search is performed not over the macro-architecture but at the level of block-level operations and connections, constraining the space to balance expressiveness and practical search efficiency.

2. Assisted Tabu Search (ATS) Methodology

ATS introduces memory-based, iterative exploration for neural architecture optimization, inspired by classical tabu search. The process commences with an initial scoring and pre-selection of parent architectures. Children architectures are generated by applying compatible mutations such as layer swaps or operation replacements. The search is regulated via a “tabu list,” recording unproductive or recently explored paths to avoid local minima and repetitive evaluation.

Quantitatively, the reward for exploring from parent $m_i$ to child $m_j$ is defined by:

$$R(m_i, m_j) = \alpha \cdot \left(\frac{\text{score}(m_j)}{\text{score}(m_i)}\right) + (1 - \alpha) \cdot \left(\frac{P}{P(m_j)}\right)^r$$

where $P(m_j)$ is the parameter count of the child, $P$ the parameter budget target, $\alpha$ a hyperparameter balancing score and compactness, and $r = 0$ if $P(m_j) \leq P$, else $r = 1$. Each candidate's performance proxy, $\text{score}(\cdot)$, is computed with a kernel matrix derived from binary activation codes, serving as an early stopping criterion for promising architectures.

Final selection is based on a multi-objective grade:

$$G(m) = \alpha \cdot A(m) + (1 - \alpha) \cdot \left(\frac{P}{P(m)}\right)^r$$

where $A(m)$ is task-specific accuracy (for classification), PSNR (for super-resolution), or other suitable metrics.

3. Search Space and Efficiency Strategies

Compared to many traditional NAS methods, which train thousands of candidate models or rely on exhaustive reinforcement learning or differentiable search in very broad spaces (resulting in significant computational costs), LDP constrains the search space through its multi-scale backbone and targets only intra-block diversification. This allows for a substantial reduction in search time—ATS typically converges within approximately 4.3 GPU days, in contrast to the thousands of GPU hours required by older approaches.

Initial evaluation of child architectures uses proxy scores (derived from activation pattern distances) to minimize unnecessary full trainings, ensuring computational efficiency and scalability to large datasets or hardware-constrained deployment scenarios.

4. Performance Across Dense Prediction Tasks

LDP's search framework has been evaluated on a wide range of dense prediction tasks:

• Monocular Depth Estimation: On NYU-Depth-v2 and KITTI, LDP-derived models outperform prior lightweight baselines in mean absolute relative error (REL) and root mean square error (RMSE), while achieving greater compactness in parameter count.
• Semantic Segmentation: On Cityscapes and COCO-stuff, LDP models maintain competitive mean Intersection-over-Union (mIoU) and pixel accuracy, but with significantly smaller footprints than manually designed or classical NAS-derived networks.
• Image Super-Resolution: On datasets such as DIV2K, Set5, Set14, BSD100, and Urban100, LDP-based architectures match or exceed state-of-the-art benchmarks on peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM), despite parameter counts reduced by up to an order of magnitude.

Experiments demonstrate that LDP yields networks 5%–315% more compact than alternatives, while retaining strong accuracy. Real-time inference on hardware such as the Google Pixel 3a further attests to the framework's efficiency.

Dense Prediction Task Results Table

Task	Metric	LDP Improvement
Monocular Depth	REL, RMSE	Lower error, fewer params
Semantic Segmentation	mIoU, Pixel Acc.	Compact & competitive
Super-Resolution	PSNR, SSIM	Matches SoA, smaller net

5. Technical Aspects and Proxy Evaluation

The LDP search protocol employs proxy evaluation to rapidly prioritize candidates:

• Kernel Matrix Score: $\text{score}(f) = \log |\mathbf{K}_H|$ where each entry $[\mathbf{K}_H]_{i,j} = N_A - d_H(c_i, c_j)$, with $d_H$ denoting Hamming distance of binary activation codes $c_i$, $N_A$ the number of activations.
• Mutation Validation: Offspring architectures are validated by reward $R$ and only the most promising undergo full training and evaluation via the multi-objective grade $G$.

This proxy evaluation approach provides a computational bottleneck reduction and enables ATS to balance compactness and representational power efficiently.

6. Applications, Implications, and Future Work

Given its substantial gains in parameter and runtime efficiency, LDP is well-suited for resource-constrained environments such as mobile platforms, embedded systems, and edge devices commonly used in robotics and autonomous driving. Concrete use cases include real-time depth sensing for AR/robotics, scene understanding in advanced driver-assistance systems, and affordable high-quality medical and surveillance imaging.

LDP’s approach—premised on backbone-guided search and multi-objective optimization—suggests a general strategy for automated NAS tailored to dense prediction, making LDP a reference point for future research. Potential research directions include integrating hardware-aware objective functions, further search-space compression, and extending the method to additional dense estimation modalities.

7. Comparison with Related Approaches

Unlike earlier NAS frameworks generally targeted at image classification or one dense prediction subtask (often at great computational expense), LDP demonstrates flexibility across dense estimation, segmentation, and reconstruction. Its backbone-based search space, along with the reward-driven ATS, enables both parameter compactness and strong competitive accuracy, while maintaining search efficiency and broad applicability. This versatility distinguishes it from approaches where architecture search is tightly coupled to either specific tasks or architectural motifs.

A plausible implication is the feasibility of rapidly adapting dense prediction CNNs for new device constraints and deployment platforms, by re-running ATS within the LDP framework and selecting architectures that optimize performance within given resource budgets.