Latency-Aware Convolutions

• Latency-aware convolutions are convolution operators optimized to minimize real-world inference latency using device-specific measurements and constrained optimization techniques.
• They integrate empirical profiling, analytical modeling, and frameworks like ILP, differentiable NAS, and dynamic programming to achieve precise accuracy–latency trade-offs.
• These methods have demonstrated significant reductions in latency on diverse platforms, including GPUs, CPUs, microcontrollers, and optical hardware.

Latency-aware convolutions refer to convolutional operators and architectures specifically optimized to minimize real-world inference latency under explicit hardware and system constraints. These methods integrate device-specific latency measurements, analytical or empirical latency models, and constrained optimization frameworks to deliver Pareto-optimal accuracy–latency trade-offs, often under hard real-time budgets. Unlike indirect proxies such as floating-point operations (FLOPs) or parameter count, latency-aware convolutions directly target the wall-clock latency experienced on CPUs, GPUs, microcontrollers, or even optical hardware, often using hardware-in-the-loop measurements or predictive analytical modeling. This article surveys foundational approaches, algorithms, and empirical outcomes in latency-aware convolution design.

1. Principles of Latency-Aware Convolution Design

Latency-aware convolutional methods fundamentally differ from FLOPs- or model-size-centric optimizations by focusing on achieving specified end-to-end latency constraints. The canonical formulation is a constrained optimization problem: $$\min_{\mathcal{A}} \,\, \text{Accuracy Loss}(\mathcal{A}) \quad \text{s.t.} \quad \text{Latency}(\mathcal{A}) \leq \text{Budget}$$ where $\mathcal{A}$ denotes the set of architectural choices or convolution parameters. This direct targeting of latency is implemented through several mechanisms:

• Empirical Latency Profiling: Building per-operator or per-layer lookup tables (LUTs) of measured latencies on the deployment hardware for all candidate convolution variants. This approach is prevalent in methods such as LANA (Molchanov et al., 2021) and MicroNAS (King et al., 2023).
• Analytical Latency Modeling: Deriving closed-form or piecewise-linear models that incorporate device architectural features, memory transfer, scheduling strategies, and compute throughput (e.g., LASNet (Han et al., 2022)).
• Integrated Optimization: Incorporating latency proxies into neural architecture search (NAS), constrained pruning, or layer fusion using formulation such as integer linear programming (ILP), alternating direction method of multipliers (ADMM), or dynamic programming (DP), as detailed in subsequent sections.

These methods often guarantee end-to-end latency compliance while operating over rich libraries of convolutional alternatives, including depthwise separable, inverted bottleneck, stacked convolutions, ViT/Mixer blocks, and sparse or dynamic operators.

2. Algorithmic Frameworks and Optimization Techniques

Several optimization strategies have been proposed for constructing latency-aware convolutions:

LANA: ILP-based Layer Replacement

The LANA framework frames one-shot layer-wise convolution replacement as a large-scale ILP. Given $N$ layers and $M$ candidate ops per layer, LANA trains each op with feature-map distillation and populates vectors $a_{i,j}$ (accuracy change) and $b_{i,j}$ (hardware-measured latency). The ILP is formulated as: $$\begin{aligned} & \min_{z} \quad \sum_{i=1}^N a_i^T z_i \ & \text{s.t.} \quad \sum_{i=1}^N b_i^T z_i \leq B, \quad 1^T z_i = 1,\ z_{i,j}\in\{0,1\} \end{aligned}$$ Empirically, LANA searches a $197^{50}\sim10^{100}$ architecture space within seconds and achieves $2-5\times$ latency reduction with minimal accuracy loss on EfficientNet/ResNeST architectures (Molchanov et al., 2021).

Differentiable NAS with Soft Latency Constraints

MicroNAS integrates per-operator latency from LUTs into a differentiable NAS (DNAS) objective by interpolating expected latency for each dynamic convolution primitive using Gumbel-Softmax architectural weights. The objective incorporates a soft hinge penalty: $$\mathcal{L} = \mathcal{L}_{\text{task}}(w, \alpha) + \gamma_{\mathrm{lat}} \log\left(\frac{L_{\mathrm{acc}}(\alpha)}{L_{\mathrm{budget}}}\right) [L_{\mathrm{acc}}(\alpha)\ge L_{\mathrm{budget}}]$$ This enables smooth accuracy–latency trade-off curves and strict adherence to MCU timing constraints (King et al., 2023).

Layer and Depth Fusion via Dynamic Programming

Depth compression approaches cast the problem as an NP-hard subset selection of which activations to linearize (identity) and which conv chains to merge. An exact two-stage dynamic programming approach solves a surrogate optimization using pre-measured block latencies and block importance scores: $$\max_{A\subseteq S\subseteq [L-1]} \sum_{a_{k-1},a_k} I[a_{k-1},a_k] \quad\text{s.t.}\quad \sum_{s_{k-1},s_k} T[s_{k-1},s_k]<T_0.$$ TensorRT profiled per-block measurements ensure hardware-consistent latency minimization (Kim et al., 2023).

Latency-Constrained Sparse Pruning

Pruning-based frameworks such as ALCS enforce SIMD-structured group sparsity while directly constraining measured runtime via a piecewise-linear latency model. The optimization is solved by ADMM: $$\min_{W}\;\mathcal L(W) \quad\text{s.t.}\;\mathcal T(W)\le T_\mathrm{budget}$$ Resulting models preserve SIMD efficiency, granting $1.7\times$–$2.7\times$ latency improvements at negligible accuracy loss (Zhao et al., 2021).

3. Hardware-Informed Latency Models and Empirical Profiling

Latency-aware convolutional design requires accurate, device-specific latency prediction:

• LUTs and On-Device Profiling: Both LANA and MicroNAS rely on compact tables mapping each $$(\text{kernel size}, \text{stride}, C_{\text{in}}, C_{\text{out}})$$ tuple to observed latency, covering the complete convolutional search space on target hardware (Molchanov et al., 2021, King et al., 2023).
• Piecewise Linear Latency Models: For structured sparse convolutions (ALCS), layer latency is modeled as locally linear in the count of nonzeros. Only 11 calibrations per layer suffice for percent-level accuracy in surrogate predictive models (Zhao et al., 2021).
• Analytical and Memory-Aware Models: LASNet introduces a predictor summing data movement and computation latencies:

$\ell = \ell_{\mathrm{data}} + \ell_{\mathrm{comp}}$

Memory hierarchy, compute tiles, activation sparsity, and scheduling strategies are incorporated, allowing stage-wise granularity search to optimize for predicted real-world latency rather than FLOPs alone (Han et al., 2022).

• Empirical Validation: Cross-validation on platforms such as Nvidia Tesla V100, Jetson TX2/Nano, and Cortex-M MCUs demonstrates $<$5% modeling error and successful latency constraint adherence.

4. Architectures and Primitives for Low-Latency Convolution

Latency-aware convolution methods target diverse architectural levels:

• Layerwise Op Replacement: LANA searches per-layer pools including depthwise, inverted bottleneck, and transformer/Mixer blocks.
• Depthwise and Channelwise Pruning: SIMD-structured group pruning enables sparsity patterns mapped neatly to CPU/GPU vectorization units (Zhao et al., 2021).
• Spatialwise Dynamic Convolutions: LASNet and related dynamic networks enable patch-level masking; only informative regions are convolved at coarse spatial granularity $S \gg 1$ to minimize memory-bound latency (Han et al., 2022).
• Layer Fusion/Chain Collapse: Dynamic programming-based merging combines adjacent linearized conv chains into deeper kernels for optimized execution within a hard budget (Kim et al., 2023).
• Operator Fusion and Scheduling: Highly optimized kernel fusion (masker-conv, gather-conv, scatter-add) and tile size autoselection, as well as implicit-GEMM or Winograd mapping, further reduce operator and data movement bottlenecks (Galvez et al., 30 Sep 2025, Han et al., 2022).
• Optical/Physical Implementations: Beyond electronic architectures, all-optical JTC-based convolution exploits $O(n^2)$ scaling and nearly speed-of-light latency for edge-AI applications (George et al., 2022).

5. Quantitative Benchmarks and Trade-offs

Latency-aware convolutions consistently deliver superior empirical latency under fixed- or minimum-accuracy constraints:

Method / Model	Device	Top-1 / F1	Latency	Speedup over Baseline
LANA 0.45×B2	V100 GPU	79.71%	16.2 ms	2.4×
LANA 0.25×B4	V100 GPU	81.83%	30.4 ms	4.3×
MicroNAS (UCI-HAR, 10ms)	Cortex-M33	F1 ≈ 88%	10 ms	N/A
MicroNAS (UCI-HAR, 200ms)	Cortex-M33	F1 ≈ 95%	200 ms	N/A
ALCS (ResNet-50)	ARM CPU	-	1.7×–2.7×	1.7×–2.7×
LASNet (ResNet-101, t=0.6)	V100	76.9%	25.3 ms	33% ↓
LASNet (RegNetY)	Nano GPU	-	20–30% ↓	Up to 2×
Depth Compression (MobileNetV2)	RTX2080Ti	72.83%	13.67 ms	1.08×–1.41×

Latency-aware approaches routinely demonstrate strict latency budget enforcement and improved or preserved accuracy relative to channel pruning or theoretically optimized (FLOP-centric) alternatives (Molchanov et al., 2021, King et al., 2023, Kim et al., 2023). Methods designed without direct latency measurement frequently fail to meet constraints when mapped to deployment hardware, especially on memory-bound platforms.

6. Special Topics: Beyond Classical Implementations

Energy-constrained and Physical-Domain Latency-Aware Convolution

• CPU-Specific Optimizations: Empirical benchmarks show that implicit-lowering GEMM convolutions, as implemented in OneDNN, yield minimal end-to-end latency and energy on multi-core CPUs where cache bandwidth, SIMD width, and physical core count are optimized. Winograd F(2,3) kernels can deliver further $~20\%$ latency reductions under suitable memory and SIMD conditions (Galvez et al., 30 Sep 2025).
• Optical Joint Transform Correlators: Nonlinear optical architectures implementing convolution via four-wave mixing achieve $O(n^2)$ scaling and sub-nanosecond aggregate latency, offering a fundamentally different means for edge or cloud AI acceleration where classical electronics cannot compete for single-batch throughput (George et al., 2022).

7. Limitations, Best Practices, and Open Directions

Current solutions depend critically on high-fidelity latency models and exhaustive on-device profiling or accurate analytical predictors. Challenges include:

• Modeling inter-operator memory effects and multi-threaded scheduling beyond per-layer additivity.
• Generalizing operator- and hardware-specific latency proxies to emerging accelerators (NPUs, FPGAs, optical ICs).
• Incorporating more expressive nonlinearity and activation patterns in depth fusion and compression (Kim et al., 2023).
• Extending latency-aware design to transformer blocks and non-convolutional operators under tight real-time constraints.

Best practices include integrating hardware-in-the-loop profiling, leveraging operator fusion and kernel scheduling, fusing batch normalization and activation where possible, and directly enforcing latency constraints at all optimization stages. On CPUs, GEMM-based schemes are preferable under standard conditions, with Winograd reserved for conv-heavy, wide-SIMD scenarios and direct methods only for nonstandard shapes (Galvez et al., 30 Sep 2025).

Latency-aware convolution methods thus comprise a spectrum of hardware-cognizant approaches that directly address system-level latency, yielding architectures that reliably meet the deployment constraints of contemporary and next-generation computing platforms.