Hardware-Customized Latency Predictor
- Hardware-customized latency predictors are models that estimate execution delay by integrating neural architecture details with device-specific metrics.
- They employ few-shot adaptation and hardware embeddings to overcome the limitations of traditional FLOPs metrics, capturing parallelism, memory access, and runtime scheduling impacts.
- Applications span neural architecture search, compiler cost modeling, and online service monitoring, demonstrating sample efficiency and scalability across diverse hardware.
A hardware-customized latency predictor is a model, estimator, or analytical cost formulation that predicts the execution latency of a neural architecture, tensor program, or serving pipeline on a specified hardware target by conditioning on architecture structure together with device-dependent information. The concept arose in hardware-aware neural architecture search (NAS), where FLOPs and parameter count are often inadequate because real latency depends on parallelism, memory access, runtime scheduling, fusion, and platform-specific operator implementations, and it has since expanded to compiler cost models, edge-runtime prediction, accelerator design-space exploration, and online service monitoring (Lee et al., 2021, Chai et al., 2023, Le et al., 28 Feb 2026).
1. Problem setting and motivation
The basic motivation is that latency is both indispensable and expensive to obtain. Hardware-aware NAS, deployment-time model selection, and compiler optimization all require rapid latency estimates for many candidates, yet direct profiling on the target device is slow enough to become the dominant systems bottleneck. One concrete illustration is MAPLE’s example: if average latency is $50$ ms, $50$ runs are used to reduce timing variance, and a NAS method evaluates architectures, latency collection alone reaches $750$ hours (Abbasi et al., 2021). In the same spirit, proxy-device work on hardware-aware NAS notes that building a latency predictor for each target device scales poorly, citing examples such as k measurements for one mobile device, 350k records for one device, and up to 4.4 days of measurement for one edge device (Lu et al., 2021).
A second motivation is that additive or hardware-agnostic surrogates frequently break down. HELP argues that FLOPs are often inaccurate because real latency depends not only on arithmetic count but also on parallelism, memory access, and execution effects on the actual hardware (Lee et al., 2021). This mismatch is amplified on optimized edge runtimes. MAPLE-Edge shows that TensorRT can alter the graph through kernel auto-tuning, layer fusion, tensor fusion, and elimination of unused outputs, so a lookup table that sums layerwise latencies can substantially overestimate actual runtime (Nair et al., 2022). A related operation-wise study for mobile inference on TFLite reports that GPU prediction must be kernel-wise rather than graph-op-wise because the executed unit after compilation may be a fused kernel or a different convolution implementation such as Winograd or optimized grouped convolution (Li et al., 2022).
The topic therefore sits at the intersection of hardware variability and workload variability. For NAS, the central problem is usually to estimate end-to-end latency of candidate architectures on an unseen device with a small number of target measurements. For compilers, the task shifts toward tensor-program or kernel latency on a given accelerator. For online systems, the task becomes predicting per-request or per-batch latency under changing resource contention. Across these settings, “hardware-customized” means that latency is treated as a function of both the computational object and the execution substrate, rather than of the architecture alone.
2. Formal problem statements and hardware representation
Most formulations generalize architecture-only regression into a hardware-aware map . In HELP, each device is a task , and the predictor is explicitly reformulated from to , with few-shot adaptation on unseen hardware as the core regime (Lee et al., 2021). In the OFA-oriented generalized latency predictor, the analogous distinction is between hardware-specific $50$0 and hardware-generalized $50$1, where $50$2 is a hardware-parameter vector (Syed et al., 2021). MAPLE uses the same general form $50$3, where $50$4 is an architecture encoding and $50$5 is a quantitative hardware descriptor (Abbasi et al., 2021).
The central representational issue is how to encode hardware. HELP avoids hardware internals entirely and treats each device as a black-box latency function. Its hardware embedding $50$6 is a standardized 10-dimensional vector of measured latencies on a fixed set of reference architectures, making the device representation operational rather than structural (Lee et al., 2021). MAPLE uses a more explicit microarchitectural description: it measures 10 Linux perf counters on each of 15 NAS-Bench-201 primitive operators, yielding a $50$7-dimensional descriptor; operator latency is also included so that end-to-end latency can be learned from a hardware signature grounded in microprocessor behavior (Abbasi et al., 2021). MAPLE-Edge reduces this to six counters that are available across Linux kernels—CPU cycles, instructions, cache references, cache misses, L1 data cache loads, and L1 data cache load misses—and then normalizes them by operator latency so the descriptor becomes a rate-based hardware-runtime signature instead of a vector of absolute counts (Nair et al., 2022).
Other work adopts simpler or more explicit hardware metadata. In the OFA hardware-generalization study, the generalized GPU model appends $50$8 to the architecture vector, increasing the input length from 128 to 131 (Syed et al., 2021). MAPLE-X supplements MAPLE’s counter-based descriptor with an explicit device-specification vector consisting of core count, maximum clock speed, and thermal design power, and uses Euclidean distance in that 3D space to weight source-device samples during transfer (Abbasi et al., 2022). By contrast, PerfSAGE does not use an explicit hardware descriptor at all; hardware customization is achieved by training a separate graph-level predictor for each target metric and platform rather than by conditioning a single model on device features (Chai et al., 2023). Analytical accelerator predictors take yet another route: DNN-Chip Predictor parameterizes hardware via PE count, NoC design, memory hierarchy capacities and bandwidths, unit MAC delay, and technology-dependent costs, so latency becomes a direct function of workload dimensions, hardware architecture, and dataflow mapping (Zhao et al., 2020).
This diversity implies that “hardware representation” is not a settled concept. Some methods encode measured behavior, some encode public specifications, some rely on compiler-level metadata plus target fine-tuning, and some bypass learned embeddings in favor of direct architectural parameters. A plausible implication is that representation choice is inseparable from the prediction granularity: device descriptors that work for architecture-level NAS are not necessarily those that work for kernel-level compiler cost models or runtime telemetry-conditioned predictors.
3. Modeling paradigms
The literature clusters into a small number of recurrent modeling paradigms.
| Paradigm | Hardware customization mechanism | Representative papers |
|---|---|---|
| End-to-end learned regressors | Architecture encoding plus device embedding or per-target training | (Xu et al., 2020, Syed et al., 2021, Lee et al., 2021, Chai et al., 2023) |
| Compositional layer/op/kernel models | Per-component prediction with sum, longest-path, or graph replay aggregation | (Li et al., 2022, Huai et al., 8 Jul 2026, Hu et al., 2023, Le et al., 28 Feb 2026) |
| Analytical hardware models | Explicit bandwidth, memory-hierarchy, tiling, dataflow, or schedule search | (Zhao et al., 2020, Han et al., 2022) |
| Transfer-by-ranking or proxy reuse | Cross-device latency monotonicity and lightweight adaptation | (Lu et al., 2021, Akhauri et al., 2024) |
End-to-end learned regressors map a whole-architecture representation directly to latency. Examples include the DARTS latency prediction module, which encodes a normal cell as a 112-dimensional binary vector and predicts latency with an MLP integrated into a differentiable NAS objective (Xu et al., 2020); the OFA 3-layer MLP regressor over a 128-dimensional architecture vector or 131-dimensional architecture-plus-hardware vector (Syed et al., 2021); HELP’s architecture encoder plus device encoder plus prediction head, with a hardware-adaptive modulation vector $50$9 used to condition the initialization (Lee et al., 2021); and PerfSAGE’s graph-level GNN over arbitrary TFLite DAGs with categorical feature encoding and graph pooling (Chai et al., 2023).
Compositional models decompose latency into smaller units. The mobile edge study predicts CPU latency at the operation level and GPU latency at the post-compilation kernel level, then sums component predictions and adds a device-specific overhead term (Li et al., 2022). ZeroBN trains a three-layer BP network on parameterized single-layer models and reconstructs whole-model latency by finding the latency-weighted longest path in the network topology (Huai et al., 8 Jul 2026). CDMPP predicts tensor-program latency from compact ASTs plus hardware metadata, then composes program-level predictions into end-to-end DNN latency with a DFG replayer (Hu et al., 2023). PM2Lat decomposes a model into kernels, differentiates kernel configurations explicitly, predicts compute-intensive kernels through throughput interpolation and memory-bound kernels through regression on implementation-level proxy metrics, and then sums kernel latencies under a sequential CUDA execution assumption (Le et al., 28 Feb 2026).
Analytical hardware models replace black-box regression with a parameterized systems model. DNN-Chip Predictor expresses accelerator latency as
0
where memory latencies are derived from refresh counts, per-refresh transfer volumes, and hierarchy bandwidths under a loop/dataflow abstraction (Zhao et al., 2020). LASNet uses a latency model for dynamic spatial inference that decomposes operator cost into data movement and computation,
1
and searches tile shapes while conditioning on block configuration, activation rate, spatial granularity, and hardware parameters such as number of processing engines, frequency, bandwidth, and FP32 capability (Han et al., 2022).
A separate line of work reframes the task around ranking transfer instead of absolute latency regression. “One Proxy Device Is Enough” formalizes latency monotonicity and shows that if two devices preserve architecture latency orderings, they share the same Pareto-optimal set for latency-constrained NAS; in the absence of strict monotonicity, it adapts a proxy predictor with a sparse correction over operator-level weights (Lu et al., 2021). “On Latency Predictors for NAS” argues that predictor robustness depends not only on architecture but also on training/test hardware partitioning, and finds that operation-specific hardware embeddings, diversity-based target sample selection, and supplementary architecture encodings improve few-shot transfer on low-correlation device splits (Akhauri et al., 2024).
4. Few-shot adaptation and sample efficiency
Few-shot adaptation is the defining property of the modern hardware-customized latency predictor. HELP treats each device as a task in a meta-learning problem and reports that with only 10 target-device measurements on NAS-Bench-201 it reaches Spearman correlations of 2 on unseen GPU, 3 on unseen CPU, 4 on Pixel2, 5 on Raspi4, 6 on ASIC, and 7 on FPGA, with mean 8. The same table reports mean 9 for FLOPs, $750$0 for a layer-wise predictor where applicable, $750$1 for BRP-NAS with 900 samples, and $750$2 for BRP-NAS with 3200 samples, so the main empirical claim is sample efficiency rather than only asymptotic accuracy (Lee et al., 2021).
MAPLE intensifies that emphasis on sample efficiency. In leave-one-out evaluation across eight devices, the paper reports mean $750$3 error-bound accuracy of $750$4 for HELP with 10 samples, $750$5 for MAPLE with 3 samples, and $750$6 for MAPLE with 10 samples, while using CPU-side performance counters as descriptors even for GPU prediction (Abbasi et al., 2021). MAPLE-Edge addresses the harder mixed-runtime case and reports that under a combined TensorRT+TFLite training pool, mean accuracy on Jetson TX1 TensorRT rises from $750$7 for MAPLE to $750$8 for MAPLE-Edge, and on Jetson TX2 TensorRT from $750$9 to 0, still with only 10 adaptation measurements and a normalized six-counter descriptor (Nair et al., 2022). MAPLE-X adds explicit hardware-based sample weighting and neighborhood-based virtual examples, and reports mean 1 accuracy 2 versus 3 for MAPLE and 4 for HELP while using only 3 target-device adaptation samples (Abbasi et al., 2022).
Not all few-shot transfer is meta-learning. In the OFA setting, a hardware-specific base predictor can be adapted by freezing the first two layers and fine-tuning on 700 target-device examples, which the paper describes as about 80% smaller than the training size needed when training from scratch, with the “elbow” for full training appearing around 4000 samples (Syed et al., 2021). The proxy-device line pushes further toward rank-based reuse: it reports that 30–50 sample architectures are enough to estimate SRCC reliably, and in the majority of experiments 50 latency measurements are enough to recover strong monotonicity after proxy adaptation, avoiding a full per-device predictor build (Lu et al., 2021). This suggests that “few-shot” spans a continuum, from 3–10 samples in meta-learned or descriptor-rich models to a few hundred in transfer-learning regimes, with the required budget largely determined by how informative the hardware representation is.
5. Integration into NAS and deployment systems
The operational significance of a hardware-customized latency predictor is most visible when it is inserted into a larger search or optimization loop. HELP is explicitly evaluated as a drop-in replacement for device-specific predictors inside multiple NAS systems. In MetaD2A on NAS-Bench-201/CIFAR-100, replacing BRP-NAS with HELP reduces the target-device sample count from 900 to 10 and cuts latency-model build time from around 1120 s to 25 s on Pixel2 and from 940 s to 11 s on Titan RTX; corresponding total NAS time drops from about 1220 s to 125 s and from 1040 s to 111 s, respectively. In OFA on the MobileNetV3 search space, OFA plus layer-wise predictor can take 15 h on Titan RTX whereas OFA plus HELP takes 26 s with 10 samples; on WMT’14 En-De with HAT, HELP replaces HAT’s end-to-end predictor and uses 10 samples instead of 2000 (Lee et al., 2021).
The OFA-oriented generalized latency-prediction work reaches a similar systems conclusion from a different modeling direction. It reports that learned predictors can match the NAS performance of the lookup-table baseline approach, and that the forward-pass cost of a predictor is negligible relative to direct measurement: on RTX 2080 Ti, prediction takes approximately 5–6 ms versus 7 ms for direct latency measurement, and on Intel Xeon CPU approximately 8–9 ms versus 0 ms (Syed et al., 2021). ZeroBN extends the deployment narrative beyond NAS and uses a per-device BP latency regressor to enforce a hard 34 ms latency target during one-shot architecture adaptation on Jetson edge devices. The reported predictor error on real models averages 6.12%, and the hardware-constrained optimization reduces GoogLeNet on Jetson Nano from 40.32 ms to 34 ms with only 0.14% Top-1 accuracy reduction, while on Jetson TX2 it compresses VGG-19 from 119.98 ms to 34 ms and improves accuracy by 0.5% (Huai et al., 8 Jul 2026).
Dynamic-network and arbitrary-graph predictors extend the same idea to other optimization problems. LASNet uses a hardware-aware latency model to choose spatial granularity and scheduling strategy for dynamic CNNs, reporting that average latency of ResNet-101 on ImageNet can be reduced by 36% on Nvidia Tesla-V100 and 46% on Nvidia Jetson TX2 GPU without sacrificing accuracy (Han et al., 2022). PerfSAGE shows that full-graph modeling can reach specialized-model MAPE values as low as 1.098% for CNN-ImageNet CPU latency and remains below 5% across all reported targets and search spaces, making it suitable for arbitrary TFLite graphs rather than only hand-designed NAS spaces (Chai et al., 2023). The common pattern is that latency prediction becomes useful when it is cheap enough to sit inside a repeated optimization loop and accurate enough to preserve ranking or feasibility under device constraints.
6. Extensions beyond offline NAS latency prediction
The topic has broadened well beyond offline architecture scoring. At compiler level, PM2Lat predicts GPU execution latency by modeling actual kernel configurations rather than logical operators, using throughput interpolation for compute-intensive kernels and linear regression on implementation-level proxy metrics for memory-bound utility kernels; it reports error rates below 10% across different data types and hardware platforms on Transformer models, with 3–8% error on diverse kernels such as Triton, Flash Attention, and Cutlass Attention (Le et al., 28 Feb 2026). CDMPP addresses the cross-model and cross-device tensor-program setting with a Compact AST representation, explicit hardware metadata, CMD-based fine-tuning, and KMeans-based task sampling, reporting 14.03% prediction error for cross-model prediction and 10.85% for cross-device prediction while supporting both training and inference accelerators (Hu et al., 2023). These works shift hardware customization from NAS search spaces to compiler-generated programs and kernel-level execution.
A second extension is toward online service latency prediction under heterogeneous deployment conditions. Morpheus predicts request RTT from pre-submission time-series monitoring data in a Kubernetes-managed heterogeneous GPU cluster and deploys a separate predictor for each application-server pair, reporting up to 95% accuracy while keeping prediction delay within 10% of application RTT (Giannakopoulos et al., 23 Oct 2025). LatencyPrism models expected LLM decode latency as a function of workload, monitors positive prediction error rather than raw latency, and reports that it distinguishes workload-driven latency variation from anomalies with an F1-score of 0.98 while operating in a zero-intrusion multi-platform setting (Yin et al., 14 Jan 2026). “Breaking the Ice” further extends the scope to cold-start prediction: it decomposes vLLM startup into six steps, shows that startup is predominantly CPU bound, and develops a lightweight analytical model that predicts startup latency for a given hardware configuration (Kabakibo et al., 5 Jun 2026). This broader usage suggests that the notion of a hardware-customized latency predictor now includes startup modeling, online telemetry-conditioned prediction, and service-level control, not only NAS-time architecture evaluation.
7. Limitations, misconceptions, and open questions
A persistent misconception is that hardware customization can be replaced by a universal analytic proxy such as FLOPs or by a single fixed lookup table. Multiple strands of evidence argue otherwise. FLOPs-based ranking is consistently weaker than device-aware prediction in HELP (Lee et al., 2021); TensorRT graph optimization breaks layer-additive LUT assumptions in MAPLE-Edge (Nair et al., 2022); and the TFLite GPU study shows that kernel fusion and kernel selection materially change what should be predicted (Li et al., 2022). PerfSAGE reaches strong accuracy precisely by abandoning layerwise summation in favor of full-graph regression on arbitrary TFLite DAGs (Chai et al., 2023).
A second open question concerns how much explicit hardware knowledge is necessary. Descriptor-rich methods such as MAPLE, MAPLE-Edge, and MAPLE-X show that performance counters or simple hardware specifications can strongly improve adaptation, yet these descriptors are themselves task- and platform-dependent (Abbasi et al., 2021, Abbasi et al., 2022). Conversely, the OFA GPU-generalization study shows that appending a small hardware vector to architecture one-hots is not automatically sufficient: with data from only two GPUs, the generalized model’s loss does not converge and RMSE remains around 7–11 ms (Syed et al., 2021). PM2Lat takes the opposite stance and argues that since many important GPU properties are unobservable, a per-device kernel-aware model is safer than a universal cross-device regressor; it remains NVIDIA-focused, does not yet support convolution, and still exhibits failure modes under thermal throttling and low-utilization BF16 regimes (Le et al., 28 Feb 2026).
A third issue is that few-shot transfer is not zero-shot transfer. HELP, MAPLE, MAPLE-Edge, MAPLE-X, proxy-device adaptation, and NASFLAT-style transfer all reduce target-device profiling cost, but none eliminate the need for at least a small calibration set on the target (Lee et al., 2021, Lu et al., 2021, Akhauri et al., 2024). Predictor robustness is also highly sensitive to evaluation protocol. “On Latency Predictors for NAS” argues that prior evaluations often relied on hand-crafted source/target device sets and that existing transfer and meta-learning methods exhibit significant performance variability; its automated low-correlation partitions make the task materially harder and change which design choices appear robust (Akhauri et al., 2024). A plausible implication is that progress on hardware-customized latency prediction depends as much on principled benchmarking and hardware split construction as on new predictor architectures.
Taken together, the literature portrays hardware-customized latency prediction as a family of methods rather than a single design. At one extreme are per-device analytical models grounded in memory hierarchy, tiling, and dataflow; at the other are end-to-end learned regressors equipped with hardware embeddings, counter-based descriptors, or few-shot transfer. Between them lie proxy ranking transfer, graph-level arbitrary-model predictors, compiler cost models, and online telemetry-conditioned service predictors. The unifying principle is consistent: latency must be modeled as a joint property of computation and execution substrate, and the quality of that modeling determines whether hardware-aware optimization remains practical.