--- title: 'DeepRecInfra: Optimizing Neural Recommendation Inference' url: https://www.emergentmind.com/topics/deeprecinfra type: topic --- # DeepRecInfra: Optimizing Neural Recommendation Inference DeepRecInfra is an end-to-end modeling, scheduling, and evaluation framework designed to optimize neural recommendation inference at datacenter scale. Developed with a co-design methodology, DeepRecInfra integrates a suite of industry-representative models, a dynamic workload generator modeled after real-world traffic, a scheduler (DeepRecSched) for hardware- and workload-aware batch and accelerator decision-making, and implementations tailored for modern CPU and GPU server nodes. The platform targets high-throughput, low-latency serving of large-scale deep learning recommendation models (DLRMs), facilitating rapid exploration and deployment of optimizations in production environments [2001.02772]. ## 1. Architecture and Modeling Scope DeepRecInfra explicitly targets the diverse, stringent requirements of production recommendation systems. It is structured around three core pillars: - **Model Suite:** Implements and benchmarks eight state-of-the-art neural recommendation models, including NCF, Wide & Deep (WnD), MT-WnD, Facebook DLRM-RMC1/2/3, and Alibaba DIN/DIEN, covering a spectrum of real-world topologies in Caffe2 with CPU (Intel MKL) and GPU (cuDNN) backends. - **Latent Service-Level Objectives (SLOs):** Incorporates strict tail-latency (p95) constraints, with real service SLOs collected for each model. Latency targets are swept across three ranges (Low, Medium, High, ±50% relative). - **Workload Generator:** Employs a Poisson process for inter-arrival times, with query working-sets extracted to match the heavy-tailed distributions observed in production datacenters. This explicit modeling of both the model and operational context allows DeepRecInfra to match the per-query behavior and statistical risk factors critical to large-scale commercial deployments [2001.02772]. ## 2. Dynamic Scheduling: DeepRecSched Formalism At the heart of DeepRecInfra is DeepRecSched, a query-batch scheduling and accelerator offload algorithm designed to maximize throughput under SLO constraints. The scheduling objective is formalized as: $$ \begin{aligned} & \text{Maximize}\quad QPS(b,s) \\ & \text{Subject to:}\quad \text{Latency}_{p95}(b, s) \leq L_{SLO} \\ &\quad 1 \leq b \leq C_{max} \\ &\quad 0 \leq s \leq Q_{max} \end{aligned} $$ where $b$ is per-request batch size, $s$ the query-size threshold for GPU offload, $QPS(b,s)$ achieved queries/s, and $L_{SLO}$ the p95 latency bound. The optimization is performed in two phases: (1) CPU batch-size tuning for small queries, and (2) offload threshold tuning to route large queries to GPU. The system models queries as a mixture drawn from measured heavy-tailed production traces, and the scheduler leverages a hill climbing search to select $(b, s)$ maximizing throughput while remaining within latency SLO bounds. Hardware heterogeneity, such as SIMD width and memory architecture, is incorporated via benchmarking and modeling of per-device performance curves [2001.02772]. ## 3. Hardware-Aware Model Implementations DeepRecInfra supports both CPU and GPU execution, with model kernels engineered for hardware efficiency: - **CPU Optimizations:** MLPs are vectorized using Intel MKL, exploiting AVX-256 (Broadwell) and AVX-512 (Skylake) instructions; embedding lookups use table reads and pooling to minimize indirection and saturate DRAM bandwidth. - **GPU Kernels:** Models leverage cuDNN and custom data-copy streams, explicitly modeling and measuring PCIe/GPU memory transfer bottlenecks. The GPU path is invoked for compute-heavy, large queries exceeding a device-specific threshold. The system dynamically maps small queries to the CPU (to minimize copy overhead and exploit thread-level parallelism) and large queries to the GPU (to maximize compute and memory bandwidth utility), with empirical batch size selection per device and workload. For embedding-heavy models, large batch sizes are favored on the CPU, while compute-bound models benefit from aggressive GPU offload [2001.02772]. ## 4. Empirical Evaluation and Datacenter Deployment DeepRecInfra has been validated on dual-socket Broadwell and Skylake machines (28/40 cores, AVX-256/512) and a GPU emulator (GTX 1080Ti). The system uses the eight-model suite with measured production traffic patterns and SLO targets. Key experimental findings include: - **Throughput:** DeepRecSched-CPU yields 1.7×–2.7× QPS improvement over a static scheduler; DeepRecSched-GPU further raises throughput to 4.0×–5.8× due to effective offload of large queries. - **Power Efficiency:** QPS/W increases by 1.7–2.7× for the CPU scheduler; incorporating GPU yields 2.0–2.9× improvements, though only compute-dominated models consistently benefit. - **Latency:** Production deployment across hundreds of datacenter servers resulted in p95 latency reduction by 1.39× and p99 by 1.31×, directly enabling higher QPS or headcount reduction. A notable design finding is that a small cluster can replicate datacenter tail-latency statistics within ∼10%, significantly accelerating prototyping without risking production SLOs [2001.02772]. ## 5. Integration with Emerging Storage and Computation Infrastructures DeepRecInfra has become a foundational suite for evaluating and co-designing scheduling and hardware solutions for recommendation models. For instance: - **RecSSD**: Employs DeepRecInfra’s benchmarks to demonstrate that near-data processing at the SSD level yields up to 2× end-to-end inference latency improvement versus commodity SSDs, with fully NVMe-compatible host integration [2102.00075]. - **SCRec**: Measures ≥10.4×–55.8× throughput and up to 13.35× energy efficiency advantage for DLRM inference with SmartSSD+FPGA computational storage, using DeepRecInfra’s Criteo/Meta-MELS test suite [2504.00520]. This demonstrates that DeepRecInfra is not limited to software runtime scheduling but extends to full-stack co-design and hardware-in-the-loop benchmarking. ## 6. Practical Observations, Limitations, and Future Directions Observations from DeepRecInfra’s design and deployment include: - **Query Size Distributions:** Production workloads are heavier tailed than log-normal; assuming classical distributions underestimates tail-impact and depresses attainable throughput by up to 1.7×. - **Batching Strategies:** The optimal batch size is jointly determined by model bottleneck (memory vs. compute), SLO regime, and hardware microarchitecture. - **GPU Offloading:** Offload is not universally optimal; gains occur when queries are both large and compute-heavy, and SLOs are sufficiently tight to amortize copy costs. - **Proxy Fleets:** Mini-fleet experimentation enables routine exploration of scheduling/design improvements without production risk. - **Open Source Availability:** DeepRecInfra’s release with load generators, model implementations, hardware models, and scheduler enables reproducibility and community extension. Limitations include a focus on batch size, SLO, and power analysis, with fine-grained energy modeling and SLA-adaptive mechanisms left for future work. The GPU “accelerator” is a simulation/emulation model; adaptation to next-generation AI ASICs or online feedback-based adaptation to diurnal load shifts is an open direction [2001.02772]. ## 7. Related Methodologies and System Comparison The DeepRecInfra approach, emphasizing algorithm-system co-design and explicit SLO-awareness, contrasts with single-device or static scheduling frameworks. Integration with hardware-aware deduplication (e.g., RecD for end-to-end feature de-duplication during DLRM training [2211.05239]), near-data embedding aggregation (RecSSD [2102.00075]), and compute-in-storage paradigms (SCRec [2504.00520]) shows that DeepRecInfra constitutes both a benchmarking suite and a methodology driving state-of-the-art efficiency in datacenter-scale recommendation systems. | System/Project | Role in DeepRecInfra Ecosystem | Key Contribution | |------------------|----------------------------------------|---------------------------------------------------| | DeepRecInfra | Scheduling, model suite, load generation| End-to-end datacenter-scale neural recommendation | | RecSSD | Storage-layer NDP, relies on DeepRecInfra benchmarks | 2× speedup embedding-dominated inference | | SCRec | Compute/storage co-design on SmartSSD, tested on DeepRecInfra workloads | >10× throughput, >6× energy efficiency over GPUs | | RecD | Deduplication for DLRM training | 2×–3× training throughput; batch-level feature compression | These results collectively demonstrate that DeepRecInfra provides a rigorous, extensible analytic foundation for both software-centric and hardware-centric optimization in large-scale recommendation inference. Source: https://www.emergentmind.com/topics/deeprecinfra