Serving deep learning models in a serverless platform (1710.08460v2)

Abstract: Serverless computing has emerged as a compelling paradigm for the development and deployment of a wide range of event based cloud applications. At the same time, cloud providers and enterprise companies are heavily adopting machine learning and Artificial Intelligence to either differentiate themselves, or provide their customers with value added services. In this work we evaluate the suitability of a serverless computing environment for the inferencing of large neural network models. Our experimental evaluations are executed on the AWS Lambda environment using the MxNet deep learning framework. Our experimental results show that while the inferencing latency can be within an acceptable range, longer delays due to cold starts can skew the latency distribution and hence risk violating more stringent SLAs.

• The paper evaluates serverless platforms like AWS Lambda for serving deep learning models, focusing on inferencing latency.
• Warm starts show feasible latency for inference, especially with sufficient memory allocation.
• Cold starts present a significant challenge due to high latency overheads from container initialization and reveal resource allocation inefficiencies.

Serving Deep Learning Models in a Serverless Platform: An Analytical Review

The paper "Serving Deep Learning Models in a Serverless Platform" evaluates the viability of serverless computing environments, particularly focusing on AWS Lambda, for inferencing large neural network models. The authors, Ishakian, Muthusamy, and Slominski, investigate whether serverless environments, which are typically lauded for their scalability and cost-effectiveness, can effectively support the demands of neural network inferencing without violating stringent service-level agreements (SLAs) due to latency issues.

Evaluation Framework and Methodology

The paper is grounded in empirical analysis, leveraging the AWS Lambda platform alongside the MXNet deep learning framework. The authors choose three model architectures for the evaluation—SqueezeNet, ResNet-18, and ResNeXt-50. These models span a range of complexities and sizes, from SqueezeNet's relatively small 5MB model to the larger 98MB ResNeXt-50, enabling an examination of performance across diverse inferencing demands.

The experimental setup is structured to segregate warm and cold start performance to accurately assess latency metrics. The cold start evaluates the time required to launch and initialize the container upon the first invocation of a lambda function, which inherently impacts observed response time. Conversely, the warm start leverages the reuse of already initialized containers to simulate continuous inferencing loads.

Key Findings

The analysis reveals several critical insights:

• Warm Start Efficiency: The latency from warm starts demonstrates feasibility within acceptable ranges for user experience, particularly at memory allocations exceeding 1024MB. As memory size increases, the latency and inferencing time decrease proportionally, suggesting resource allocation scales with demand up to a point.
• Cold Start Challenges: The cold start latency presents a significant challenge, with overheads attributable to container initialization. The variance between warm and cold start latencies could jeopardize adherence to SLAs that do not account for bimodal latency distributions.
• Resource Allocation Costs: Interestingly, the paper shows an inconsistency between increased allocated memory and proportional performance gains, highlighting inefficiencies where higher resource allocation does not necessarily equate to improved cost-performance. The correlation between execution costs and memory size is not linear, underscoring the complexity in optimizing serverless functions for both cost and performance.

Implications and Future Research Directions

The implications of this research are twofold:

1. Technical Ecosystem Enhancement: There's a discernible need for enhancing serverless platforms to accommodate AI workloads, particularly through access to specialized resources such as GPUs, which could mitigate cold start overheads and improve cost-efficiency. Additionally, creating platforms that support a more stateful workload could leverage the non-serverless platform capabilities while maintaining the benefits inherent to serverless solutions.
2. Optimizing SLA Frameworks: Service architectures could benefit from renewed SLA frameworks that incorporate the latency variability intrinsic to serverless environments. Developing predictive models or adaptive architectures that dynamically allocate resources based on workload needs might mitigate the observed latency spikes.

Looking forward, extending the analysis to alternative frameworks like Tensorflow and expanding experimentations across different serverless providers could provide a broad spectrum of performance data. Moreover, exploring potential integrations between on-demand virtual machines with fine-grained billing and serverless solutions offers promising optimization avenues for AI scalability.

In conclusion, while the paper effectively elucidates serverless computing's potential for deep learning model inferencing, it also presents a roadmap for overcoming the challenges associated. As serverless platforms continue evolving with technological advancements, particularly in AI and machine learning, resolving the highlighted issues will be pivotal to harnessing their full potential in commercial and enterprise applications.