Hierarchical Prediction-based Management for LMaaS Systems

Abstract: LLMs have revolutionized fields such as natural language processing and software engineering, fueling the growth of Language-Model-as-a-Service (LMaaS) platforms hosted by industry leaders like OpenAI. These platforms handle millions of queries daily, requiring efficient management to reduce serving latency and meet Service Level Objectives (SLOs) while optimizing resource utilization. However, conventional cloud service management techniques, originally designed for traditional workloads, are suboptimal for LMaaS due to its dynamic service workloads and variable request loads. To address this, we propose PreServe, a tailored LMaaS management framework centered on hierarchical prediction. PreServe incorporates a service workload predictor to estimate periodic token density at a coarse granularity and a novel request load predictor to assess the resource demand of individual LLM requests, enabling the construction of a load anticipator for each LLM instance. By integrating both long-term and short-term predictions, PreServe adjusts resource allocation in advance, mitigating the risks of instance under- or over-provisioning. Moreover, PreServe optimizes request routing by considering both current and anticipated future instance loads, ensuring balanced load distribution across instances. Evaluations on real-world LMaaS production datasets demonstrate that \nm outperforms state-of-the-art approaches, achieving over 45.9% reduction in tail latency, an average 44.5% decrease in resource consumption, while incurring only 0.23% additional overhead.