Cause of throughput decline with longer outputs in vLLM adapter serving

Determine the exact cause of the observed reduction in maximum throughput and decrease in the number of served adapters as output sequence lengths increase when serving LoRA adapters with the vLLM framework, and ascertain whether preemption arising from vLLM’s greedy memory allocation strategy is responsible for disproportionately affecting longer requests.

Background

In the performance analysis of adapter-serving workloads, the authors observe that increasing output sequence lengths leads to reduced maximum throughput and fewer adapters being served at the optimal placement. This effect appears alongside other overheads such as increased memory usage from adapter weights and computational overhead from mixing adapters in batches.

The paper uses vLLM’s online batching with a greedy memory allocation strategy. The authors hypothesize that preemption introduced by this strategy may disproportionately affect longer requests, but they explicitly note that the exact cause of the throughput and placement degradation with longer outputs is not fully understood.