Precisely localize the queue-saturation knee

Determine the precise concurrency threshold at which latency saturation begins on the tested NVIDIA A100 and vLLM 0.9.1 stack by conducting finer-grained experiments between 200 and 500 concurrent users.

Background

The benchmark evaluates concurrency levels of 200 and 500 users but does not sample intermediate loads. Although the measurements reveal a marked latency transition across this interval, they do not establish the exact point at which the system enters saturation. Identifying this knee would improve capacity characterization for the tested hardware, software, and workload configuration, while avoiding the interpretation of 200 users as a universal production-capacity threshold.

References

Because the load-testing matrix does not sample intermediate points between 200 and 500 concurrent users, the precise saturation knee cannot be identified from the present experiments.

— Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing  (2610.09778 - Olympio et al., 7 Oct 2026) in Section 4.1, “Queue Saturation Discovery”