Clairvoyant: Predictive SJF Scheduling to Mitigate Head-of-Line Blocking in Serial LLM Backends

Abstract: Serial LLM inference backends -- such as Ollama -- process requests one at a time under FCFS admission, causing Head-of-Line Blocking (HOLB) under mixed workloads at high utilisation: short factual queries can be delayed by minutes behind long generation jobs. While cloud-scale deployments mitigate HOLB via continuous batching (vLLM, Orca), these solutions require tens of GB of VRAM for concurrent KV-caches -- infeasible for memory-constrained edge and local deployments that rely on serial request dispatch. We present \clairvoyant, a drop-in sidecar proxy for any serial OpenAI-compatible backend (e.g., Ollama, llama.cpp). \clairvoyant predicts response length from 19 lightweight lexical features via an ONNX-exported XGBoost classifier, achieving 0.029\,ms per-request latency (four orders of magnitude below typical generation time). Because admission scheduling depends on relative ordering rather than exact prediction, the system optimises ranking fidelity, achieving 62--96\% in-distribution and 52--66\% cross-distribution accuracy across natural conversation datasets. We find that curated instruction datasets are degenerate training sources for length prediction: GPT-imposed brevity constraints reduce Long-class representation to under 0.02\% of examples, making natural conversation logs the only viable training source. End-to-end GPU benchmarks on an RTX~4090 show 70--76\% P50 latency reduction for short requests under maximum queue pressure (100 concurrent requests) and 17\% under steady-state Poisson arrivals ($ρ=0.74$). \clairvoyant is open-source and requires no modifications to the inference backend.

• The paper presents Clairvoyant, a sidecar proxy that uses predictive SJF scheduling to mitigate head-of-line blocking in serial LLM backends.
• It employs a 3-class ONNX-exported XGBoost classifier with 19 lightweight lexical features to predict output length and prioritize requests.
• Empirical results demonstrate up to 76% latency reduction for short jobs while maintaining minimal overhead and controlled delay for long requests.

Clairvoyant: Predictive SJF Scheduling to Mitigate Head-of-Line Blocking in Serial LLM Backends

Motivation and Problem Setting

Serial LLM inference backends, such as Ollama and llama.cpp, operate under FCFS (First-Come-First-Served) admission control, processing one request at a time. In practical edge or resource-constrained environments, highly variable output-generation times—ranging from sub-10-token factual queries (< 2 s) to multi-thousand-token generative tasks (> 1 min)—result in significant head-of-line blocking (HOLB): short requests are forced to wait behind long jobs, often accumulating excessive latency. While continuous batching engines (e.g., vLLM, Orca) mitigate HOLB in high-concurrency cloud settings, their memory requirements (tens of GBs of VRAM for concurrent KV-caches) are prohibitive for edge/local deployments.

The paper introduces Clairvoyant, a drop-in, sidecar proxy that addresses queue-level HOLB for serial OpenAI-compatible backends via non-preemptive Shortest-Job-First (SJF) scheduling. The system intercepts API requests, predicts output length using low-latency lexical features and an ONNX-exported XGBoost classifier, and dispatches requests in SJF order, incorporating a starvation timeout to prevent indefinite deferral of long jobs.

Figure 2: System architecture: API requests are intercepted, efficiently featurized and predicted for output length, and then dispatched in SJF order, with starvation safeguards.

Scheduling Architecture and Predictor Design

Architectural Overview

Clairvoyant operates as a stateless sidecar proxy, introducing no modifications to upstream inference engines. The system comprises three tightly integrated components:

• Lightweight Feature Extraction: Nineteen lexical features—spanning prompt length approximations, code/format keyword presence, length instructions, syntactic complexity, and instruction verb type—are extracted using pure string manipulation with sub-microsecond cost per request.
• ONNX-Based Length Prediction: A 3-class XGBoost classifier, exported to ONNX, predicts $$[\mathrm{P}(\mathrm{Short}), \mathrm{P}(\mathrm{Medium}), \mathrm{P}(\mathrm{Long})]$$ with a latency of 0.015–0.029 ms on CPU, orders of magnitude lower than embedding-based alternatives or LLM generation costs.
• Min-Heap SJF Scheduling with Starvation Timeout: Requests are queued and prioritized by ascending $\mathrm{P}(\text{Long})$; starvation is controlled by promoting any request exceeding the calibrated timeout $\tau=3\times\mu_{\mathrm{short}}$ to the head.

This admission queue strictly addresses queue-level (Layer 1) HOLB, not token-level (Layer 2) blocking remediated in continuous-batching backends.

Data, Model Training, and Predictive Insights

Dataset Curation and Key Finding

The efficacy of length-predictive SJF scheduling fundamentally depends on the availability of diverse, representative long-response examples in training data. The authors identify that curated instruction datasets (Alpaca, CodeAlpaca, Dolly) are degenerate for this purpose, containing virtually no (<<0.1%) long-output instances due to upstream brevity constraints from instruction-following model construction. Only natural conversation logs (ShareGPT, LMSYS-Chat-1M, OASST1) exhibit sufficient Long-class diversity required for stable stratified training. This dataset analysis is a robust, distributional assertion documented with explicit class breakdowns across all public LLM datasets considered.

Model Training and Scheduling Formulation

Output-length prediction is cast as a 3-class softmax classification (Short: <200 tokens, Medium: 200–799, Long: ≥800), but for scheduling, the system optimizes and evaluates pairwise ranking accuracy (fraction of $(\text{Short},\text{Long})$ pairs where Long is ranked above Short) rather than plain classification accuracy. This ranking metric aligns directly with SJF's operational objectives and consistently outperforms 3-class classification by 21–29 percentage points in test conditions.

The predictor is an XGBoost model (300 trees, depth 6), trained on balanced, stratified subsets of natural conversation datasets. Ablation analysis reveals that prompt token length and instruction verb are key, while code and format indicators may be distributionally dependent.

Empirical Results

Latency and Scheduling Gains

On an RTX 4090 testbed (serial Ollama backend, Gemma3:4b and Llama3.1:8b models, $n=250$ per cell):

• Short-request P50 latency reductions of 70–76% under burst arrival, with P95/P99 gains of comparable scale.
• Long-request latency increases by 21–27%, a deliberate artifact of SJF ordering.
• Steady-state Poisson arrival experiments at $\rho=0.74$ show short-request P50 latency reduced by 17%, with the benefit peaking at mid-to-high utilization.
• At low queue utilization ($\rho\lesssim0.50$), HOLB is rare and SJF gains are negligible—therefore FCFS suffices.

Predictor and System Overhead

The ONNX classifier's $<$0.03 ms per-request latency is negligible compared to generation, and is four orders of magnitude faster than embedding-based approaches, which are measured at 12–149 ms P50–mean latency per request, and thus impractical for edge deployment.

Scheduling Efficacy

Clairvoyant achieves 62–96% in-distribution and 52–66% cross-distribution Short-vs-Long ranking accuracy, where random and optimized prompt-length rules offer only 50–56%. Heuristic keyword-based schedules degrade under cross-domain shifts.

Starvation Management and Practical Deployment

To prevent starvation of long requests, a starvation-promoting threshold $\tau=3\times\mu_{\mathrm{short}}$ is recommended, derived empirically and shown via Pareto analysis to optimally balance short-request latency reduction and bounded long-request inflation. This configuration is hardware- and workload-adaptive.

The system's operational envelope is precisely characterized: Clairvoyant provides measurable benefit for serial or moderately concurrent deployments where VRAM precludes continuous batching. It is unnecessary for cloud-scale backends with Layer 2 token-level scheduling or under low queue utilization.

Comparison to Related Work

Clairvoyant complements, rather than supplants, high-concurrency scheduling engines (vLLM (Kwon et al., 2023), Orca (Joseph et al., 2022), FastServe (Wu et al., 2023)), which inherently eliminate Layer 1 HOLB at the cost of significant VRAM consumption. It also targets a different layer than LTR/learning-to-rank scheduling (Fu et al., 2024), which is optimized for preemptive/batched backends and relies on higher-latency prompt embedding features. Compared to prior proxy-model approaches (Qiu et al., 2024, Xie et al., 12 Feb 2026), Clairvoyant achieves lower prediction latency and system complexity, and uniquely quantifies the failure of instruction datasets for real-world scheduling.

Limitations and Extensions

Clairvoyant is explicitly scoped to OpenAI-compatible, serial backends. Its cross-distribution accuracy declines outside the training domain, mandating deployment-time fine-tuning with production logs for optimal ranking fidelity. The system currently prioritizes mean short-request latency over per-tenant fairness; extensions for multitenancy are proposed but not implemented. Hardware-dependent tuning (i.e., accurate measurement of $\mu_{\mathrm{short}}$) is necessary for optimal starvation calibration.

Implications and Future Extensions

Clairvoyant operationalizes queue-theoretic SJF scheduling for latency-sensitive, resource-constrained LLM deployments. It empirically demonstrates that lightweight lexical features and low-latency models suffice for practical SJF ordering, and that more complex, data-hungry proxy models are unnecessary for the target regime. The clear class degeneracy finding in instruction datasets has immediate ramifications for scheduling research: only natural usage logs yield viable predictors.

Open directions include: integration of TTL-aware or per-tenant weighted prioritization for fairness, extension to non-English prompts, explicit TTFT latency instrumentation, and validation on additional serial backends.

Conclusion

Clairvoyant is an efficient, open-source system for non-preemptive, length-aware request admission control in serial LLM deployments. It reduces short-request latency by up to 76% relative to FCFS under high queue pressure, with negligible scheduling overhead, and operates entirely as a proxy layer without backend modifications. The work delineates the deployment boundary where lightweight SJF admission is both practical and beneficial, and it exposes the inadequacy of instruction-tuned datasets for real-world length-predictive applications. The methodology and open-science approach provide a robust foundation for further research in edge LLM serving optimization.