Mixed Agentic Workload Scheduling

• Mixed Agentic Workload Scheduling (MAWS) is a framework that orchestrates heterogeneous, multi-stage tasks across CPU, GPU, and memory resources.
• It integrates dynamic scheduling, resource-aware dispatch, and predictive modeling to optimize latency, throughput, and fairness in agentic AI workflows.
• MAWS enables scalable and adaptive scheduling in multi-tenant environments while addressing input-dependent uncertainty and resource coupling challenges.

Mixed Agentic Workload Scheduling (MAWS) refers to a family of systems, algorithms, and principled frameworks designed to orchestrate the efficient, fair, and scalable execution of workloads comprising heterogeneous, multi-stage, and multi-agentic tasks—typically in environments serving LLM agents, coupled tool chains, and diverse user or service requirements. MAWS addresses the unique coupling of CPU, GPU, memory, and scheduling resources in agentic pipelines, which blend GPU-bound inference (text generation, multimodal reasoning) with CPU-heavy tool invocations (retrieval, scripting, analytics). This paradigm spans request-level scheduling, system co-design, cluster-wide orchestration, and OS-level policy mixtures, unifying advances in latency-sensitive agentic AI serving, cross-resource coordination, and adaptive online optimization under tight Service-Level Objectives (SLOs).

1. Problem Landscape and Motivation

Agentic AI workloads sharply deviate from classic, homogeneous inference because they execute iterative, dynamic workflows in which LLM calls, tool invocations, and orchestration logic interleave at fine granularity. Pipeline stages vary in computational intensity, memory footprint, and resource type; the composition and structure of the sequence typically depends on prompt semantics, user context, or upstream decision making. Key challenges arise from:

• Heterogeneity of Task Types: Mixes of long-running LLM reasoning, short interactive queries, tool calls, and multistage refinement exhibit heavy-tailed latency and variable critical paths.
• Resource Coupling: Simultaneous contention on GPU memory (KV cache), parallel decode slots, CPU tool threads, and process memory creates joint constraints requiring global view to avoid oversubscription or idling (Wang et al., 14 Apr 2026).
• Input-Dependent Uncertainty: The model call graph, output length, and per-stage latency depend on prompt features or agent plan, requiring on-the-fly estimation for effective scheduling (Huang et al., 19 Jun 2026, Wang et al., 11 Jun 2026).
• Multi-Tenant Environments: Cluster-scale, shared infrastructure magnifies fairness, isolation, and SLO guarantee requirements, especially under high-load or multi-tenant regimes (Wang et al., 11 Jun 2026, Peng et al., 8 May 2025).

2. Core MAWS Methodologies

MAWS systems employ several interlocking components to address the above challenges:

A. Actionable Scheduling Abstractions

• Request/Workflow Decomposition: Work is structured as DAGs or chains of LLM/tool stages, each described by dependency, compute, and resource demand signatures (Peng et al., 8 May 2025, Wang et al., 11 Jun 2026).
• Heterogeneity-Aware Dispatch: Schedulers must assign tasks to hardware matching not only queue length but also per-device capability, memory headroom, and expected affinity with query type or agent role (Wang et al., 11 Jun 2026, Peng et al., 8 May 2025).

B. Resource-Aware & Critical-Path Scheduling

• Holistic Admission Control: Closed-loop controllers (e.g., AIMD) jointly monitor GPU and CPU phase boundaries, dynamically calibrating the number of concurrent sessions to avoid overloads (Wang et al., 14 Apr 2026).
• Queueing and Preemption: Dynamic, urgency-based (SRPT, density-weighted) schemes order both global and local queues to prioritize latency-sensitive continuations, minimize head-of-line blocking, and backfill batch work (Wang et al., 11 Jun 2026, Huang et al., 19 Jun 2026, Sidik et al., 29 Jan 2026).
• Multi-Stage Slack Allocation: Per-request or per-stage urgency is computed based on remaining SLO slack, compute estimates, and current queue age, enabling deadline-aware prioritization at both global and replica level (Peng et al., 8 May 2025, Wang et al., 11 Jun 2026).

C. Predictive Modeling and Distributional Estimation

• Prompt Semantics Extraction: Deploy reduced-parameter semantic models or feature encoders to estimate token count, output length, tool-call intent, and model call topology per request; predictions are integrated into dispatch and scaling decisions (Huang et al., 19 Jun 2026, Wang et al., 11 Jun 2026).
• Quantile/Distributional Forecasting: MLP heads produce quantile sketches of expected latency or call count, which are preserved until the cost evaluation and decision stage for tail-aware optimization (Huang et al., 19 Jun 2026).
• Online Adaptation: OOD drift is detected by tail-quantile loss, triggering focused retraining of only the affected MLPs when prediction error breaches a threshold (Huang et al., 19 Jun 2026).

D. Agentic or Meta-Scheduling Policy

• Scheduler as Agent: At the process or cluster level, the scheduling policy itself is cast as an agent with a sense–think–act loop: runtime metrics inform workload pattern classification, which in turn triggers expert policy selection (e.g., SJF, DRF, FCFS) via mapping tables or time-weighted probability voting (Wang et al., 7 Nov 2025).
• Mixture of Schedulers: Rather than a monolithic scheduler, a portfolio of expert sub-schedulers is maintained, with online selection depending on the current workload class and system feedback (Wang et al., 7 Nov 2025).

3. Concrete Architectures and Algorithms

Table: Representative MAWS Systems and Their Distinct Approaches

System	Scheduling Principle	Resource Span
SwarmX	Neural predictors + tail quantile cost; adaptive agent framework	Multi-tenant, GPU/CPU (Huang et al., 19 Jun 2026)
Maestro	Role/prompt semantic prediction with hierarchical node/cluster scheduling; workflow-SRPT, memory-aware routing	Multi-cluster, multi-model KV cache (Wang et al., 11 Jun 2026)
MARS	Unified information stream; admission-execution decoupling; agent-centric MLFQ with prioritized resumption	GPU/CPU co-scheduling, agent sessions (Wang et al., 14 Apr 2026)
HEXGEN-TEXT2SQL	Hierarchical scheduler (global dispatcher + local urgency queue); simulation-based param tuning	Heterogeneous GPUs, stage DAGs (Peng et al., 8 May 2025)
MOSAIC	ILP expert placement; per-worker prompt assignment; adaptive aggregation gate	MoA (multi-agent), multi-GPU (Mitra et al., 2 Jun 2026)
ASA	Sched-agent selects expert policy via ensemble ML, mapping, and time-weighted voting	OS-level, process/batch (Wang et al., 7 Nov 2025)
EWSJF	Density-weighted SJF-inspired partitioning, meta-optimized online	LLM serving (vLLM), intra-batch (Sidik et al., 29 Jan 2026)

Each architecture is characterized by modules for: (1) feature extraction (semantic/prompt/role), (2) distributional cost prediction, (3) dynamic, multi-level queueing or admission, (4) online adaptation, and (5) robust plugin interfaces for integration with cluster managers or OS-level hooks.

4. Empirical Impact and Quantitative Gains

MAWS approaches report consistent, significant benefits across a variety of metrics, workloads, and deployment scenarios:

• Latency Improvement: SwarmX reduces P95/P99 latency by 11–61% across structured and agentic pipelines, with production P99 tail reduction up to 52% on CPU-only services (Huang et al., 19 Jun 2026). MARS achieves 5.9× lower mean latency versus FCFS in heterogeneous co-scheduling (Wang et al., 14 Apr 2026). Maestro increases SLO attainment by +23.6 pp versus EDF and reduces interactive queueing delays by >80% (Wang et al., 11 Jun 2026).
• Throughput and Goodput: Up to 2× sustainable throughput improvement (at fixed SLO) in GPU clusters with SwarmX (Huang et al., 19 Jun 2026), up to 1.75× throughput (vs. vLLM) in HEXGEN-TEXT2SQL for Text-to-SQL agentic pipelines (Peng et al., 8 May 2025), and 41–54% token throughput boost over FCFS in EWSJF (Sidik et al., 29 Jan 2026).
• Adaptive Overhead: Online adaptation in SwarmX detects severe regime shifts within ≲ 100 s (Huang et al., 19 Jun 2026); MOSAIC solves ILP placement in milliseconds for expert batches (Mitra et al., 2 Jun 2026).
• Multi-Tenant and Cross-Cluster: Maestro achieves a 67% reduction in KV-reserved HBM via elastic provisioning and overcommitment, enabling co-location of five model variants on a single A100 GPU; cross-cluster routing ablation shows latency cut from 36.99 s (baseline) to 30.77 s at λ=2.0 req/s (Wang et al., 11 Jun 2026).

5. Generalization, Extensions, and Practical Considerations

• Resource Mixes: MAWS policies can be tuned for environments where resource coupling is tight (e.g. GPU+CPU as in MARS), or where the main challenge is memory fragmentation and model co-location (Maestro) or agentic logic with LLM/Tool alternation (SwarmX, HEXGEN-TEXT2SQL).
• DAG-Structured and Nonlinear Workflows: Many-agent or chain-of-thought scenarios introduce trees or graphs of dependent stages, necessitating slack allocation, branch-aware prioritization, and minimum-impact memory coordination (Wang et al., 14 Apr 2026, Wang et al., 11 Jun 2026).
• Mixture-of-Schedulers for General OS Workloads: The ASA framework treats the OS scheduler as an agent that dynamically maps workload patterns to the best scheduler within a portfolio, leveraging ensemble ML and online voting (Wang et al., 7 Nov 2025).
• Multi-Agent RL Formulation: For geo-distributed scheduling, the MAWS paradigm can be cast as a fully cooperative Multi-Agent MDP with multi-objective reward (utility, energy, and carbon), optimized via actor-critic RL (MASAC), achieving +28.6% utility vs. local baselines (Zhang et al., 2023).
• Integration with Existing Infrastructure: Many MAWS systems are designed to be plug-ins for Ray, Kubernetes, Linux sched_ext, or vLLM, exposing a minimal, safe Action Set and requiring no user-facing API changes (Huang et al., 19 Jun 2026, Wang et al., 7 Nov 2025, Sidik et al., 29 Jan 2026).

6. Limitations and Open Challenges

• Prediction Granularity: Systems like SwarmX and Maestro rely on accurate per-request or per-stage cost forecasting; misclassification or inaccurate modeling (e.g., ratio of CPU to GPU work in tightly coupled workloads) can degrade performance (Huang et al., 19 Jun 2026, Raj et al., 1 Nov 2025).
• Resource Constraints: Multi-processing is not cost-effective when memory overcommitment triggers swapping; memory-bound KV cache usage at scale remains a primary limiter (Raj et al., 1 Nov 2025, Wang et al., 11 Jun 2026).
• Fairness and Starvation: While learning-based and priority-based policies outperform FCFS in both throughput and tail latency, they require further progress-aware adjustments for strong fairness or rate guarantees in adversarial and multi-tenant settings (Wang et al., 14 Apr 2026, Sidik et al., 29 Jan 2026).
• Extension to Cross-Domain, Cross-OS, and Heterogeneous Hardware: Instantiation of MAWS for Windows, BSD, ARM big.LITTLE, or FPGAs requires custom metric probes, scheduler-switching APIs, and mapping-table retraining (Wang et al., 7 Nov 2025).

7. Summary and Prospects

MAWS is a foundational abstraction for orchestrating heterogeneous, agentic pipelines, integrating advances in neural prediction, online adaptation, and agent-based policy mixture to address SLO-driven, high-throughput, low-latency LLM inference and multi-agent execution. The paradigm has practical instantiations in production-scale systems—SwarmX, Maestro, MARS, HEXGEN-TEXT2SQL, MOSAIC, EWSJF, and the Adaptive Scheduling Agent—demonstrating that agentic and workload-conscious scheduling can achieve significant tail-latency reductions, throughput gains, and robust adaptation across dynamic workload and infrastructure regimes (Huang et al., 19 Jun 2026, Wang et al., 11 Jun 2026, Wang et al., 14 Apr 2026, Peng et al., 8 May 2025, Mitra et al., 2 Jun 2026, Raj et al., 1 Nov 2025, Sidik et al., 29 Jan 2026, Wang et al., 7 Nov 2025, Zhang et al., 2023). Open research challenges include fully generalized fairness-aware scheduling for multi-tenant agentic workloads, efficient support for arbitrary DAG-structured agent plans, and real-time cross-cluster coordination under strict resource and latency constraints.