Two-layer Scheduling: Hierarchical Task Management

• Two-layer scheduling is a hierarchical framework that employs an upper layer for coarse-grained task grouping and a lower layer for detailed resource allocation.
• It enhances system efficiency by coordinating global strategies with localized scheduling, addressing data locality, load imbalance, and time-sensitive constraints.
• Applications span graph analytics, HPC, LLM inference, and edge–cloud computing, delivering notable gains in throughput, latency, and scalability.

Two-layer scheduling refers to the use of two hierarchically organized scheduling mechanisms, each handling a distinct level of granularity or system abstraction, with coordinated or sequential assignment of work at each level. This paradigm is prominent in diverse domains, including concurrent graph analytics, high-performance computing clusters, LLM inference serving, hierarchical edge–cloud computing, time-triggered networking, neural network training, and hierarchical parallel machines. In a canonical arrangement, the upper layer decides task grouping, placement, or prioritization on a coarse scale (cluster, batch, global micro-window), while the lower layer resolves finer resource allocation, ordering, or batching for sub-tasks or sub-units within the allocations received from above. Two-layer scheduling enables exploitation of system-specific structure—such as data locality, load imbalance, resource heterogeneity, or time-sensitive constraints—and has demonstrated substantial advantages in throughput, latency, scalability, and fairness.

1. Foundational Models and Formalism

Two-layer scheduling frameworks can be mathematically abstracted as sequential compositions of two mapping stages. For a generic system:

• Let $T$ denote the set of tasks, $R_1$ the set of upper-layer resources or allocations (e.g., clusters, time-windows, layers, processing elements), and $R_2(r_1)$ the set of lower-layer resources or subdivisions within upper-layer allocation $r_1$. The first layer computes an assignment or partition

$\pi_1 : T \rightarrow R_1$

followed by the second layer, which processes all tasks assigned to any $r_1$ via

$\pi_2^{(r_1)} : T_{r_1} \rightarrow R_2(r_1)$

where $T_{r_1} = \{ t \in T : \pi_1(t) = r_1 \}$.

• Objectives and constraints are typically split: Layer 1 handles global budget, placement, or latency/capacity envelope; Layer 2 addresses local resource contention, micro-scheduling, jitter isolation, batching, or convergence.

Classic examples instantiate this with different semantics—for graph concurrency, blocks and jobs; for HPC, batch and loop/task levels; for networking, global time windows and per-link offset schedulers; and so on.

2. Principal Mechanisms in Representative Systems

Concurrent Graph Processing ("Efficient Two-Level Scheduling for Concurrent Graph Processing" (Zhao, 2018))

• Layer 2—Multiple Priority-Based Data Scheduling (MPDS): Establishes a global queue of prioritized graph data blocks by merging per-job local priority queues (using a dual-factor order over summary statistics of non-converged importance, with linear sampling and merge), selecting blocks to load into cache based on highest projected aggregate benefit across all jobs.
• Layer 1—Correlations-Aware Job Scheduling (CAJS): When a block is in cache, only dispatches jobs active on that block (have not converged on it), so all their updates proceed in parallel, maximizing cache efficiency and eliminating redundant reloads.
• Formal model: Block priorities $S_j(B_b) = (n_j(b),\mu_j(b))$, global merge $G(B_b) = \sum_{j=1}^N \varphi_j(B_b)$, cache-state indicator $R_1$0.
• Quantitative impact: Reduces LLC miss rate by 44%, convergence iterations by 32%, and roughly doubles throughput compared to single-level baselines.

High-Performance Computing (Two-Level Simulation, Batch and Application) (Eleliemy et al., 2018)

• Batch-Level Scheduling (BLS): Assigns jobs (each possibly parallel) to clusters/nodes, sequencing job launches for system-level objectives (e.g., First-Come-First-Serve, Earliest-Deadline-First).
• Application-Level Scheduling (ALS): Decomposes each job into parallelizable tasks (iterations, blocks) and schedules within node allocations for local load balance (STATIC, Self-Scheduling, Guided Self-Scheduling, Factoring).
• Inter-layer interaction: Poor ALS can delay job completion, which in turn keeps cluster nodes occupied, degrading global throughput. Simultaneous simulation reveals that tightly coupled BLS+ALS policies yield up to 25% improvements in makespan and resource utilization.

LLM Inference Serving ("A Predictive and Synergistic Two-Layer Scheduling Framework" (Zhang et al., 27 Sep 2025))

• Cluster layer (PRISM): Routing layer that assigns inference requests to backends using real-time, predictive metrics—estimated per-step latency, workload, free KV-cache, and heterogeneous capacity—computed from engine-internal performance models.
• Engine layer (LENS): SLO-aware batching layer per engine; dynamically adapts batch and token scheduling in response to incoming SLOs and workload characteristics, optimizing for tail-latency and throughput under fluctuating conditions.
• Performance: Achieves 43% higher SLO attainment rate and up to 3× throughput increase versus leading prior serving baselines.

Edge–Cloud Workflow Scheduling ("Cloud-Fog-Edge Collaborative Computing for Sequential MIoT Workflow" (Fu et al., 24 Oct 2025))

• Tier 1 (Global DDPG Agent): Selects which compute layer (edge, fog, cloud) to assign each workflow stage, balancing communication overhead versus compute.
• Tier 2 (Local DDPG Agents): Assigns specific node within the chosen layer, load-balancing and observing node-level memory and execution constraints.
• Hierarchical policy enables: Decomposition of the NP-hard optimal assignment, yielding up to 5.3% improvement over greedy approaches for long, complex medical workflows.

3. Algorithms and Scheduling Complexity

Two-layer scheduling approaches consistently exploit decomposability to mitigate computational complexity and exploit modularity:

• Graph analytics (MPDS+CAJS): Per-job and global queue construction requires $R_1$1 time per iteration; cache-optimized scheduling inside blocks eliminates $R_1$2 repeated loads.
• Hierarchical networks (H2S (Geppert et al., 2023)): Global phase partitions $R_1$3 streams into $R_1$4 micro-windows in $R_1$5, with local (per-window) subproblems (offset assignment) reduced to small-size ILP or greedy heuristics; leads to total scheduling time (per $R_1$6 streams) of under $R_1$7 second.
• Parallel neural network training (LayerPipe (Unnikrishnan et al., 2021)): Schedules intra-layer (activation/weight gradients) and inter-layer (channel block "borrowing") concurrency via a target per-processor budget, exploiting fine-grained pipelining, with $R_1$8 assignment time and a 25–80% reduction in end-to-end compute cycles.
• HPC batch-application simulation: Each job's ALS is simulated in parallel, with feedback injected into the BLS event queue, requiring only coordination via connection-layer synchronization.

4. Performance, Scaling, and Trade-offs

Two-layer scheduling frameworks demonstrate favorable scaling, efficiency, and operational trade-offs:

• Concurrent graph processing: 2.1× throughput gain at 8-way concurrency, with scaling remaining sub-linear under increasing parallel jobs; CAJS+MPDS incurs an overhead of less than 5% of runtime, amortized across many blocks (Zhao, 2018).
• Hierarchical time-triggered networks: Admission ratios exceed 97% for 45,000 streams; per-hop jitter is zero, and end-to-end latency is tightly controlled within 1μs of bound (Geppert et al., 2023).
• LLM serving: Full two-layer synergy (PRISM+LENS) outperforms either component alone (3.2× improvement vs. 2.5× and 1.4× for engine-only and router-only, respectively) (Zhang et al., 27 Sep 2025).
• Edge–cloud scheduling: Deeper sequential workflows benefit from global/local division, with the gap to HEFT narrowing for long chains, suggestive of increasing efficacy with scale and complexity (Fu et al., 24 Oct 2025).
• Neural network training: Up to 80% speedup over PipeDream with 9 processors; communication overhead per split is negligible relative to base inter-layer activation exchange (Unnikrishnan et al., 2021).
• HPC two-level simulation: Denser ALS policies (SS, GSS, FAC) robustly insulate against workload variability, containing makespan increase to under 10% under large-per-task-variation; static chunking can degrade performance by 20–30% in the presence of skew (Eleliemy et al., 2018).

System/Domain	Upper Layer Policy	Lower Layer Policy	Primary Gain
Graph analytics (Zhao, 2018)	Block selection (MPDS)	Job scheduling in block (CAJS)	2.1× throughput, 44% reduction in cache misses
HPC (Eleliemy et al., 2018)	Batch scheduling (BLS)	Loop/task partitioning (ALS)	8–36% makespan drop with good pairing
LLM serving (Zhang et al., 27 Sep 2025)	Request routing (PRISM)	SLO-aware batching (LENS)	43% SLO attainment, 3× throughput
Hierarchical net (Geppert et al., 2023)	Micro-window allocation	Per-link offset scheduling	Sub-second planning, 97–99% admitted
Edge–cloud (Fu et al., 24 Oct 2025)	Layer selection (global DDPG)	Node assignment (local DDPG)	5.3% gain vs. greedy, stable scaling
DNN Training (Unnikrishnan et al., 2021)	Inter-layer split (scheduling)	Intra-layer split (pipelining)	25–80% speedup, negligible overhead

5. Generalizations, Applicability, and Limitations

The two-layer scheduling paradigm generalizes across (i) work domains with multi-level hardware or logical resources, (ii) data sharing and contention phenomena, and (iii) temporal or fairness constraints. Applicability is demonstrated for:

• Concurrent iterative graph workloads, stochastic gradient descent pipelines, key–value store request routing, real-time network flows, batch + intra-application loop/task assignment, and deep neural network training partitions (Zhao, 2018, Eleliemy et al., 2018, Zhang et al., 27 Sep 2025, Geppert et al., 2023, Unnikrishnan et al., 2021).
• Iterative schemes benefit particularly from synchronized attention to data-access correlation and resource sharing, e.g., blocks in memory or cache, or token slots on GPU.
• Hierarchical approaches, as in cloud-fog-edge or multi-tier networks, mitigate the combinatorial action spaces encountered in monolithic (single-layer) formulations.

Limitations and trade-offs:

• Block granularity and window sizing may introduce short-latency delays or underutilization at the micro-scale, though these are generally outweighed by upstream efficiency.
• Overhead introduced by queue computation, model updating, or local window assignment is minimal in evaluated systems (typically 1–7% of total runtime).
• Effectiveness can depend on workload properties; optimal parameter boundaries (e.g., queue lengths, window caps, block sizes) must be calibrated to match system dynamics and workload scale.

6. Design Insights and Future Directions

Key guidelines for designing two-layer scheduling systems:

• Mutual awareness is crucial: Inter-layer feedback, such as exposing top-down SLOs/priority estimates or propagating bottom-up state summaries, enhances efficiency and responsiveness (e.g., PRISM/LENS; BLS/ALS with simulated feedback).
• No single optimal pairing: The best inter-layer algorithmic match depends on job/task variability, node heterogeneity, and service objectives; adaptive selection is preferable (Eleliemy et al., 2018).
• Decomposition for scalability: Hierarchical decomposition is effective for large-scale instances, converting intractable exact scheduling to tractable per-window or per-group subproblems (e.g., H2S network scheduling).
• Cross-layer synergy: Predictive or learning-based frameworks (e.g., SynergySched, DDPG-based MIoT) enable anticipatory scheduling, outperforming reactive or static policies as system scale and heterogeneity grow.

A plausible implication is that as distributed and multi-level systems proliferate (heterogeneous clusters, neurocomputing, IoT, hybrid cloud/edge), two-layer scheduling architectures will increasingly underpin practical high-performance, fair, and robust resource orchestration.