- The paper introduces VR-Ly-EXP4, combining Lyapunov optimization with task-conditioned control-variate estimators to reduce depth-amplified variance under policy-dependent oracle feedback.
- The method achieves sublinear regret and long-term resource-constraint satisfaction with forced exploration, while its guarantees depend on baseline quality, strict feasibility, and finite threshold discretization.
- On 3-, 4-, and 5-layer benchmarks, VR-Ly-EXP4 delivers the best results, reaching a 0.4456 hit rate and 0.2923 error rate in five-layer systems.
Problem setting and motivation
Hierarchical inference (HI) systems route inference jobs through a sequence of computing layers, where each node either accepts its local prediction or offloads the job to a more capable upstream node. The paper studies online routing in such systems when the hierarchy is deep, offloading is multi-destination, and supervision is available only at a terminal oracle layer. This combination produces a learning problem with three coupled difficulties: the per-job loss is defined recursively along the routing path; feedback on prediction error is observed only if and when a job reaches the oracle layer, making observability policy-dependent and depth-sensitive; and routing must satisfy long-term per-node resource constraints alongside memory-limited model placement. The authors argue that existing HI formulations—largely two-layer, single-destination, or statically optimized—do not capture this recursive, policy-dependent feedback structure.
The key structural observation is that the marginal probability ρnπ(j) that a job reaching node n eventually reaches the oracle decays multiplicatively with routing depth. Consequently, importance-weighted estimators of the form $\mathbbm{1}_{\text{fb},n}(j) f / \rho_n^{\boldsymbol{\pi}}(j)$ are unbiased but suffer variance scaling as (1−ρ)/ρ, which grows exponentially in depth. In a 5-layer hierarchy, static baselines exhibit oracle feedback rates as low as 0.0002, so naive bandit learning receives almost no usable signal. This depth-amplified variance is the central technical obstacle the paper addresses.
System model
The network consists of K layers of nodes; layer 1 contains entry devices generating jobs, intermediate layers perform local inference and decide between termination and multi-destination offloading, and layer K is an error-free oracle. Each job carries a task type y(j) and a confidence score zn(j) realized at each visited node. Routing at node n is governed by a randomized policy πn over actions n0, where action 0 denotes local termination. Offloading incurs deterministic resource cost n1 combining communication and computation, regulated by long-term constraints n2. Nodes also periodically update their loaded model sets n3 subject to memory capacity n4. Inference error is Bernoulli per model, with unsupported modalities incurring error 1 deterministically. Arrivals are assumed i.i.d. with bounded first and second moments (Assumption 1), which underpins all theoretical guarantees.
Algorithm design
The proposed method, VR-Ly-EXP4, decomposes the problem across two timescales. On the fast timescale, Lyapunov drift-plus-penalty converts the long-term resource constraints into virtual queue stability: per-node queues n5 accumulate budget violations, and Proposition 3.1 shows mean-rate stability implies constraint satisfaction. The per-slot objective couples queue-weighted routing costs with the scaled inference error n6, where n7 trades accuracy against transient resource deviation.
On the learning side, each node runs EXP4 over a joint expert space n8, where experts pair confidence thresholds n9 with destination nodes $\mathbbm{1}_{\text{fb},n}(j) f / \rho_n^{\boldsymbol{\pi}}(j)$0. The main contribution is the variance-reduced loss estimator:
$\mathbbm{1}_{\text{fb},n}(j) f / \rho_n^{\boldsymbol{\pi}}(j)$1
The task-conditioned baseline $\mathbbm{1}_{\text{fb},n}(j) f / \rho_n^{\boldsymbol{\pi}}(j)$2 estimates the conditional expected expert loss for task type $\mathbbm{1}_{\text{fb},n}(j) f / \rho_n^{\boldsymbol{\pi}}(j)$3 via an exponential moving average updated only when terminal feedback arrives, itself divided by $\mathbbm{1}_{\text{fb},n}(j) f / \rho_n^{\boldsymbol{\pi}}(j)$4 to correct partial observability. Because the baseline is subtracted inside the importance-weighted term and added back outside it, unbiasedness holds regardless of baseline accuracy (Lemma A.1). Lemma 4.2 establishes variance reduction under the condition $\mathbbm{1}_{\text{fb},n}(j) f / \rho_n^{\boldsymbol{\pi}}(j)$5: the exact variance difference equals $\mathbbm{1}_{\text{fb},n}(j) f / \rho_n^{\boldsymbol{\pi}}(j)$6, which is non-negative precisely when the baseline neither vanishes nor overestimates twice the true loss. This condition is an assumption the paper concedes rather than proves globally; it depends on baseline quality, which in turn depends on feedback frequency.
Model placement is handled by a greedy marginal-density rule over a knapsack-constrained submodular utility combining expected local accuracy with a switching penalty. The paper proves submodularity but notes the utility is not necessarily monotone due to switching costs, so the classical $\mathbbm{1}_{\text{fb},n}(j) f / \rho_n^{\boldsymbol{\pi}}(j)$7 guarantee of Sviridenko does not directly apply—a limitation stated explicitly.
Theoretical guarantees
Theorem 4.5 bounds node-level regret relative to the best fixed threshold-destination expert by $\mathbbm{1}_{\text{fb},n}(j) f / \rho_n^{\boldsymbol{\pi}}(j)$8 plus a variance proxy term scaling with squared estimator magnitudes. Since the estimator variance enters this proxy directly, variance reduction tightens the regret bound. With $\mathbbm{1}_{\text{fb},n}(j) f / \rho_n^{\boldsymbol{\pi}}(j)$9, regret is (1−ρ)/ρ0 and virtual queues are mean-rate stable. Corollary 4.6 gives system-level near-optimality against the true continuous optimal policy:
(1−ρ)/ρ1
decomposing the gap into the Lyapunov trade-off, vanishing learning regret, and irreducible threshold discretization error (1−ρ)/ρ2. Finiteness of (1−ρ)/ρ3 relies structurally on the forced uniform exploration mix ((1−ρ)/ρ4), which lower-bounds (1−ρ)/ρ5 away from zero; without exploration, the guarantees would fail at depth. The analysis makes no stochastic assumptions on induced losses beyond arrival stationarity, relying instead on conditional unbiasedness and boundedness of the estimator.
Empirical results
Experiments use RouterBench and VL-RouterBench data: 79,988 samples spanning 114 task types over language and vision–language workloads, evaluated on 23 models across 3-, 4-, and 5-layer hierarchies with 20,000-job traces averaged over 5 runs. Key results:
| Method |
Hit rate (5-layer) |
Error rate (5-layer) |
| Random |
0.0 |
0.4705 |
| Round-Robin |
0.0 |
0.4693 |
| Pure Local |
0.0 |
0.4680 |
| Ly-EXP4 |
0.3949 |
0.3222 |
| VR-Ly-EXP4-LocalLoss |
0.4346 |
0.3103 |
| VR-Ly-EXP4 |
0.4456 |
0.2923 |
VR-Ly-EXP4 achieves the lowest error and highest hit rate (fraction of inherently unsolvable tasks routed to the oracle) at every depth, maintaining hit rates above 0.44 while static heuristics score exactly 0.0. The gap over standard Ly-EXP4 persists or widens with depth, consistent with the variance-reduction mechanism compensating for sparser feedback. Expert-weight entropy decays faster with variance reduction, indicating accelerated policy convergence. The LocalLoss ablation confirms that recursive upstream expected loss terms are necessary for accurate destination evaluation. An additional ablation shows greedy placement outperforms random-fixed and layer-diverse placement once adaptive routing is enabled, though it underperforms under purely random routing due to reduced exploration diversity—an interaction worth noting.
Limitations and open questions
Several conditions are conceded rather than resolved. The variance-reduction guarantee requires the baseline to satisfy (1−ρ)/ρ6 per job, which cannot be verified a priori when feedback is sparse; behavior under systematically poor baselines is unanalyzed. The near-optimality bound assumes Slater's condition for strict feasibility and inherits the discretization floor (1−ρ)/ρ7 from the finite threshold grid. The greedy placement scheme lacks the monotone-submodular approximation guarantee because of switching penalties. Confidence scores are modeled as Gaussian proxies centered at best-model error rates rather than measured uncertainty from deployed models, leaving open how real predictive-uncertainty signals affect routing quality. Finally, the regret analysis covers only fixed-expert comparators; competing against adaptive downstream policies remains open.
Conclusion
The paper formalizes online routing in deep hierarchical inference as a partial-feedback problem with recursively defined, policy-dependent observability, identifies depth-amplified estimator variance as the core instability, and resolves it with a task-conditioned control-variate estimator integrated into Lyapunov-constrained EXP4. It delivers sublinear regret, constraint satisfaction under stochastic arrivals, and empirical dominance over importance-weighted baselines on large-scale multi-task workloads, while leaving baseline-quality conditions and non-monotone placement guarantees as explicit open points.