EMA-Guided Router in ORCH Framework

• EMA-Guided Router is an adaptive mechanism that uses exponential moving averages to prioritize LLM agents based on historical accuracy, latency, cost, and stability.
• It deterministically assigns roles within the ORCH multi-agent pipeline by selecting top-K agents and balancing performance trade-offs through weighted scoring.
• Empirical evaluations on benchmarks like MMLU-Pro show notable accuracy improvements with modest latency overhead, underscoring its value in controlled evaluation scenarios.

An EMA-guided router is an adaptive agent selection mechanism within the ORCH multi-agent framework for discrete-choice reasoning with LLMs. It employs per-agent exponential moving averages (EMAs) of accuracy, latency, cost, and stability, producing agent-specific routing scores to deterministically select which LLMs execute each role in the orchestration pipeline. This approach enables dynamic prioritization of agents exhibiting superior historical performance while penalizing slow, costly, or unreliable endpoints, and is intended primarily for benchmarking, controlled evaluation, or delayed-feedback scenarios rather than live deployments lacking immediate gold labels (Zhou et al., 2 Feb 2026).

1. Formal Structure and Scoring Mechanism

Consider a set $A$ of LLM agents. For each agent $a\in A$ and query timestep $t$, four scalar feedback signals are monitored:

• $q_t^a \in \{0, 1\}$: correctness (1 if agent's answer matches the gold label, else 0)
• $\ell_t^a \in \mathbb{R}_+$: measured API latency (e.g., seconds)
• $c_t^a \in \mathbb{R}_+$: call cost estimate
• $s_t^a \in \{0, 1\}$: stability (1 if no error/timeout/format violation, else 0)

For each signal $s \in \{q, \ell, c, s\}$ and agent $a$, an EMA is maintained:

$$\text{EMA}_s^a(t) = \alpha \cdot s_t^a + (1 - \alpha)\cdot \text{EMA}_s^a(t-1)$$

where $a\in A$0 is a smoothing parameter controlling the recency bias. Initialization for $a\in A$1 uses neutral priors (e.g., $a\in A$2 for quality/stability; average values for latency/cost).

A routing score $a\in A$3 for each agent is then calculated to reward high accuracy and stability while penalizing high latency and cost:

$a\in A$4

with tunable, non-negative weights $a\in A$5 specifying the trade-offs.

2. Routing Algorithm and Pipeline Integration

Within ORCH, the EMA router operates before the "many analyses, one merge" pipeline. For each test instance, it follows a sequence:

1. Routing Score Computation: Evaluate $a\in A$6 for each agent using current EMAs.
2. Top-K Agent Selection: Select the agents with the top $a\in A$7 scores (default $a\in A$8; $a\in A$9 for cost-sensitive variants), denoted $t$0.
3. Role Assignment:
   • The highest-scoring agent in $t$1 is assigned the decomposition ("dispatcher") role, responsible for generating subquestions.
   • The next highest-scoring agent becomes the merger, synthesizing the final answer.
   • All agents in $t$2 perform their respective analyses in parallel.
4. Delayed Feedback Update: Once gold labels are received, EMAs for each participating agent are updated with the observed $t$3 values, ensuring that the routing policy incorporates recent performance.

Throughout, deterministic procedures—fixed prompts, temperature 0, and tie-breaking rules—ensure reproducibility given fixed evaluation histories.

3. Hyperparameter Configuration

The EMA-guided routing layer exposes several critical hyperparameters:

• Smoothing Factor $t$4: Determines EMA memory horizon. Empirically, $t$5 balances responsiveness and noise.
• Combination Weights ($t$6): Typically $t$7, $t$8, $t$9; these values follow a grid search on held-out MMLU data to maximize accuracy under cost constraints.
• Top-K Agents: Defaults to $q_t^a \in \{0, 1\}$0 for parity with baseline; $q_t^a \in \{0, 1\}$1 variants reduce cost/latency by $q_t^a \in \{0, 1\}$215% at an accuracy trade-off of $q_t^a \in \{0, 1\}$33 percentage points on MMLU-Pro.

All hyperparameters can be adjusted to reflect changing priorities such as stricter cost budgets or latency caps.

4. Handling Delayed Feedback and Cold Start

Agent-level correctness feedback ($q_t^a \in \{0, 1\}$4) depends on reference gold labels, which are only observed after final answer synthesis. The router defers EMA updates until these become available, simply buffering feedback per agent per query. For cold start, EMAs are seeded to neutral priors (e.g., $q_t^a \in \{0, 1\}$5, $q_t^a \in \{0, 1\}$6) to avoid penalizing agents with no prior data. In live applications lacking immediate gold, one would substitute proxy feedback metrics (e.g., user votes, verifier confidence) while maintaining structural consistency.

5. Empirical Performance and Statistical Evaluation

On controlled benchmarks, EMA-guided routing yields consistent gains, particularly on more difficult tasks:

Benchmark	ORCH (fixed)	ORCH + EMA	Δ accuracy	Latency (ms/q)	b (correct only EMA)	c (correct only fixed)	McNemar p-value
MMLU (4-way)	0.813	0.820	+0.7 pp	11,775 → 12,315	15	13	≈0.71 (n.s.)
MMLU-Pro (10-way)	0.733	0.753	+2.0 pp	18,263 → 19,646	75	18	≈3.4×10⁻⁹ (≪0.05)

• On MMLU, the accuracy gain is not statistically significant (paired McNemar test, $q_t^a \in \{0, 1\}$7).
• On MMLU-Pro, the accuracy gain is statistically significant ($q_t^a \in \{0, 1\}$8).
• Latency overhead is marginal (5–7%) due to added computation and ranking.

6. Practical Considerations and Limitations

Convergence of the EMA is governed by $q_t^a \in \{0, 1\}$9 and the stationarity of agent performance. Higher $\ell_t^a \in \mathbb{R}_+$0 offers faster adaptation but increased noise, which may be necessary in dynamic environments (e.g., post-upgrade). The greedy, deterministic top-K selection can potentially eliminate feedback for lower-ranked agents, risking suboptimal long-term performance; periodic exploration strategies (e.g., $\ell_t^a \in \mathbb{R}_+$1-greedy selection) can mitigate this.

The EMA router is lightweight in operational overhead (four scalars per agent and per-query sorting) relative to LLM inference latency. In scenarios lacking gold labels, reliance on proxy feedback incurs bias risks, underscoring the importance of continual monitoring and periodic calibration.

7. Theoretical Guarantees and Deployment Scope

Each agent’s EMA for a given signal converges (in expectation) to the true metric, assuming stationarity. However, non-stationary conditions or feedback delays can introduce transient discrepancies. The EMA-guided router is designed primarily for evaluation-and-benchmarking settings with delayed feedback, and is not recommended for open-ended live environments unless robust proxy supervision is available.

The method does not introduce stochastic elements into ORCH’s core pipeline. All routing and aggregation remain deterministic, conditioned on the sequence of observed EMAs and deterministic agents’ outputs. The mechanism enables transparent, history-adaptive agent orchestration while maintaining reproducibility and interpretability of multi-agent LLM systems for discrete-choice reasoning (Zhou et al., 2 Feb 2026).