---
title: Deep Ensemble Router (DER)
url: https://www.emergentmind.com/topics/deep-ensemble-router-der
type: topic
---

# Deep Ensemble Router (DER)

The Deep Ensemble Router (DER) is a framework designed for dynamic ensemble reasoning over a pool of large language model (LLM) experts. It integrates multiple LLMs in a sequential decision process, optimizing both output quality and computational efficiency by adaptively routing queries through selected experts and leveraging knowledge transfer between them. DER models the ensemble reasoning process as a Markov Decision Process (MDP), with a dedicated agent responsible for expert selection and answer refinement at each stage, trained using Proximal Policy Optimization (PPO) with explicit cost and quality awareness [2412.07448].

## 1. Markov Decision Process Formulation

DER represents the routing and refinement of answers as an episodic MDP $\langle\S,\A,\T,\R,\pi_\theta\rangle$. At each step, the agent decides which expert model from the set $\{\M_1,\dots,\M_N\}$ to invoke, aiming to improve the current answer using as few computational resources as possible.

### MDP Components

- **State Space ($\mathcal{S}$):** At time $t$, the state is $s_t = [ Q: x, \; A: \hat y_{t-1}]$, encoding the original question $x$ and the best answer $\hat y_{t-1}$ so far.
- **Action Space ($\mathcal{A}$):** Discrete set $\{1,2,\dots,N\}$, selecting the next expert to query.
- **Transition Function ($\mathcal{T}$):** On choosing action $a_t$, the agent feeds a Knowledge Transfer Prompt (KTP) to expert $\M_{a_t}$, producing $\hat y_t$. The next state is $s_{t+1} = [Q:x, A:\hat y_t]$.
- **Termination:** The episode ends either when an automated Terminator (a trained classifier) deems the answer satisfactory, or after reaching a pre-defined maximum step $T_{\max}$.
- **Reward ($\mathcal{R}$):** At step $t$,
  - For $t=0$: $r_t = P(\hat y_t) - \alpha\,C(\M_{a_t})$
  - For $t>0$: $r_t = P(\hat y_t) + \beta\,\Delta P_t - \alpha\,C(\M_{a_t})$, with $\Delta P_t = P(\hat y_t)-P(\hat y_{t-1})$
  - Terminal adjustment: $+\gamma$ bonus if success ($P(\hat y_T)\geq p_0$), $-\gamma$ penalty otherwise.
  - Example hyperparameters: $\alpha=0.001$, $\beta=5.0$, $\gamma=2.0$.

This formalism enables DER to optimize answer quality (measured by BERTScore $P(\cdot)$) while minimizing overall compute, measured as expert parameter counts.

## 2. DER-Agent Architecture

The DER agent consists of an encoder, policy head (actor), and value head (critic):

- **Input Encoding:** Each state $s_t$ is serialized as $[Q:x, A:\hat y_{t-1}]$, which is input to a pre-trained OPT-125M transformer encoder.
- **Policy Head:** Two linear layers are stacked atop the encoder's last hidden state, producing logits $f_\theta(s_t) \in \mathbb{R}^N$. The probability of selecting action $a_t=i$ is
  $$
  \pi_\theta(a_t=i|s_t) = \frac{\exp(f_\theta(s_t)_i)}{\sum_{j=1}^N \exp(f_\theta(s_t)_j)}
  $$
- **Value Head:** A copy of the encoder plus two linear layers yields a scalar estimate $V_\phi(s_t)$ for value prediction.
- **Terminator:** A lightweight OPT-125M-based classifier predicts whether the current answer meets the BERTScore threshold $p_0$ for early stopping.

The total routing infrastructure is compact (approximately 125M parameters each for policy and value networks), allowing lightweight autonomous control.

## 3. Training Procedure and Objective

DER employs PPO for policy optimization:

- **Trajectory Collection:** Rollouts of $\tau=(s_0,a_0,r_0,\ldots,s_T)$ are sampled using the current policy, collecting states, actions, and rewards.
- **Advantage Estimation:** Computed via $A_t = r_t + \gamma V_{\phi_{\rm old}}(s_{t+1}) - V_{\phi_{\rm old}}(s_t)$.
- **Actor Objective:** PPO's clipped objective:
  $$
  \mathcal{L}_\text{actor}
  = \mathbb{E}_t\left[\min\left(\rho_t A_t, \; \mathrm{clip}(\rho_t,1-\epsilon,1+\epsilon)A_t\right)\right]
  $$
  with $\rho_t = \pi_\theta(a_t|s_t)/\pi_{\theta_{\rm old}}(a_t|s_t)$, $\epsilon=0.2$.
- **Critic Update:** Minimizes TD-error $\delta_t = (r_t + \gamma V_\phi(s_{t+1})) - V_\phi(s_t)$.
- **Early Termination Mechanism:** No formal curriculum; the episode self-terminates on easier samples, thus naturally economizing computation.

This joint PPO framework is central to optimizing the quality-cost tradeoff and supports generalization across diverse question types and ensemble expert sets.

## 4. Knowledge Transfer Prompt (KTP)

The Knowledge Transfer Prompt is a template mechanism provided to each expert at every step to ensure the newly invoked LLM leverages prior answer information constructively:

```
[Question: x]
There is an answer to the question from another student:
[Previous Answer: ŷ_{t−1}]
Using another student's answer as additional advice, you need to give a more satisfactory answer directly. DO NOT mention other students.
```

This prompt construction enforces that the expert acts as a student solicited to improve upon a previous answer, integrating but not merely restating, the prior solution. This mechanism fosters cumulative answer refinement and demonstrably boosts downstream quality metrics. Ablation studies indicate that removal of KTP degrades BERTScore performance (from 75.0 to 74.3), highlighting its integral role [2412.07448].

## 5. Inference Procedure and Hardware Efficiency

During inference, DER operates as follows:

- For up to $T_{\max}$ steps, the current state $[Q:x, A:\hat y_{t-1}]$ is encoded;
- The policy selects the next expert to invoke;
- The KTP is constructed and presented to the chosen LLM;
- If the Terminator signals satisfactory quality, the episode halts early; otherwise, the sequence continues.

Pseudocode:

```python
function DER_Inference(x, π, Terminator, T_max, p0):
  ŷ ← None
  for t in 0..T_max-1:
    s ← Encode([Q:x, A:ŷ])
    a ← argmax_i π(a=i | s)  # or sample
    prompt ← KTP(x, ŷ)
    ŷ_new ← M_a(prompt)      # call expert
    if Terminator([Q:x, A:ŷ_new]) == "stop":
      return ŷ_new
    ŷ ← ŷ_new
  return ŷ  # last answer if no early stop
```

Empirical results demonstrate that DER infers using only 15–20B parameter-inference per sample on average, a substantial reduction compared to 117B for PairRanker and 234B for full debate-based ensembles. Over half of all episodes conclude within two steps, amplifying hardware savings.

## 6. Experimental Performance and Comparative Analysis

DER established significant improvements in both efficiency and output quality on standardized benchmarks:

| Method               | Average Inference Cost (B Params) | BERTScore | GSM8K Accuracy |
|----------------------|:----------------------------------:|:---------:|:--------------:|
| Vicuna-13B           | 13                                | ≈69.6     | 33.74%         |
| PairRanker           | 117                               | ≈73.0     |  –             |
| Full Debate Ensemble | 234                               | –         |  –             |
| DER                  | 17 (MixInstruct), 26 (GSM8K)      | ≈75.0     | 34.98%         |

DER's BERTScore of ≈75.0 on the MixInstruct test set outperforms single best open-source LLMs and PairRanker, while utilizing an order of magnitude fewer parameters. On GSM8K, DER improves the Vicuna baseline by over 1 percentage point in accuracy, concurrent with a fivefold reduction in compute cost.

Ablation findings establish the significance of both the KTP and the incremental/terminal rewards to DER's effectiveness; without these components, BERTScore drops by up to 3% [2412.07448].

## 7. Significance and Context within Ensemble Reasoning

DER reframes LLM ensemble reasoning as a computationally-aware, sequential improvement task, moving beyond static voting or pairwise ranking approaches. It demonstrates that a lightweight router can dynamically leverage complementary expertise and knowledge, achieving higher answer quality under stringent computational budgets. The framework's efficient policy-and-critic design, sequential reasoning via MDP, and knowledge transfer mechanism collectively render it effective for a range of multi-expert LLM scenarios.

A plausible implication is that sequential refinement ensembles with explicit cost-quality tradeoffs, as pioneered by DER, are likely to set new standards for both data efficiency and model utilization in multi-agent language reasoning architectures [2412.07448].

Source: https://www.emergentmind.com/topics/deep-ensemble-router-der