---
title: Mixture of Grounding Experts
url: https://www.emergentmind.com/topics/mixture-of-grounding-experts
type: topic
---

# Mixture of Grounding Experts

Searching arXiv for the specified papers and closely related work on Mixture of Grounding Experts.
In one formulation, **Mixture of Grounding Experts** is realized by letting each local expert be either a physics-grounded model (grey-box) or a purely data-driven model (black-box) and learning how to blend them via smooth time-varying convex weights [2401.17118]. In a later web-agent formulation, the same term denotes a two-tier pipeline in which a multimodal visual grounding module is tried first and a structural DOM-based module is invoked as fallback [2602.02468]. Related work on sparse Mixture-of-Experts language models uses **grounding experts** to denote **context-faithful experts** whose routing and internal computations amplify attention to relevant context spans [2508.19594]. Taken together, these usages center on a common objective: improving grounding by exploiting complementary expert specializations rather than a single undifferentiated predictor.

## 1. Conceptual scope and terminological usage

The term emerged explicitly in explainable data-driven modeling as a response to a familiar asymmetry: traditional models grounded in first principles often struggle with accuracy as the system’s complexity increases, whereas machine learning approaches face challenges in interpretability and in handling physical constraints. The proposed answer is a “mixture of experts” rationale in which diverse local models are fused in a data-driven way while preserving first-principle-based priors when available [2401.17118].

A distinct but compatible usage appears in autonomous web agents. There, Mixture of Grounding Experts is not a convex ensemble of predictive functions but a hierarchical control mechanism for action grounding: a visual-canvas expert handles the default pathway, and a semantic-structural expert handles failure cases or operations that demand ultra-fine control [2602.02468]. The two experts are arranged hierarchically, visual first and structural second, to achieve both robustness and efficiency.

A third usage, in sparse MoE LLMs, does not present a framework named MoGE but studies experts that are specialized for context utilization. In that setting, grounding experts are those experts whose selection and internal computations systematically amplify attention to the truly relevant context spans, thereby driving context-grounded generation [2508.19594]. This suggests a broader interpretation of the phrase: “grounding” refers not only to physical priors, but also to reliable anchoring in context, interface state, or structured metadata.

## 2. Mathematical formulation in explainable data-driven modeling

The canonical formalization approximates an unknown nonlinear map
$$
y(t)=f^0(x(t))+e(t)
$$
by a convex combination of $M$ local experts:
$$
\hat y(t)=\sum_{i=1}^M \omega_i(t)\,f_i(x(t);\theta_i).
$$
Here, $f_i(\cdot;\theta_i)$ can be any local model, grey-box or black-box, embedding first-principle priors when available, and $\omega_i(t)\in[0,1]$ satisfy $\sum_i \omega_i(t)=1$ and represent the gating weights or trust levels [2401.17118].

The joint learning problem is written as
$$
J(\Theta,\Omega)=\ell(X,Y;\Theta,\Omega)+r(\Theta)+\mathcal L(\Omega),
$$
where $\Theta=[\theta_1,\dots,\theta_M]$ and $\Omega=\{\Omega(t)\}$. The data-fitting term can be expressed at the mixture level,
$$
\ell^{\text{mix}}=\sum_{t=1}^T c\cdot \left\|y(t)-\sum_i \omega_i(t)f_i(x(t);\theta_i)\right\|_2^2,
$$
at the local level,
$$
\ell^{\text{loc}}=\sum_{t,i}\omega_i(t)\cdot c_i\cdot \|y(t)-f_i(x(t);\theta_i)\|_2^2,
$$
or in a combined cooperative-plus-competitive form,
$$
\ell=\sum_{t=1}^T \left[\|y(t)-\Omega(t)^T f(x(t);\Theta)\|^2+\beta\sum_{i=1}^M \omega_i(t)\cdot \|y(t)-f_i(x(t);\theta_i)\|^2\right].
$$

The parameter regularizer is
$$
r(\Theta)=\lambda_\theta\sum_{i=1}^M r_i(\theta_i),
$$
described as a group regularizer on expert parameters, for example $\ell_2$-shrinkage. The gating regularizer shapes temporal evolution:
$$
\mathcal L(\Omega)=\eta\sum_{t=2}^T \|\Omega(t)-\Omega(t-1)\|_2^2,
$$
with $\eta>0$ penalizing abrupt gating changes and enforcing smooth switches among experts. This smoothness term is also presented as
$$
\mathcal L^{\text{trans}}(\Omega(t),\Omega(t-1))=\eta\|\Omega(t)-\Omega(t-1)\|_2^2,
$$
and is interpreted as encouraging $\Omega(t)$ to vary smoothly in time under a Markov prior while avoiding physically implausible rapid regime-switching [2401.17118].

Grey-box experts can embed explicit physics. One example is a single-track vehicle-dynamics model,
$$
\dot y(t)=A(\nu_x(t),c_f,c_r,\dots)\,y(t)+B(\nu_x(t))\,s(t),
$$
with $\{c_f,c_r,m,J_z,l_f,l_r\}$ as parameters $\theta_i$ learned from data. In that case, $r_i(\theta_i)$ can enforce physical plausibility, including positivity and physical bounds, via constraints or penalties.

## 3. Optimization, specialization, and interpretability

Training is organized as alternating coordinate descent. Algorithm 1 initializes $\Omega^1$—for example uniformly or based on domain “trust”—and then repeats three steps until convergence: an expert-fit step
$$
\Theta^{k+1}=\arg\min_\Theta \ell(X,Y;\Theta,\Omega^k)+r(\Theta),
$$
a weight-fit step
$$
\Omega^{k+1}=\arg\min_{\Omega\in\text{simplex}} \ell(X,Y;\Theta^{k+1},\Omega)+\mathcal L(\Omega),
$$
and a convergence check based on $\|\Theta^{k+1}-\Theta^k\|$, $\|\Omega^{k+1}-\Omega^k\|$, or $\Delta J<\epsilon$ [2401.17118]. When the loss uses only local fits, the $\Theta$-update can split across experts, enabling parallel grey-box fitting or black-box training. The $\Omega$-update is a convex quadratic program over $T$ probability vectors.

The hyper-parameter $\beta$ governs the collaborative-versus-competitive regime. When $\beta\to 0$, all emphasis is placed on mixture-level fit, so experts collaborate to minimize global error. Large $\beta$ emphasizes local fits, so experts specialize and “compete” on their favored operating regimes. This parameterization makes specialization an explicit design choice rather than an incidental by-product.

Interpretability is addressed directly rather than retrospectively. The recommended visualization is to plot $\omega_i(t)$ against time to show which expert is trusted in different regimes. Quantitative output metrics are
$$
\text{MAE}= \frac{1}{T_v}\sum |y_v(t)-\hat y_v(t)|
$$
and
$$
\text{GoF}=1-\frac{\sum (y-\hat y)^2}{\sum (y-\bar y)^2}.
$$
Sensitivity analyses on hyper-parameters $(\lambda_\theta,\eta,\rho)$ and noise level (SNR) are used to demonstrate robustness [2401.17118].

A common misconception is that every “mixture of grounding experts” must be a jointly trained softmax-gated MoE. The explainable modeling formulation does use convex weights and explicit optimization over gating variables, but later applications show that the phrase also covers deterministic routing schemes and expert identification procedures with no jointly trained MoE loss. The shared element is grounded specialization, not a single mandatory training recipe.

## 4. Empirical validation in system identification and vehicle dynamics

The numerical example uses $M=2$ linear experts with true experts
$$
\theta_1=(0.50,-0.30,0.90,-0.80),\qquad \theta_2=(0.10,0.40,-0.60,-0.50),
$$
a PRBS input, $\text{SNR}\approx 20$ dB, and $T=6000$. Under this setup, the algorithm recovers $\theta_i$ within 1–2% and produces $\omega_i(t)$ matching ground-truth in the reported plots. Sensitivity studies vary $\lambda_\theta$, $\eta$, and $\rho$ in Figures 3a–c and noise level in Figure 4 [2401.17118].

A real-world case study addresses side-slip estimation from vehicle tests conducted in summer and winter. The features are
$$
x(t)=[\text{yaw acc},\ \text{yaw\_rate}/\text{vel},\ \text{steering}].
$$
Three expert sets are considered:

| Configuration | Experts |
|---|---|
| 2ST | 2 single-track models (2ST, grey-box × 2) |
| ST+RF | 1 single-track + RandomForest(50 trees, depth 3) |
| ST+P | 1 single-track + 3rd-order polynomial |

The gating predictor is an RF regressor with 35 trees and depth 25 trained to map $x\to\Omega$. On the test set, the reported results are:

| Configuration | MAE / GoF |
|---|---|
| 2ST | 1.573 deg / 0.815 |
| ST+RF | 1.601 / 0.702 |
| ST+P | 1.599 / 0.591 |

The paper also reports comparison against state-of-art approaches described as serial, parallel, and LIME+SHAP in Table 6 and Figure 7 [2401.17118]. A plausible implication is that, in this setting, expert blending does not automatically favor the more flexible hybrid combinations; the two-grey-box configuration is the strongest among the listed variants, which underscores that “grounding” can be operationally valuable rather than merely interpretable.

Implementation guidance is correspondingly modular. The experts may be single-track models, parameterized ODEs, linear dynamic models, random forests, shallow neural networks, or polynomials. The gating module may be any regression or classifier mapping $x\to\Omega(x)$, such as an RF regressor or a soft-max logistic gating NN. For large $T$, the $\Omega$-update can be solved on overlapping windows of length $W$, hyper-parameters can be selected by grid-search on validation MAE and GoF, $\Omega^1$ can be initialized using domain knowledge, and the number of experts $M$ can be chosen by cross-validation [2401.17118].

## 5. Hierarchical Mixture of Grounding Experts in web agents

In Avenir-Web, Mixture of Grounding Experts is an inference-time grounding mechanism embedded in the execution loop of a multimodal web agent. The system uses exactly two experts. The **visual-canvas expert** is the default visual expert, based on a general-purpose multimodal LLM such as Qwen-3-VL or Gemini 3 Pro; it takes a screenshot of the full viewport, including nested iframes, together with a textual action intent, and outputs either a precise $(x,y)$ coordinate in normalized $[0,1000]$ space or a visible text label to click, along with auxiliary parameters such as `press_enter_after=true` [2602.02468]. It uses Set-of-Mark–style overlay prompting to annotate interactive regions with short numeric tokens, enabling the LLM to pick a target in a single forward pass.

The **semantic-structural expert** is a fallback DOM expert. It operates purely over the parsed DOM or accessibility tree and searches by ARIA roles, CSS selectors, or visible text inside the DOM, mapping the result to a browser action such as clicking an element or filling an input. It is used only in two edge-case scenarios: when the visual expert’s confidence is low or it produces a non-responsive click with no state change detected, or when the operation demands ultra-fine control, such as selecting an option from a native `<select>` dropdown where direct script-assignment is possible [2602.02468].

The gating mechanism is explicitly not a learned gating network or softmax-based mixture. Instead, the control flow is deterministic: try the visual expert first, then fallback to the structural expert if verification fails. The paper formalizes this with a confidence gate:
$$
w_V=
\begin{cases}
1, & c_V\ge \tau,\\
0, & c_V<\tau,
\end{cases}
\qquad
w_S=1-w_V,
$$
and
$$
a_t=w_V\,a_t^{(V)}+w_S\,a_t^{(S)}.
$$
In practice, this means that if the visual click “takes” and a state-change verifier passes, the system proceeds; otherwise the structural expert is immediately invoked.

The per-turn workflow is correspondingly concrete: the core agent model outputs an abstract operation intent $z_t$; MoGE receives $z_t$; the visual expert proposes $a_t^{(V)}$; the browser executes it; a state-change verifier checks whether the DOM or screenshot changed meaningfully; if verification fails, the structural expert produces $a_t^{(S)}$; the outcome is then fed back into Adaptive Memory and the Task-Tracking Checklist for the next reasoning step [2602.02468]. The paper states that the visual expert succeeds in more than 99% of routine clicks and typing tasks across modern web layouts, that one-shot grounding covers approximately 90–95% of operations, and that the structural fallback recovers many of the remaining 5–10% of cases.

The training story is intentionally minimal. No new jointly trained MoE or gating loss is introduced. The visual expert is a frozen or fine-tuned MLLM trained with standard cross-entropy on multimodal instruction data, the structural fallback is un-trained rule-based code, and there is no extra loss term for MoGE itself [2602.02468]. On a 50-task subset, the reported ablation gives 48.0% success for full Avenir-Web with MoGE and 40.0% success without Mixture of Grounding Experts. The paper therefore presents MoGE here not as a classical trainable MoE but as a pragmatic two-tier grounding pipeline.

## 6. Grounding experts in sparse neural architectures and related specialization results

In sparse MoE LLMs, grounding experts are studied under the label **context-faithful experts**. Context faithfulness is defined as the model’s ability to accurately attend to and integrate relevant parts of a provided context $c$ when generating or predicting an output $y$ for a query $q$, within a task
$$
f:\mathcal Q\times \mathcal C\to \mathcal Y,\qquad y=f(q,c).
$$
A model is context-faithful if its predictions change appropriately when the supporting context is modified, for example swapped or counterfactualized. In a Mixture-of-Experts layer, each token input $u_t^{(\ell)}$ is routed to a small subset of $k$ experts out of $N_e$ available; grounding experts are those whose selection and internal computations systematically amplify attention to truly relevant context spans [2508.19594].

The identification procedure, Router Lens, has two phases. First, only the router parameters $\theta_r$ are fine-tuned while all other weights $\theta_o$ are frozen:
$$
\min_{\theta_r}\;\mathcal L_{\text{task}}\bigl(f(x;\theta_r,\theta_o)\bigr),
\qquad
\theta_r\leftarrow \theta_r-\eta\nabla_{\theta_r}\mathcal L_{\text{task}}.
$$
Second, each expert’s **Context-Dependence Ratio** is measured:
$$
r_i^{(\ell)}
=\frac{1}{N_s}\sum_{j=1}^{N_s}\frac{1}{L_j}\sum_{t=1}^{L_j}
\frac{\mathds{1}(g_{i,t}^{(\ell,j)}>0)}{k},
$$
and the top-$k$ experts by $r_i^{(\ell)}$ in each layer are selected as grounding experts [2508.19594].

The reported interpretive result is a layer-wise “think-twice” mechanism. Relative to the base model, router-tuned models allocate more attention to context tokens in middle layers and then concentrate more on answer spans in deeper layers, as measured by Context Attention Gain, Answer Attention Gain, and Answer Probability Gain. The paper further reports that router tuning alone lifts base MoEs from $\text{EM}\approx 20$–45% up to 50–85% across QA tasks; masking grounding experts in the router-tuned model causes drastic performance drops, including a reported $-73$% EM on NQ-Swap; and higher answer-attention gains correlate strongly with EM improvements, with $r\approx 0.95$ [2508.19594].

The downstream optimization method, **Context-faithful Expert Fine-Tuning (CEFT)**, freezes all non-selected parameters and fine-tunes only the identified grounding experts on the same task loss. For OLMoE-1B-7B, fully fine-tuning all parameters activates approximately 6.9 B trainable weights, whereas CEFT tunes approximately 0.5 B, a 13.8× reduction. The paper states that CEFT matches or exceeds full fine-tuning on all benchmarks listed in Table 4 and exhibits far less catastrophic forgetting on out-of-domain MMLU [2508.19594]. This suggests that, in sparse LLMs, grounding may be concentrated in a restricted expert subpopulation rather than uniformly distributed across the entire parameter set.

A related result from open-vocabulary object detection is not presented as Mixture of Grounding Experts, but it sharpens the general picture of specialization in grounding-oriented MoE systems. Dynamic-DINO extends Grounding DINO 1.5 Edge from a dense model to a dynamic inference framework via MoE-Tuning, replacing each decoder FFN by a sparse MoE layer with granularity decomposition, top-$k$ gating, and initialization chosen so that the router initially selects experts from the same parent FFN and the subnet reproduces the original FFN at epoch 0 [2507.17436]. The reported co-selection analysis shows that shallow MoE layers “mix broadly,” while later layers form stable cliques of size 2–3 specializing on particular semantics. A plausible implication is that expert specialization in grounding tasks can evolve from exploratory collaboration in early processing stages to stable semantic specialization in deeper stages.

Across these formulations, Mixture of Grounding Experts does not designate a single architectural template. It names a recurring design principle: grounding is improved by combining experts that differ in prior structure, representational modality, or context sensitivity, and the combination rule may be a smooth convex mixture, a deterministic fallback policy, or a router-mediated sparse activation pattern.

Source: https://www.emergentmind.com/topics/mixture-of-grounding-experts