Mixture of Grounding Experts

• The paper introduces a dynamic blending of physics-grounded (grey-box) and data-driven (black-box) models using smooth time-varying convex weights to improve grounding.
• The methodology employs a hierarchical design where a primary multimodal visual grounding module is paired with a structural fallback to ensure robust performance.
• Empirical validations in vehicle dynamics and language models demonstrate enhanced interpretability, context-specific specialization, and efficient training with reduced parameter tuning.

Searching arXiv for the specified papers and closely related work on Mixture of Grounding Experts. In one formulation, Mixture of Grounding Experts is realized by letting each local expert be either a physics-grounded model (grey-box) or a purely data-driven model (black-box) and learning how to blend them via smooth time-varying convex weights (Leoni et al., 2024). In a later web-agent formulation, the same term denotes a two-tier pipeline in which a multimodal visual grounding module is tried first and a structural DOM-based module is invoked as fallback (Li et al., 2 Feb 2026). Related work on sparse Mixture-of-Experts LLMs uses grounding experts to denote context-faithful experts whose routing and internal computations amplify attention to relevant context spans (Bai et al., 27 Aug 2025). Taken together, these usages center on a common objective: improving grounding by exploiting complementary expert specializations rather than a single undifferentiated predictor.

1. Conceptual scope and terminological usage

The term emerged explicitly in explainable data-driven modeling as a response to a familiar asymmetry: traditional models grounded in first principles often struggle with accuracy as the system’s complexity increases, whereas machine learning approaches face challenges in interpretability and in handling physical constraints. The proposed answer is a “mixture of experts” rationale in which diverse local models are fused in a data-driven way while preserving first-principle-based priors when available (Leoni et al., 2024).

A distinct but compatible usage appears in autonomous web agents. There, Mixture of Grounding Experts is not a convex ensemble of predictive functions but a hierarchical control mechanism for action grounding: a visual-canvas expert handles the default pathway, and a semantic-structural expert handles failure cases or operations that demand ultra-fine control (Li et al., 2 Feb 2026). The two experts are arranged hierarchically, visual first and structural second, to achieve both robustness and efficiency.

A third usage, in sparse MoE LLMs, does not present a framework named MoGE but studies experts that are specialized for context utilization. In that setting, grounding experts are those experts whose selection and internal computations systematically amplify attention to the truly relevant context spans, thereby driving context-grounded generation (Bai et al., 27 Aug 2025). This suggests a broader interpretation of the phrase: “grounding” refers not only to physical priors, but also to reliable anchoring in context, interface state, or structured metadata.

2. Mathematical formulation in explainable data-driven modeling

The canonical formalization approximates an unknown nonlinear map

$y(t)=f^0(x(t))+e(t)$

by a convex combination of $M$ local experts:

$$\hat y(t)=\sum_{i=1}^M \omega_i(t)\,f_i(x(t);\theta_i).$$

Here, $f_i(\cdot;\theta_i)$ can be any local model, grey-box or black-box, embedding first-principle priors when available, and $\omega_i(t)\in[0,1]$ satisfy $\sum_i \omega_i(t)=1$ and represent the gating weights or trust levels (Leoni et al., 2024).

The joint learning problem is written as

$$J(\Theta,\Omega)=\ell(X,Y;\Theta,\Omega)+r(\Theta)+\mathcal L(\Omega),$$

where $\Theta=[\theta_1,\dots,\theta_M]$ and $\Omega=\{\Omega(t)\}$. The data-fitting term can be expressed at the mixture level,

$$\ell^{\text{mix}}=\sum_{t=1}^T c\cdot \left\|y(t)-\sum_i \omega_i(t)f_i(x(t);\theta_i)\right\|_2^2,$$

at the local level,

$M$0

or in a combined cooperative-plus-competitive form,

$M$1

The parameter regularizer is

$M$2

described as a group regularizer on expert parameters, for example $M$3-shrinkage. The gating regularizer shapes temporal evolution:

$M$4

with $M$5 penalizing abrupt gating changes and enforcing smooth switches among experts. This smoothness term is also presented as

$M$6

and is interpreted as encouraging $M$7 to vary smoothly in time under a Markov prior while avoiding physically implausible rapid regime-switching (Leoni et al., 2024).

Grey-box experts can embed explicit physics. One example is a single-track vehicle-dynamics model,

$M$8

with $M$9 as parameters $$\hat y(t)=\sum_{i=1}^M \omega_i(t)\,f_i(x(t);\theta_i).$$0 learned from data. In that case, $$\hat y(t)=\sum_{i=1}^M \omega_i(t)\,f_i(x(t);\theta_i).$$1 can enforce physical plausibility, including positivity and physical bounds, via constraints or penalties.

3. Optimization, specialization, and interpretability

Training is organized as alternating coordinate descent. Algorithm 1 initializes $$\hat y(t)=\sum_{i=1}^M \omega_i(t)\,f_i(x(t);\theta_i).$$2—for example uniformly or based on domain “trust”—and then repeats three steps until convergence: an expert-fit step

$$\hat y(t)=\sum_{i=1}^M \omega_i(t)\,f_i(x(t);\theta_i).$$3

a weight-fit step

$$\hat y(t)=\sum_{i=1}^M \omega_i(t)\,f_i(x(t);\theta_i).$$4

and a convergence check based on $$\hat y(t)=\sum_{i=1}^M \omega_i(t)\,f_i(x(t);\theta_i).$$5, $$\hat y(t)=\sum_{i=1}^M \omega_i(t)\,f_i(x(t);\theta_i).$$6, or $$\hat y(t)=\sum_{i=1}^M \omega_i(t)\,f_i(x(t);\theta_i).$$7 (Leoni et al., 2024). When the loss uses only local fits, the $$\hat y(t)=\sum_{i=1}^M \omega_i(t)\,f_i(x(t);\theta_i).$$8-update can split across experts, enabling parallel grey-box fitting or black-box training. The $$\hat y(t)=\sum_{i=1}^M \omega_i(t)\,f_i(x(t);\theta_i).$$9-update is a convex quadratic program over $f_i(\cdot;\theta_i)$0 probability vectors.

The hyper-parameter $f_i(\cdot;\theta_i)$1 governs the collaborative-versus-competitive regime. When $f_i(\cdot;\theta_i)$2, all emphasis is placed on mixture-level fit, so experts collaborate to minimize global error. Large $f_i(\cdot;\theta_i)$3 emphasizes local fits, so experts specialize and “compete” on their favored operating regimes. This parameterization makes specialization an explicit design choice rather than an incidental by-product.

Interpretability is addressed directly rather than retrospectively. The recommended visualization is to plot $f_i(\cdot;\theta_i)$4 against time to show which expert is trusted in different regimes. Quantitative output metrics are

$f_i(\cdot;\theta_i)$5

and

$f_i(\cdot;\theta_i)$6

Sensitivity analyses on hyper-parameters $f_i(\cdot;\theta_i)$7 and noise level (SNR) are used to demonstrate robustness (Leoni et al., 2024).

A common misconception is that every “mixture of grounding experts” must be a jointly trained softmax-gated MoE. The explainable modeling formulation does use convex weights and explicit optimization over gating variables, but later applications show that the phrase also covers deterministic routing schemes and expert identification procedures with no jointly trained MoE loss. The shared element is grounded specialization, not a single mandatory training recipe.

4. Empirical validation in system identification and vehicle dynamics

The numerical example uses $f_i(\cdot;\theta_i)$8 linear experts with true experts

$f_i(\cdot;\theta_i)$9

a PRBS input, $\omega_i(t)\in[0,1]$0 dB, and $\omega_i(t)\in[0,1]$1. Under this setup, the algorithm recovers $\omega_i(t)\in[0,1]$2 within 1–2% and produces $\omega_i(t)\in[0,1]$3 matching ground-truth in the reported plots. Sensitivity studies vary $\omega_i(t)\in[0,1]$4, $\omega_i(t)\in[0,1]$5, and $\omega_i(t)\in[0,1]$6 in Figures 3a–c and noise level in Figure 1 (Leoni et al., 2024).

A real-world case study addresses side-slip estimation from vehicle tests conducted in summer and winter. The features are

$\omega_i(t)\in[0,1]$7

Three expert sets are considered:

Configuration	Experts
2ST	2 single-track models (2ST, grey-box × 2)
ST+RF	1 single-track + RandomForest(50 trees, depth 3)
ST+P	1 single-track + 3rd-order polynomial

The gating predictor is an RF regressor with 35 trees and depth 25 trained to map $\omega_i(t)\in[0,1]$8. On the test set, the reported results are:

Configuration	MAE / GoF
2ST	1.573 deg / 0.815
ST+RF	1.601 / 0.702
ST+P	1.599 / 0.591

The paper also reports comparison against state-of-art approaches described as serial, parallel, and LIME+SHAP in Table 6 and Figure 2 (Leoni et al., 2024). A plausible implication is that, in this setting, expert blending does not automatically favor the more flexible hybrid combinations; the two-grey-box configuration is the strongest among the listed variants, which underscores that “grounding” can be operationally valuable rather than merely interpretable.

Implementation guidance is correspondingly modular. The experts may be single-track models, parameterized ODEs, linear dynamic models, random forests, shallow neural networks, or polynomials. The gating module may be any regression or classifier mapping $\omega_i(t)\in[0,1]$9, such as an RF regressor or a soft-max logistic gating NN. For large $\sum_i \omega_i(t)=1$0, the $\sum_i \omega_i(t)=1$1-update can be solved on overlapping windows of length $\sum_i \omega_i(t)=1$2, hyper-parameters can be selected by grid-search on validation MAE and GoF, $\sum_i \omega_i(t)=1$3 can be initialized using domain knowledge, and the number of experts $\sum_i \omega_i(t)=1$4 can be chosen by cross-validation (Leoni et al., 2024).

5. Hierarchical Mixture of Grounding Experts in web agents

In Avenir-Web, Mixture of Grounding Experts is an inference-time grounding mechanism embedded in the execution loop of a multimodal web agent. The system uses exactly two experts. The visual-canvas expert is the default visual expert, based on a general-purpose multimodal LLM such as Qwen-3-VL or Gemini 3 Pro; it takes a screenshot of the full viewport, including nested iframes, together with a textual action intent, and outputs either a precise $\sum_i \omega_i(t)=1$5 coordinate in normalized $\sum_i \omega_i(t)=1$6 space or a visible text label to click, along with auxiliary parameters such as press_enter_after=true (Li et al., 2 Feb 2026). It uses Set-of-Mark–style overlay prompting to annotate interactive regions with short numeric tokens, enabling the LLM to pick a target in a single forward pass.

The semantic-structural expert is a fallback DOM expert. It operates purely over the parsed DOM or accessibility tree and searches by ARIA roles, CSS selectors, or visible text inside the DOM, mapping the result to a browser action such as clicking an element or filling an input. It is used only in two edge-case scenarios: when the visual expert’s confidence is low or it produces a non-responsive click with no state change detected, or when the operation demands ultra-fine control, such as selecting an option from a native <select> dropdown where direct script-assignment is possible (Li et al., 2 Feb 2026).

The gating mechanism is explicitly not a learned gating network or softmax-based mixture. Instead, the control flow is deterministic: try the visual expert first, then fallback to the structural expert if verification fails. The paper formalizes this with a confidence gate:

$\sum_i \omega_i(t)=1$7

and

$\sum_i \omega_i(t)=1$8

In practice, this means that if the visual click “takes” and a state-change verifier passes, the system proceeds; otherwise the structural expert is immediately invoked.

The per-turn workflow is correspondingly concrete: the core agent model outputs an abstract operation intent $\sum_i \omega_i(t)=1$9; MoGE receives $$J(\Theta,\Omega)=\ell(X,Y;\Theta,\Omega)+r(\Theta)+\mathcal L(\Omega),$$0; the visual expert proposes $$J(\Theta,\Omega)=\ell(X,Y;\Theta,\Omega)+r(\Theta)+\mathcal L(\Omega),$$1; the browser executes it; a state-change verifier checks whether the DOM or screenshot changed meaningfully; if verification fails, the structural expert produces $$J(\Theta,\Omega)=\ell(X,Y;\Theta,\Omega)+r(\Theta)+\mathcal L(\Omega),$$2; the outcome is then fed back into Adaptive Memory and the Task-Tracking Checklist for the next reasoning step (Li et al., 2 Feb 2026). The paper states that the visual expert succeeds in more than 99% of routine clicks and typing tasks across modern web layouts, that one-shot grounding covers approximately 90–95% of operations, and that the structural fallback recovers many of the remaining 5–10% of cases.

The training story is intentionally minimal. No new jointly trained MoE or gating loss is introduced. The visual expert is a frozen or fine-tuned MLLM trained with standard cross-entropy on multimodal instruction data, the structural fallback is un-trained rule-based code, and there is no extra loss term for MoGE itself (Li et al., 2 Feb 2026). On a 50-task subset, the reported ablation gives 48.0% success for full Avenir-Web with MoGE and 40.0% success without Mixture of Grounding Experts. The paper therefore presents MoGE here not as a classical trainable MoE but as a pragmatic two-tier grounding pipeline.

6. Grounding experts in sparse neural architectures and related specialization results

In sparse MoE LLMs, grounding experts are studied under the label context-faithful experts. Context faithfulness is defined as the model’s ability to accurately attend to and integrate relevant parts of a provided context $$J(\Theta,\Omega)=\ell(X,Y;\Theta,\Omega)+r(\Theta)+\mathcal L(\Omega),$$3 when generating or predicting an output $$J(\Theta,\Omega)=\ell(X,Y;\Theta,\Omega)+r(\Theta)+\mathcal L(\Omega),$$4 for a query $$J(\Theta,\Omega)=\ell(X,Y;\Theta,\Omega)+r(\Theta)+\mathcal L(\Omega),$$5, within a task

$$J(\Theta,\Omega)=\ell(X,Y;\Theta,\Omega)+r(\Theta)+\mathcal L(\Omega),$$6

A model is context-faithful if its predictions change appropriately when the supporting context is modified, for example swapped or counterfactualized. In a Mixture-of-Experts layer, each token input $$J(\Theta,\Omega)=\ell(X,Y;\Theta,\Omega)+r(\Theta)+\mathcal L(\Omega),$$7 is routed to a small subset of $$J(\Theta,\Omega)=\ell(X,Y;\Theta,\Omega)+r(\Theta)+\mathcal L(\Omega),$$8 experts out of $$J(\Theta,\Omega)=\ell(X,Y;\Theta,\Omega)+r(\Theta)+\mathcal L(\Omega),$$9 available; grounding experts are those whose selection and internal computations systematically amplify attention to truly relevant context spans (Bai et al., 27 Aug 2025).

The identification procedure, Router Lens, has two phases. First, only the router parameters $\Theta=[\theta_1,\dots,\theta_M]$0 are fine-tuned while all other weights $\Theta=[\theta_1,\dots,\theta_M]$1 are frozen:

$\Theta=[\theta_1,\dots,\theta_M]$2

Second, each expert’s Context-Dependence Ratio is measured:

$\Theta=[\theta_1,\dots,\theta_M]$3

and the top-$\Theta=[\theta_1,\dots,\theta_M]$4 experts by $\Theta=[\theta_1,\dots,\theta_M]$5 in each layer are selected as grounding experts (Bai et al., 27 Aug 2025).

The reported interpretive result is a layer-wise “think-twice” mechanism. Relative to the base model, router-tuned models allocate more attention to context tokens in middle layers and then concentrate more on answer spans in deeper layers, as measured by Context Attention Gain, Answer Attention Gain, and Answer Probability Gain. The paper further reports that router tuning alone lifts base MoEs from $\Theta=[\theta_1,\dots,\theta_M]$6–45% up to 50–85% across QA tasks; masking grounding experts in the router-tuned model causes drastic performance drops, including a reported $\Theta=[\theta_1,\dots,\theta_M]$7% EM on NQ-Swap; and higher answer-attention gains correlate strongly with EM improvements, with $\Theta=[\theta_1,\dots,\theta_M]$8 (Bai et al., 27 Aug 2025).

The downstream optimization method, Context-faithful Expert Fine-Tuning (CEFT), freezes all non-selected parameters and fine-tunes only the identified grounding experts on the same task loss. For OLMoE-1B-7B, fully fine-tuning all parameters activates approximately 6.9 B trainable weights, whereas CEFT tunes approximately 0.5 B, a 13.8× reduction. The paper states that CEFT matches or exceeds full fine-tuning on all benchmarks listed in Table 4 and exhibits far less catastrophic forgetting on out-of-domain MMLU (Bai et al., 27 Aug 2025). This suggests that, in sparse LLMs, grounding may be concentrated in a restricted expert subpopulation rather than uniformly distributed across the entire parameter set.

A related result from open-vocabulary object detection is not presented as Mixture of Grounding Experts, but it sharpens the general picture of specialization in grounding-oriented MoE systems. Dynamic-DINO extends Grounding DINO 1.5 Edge from a dense model to a dynamic inference framework via MoE-Tuning, replacing each decoder FFN by a sparse MoE layer with granularity decomposition, top-$\Theta=[\theta_1,\dots,\theta_M]$9 gating, and initialization chosen so that the router initially selects experts from the same parent FFN and the subnet reproduces the original FFN at epoch 0 (Lu et al., 23 Jul 2025). The reported co-selection analysis shows that shallow MoE layers “mix broadly,” while later layers form stable cliques of size 2–3 specializing on particular semantics. A plausible implication is that expert specialization in grounding tasks can evolve from exploratory collaboration in early processing stages to stable semantic specialization in deeper stages.

Across these formulations, Mixture of Grounding Experts does not designate a single architectural template. It names a recurring design principle: grounding is improved by combining experts that differ in prior structure, representational modality, or context sensitivity, and the combination rule may be a smooth convex mixture, a deterministic fallback policy, or a router-mediated sparse activation pattern.