Mixture of Calibrated Experts (MoCaE)

• MoCaE is a framework that calibrates individual expert outputs to ensure their scores are commensurate before aggregation, thereby enhancing overall system performance.
• In object detection, MoCaE refines detector confidences through post-hoc calibration and merging, preventing overconfident predictions from degrading detection quality.
• For LLM alignment, calibrated routing fuses task-specific experts (helpfulness, harmlessness, and honesty) to mitigate negative transfer and catastrophic forgetting.

Searching arXiv for papers on Mixture of Calibrated Experts (MoCaE) and closely related calibrated MoE work. Mixture of Calibrated Experts (MoCaE) denotes a class of mixture-of-experts constructions in which calibration is treated as a first-class mechanism for combining specialized predictors. The term appears explicitly in object detection, where MoCaE calibrates detector confidences before mixture and post-processing (Oksuz et al., 2023), and in LLM alignment, where a Mixture of Calibrated Experts is used as a calibrated, task-adaptive routing module inside a Transformer to fuse separately trained experts for Helpfulness, Harmlessness, and Honesty (HHH) (Kashyap et al., 10 Sep 2025). Across these settings, the common motivation is that naïve expert combination can be degraded by miscalibration: experts may emit scores on incompatible scales, routers may over-select inappropriate experts, or aggregate confidence may fail to reflect empirical reliability, especially under distribution shift (Wong et al., 18 Jun 2026). The phrase therefore refers less to a single universal architecture than to a design principle: expert specialization is preserved, but combination is constrained by calibration so that mixture weights or scores better reflect actual performance.

1. Origins and motivating problem

MoCaE was introduced to address a recurring failure mode of expert mixtures: combining strong experts does not reliably improve performance when the experts’ confidence signals are not comparable. In object detection, the central diagnosis is that different detectors often produce scores on different scales, so a naïve merger behaves like “the most overconfident detector,” and detection quality can decline despite expert diversity (Oksuz et al., 2023). The paper attributes this to miscalibration, defined there in a localization-aware sense: a detector’s confidence does not match its actual detection quality.

In LLM alignment, the motivating problem is different in surface form but analogous in structure. TrinityX identifies two issues: catastrophic forgetting or negative transfer across helpfulness, harmlessness, and honesty objectives, and miscalibrated expert routing in standard MoE systems (Kashyap et al., 10 Sep 2025). The proposed Mixture of Calibrated Experts is intended to preserve specialization by keeping alignment behaviors in separate expert modules while combining them through a calibrated router.

A broader theoretical framing is provided by work on calibration in MoEs under distribution shift. That analysis shows that calibration is not merely a property of individual experts; in soft-routed MoEs, expert-level calibration can fail to induce calibration of the aggregate predictor when routing configurations change at test time (Wong et al., 18 Jun 2026). This suggests that MoCaE-style methods address a general systems problem in expert aggregation rather than a task-specific anomaly.

2. Core principle: calibration as the basis of expert combination

The defining feature of MoCaE is that expert combination is mediated by calibrated quantities rather than raw confidence or unconstrained routing scores. In the object-detection formulation, each detector is calibrated post hoc so that its confidence better reflects localization quality, and only then are detections merged and refined (Oksuz et al., 2023). There is no learned gating network; instead, calibrated detector scores implicitly determine which predictions survive downstream suppression and fusion.

In TrinityX, calibration enters directly into the routing computation. The framework trains one expert for each HHH dimension—helpfulness, harmlessness, and honesty—and combines a learned input-dependent routing distribution with static task weights derived from expert vectors (Kashyap et al., 10 Sep 2025). The resulting mixture is intended to be both contextual and alignment-aware. This differs from standard MoE practice, which often optimizes routing only against task loss.

A concise comparison is useful:

Setting	Experts	Calibration role
Object detection (Oksuz et al., 2023)	Heterogeneous detectors	Post-hoc confidence calibration before merging
LLM alignment (Kashyap et al., 10 Sep 2025)	HHH-specific task-vector experts	Calibrated routing via task priors and router probabilities
Distribution-shift analysis (Wong et al., 18 Jun 2026)	Generic soft- or hard-routed experts	Studies when expert calibration does or does not calibrate the aggregate

This suggests a common abstraction: MoCaE seeks to make expert scores commensurate before aggregation. A plausible implication is that the term “calibrated” refers not only to probability calibration in the narrow statistical sense, but also to making expert contributions operationally comparable within a mixture.

3. TrinityX and MoCaE for HHH alignment

In TrinityX, MoCaE is a mixture-of-experts layer integrated at the Feed-Forward Network layers of the Transformer (Kashyap et al., 10 Sep 2025). The base model is a frozen pretrained LLaMA-2-7B with parameters $\theta_0$, and each alignment objective $i$ is represented by a low-rank task vector

$$\mathcal{T}_i \in \mathbb{R}^{r \times d}, \quad r \ll d,$$

with $n=3$ objectives and $d=4096$ for LLaMA-2. The experts are trained independently on task-specific datasets: $$\Theta = \{\mathcal{T}_1,\mathcal{T}_2,\ldots,\mathcal{T}_n\}.$$

The paper explicitly avoids naïve weight merging of the form

$$\theta_{\text{merged}} = \theta_0 + \sum_{i=1}^n \Delta \theta_i, \quad \text{where } \Delta \theta_i = \mathcal{T}_i,$$

because such merging risks interference and catastrophic forgetting (Kashyap et al., 10 Sep 2025). It also notes that task vectors can be negated,

$\mathcal{T}_{\text{new}} = -\mathcal{T}_j,$

as a way to invert undesirable behavior.

Before routing, TrinityX computes static importance scores relative to a reference task vector: $$\gamma_i = \langle \mathcal{T}_i,\mathcal{T}_{\text{ref}} \rangle, \qquad \tilde{\gamma}_i = \frac{\gamma_i}{\sum_{j=1}^n \gamma_j},$$ so that $\sum_{i=1}^{n}\tilde{\gamma}_i = 1$. These normalized scores act as a task-adaptive prior (Kashyap et al., 10 Sep 2025).

Given a hidden state $i$0, the router computes

$i$1

followed by a temperature-scaled softmax

$i$2

with $i$3 (Kashyap et al., 10 Sep 2025). Each expert produces

$i$4

and the calibrated mixture weights are

$i$5

The aggregate representation is then

$i$6

The final representation is also denoted

$i$7

For the HHH case, the fused representation can be interpreted as

$i$8

This realizes a unified alignment-aware representation in which the model does not choose only one behavior, but fuses helpfulness, harmlessness, and honesty in a calibrated composition (Kashyap et al., 10 Sep 2025).

4. Regularization, routing stability, and calibration logic

TrinityX augments the primary task loss with routing regularizers: $i$9

$$\mathcal{T}_i \in \mathbb{R}^{r \times d}, \quad r \ll d,$$0

and

$$\mathcal{T}_i \in \mathbb{R}^{r \times d}, \quad r \ll d,$$1

with $$\mathcal{T}_i \in \mathbb{R}^{r \times d}, \quad r \ll d,$$2 and $$\mathcal{T}_i \in \mathbb{R}^{r \times d}, \quad r \ll d,$$3 (Kashyap et al., 10 Sep 2025). The paper also reports a gating loss coefficient of $$\mathcal{T}_i \in \mathbb{R}^{r \times d}, \quad r \ll d,$$4 used to encourage sparse expert routing.

These terms are intended to stabilize routing and avoid degenerate expert selection. Entropy regularization encourages diversity in expert usage, while the KL term penalizes abrupt routing shifts across steps (Kashyap et al., 10 Sep 2025). Ablation results indicate that gating loss improves expert selection and boosts performance, especially win rate and truthfulness, and that entropy/KL regularization improves safety and truthfulness by preventing overconfident or unstable routing.

Theoretical work on MoE calibration under shift clarifies why this stabilization problem is nontrivial. For a general soft-routed MoE,

$$\mathcal{T}_i \in \mathbb{R}^{r \times d}, \quad r \ll d,$$5

expert calibration alone is only a marginal constraint and does not guarantee calibration of the final mixture under changes in routing-configuration prevalence (Wong et al., 18 Jun 2026). The key contrast is that hard-routed MoEs can inherit calibration under a broad class of shifts, whereas soft-routed MoEs generally cannot. Since TrinityX is a soft mixture, this broader theory suggests that MoCaE’s routing calibration and regularization are addressing precisely the part of the problem that standard expert-wise calibration leaves unresolved.

5. Object detection formulation of MoCaE

The paper "MoCaE: Mixture of Calibrated Experts Significantly Improves Object Detection" defines MoCaE in a distinct but closely related way (Oksuz et al., 2023). Here the experts are off-the-shelf object detectors, often heterogeneous in architecture and training objective. The problem is that raw confidence scores are not directly comparable across detectors, so simple concatenation followed by standard NMS can degrade performance.

MoCaE therefore uses a four-stage pipeline (Oksuz et al., 2023):

1. Each expert detector runs independently.
2. Each detector is calibrated post hoc so that its score better reflects box quality.
3. The calibrated detections are merged.
4. The merged detections are passed through Refining NMS, which combines Soft NMS and Score Voting.

The calibration target aligns confidence with IoU to the matched ground truth: $$\mathcal{T}_i \in \mathbb{R}^{r \times d}, \quad r \ll d,$$6 Calibration is learned as a post-hoc regression problem via a calibrator $$\mathcal{T}_i \in \mathbb{R}^{r \times d}, \quad r \ll d,$$7, with class-agnostic isotonic regression as the main practical choice and linear regression used in some analyses (Oksuz et al., 2023).

The principal calibration metric is Localisation-aware Expected Calibration Error (LaECE), based on the localization-aware condition

$$\mathcal{T}_i \in \mathbb{R}^{r \times d}, \quad r \ll d,$$8

with per-class measure

$$\mathcal{T}_i \in \mathbb{R}^{r \times d}, \quad r \ll d,$$9

The paper also defines LaACE and LaMCE (Oksuz et al., 2023).

Refining NMS adds stronger post-processing than conventional suppression. Score Voting refines box coordinates as

$n=3$0

with overlap weights

$n=3$1

The intended effect is that once confidence reflects localization quality, ranking and suppression become meaningful across detectors, so better-localized predictions from less overconfident experts are not systematically discarded (Oksuz et al., 2023).

6. Empirical findings across domains

In TrinityX, extensive experiments are reported on Alpaca, BeaverTails, and TruthfulQA (Kashyap et al., 10 Sep 2025). The paper states relative improvements of 32.5% in win rate, 33.9% in safety score, and 28.4% in truthfulness, and reports that memory usage and inference latency are reduced by over 40% compared to prior MoE-based approaches. On the base LLaMA-2-7B setting, TrinityX achieves WR 36.75, SS 41.03, TI 40.66, and Avg 12.12, compared with H$n=3$2Fusion at WR 13.79, SS 42.00, TI 18.82, and Avg -3.13 (Kashyap et al., 10 Sep 2025). In a MoCaE-specific ablation, MoCaE with LLaMA-2-7B is reported at WR 93.33 vs 72.00, SS 23.17 vs 30.40, TI 75.00 vs 39.85, and Avg 48.38 vs 26.70 relative to H$n=3$3Fusion. The paper also reports near-perfect calibration in some settings, including ECE = 0.00 and Brier Score = 0.4988.

The object-detection MoCaE reports improvements on five detection tasks and 15 detectors (Oksuz et al., 2023). The paper states that it improves detectors on COCO and LVIS by up to approximately 2.5 AP, reaches 65.1 AP on COCO test-dev, reaches 82.62 AP50 on DOTA, and improves open-vocabulary detection as well. On COCO minitest, an RS R-CNN + ATSS + PAA MoCaE reaches 45.5 AP compared with 43.2 AP for the best single model; on COCO test-dev with strong detectors, it reaches 59.0 AP, improving the best single by +2.4 AP (Oksuz et al., 2023). Ablations in that work identify calibration as the main driver of improvement: uncalibrated mixtures perform poorly, Soft NMS alone offers only minor improvement, Score Voting helps more, and the largest gain comes from calibrating the experts.

The distribution-shift study offers a different empirical message. It includes a baseline labeled MoCaE, defined there as post-hoc calibration of experts before combining them, and shows that this improves ECE only modestly relative to Vanilla MoE (Wong et al., 18 Jun 2026). On the CIFAR-10H hard subset, Hard ECE values are 0.281 for Vanilla MoE, 0.262 for MoCaE, 0.074 for Robust MoE, and 0.065 for FGR+Robust; on the CivilComments hard subset, Hard ECE values are 0.108 for Vanilla MoE, 0.101 for MoCaE, 0.037 for Robust MoE, and 0.040 for Robust Filtered (Wong et al., 18 Jun 2026). The paper uses these results to argue that calibrating experts individually is insufficient to resolve mixture-level calibration in soft-routed MoEs under shift.

7. Interpretation, misconceptions, and related uses

A common misconception is that MoCaE denotes a single standard architecture. The literature in the supplied corpus does not support that interpretation. In object detection, MoCaE refers to post-hoc calibration plus calibrated merging of heterogeneous detectors, without a learned routing network (Oksuz et al., 2023). In TrinityX, MoCaE is an internal Transformer module with calibrated task-adaptive routing over HHH-specific experts (Kashyap et al., 10 Sep 2025). The shared concept is calibration-aware expert aggregation, not architectural identity.

A second misconception is that calibrating each expert is sufficient to make the whole MoE calibrated. The distribution-shift analysis explicitly rejects this for soft-routed models: expert calibration is sufficient for overall calibration in hard-routed models under a broad class of distribution shifts, but not sufficient for soft-routed models (Wong et al., 18 Jun 2026). This result places an important caveat on MoCaE-style designs that rely only on expert-wise post-hoc calibration.

A third misconception is that calibration is only a confidence-reporting nicety. In all three strands of work, calibration is tied directly to system behavior. In detection, it determines which boxes survive NMS-like procedures (Oksuz et al., 2023). In TrinityX, calibrated routing determines how helpfulness, harmlessness, and honesty are composed in the hidden representation (Kashyap et al., 10 Sep 2025). Under shift, calibration affects whether aggregate confidence remains trustworthy on difficult subsets (Wong et al., 18 Jun 2026).

Related work in survival analysis illustrates a neighboring idea without explicitly using the term MoCaE. The paper "Let the Experts Speak: Improving Survival Prediction & Calibration via Mixture-of-Experts Heads" does not define a model named Mixture of Calibrated Experts, but it studies MoE heads where greater expert expressiveness improves calibration and predictive accuracy while preserving clustering ability (Morrill et al., 11 Nov 2025). The closest analogue is its Personalized MoE, which yields patient-specific expert outputs and is evaluated with equal-mass expected calibration error, time-dependent Brier score, and Harrell’s concordance index. This suggests that the broader research trajectory around calibrated expert mixtures includes both explicit MoCaE models and adjacent architectures motivated by the same calibration-versus-specialization tension.

Taken together, the literature presents MoCaE as a methodological family organized around a stable thesis: specialized experts are useful only to the extent that their contributions are calibrated in the aggregate. Where that calibration is achieved by post-hoc score correction, calibrated routing priors, or robust training against routing-induced shift varies by domain, but the underlying objective remains the same.