CEFT: Context-Faithful Expert Fine-Tuning

• Context-faithful Expert Fine-Tuning (CEFT) is a targeted method that enhances context grounding by selectively adapting experts within large MoE models.
• It uses a two-stage process: first, identifying specialized experts via Router Lens, then fine-tuning only those experts for context-dependent tasks.
• CEFT achieves comparable or superior performance to full fine-tuning on QA benchmarks while reducing trainable parameters and mitigating catastrophic forgetting.

Searching arXiv for papers on CEFT and closely related MoE fine-tuning methods. Context-faithful Expert Fine-Tuning (CEFT) is a lightweight optimization approach for mixture-of-experts (MoE) LLMs that selectively fine-tunes experts identified as specialized in context utilization, with the explicit goal of improving context faithfulness: the ability to ground outputs in the provided context rather than producing irrelevant or hallucinated responses. In the formulation introduced for MoE LLMs, CEFT is coupled to a diagnostic method, Router Lens, which first exposes which experts are actually critical for context-dependent reasoning and then restricts adaptation to those experts alone. Across context question answering benchmarks and multiple MoE backbones, CEFT is reported to match or surpass full fine-tuning while using far fewer trainable parameters, and to be less prone to catastrophic forgetting on out-of-domain evaluation (Bai et al., 27 Aug 2025).

1. Concept and problem setting

CEFT arises from the observation that context faithfulness remains a persistent challenge for LLMs, including MoE architectures. Although MoE models increase parameter efficiency and capacity by activating only a subset of experts per input, they still often fail to ground their outputs strictly in the provided context, which is consequential in question answering, retrieval-augmented generation, and in-context learning (Bai et al., 27 Aug 2025).

The motivating premise is that MoE sparsity may not only improve efficiency but also induce specialization. If some experts become specialized for context-relevant reasoning, then improving those experts specifically should provide a targeted and parameter-efficient route to better grounding. In this sense, CEFT is not simply another sparse adaptation method. It is an attempt to align the granularity of fine-tuning with a hypothesized functional specialization inside the pretrained MoE: “context-faithful experts.”

A central difficulty is identification. Classic approaches often rely on expert activation frequency to determine importance, but MoE pretraining typically uses load balancing, which can smooth router activations and obscure specialization. CEFT therefore begins not with expert fine-tuning but with a probing-and-selection stage designed to reveal which experts are genuinely implicated in context use (Bai et al., 27 Aug 2025).

2. Router Lens and the identification of context-faithful experts

Router Lens is the expert-discovery procedure that precedes CEFT. It is presented as a two-step method.

First, the router networks are fine-tuned on a context-dependent task while all other model weights are frozen. If model parameters are partitioned into router parameters $\theta_r$ and all other parameters $\theta_o$, the optimization is written as

$$\min_{\theta_r}\;\; \mathcal{L}_{\text{task}}\left(f(x;\theta_r,\theta_o)\right), \quad \text{s.t. } \theta_o \text{ fixed}.$$

This router-only tuning step is intended to let the router “express” which experts are actually critical for context utilization, circumventing the smoothing effect of pretraining-time load balancing (Bai et al., 27 Aug 2025).

Second, after router tuning, expert relevance is quantified by a context-dependence ratio. For expert $e_i$ in layer $\ell$, the ratio is

$r_i^{(\ell)} = \frac{1}{N_s}\sum_{j=1}^{N_s}\frac{1}{L_j}\sum_{t=1}^{L_j}\frac{\mathds{1}(g_{i,t}^{(\ell,j)} > 0)}{k},$

where $N_s$ is the number of samples, $L_j$ is the length of sample $j$, $\mathds{1}(g_{i,t}^{(\ell,j)} > 0)$ indicates whether expert $\theta_o$0 is selected for token $\theta_o$1, and $\theta_o$2 is the number of activated experts per token. The top-$\theta_o$3 experts with the highest $\theta_o$4 in each layer are then selected as context-faithful experts (Bai et al., 27 Aug 2025).

The paper reports that router tuning alone can produce dramatic gains on context question answering. In the cited example, SQuAD exact match on OLMoE-1B-7B rises from 26.6 to 80.5, which is presented as evidence that correct expert selection is crucial for context-dependent performance (Bai et al., 27 Aug 2025).

3. CEFT as a two-stage selective fine-tuning method

Once Router Lens has identified context-faithful experts, CEFT proceeds by freezing all routers and all non-selected experts, and fine-tuning only the selected experts on the target context-dependent task (Bai et al., 27 Aug 2025).

The overall procedure is explicitly two-stage:

1. Expert identification: freeze all but routers, train the router on a context task, compute the context-dependence ratio, and select top-$\theta_o$5 experts per layer.
2. Expert fine-tuning: freeze all but the selected experts and fine-tune those experts on the task.

This organization distinguishes CEFT from full fine-tuning, which updates the entire model, and from more generic expert-specialized methods that do not specifically target context faithfulness. In the reported implementation, only “a handful of experts” are trained relative to the total model size. The provided example is OLMoE-1B-7B, where full fine-tuning uses 6.9B parameters and CEFT uses 0.5B parameters, described as 13.8x fewer trainable parameters (Bai et al., 27 Aug 2025).

The method is characterized as lightweight not because it abandons expert routing, but because it restricts optimization to a sparse subset of experts that Router Lens has already linked to context utilization. A plausible implication is that CEFT treats MoE sparsity not merely as an efficiency mechanism but as a diagnostic substrate for identifying which submodules are worth adapting.

4. Mechanistic interpretation: how CEFT alters context use

The paper provides several mechanistic probes intended to explain why CEFT improves context faithfulness. One line of evidence comes from masking experiments: masking context-faithful experts causes large drops in task performance, whereas masking standard experts does not, which is presented as causal evidence that the selected experts are necessary for context-dependent behavior (Bai et al., 27 Aug 2025).

A second line of evidence comes from attention analysis. Router-tuned and CEFT-enhanced models are reported to allocate higher attention to answer and context tokens. This is quantified with Context Attention Gain (CAG), which measures the increase in attention to context after Router Tuning or CEFT:

$\theta_o$6

The paper also introduces Answer Probability Gain (APG), which measures how much more probable the correct answer becomes in intermediate layers due to context-faithful experts:

$\theta_o$7

Layer-wise tracing is described as showing a “Think Twice” pattern: the model first increases global context attention in mid layers and then narrows onto answer tokens in deeper layers (Bai et al., 27 Aug 2025).

The reported correlation analysis further links these probes to task-level gains. Figure 1 is described as showing a strong positive correlation, $\theta_o$8, between answer attention gain and improved exact-match performance. This suggests that the selected experts do not merely change output distributions; they progressively amplify attention to relevant contextual information and enhance context grounding (Bai et al., 27 Aug 2025).

5. Empirical performance, efficiency, and retention

CEFT is evaluated on OLMoE-1B-7B, DeepSeek-V2-Lite, MiniCPM-MoE-8x2B, and Mixtral-8x7B across context QA benchmarks including SQuAD, NQ, HotpotQA, NQ-Swap, and ConfiQA. The reported headline result is that CEFT matches or surpasses both full fine-tuning (FFT) and Expert-Specialized Fine-Tuning (ESFT) across the evaluated models and benchmarks (Bai et al., 27 Aug 2025).

A representative example is provided for SQuAD on OLMoE-1B-7B:

Method	EM	F1
FFT	81.6	88.8
CEFT	83.1	90.3

The paper further states that on counterfactual, conflated, and challenging benchmarks such as NQ-Swap and ConfiQA, CEFT achieves “state-of-the-art, context-faithful performances,” with exact-match scores above 90% (Bai et al., 27 Aug 2025).

Efficiency is a recurring emphasis. CEFT’s trainable-parameter reduction is highlighted visually in Figure 2, and the OLMoE-1B-7B example gives the concrete scale: 6.9B trainable parameters for FFT versus 0.5B for CEFT. The method is also described as much less prone to catastrophic forgetting than FFT on MMLU, indicating better preservation of original capabilities under selective adaptation (Bai et al., 27 Aug 2025).

An ablation reported in Table 7 shows that performance plateaus as the number of trainable experts increases, which is used to argue that careful expert selection matters more than simply increasing the number of updated experts. Table 12 is described as showing that CEFT outperforms other context-faithfulness enhancement techniques, including CFP, CAD, ContextCite, and Context-DPO (Bai et al., 27 Aug 2025).

6. Relation to adjacent methods and broader context-faithfulness research

CEFT sits at the intersection of two research lines: MoE-specific fine-tuning and context-faithfulness optimization.

Within MoE fine-tuning, ESFT identifies task-relevant experts by probing routing distributions and fine-tunes only the experts most relevant to a downstream task while freezing the others; it is motivated by the finding that within a task, activated experts are highly concentrated, while the set of top-activated experts differs significantly across tasks (Wang et al., 2024). CEFT is narrower in scope: instead of selecting experts for general task relevance, it selects experts for context utilization specifically (Bai et al., 27 Aug 2025). PERFT, by contrast, integrates PEFT modules directly into the MoE mechanism and studies routed adaptation modules such as PERFT-R and PERFT-E; the routed variants are reported to outperform MoE-agnostic PEFT baselines, and PERFT-E is described as aligned with context-faithful adaptation because it reuses the original router (Liu et al., 2024, Liu et al., 4 Aug 2025). ExpertCondenser addresses a different MoE tuning failure mode—preservation of long-tailed expert information under sparse routing—by combining bias-driven sparsification with always-active gated condenser experts, and is reported to outperform DenseMixer and ESFT with average gain of 2.5%+ on mathematical reasoning and commonsenseQA benchmarks (He et al., 24 Apr 2026).

Broader work on context faithfulness defines the problem in ways that extend beyond MoE routing. In retrieval-augmented climate QA, ClimateGPT Faithful+ is obtained by excluding unfaithful instruction-fine-tuning subsets, improving supported atomic claims from 30% to 57% according to an automatic metric (Thulke et al., 21 May 2025). In knowledge-conflict settings, SI-FACT uses self-instructed contrastive data and improves Contextual Recall Rate by 6.2% over the best baseline on ECARE_KRE while reducing dependence on internal memory (Fu, 12 Sep 2025). In self-supervised speech models, context-aware fine-tuning attaches a context module and adds an auxiliary loss encouraging similarity to surrounding-segment context vectors, outperforming a standard fine-tuning baseline and rivaling a context injection baseline without requiring surrounding segments at inference time (Shon et al., 2022). In CLIP fine-tuning, CAR-FT preserves context-aware robustness by minimizing the Kullback-Leibler Divergence between context distributions induced by original and fine-tuned models, achieving 78.5% averaged accuracy on DomainBed (Mao et al., 2022).

Taken together, these results suggest that “context faithfulness” is being operationalized through different mechanisms across domains: selective expert adaptation in MoE LLMs, contrastive learning against knowledge conflict, training-data filtering for retrieval grounding, auxiliary context losses in speech, and context-distribution regularization in vision-LLMs. CEFT’s distinctive claim within this broader landscape is that context faithfulness in MoE LLMs is at least partly localized in a subset of experts and can therefore be improved by selectively tuning those experts rather than the full model (Bai et al., 27 Aug 2025).