---
title: 'CEFT: Context-Faithful Expert Fine-Tuning'
url: https://www.emergentmind.com/topics/context-faithful-expert-fine-tuning-ceft
type: topic
---

# CEFT: Context-Faithful Expert Fine-Tuning

Searching arXiv for recent papers on CEFT and closely related MoE fine-tuning methods.
Context-faithful Expert Fine-Tuning (CEFT) is a lightweight optimization approach for mixture-of-experts (MoE) large language models that selectively fine-tunes experts identified as specialized in context utilization, with the explicit goal of improving context faithfulness: the ability to ground outputs in the provided context rather than producing irrelevant or hallucinated responses. In the formulation introduced for MoE language models, CEFT is coupled to a diagnostic method, Router Lens, which first exposes which experts are actually critical for context-dependent reasoning and then restricts adaptation to those experts alone. Across context question answering benchmarks and multiple MoE backbones, CEFT is reported to match or surpass full fine-tuning while using far fewer trainable parameters, and to be less prone to catastrophic forgetting on out-of-domain evaluation [2508.19594].

## 1. Concept and problem setting

CEFT arises from the observation that context faithfulness remains a persistent challenge for large language models, including MoE architectures. Although MoE models increase parameter efficiency and capacity by activating only a subset of experts per input, they still often fail to ground their outputs strictly in the provided context, which is consequential in question answering, retrieval-augmented generation, and in-context learning [2508.19594].

The motivating premise is that MoE sparsity may not only improve efficiency but also induce specialization. If some experts become specialized for context-relevant reasoning, then improving those experts specifically should provide a targeted and parameter-efficient route to better grounding. In this sense, CEFT is not simply another sparse adaptation method. It is an attempt to align the granularity of fine-tuning with a hypothesized functional specialization inside the pretrained MoE: “context-faithful experts.”

A central difficulty is identification. Classic approaches often rely on expert activation frequency to determine importance, but MoE pretraining typically uses load balancing, which can smooth router activations and obscure specialization. CEFT therefore begins not with expert fine-tuning but with a probing-and-selection stage designed to reveal which experts are genuinely implicated in context use [2508.19594].

## 2. Router Lens and the identification of context-faithful experts

Router Lens is the expert-discovery procedure that precedes CEFT. It is presented as a two-step method.

First, the router networks are fine-tuned on a context-dependent task while all other model weights are frozen. If model parameters are partitioned into router parameters $\theta_r$ and all other parameters $\theta_o$, the optimization is written as
$$
\min_{\theta_r}\;\; \mathcal{L}_{\text{task}}\left(f(x;\theta_r,\theta_o)\right), \quad \text{s.t. } \theta_o \text{ fixed}.
$$
This router-only tuning step is intended to let the router “express” which experts are actually critical for context utilization, circumventing the smoothing effect of pretraining-time load balancing [2508.19594].

Second, after router tuning, expert relevance is quantified by a context-dependence ratio. For expert $e_i$ in layer $\ell$, the ratio is
$$
r_i^{(\ell)} =
\frac{1}{N_s}\sum_{j=1}^{N_s}\frac{1}{L_j}\sum_{t=1}^{L_j}\frac{\mathds{1}(g_{i,t}^{(\ell,j)} > 0)}{k},
$$
where $N_s$ is the number of samples, $L_j$ is the length of sample $j$, $\mathds{1}(g_{i,t}^{(\ell,j)} > 0)$ indicates whether expert $i$ is selected for token $t$, and $k$ is the number of activated experts per token. The top-$k$ experts with the highest $r_i^{(\ell)}$ in each layer are then selected as context-faithful experts [2508.19594].

The paper reports that router tuning alone can produce dramatic gains on context question answering. In the cited example, SQuAD exact match on OLMoE-1B-7B rises from 26.6 to 80.5, which is presented as evidence that correct expert selection is crucial for context-dependent performance [2508.19594].

## 3. CEFT as a two-stage selective fine-tuning method

Once Router Lens has identified context-faithful experts, CEFT proceeds by freezing all routers and all non-selected experts, and fine-tuning only the selected experts on the target context-dependent task [2508.19594].

The overall procedure is explicitly two-stage:

1. **Expert identification**: freeze all but routers, train the router on a context task, compute the context-dependence ratio, and select top-$k$ experts per layer.
2. **Expert fine-tuning**: freeze all but the selected experts and fine-tune those experts on the task.

This organization distinguishes CEFT from full fine-tuning, which updates the entire model, and from more generic expert-specialized methods that do not specifically target context faithfulness. In the reported implementation, only “a handful of experts” are trained relative to the total model size. The provided example is OLMoE-1B-7B, where full fine-tuning uses 6.9B parameters and CEFT uses 0.5B parameters, described as 13.8x fewer trainable parameters [2508.19594].

The method is characterized as lightweight not because it abandons expert routing, but because it restricts optimization to a sparse subset of experts that Router Lens has already linked to context utilization. A plausible implication is that CEFT treats MoE sparsity not merely as an efficiency mechanism but as a diagnostic substrate for identifying which submodules are worth adapting.

## 4. Mechanistic interpretation: how CEFT alters context use

The paper provides several mechanistic probes intended to explain why CEFT improves context faithfulness. One line of evidence comes from masking experiments: masking context-faithful experts causes large drops in task performance, whereas masking standard experts does not, which is presented as causal evidence that the selected experts are necessary for context-dependent behavior [2508.19594].

A second line of evidence comes from attention analysis. Router-tuned and CEFT-enhanced models are reported to allocate higher attention to answer and context tokens. This is quantified with Context Attention Gain (CAG), which measures the increase in attention to context after Router Tuning or CEFT:
$$
\text{CAG}^{(\ell)} =
\frac{1}{N_s}\sum_{i=1}^{N_s}
\left(
\frac{\alpha_{i,\text{rt}}^{(\ell)} - \alpha_{i,\text{base}}^{(\ell)}}{\alpha_{i,\text{base}}^{(\ell)}}
\right).
$$
The paper also introduces Answer Probability Gain (APG), which measures how much more probable the correct answer becomes in intermediate layers due to context-faithful experts:
$$
\text{APG}^{(\ell)} =
\frac{1}{N_s}\sum_{i=1}^{N_s}
\left[
\frac{p_{\text{rt}}^{(\ell)}(y_i) - p_{\text{base}}^{(\ell)}(y_i)}{p_{\text{base}}^{(\ell)}(y_i)}
\right].
$$
Layer-wise tracing is described as showing a “Think Twice” pattern: the model first increases global context attention in mid layers and then narrows onto answer tokens in deeper layers [2508.19594].

The reported correlation analysis further links these probes to task-level gains. Figure 15 is described as showing a strong positive correlation, $r = 0.95$, between answer attention gain and improved exact-match performance. This suggests that the selected experts do not merely change output distributions; they progressively amplify attention to relevant contextual information and enhance context grounding [2508.19594].

## 5. Empirical performance, efficiency, and retention

CEFT is evaluated on OLMoE-1B-7B, DeepSeek-V2-Lite, MiniCPM-MoE-8x2B, and Mixtral-8x7B across context QA benchmarks including SQuAD, NQ, HotpotQA, NQ-Swap, and ConfiQA. The reported headline result is that CEFT matches or surpasses both full fine-tuning (FFT) and Expert-Specialized Fine-Tuning (ESFT) across the evaluated models and benchmarks [2508.19594].

A representative example is provided for SQuAD on OLMoE-1B-7B:

| Method | EM | F1 |
|---|---:|---:|
| FFT | 81.6 | 88.8 |
| CEFT | **83.1** | **90.3** |

The paper further states that on counterfactual, conflated, and challenging benchmarks such as NQ-Swap and ConfiQA, CEFT achieves “state-of-the-art, context-faithful performances,” with exact-match scores above 90% [2508.19594].

Efficiency is a recurring emphasis. CEFT’s trainable-parameter reduction is highlighted visually in Figure 8, and the OLMoE-1B-7B example gives the concrete scale: 6.9B trainable parameters for FFT versus 0.5B for CEFT. The method is also described as much less prone to catastrophic forgetting than FFT on MMLU, indicating better preservation of original capabilities under selective adaptation [2508.19594].

An ablation reported in Table 7 shows that performance plateaus as the number of trainable experts increases, which is used to argue that careful expert selection matters more than simply increasing the number of updated experts. Table 12 is described as showing that CEFT outperforms other context-faithfulness enhancement techniques, including CFP, CAD, ContextCite, and Context-DPO [2508.19594].

## 6. Relation to adjacent methods and broader context-faithfulness research

CEFT sits at the intersection of two research lines: MoE-specific fine-tuning and context-faithfulness optimization.

Within MoE fine-tuning, ESFT identifies task-relevant experts by probing routing distributions and fine-tunes only the experts most relevant to a downstream task while freezing the others; it is motivated by the finding that within a task, activated experts are highly concentrated, while the set of top-activated experts differs significantly across tasks [2407.01906]. CEFT is narrower in scope: instead of selecting experts for general task relevance, it selects experts for context utilization specifically [2508.19594]. PERFT, by contrast, integrates PEFT modules directly into the MoE mechanism and studies routed adaptation modules such as PERFT-R and PERFT-E; the routed variants are reported to outperform MoE-agnostic PEFT baselines, and PERFT-E is described as aligned with context-faithful adaptation because it reuses the original router [2411.08212; 2508.02587]. ExpertCondenser addresses a different MoE tuning failure mode—preservation of long-tailed expert information under sparse routing—by combining bias-driven sparsification with always-active gated condenser experts, and is reported to outperform DenseMixer and ESFT with average gain of 2.5%+ on mathematical reasoning and commonsenseQA benchmarks [2604.23036].

Broader work on context faithfulness defines the problem in ways that extend beyond MoE routing. In retrieval-augmented climate QA, ClimateGPT Faithful+ is obtained by excluding unfaithful instruction-fine-tuning subsets, improving supported atomic claims from 30% to 57% according to an automatic metric [2505.15633]. In knowledge-conflict settings, SI-FACT uses self-instructed contrastive data and improves Contextual Recall Rate by 6.2% over the best baseline on ECARE_KRE while reducing dependence on internal memory [2509.10208]. In self-supervised speech models, context-aware fine-tuning attaches a context module and adds an auxiliary loss encouraging similarity to surrounding-segment context vectors, outperforming a standard fine-tuning baseline and rivaling a context injection baseline without requiring surrounding segments at inference time [2212.08542]. In CLIP fine-tuning, CAR-FT preserves context-aware robustness by minimizing the Kullback-Leibler Divergence between context distributions induced by original and fine-tuned models, achieving 78.5% averaged accuracy on DomainBed [2211.16175].

Taken together, these results suggest that “context faithfulness” is being operationalized through different mechanisms across domains: selective expert adaptation in MoE LLMs, contrastive learning against knowledge conflict, training-data filtering for retrieval grounding, auxiliary context losses in speech, and context-distribution regularization in vision-language models. CEFT’s distinctive claim within this broader landscape is that context faithfulness in MoE LLMs is at least partly localized in a subset of experts and can therefore be improved by selectively tuning those experts rather than the full model [2508.19594].

Source: https://www.emergentmind.com/topics/context-faithful-expert-fine-tuning-ceft