---
title: 'SMEPO: Semantic Masked Expert Policy Optimization'
url: https://www.emergentmind.com/topics/semantic-masked-expert-policy-optimization-smepo
type: topic
---

# SMEPO: Semantic Masked Expert Policy Optimization

Searching arXiv for the specified SMEPO paper and related masking work to ground the article in current arXiv records.
{"query":"id:2605.25198 OR \"Hide to Guide: Learning via Semantic Masking\"","max_results":5,"sort_by":"submittedDate","sort_order":"descending"}
{"query":"id:2603.05670 OR \"TransMASK: Masked State Representation through Learned Transformation\"","max_results":5,"sort_by":"submittedDate","sort_order":"descending"}
Semantic Masked Expert Policy Optimization (SMEPO) is a guidance-level intervention for reinforcement learning with verifiable rewards (RLVR) that converts external expert traces into fill-in-the-blank style scaffolds. It preserves the expert’s decomposition, plan, and procedural structure, while masking reward-relevant semantic spans along the critical path to what the verifier checks—numbers and intermediate values in math, executable code bodies in programming, and answer-linked entities in search. By hiding these spans but keeping the structure, SMEPO improves exploration without changing the verifier, reward function, or RL objective. Across math, code, and agentic search, it yields consistent accuracy gains over vanilla GRPO and direct expert conditioning, and often reduces training time substantially [2605.25198].

## 1. Problem setting and motivation

RLVR trains a policy directly from verifiable feedback. Given a problem prompt $x$, the policy $\pi_\theta$ emits an answer $y$, and a domain-specific verifier $V$ computes a scalar reward $R(x, y)$. The standard RL objective is

$$
J(\pi_\theta) = \mathbb{E}_{x\sim D,\; y\sim \pi_\theta(\cdot|x)}[R(x,y)].
$$

In practice, GRPO-style critic-free policy optimization is used to maximize $J(\pi_\theta)$, optionally with KL regularization to a reference policy. The central bottleneck is exploration: early or weak rollouts often fail on reasoning-intensive tasks, generating sparse rewards that provide little learning signal [2605.25198].

External expert traces provide a natural source of guidance. In math, these traces may contain chain-of-thought and intermediate values; in code, complete executable programs; in search, search trajectories, entities, and query pivots. The paper identifies a failure mode in which full expert traces expose reward-relevant content along the critical path to verifier success. This creates an unintended reward hacking channel: the policy can obtain high reward by copying from or lightly adapting the exposed content instead of learning the intended reasoning, coding, or active search behavior.

A common misconception is that guided RLVR is primarily a question of how much expert information should be shown. The paper argues more narrowly that the decisive issue is which spans must be hidden. Heuristic controls such as partial prefixes or hints regulate how much of the trace is shown, but not which specific spans are reward-relevant, and are therefore insufficiently targeted.

## 2. Formalization of SMEPO

SMEPO separates reusable procedural guidance from reward-relevant content via lightweight semantic masking. Let $\tau^\star$ denote an expert trace paired with $x$. Conceptually, $\tau^\star$ contains two types of information:

$$
\tau^\star = (\tau^\star_g, \tau^\star_r),
$$

where $\tau^\star_g$ captures reusable guidance such as solution organization, decomposition, templates, and control-flow, while $\tau^\star_r$ contains content whose surface form is tightly aligned with verifier success, including final or repeated answers, intermediate numeric values, code bodies, and critical entities.

SMEPO defines a masking operator $M(\cdot)$ that preserves $\tau^\star_g$ while replacing $\tau^\star_r$ spans with placeholders:

$$
\tilde{\tau}^\star = M(\tau^\star).
$$

RL training then conditions on the masked trace,

$$
y \sim \pi_\theta(\cdot \mid x, \tilde{\tau}^\star),
$$

with the same reward $R(x,y)$. Formally, the objective under SMEPO is unchanged:

$$
\max_\theta \mathbb{E}_{(x,\tau^\star)\sim D,\; y\sim \pi_\theta(\cdot \mid x, M(\tau^\star))}[R(x,y)].
$$

This invariance of the objective is a defining property of the method. SMEPO transforms guidance in the input context but does not alter the verifier, the reward function, or the optimization algorithm. It fits into GRPO-style pipelines without code changes to the reward or loss [2605.25198].

The notion of a critical path is central. SMEPO targets semantic spans $S$ along the critical path $C$ to the verifier target: content that, if copied, materially suffices to pass $V$ or exposes near-deterministic intermediate values leading to it. In math, $C$ includes intermediate numeric values that determine the final answer. In code, it includes the executable solution block. In search, it includes the final entity, intermediate pivots, and page titles that reveal the answer route.

## 3. Semantic masking across domains

SMEPO uses lightweight rules and semantic tools to identify and replace reward-relevant spans with domain-specific placeholders while preserving procedural scaffolding [2605.25198].

| Domain | Masked spans | Preserved structure |
|---|---|---|
| Math | Numbers via regex $\rightarrow$ `[NUMBER]` | Equations, labels, decomposition, algebraic templates |
| Code | Fenced code blocks $\rightarrow$ `[CODE]` | Natural-language plan, algorithm sketches, data-structure choices, corner-case discussion |
| Agentic search | Named entities via spaCy or GLiNER, plus answer spans and emphasized terms $\rightarrow$ `[ENTITY]` | Multi-step search format and action sequence |

In math, the masking procedure hides numbers while preserving equations, labels such as “Step 1,” decomposition, and algebraic templates. The illustrative functional-equation example replaces every numeric span with `[NUMBER]`, so the policy must reconstruct the missing values and derive the scaling rule $f(2n)=2f(n)$ while following the preserved plan.

In code, the unmasked expert trace may include the full executable program. SMEPO replaces the fenced code block with `[CODE]` while keeping the surrounding procedure outline. The paper’s chain-counting example preserves the plan—enumerating offsets and counting chain starts—while requiring the policy to synthesize code rather than copy exposed implementation.

In agentic search, SMEPO masks named entities using spaCy or GLiNER, plus answer spans and emphasized terms that frequently correspond to final or intermediate entities. The Bamboogle example replaces “Rachel Jacobs” and related entity spans with `[ENTITY]`, preserving the action sequence: identify the derailment, search for the American social entrepreneur victim, and determine her age. The policy must then reformulate the query and return `<answer>39</answer>`.

This domain-by-domain design supports a specific interpretation of semantic masking: the goal is not generic corruption of the trace, but targeted suppression of verifier-aligned content while retaining reusable decomposition and procedural structure.

## 4. Training pipeline and optimization

The SMEPO pipeline has five steps. First, expert traces $\tau^\star$ are collected for each training problem $x$, for example from DeepSeek-R1 or Gemini-3-Pro, with search trajectories from MiroVerse. Second, domain-specific semantic masking is applied to obtain $\tilde{\tau}^\star = M(\tau^\star)$. Third, $x$ and $\tilde{\tau}^\star$ are concatenated as the rollout prompt, and the model generates fill-in-the-blank completions and final answers $y$. Fourth, the verifier $V$ computes reward $R(x,y)$ on $y$ only; $\tilde{\tau}^\star$ is context, not part of the scored output. Fifth, $\pi_\theta$ is optimized under GRPO with the same reward, rollout budget, and KL or entropy settings as the baseline [2605.25198].

The paper uses critic-free GRPO with advantage estimation anchored to the reward. In search, it includes a small KL penalty to a reference policy. A typical surrogate objective is

$$
L(\theta) = \mathbb{E}\big[A(x,y)\log \pi_\theta(y\mid x,\tilde{\tau}^\star)\big] - \beta \, \mathrm{KL}\big(\pi_\theta(\cdot \mid x,\tilde{\tau}^\star)\; ||\; \pi_{\mathrm{ref}}(\cdot \mid x,\tilde{\tau}^\star)\big),
$$

where $A(x,y)$ is a normalized advantage derived from $R(x,y)$, and $\beta$ is $0$ for math and code and $0.02$ for search. SMEPO does not change $R$, $A$, $\beta$, or the policy class—only the form of $\tilde{\tau}^\star$ shown to the policy during rollouts.

The reported implementation uses VeRL for math and code SFT/RL and Search-R1 for agentic search RL in a multi-turn tool environment. The main model is Qwen3-8B-Base, with Qwen2.5-7B and DeepSeek-R1-Distill-Qwen-7B used in math. Training uses a single node with $8\times$ H100 GPUs, bf16, and FSDP; 8 rollouts per prompt; 400 optimization steps; maximum response length of 12k tokens for math and code; and, in search, 2048 tokens per turn for up to 5 turns. The learning rate is $1\mathrm{e}{-6}$ for math and code and $8\mathrm{e}{-7}$ for search. Entropy is $0$ in math and code and $1\mathrm{e}{-3}$ in search. Rollout temperature is $0.8$.

The preprocessing overhead is reported as negligible: 1,024 examples take 5.96 s in math, 1.80 s in code, and 29.42 s in search on a CPU node. The implementation is released at the paper’s repository.

## 5. Empirical behavior, efficiency, and diagnostics

The evaluation covers math, code, and agentic search. Math uses GSM8K, MATH500, AIME25 (mean@32), AIME26 (mean@32), AMC, and OlympiadBench, with a verifier that extracts and matches the normalized final numeric answer. Code uses HumanEval, HumanEval+, and LiveCodeBench, with execution-based pass@1 across tests. Agentic search uses Bamboogle with Search-R1 exact-match reward on normalized answers. The paper also reports 5-shot MMLU as a sanity check, stating that there is no degradation [2605.25198].

For Qwen3-8B-Base with 1k examples per domain, the reported headline results are as follows:

| Domain | GRPO | Expert | SMEPO |
|---|---:|---:|---:|
| Math average | 47.6 | 41.0 with DeepSeek-R1 traces, 40.8 with Gemini | 50.3 |
| Code average | 56.5 | 55.7 | 59.0 |
| Agentic search | 45.6 | 23.2 | 48.8 |

In math, SMEPO improves the average from 47.6 under GRPO to 50.3, a gain of 2.7 points, while full expert conditioning hurts performance. Per-benchmark highlights include AIME25 17.6 and Olympiad 32.0 with SMEPO. In code, SMEPO reaches 59.0 compared with 56.5 for GRPO and 55.7 for direct expert conditioning, and it yields the best reported numbers on HumanEval (69.5), HumanEval+ (62.2), and LiveCodeBench (45.3). The same trend appears on the LiveCodeBench v6 subset: 29.0 for SMEPO versus 25.2 for GRPO and 24.4 for Expert. In agentic search, GRPO achieves 45.6, full expert conditioning collapses to 23.2, and SMEPO reaches 48.8, a gain of 3.2 points over GRPO.

The pattern transfers across model families. On math, Qwen2.5-7B improves from 42.4 under GRPO to 44.5 with SMEPO, while Expert degrades to 38.6. DeepSeek-R1-Distill-Qwen-7B improves from 61.8 to 62.6, with Expert at 60.4. A large-scale math experiment with 89k examples reports 47.8 for GRPO, 36.6 for Expert, and 49.4 for SMEPO.

Training efficiency is another reported effect. Time to GRPO peak mean@32 on AIME25/26 decreases from 9.90 h to 2.37 h for Qwen3-8B-Base, from 8.00 h to 4.57 h for Qwen2.5-7B, and from 14.56 h to 10.99 h for DeepSeek-R1-Distill. This is the basis for the reported “up to 4.2x” reduction in training time.

The code diagnostics directly address copying. The paper defines visible-trace overlap as

$$
\mathrm{VisibleTraceOverlap} = \frac{\mathrm{LCCS}(G_{\mathrm{rollout}}, T_{\mathrm{vis}})}{|G_{\mathrm{rollout}}|}
$$

and expert-code similarity as

$$
\mathrm{ExpertCodeSimilarity} = \frac{\mathrm{LCCS}(G_{\mathrm{code}}, T_{\mathrm{code}})}{|G_{\mathrm{code}}|}.
$$

Full-trace conditioning raises both metrics, indicating copying from visible trace content and reuse of expert code. SMEPO reduces overlap to near-zero by masking executable spans, forcing reliance on high-level plans rather than copied code.

## 6. Ablations, related methods, and limitations

The ablations are organized around the question of whether the benefit comes from targeted masking or merely from deleting text. Prefix exposure, in which 25–75% of the trace is shown, reduces harm but is non-monotonic and underperforms SMEPO. The paper attributes this to the fact that shorter traces may still leak reward-relevant spans appearing early, while truncation may remove useful later reasoning steps. Random masking at the word or sentence level, matched to SMEPO’s masking budget, underperforms full traces and is far below SMEPO, indicating that the gains are not due to corruption or removal per se. Reward-aware masking shows a progression: masking the final answer only yields moderate improvement, masking all answer uses improves further, and masking numbers throughout the trace yields the best performance. This supports the claim that reward-relevant information is distributed across intermediate values and repeats, not only in the final answer [2605.25198].

The method is positioned against guided RLVR approaches such as LUFFY, GHPO, TRAPO, POPE, Guide, and prompt-refinement approaches. Those methods regulate how much expert signal is shown or when it is shown. SMEPO instead regulates which parts are shown by masking reward-relevant spans on the critical path. The paper also relates SMEPO to masking and infilling methods in pretraining, including BERT, MASS, BART, T5, and code fill-in-the-middle. The distinction is that SMEPO uses masked traces as exploration guides under the same RL objective and reward verifier, rather than as targets for supervised reconstruction.

The reported compatibility with GHPO is additive: GHPO + SMEPO improves math from 44.0 to 45.6, and training curves on GSM8K and MATH500 are said to confirm the synergy. A plausible implication is that semantic masking can be composed with methods that modulate guidance schedule or optimization dynamics, provided the reward-aligned leakage channel remains suppressed.

The limitations are explicitly domain- and detector-dependent. Masking may be imperfect: missed spans can leak content; named-entity recognition errors may fail to catch aliases; numeric masking may under-mask or over-mask. Aggressive masking can remove too much signal and harm learning, while insufficient masking may leave residual leakage. Extending the approach to broader agentic tasks such as web navigation, mobile UI, or embodied control would require masking URLs, titles, UI labels, object names, coordinates, or action arguments while preserving action structure. The paper suggests that stronger semantic parsers or learned span taggers could improve precision, recall, and robustness.

A separate masking line, “TransMASK: Masked State Representation through Learned Transformation” [2603.05670], describes a self-supervised masking module trained jointly with an imitation learning policy to emphasize task-relevant state variables and suppress irrelevant ones. The provided account maps this mechanism to SMEPO by treating TransMASK as a drop-in mechanism for semantic masking of state representations, where a learned matrix $M$ produces $z = Ms$ and row-wise normalization aligns the mask with the policy Jacobian. This relation is conceptual rather than identical in setting: SMEPO, as defined in the RLVR paper, masks reward-relevant spans in expert traces, whereas TransMASK learns semantic state masks in imitation learning. The shared principle is selective preservation of task-relevant structure while suppressing nuisance or reward-aligned leakage channels.

Source: https://www.emergentmind.com/topics/semantic-masked-expert-policy-optimization-smepo