Hide to Guide: Learning via Semantic Masking

Abstract: Reinforcement learning with verifiable rewards (RLVR) has become a powerful paradigm for improving LLMs on reasoning-intensive tasks, but its effectiveness is often limited by exploration. For example, models often fail on hard problems, leaving little useful reward signal. External expert traces offer a natural source of guidance, yet they may also expose reward-relevant content along the critical path to the verifier target, such as final answers, intermediate values, executable implementations, or answer-related entities. This content can create an unintended reward hacking channel, allowing the policy to obtain reward by copying the trace rather than learning the underlying reasoning or agentic behavior. Existing guided-RL methods reduce this risk by using partial trajectories, but they mainly control how much expert information is shown heuristically rather than which parts should be hidden. To this end, we propose Semantic Masked Expert Policy Optimization (SMEPO), a fine-grained semantic masking strategy for expert-guided RLVR. Instead of truncating traces coarsely or revealing them unchanged, SMEPO masks reward-relevant semantic spans along the critical path while preserving the expert's decomposition, plan, and procedural structure. This turns hard problems from reasoning from scratch into a fill-in-the-blank process: the policy can follow the expert's problem-solving route, but must still reconstruct the missing values, code, or entities by itself. SMEPO is simple to apply and requires no changes to the reward function or RL objective. Across diverse domains, including math, code, and agentic search, SMEPO improves accuracy by up to 3.2 points over GRPO and reduces training time by up to 4.2x. The code is available at https://github.com/mit-han-lab/SMEPO.

• The paper introduces SMEPO, a semantic masking approach that prevents reward hacking by masking critical reward-relevant spans in expert traces.
• It demonstrates improved convergence and task accuracy, with notable gains such as a +2.7 point boost in math reasoning performance.
• SMEPO integrates effectively with other guidance methods, offering a scalable, domain-agnostic solution for robust policy optimization in diverse RLVR applications.

Semantic Masked Expert Policy Optimization: A Precise Intervention for Expert-Guided RLVR

Motivation and Problem Formulation

Reinforcement learning with verifiable rewards (RLVR) has established itself as an effective paradigm for optimizing LLMs on reasoning-intensive tasks across math, code generation, and agentic search. RLVR frameworks, such as DeepSeek-R1 and ReINFORCE++ [deepseekr1] [reinforcepp], circumvent the limitations of static imitation learning by leveraging domain-specific automated verifiers. However, exploration in RLVR is severely bottlenecked by reward sparsity: on complex tasks, initial policy rollouts almost invariably fail, yielding negligible learning signal and stalling progress. Commonly, auxiliary guidance via expert traces—rationales, full solutions, tool-use trajectories—has been introduced to aid exploration [ghpo] [luffy] [trapo] [guide]. Yet, naive conditioning on complete expert traces exposes reward-relevant content: intermediate values, code blocks, or entities directly aligned with verifier targets. This leads to reward hacking, where the policy exploits prompt-exposed answer cues rather than learning robust procedural reasoning.

Figure 1: Overview of SMEPO: naive RL rollouts incur sparse rewards; full expert traces leak reward-aligned content; SMEPO masks such spans, yielding fill-in-the-blank traces for guided exploration.

SMEPO: Semantic Masking Mechanism

SMEPO (Semantic Masked Expert Policy Optimization) is introduced to address the structural failure mode of expert-trace conditioning in RLVR. SMEPO masks semantic spans in expert traces that directly expose reward-relevant information, while preserving procedural structure and guidance—problem decomposition, equation templates, search plans. The approach reframes hard reasoning as a fill-in-the-blank process: the policy follows the expert path but must reconstruct masked content itself.

Figure 2: SMEPO pipeline: domain-specific masking rules detect reward-relevant spans in expert traces, replace them with mask placeholders, and conditional RL proceeds as usual.

SMEPO is instantiated for three core domains:

• Math reasoning: numeric spans (final/ intermediate answers) are masked, preserving decomposition steps and explanatory text.
• Code synthesis: expert code blocks are masked, retaining task, plan, and discussion.
• Agentic search: answer entities and critical pivots in search traces are masked, maintaining action sequence and trajectory logic.

SMEPO requires no changes to RL objectives or reward functions, functioning as a lightweight preprocessing step.

Empirical Results and Diagnostics

Accuracy and Convergence

Across diverse evaluation protocols (GSM8K, MATH500, HumanEval, Bamboogle), SMEPO consistently outperforms vanilla GRPO and full expert-trace conditioning. Conditioning on complete traces degrades performance due to overreliance on exposed answer content; masking reverses this trend.

• Qwen3-8B-Base (Math): SMEPO boosts accuracy from 47.6 (GRPO) to 50.3, a +2.7 point improvement; direct expert conditioning drops performance to 41.0.
• Qwen2.5-7B: SMEPO yields 44.5 vs. 38.6 for full expert-trace conditioning.
• DeepSeek-R1-Distill-Qwen-7B: SMEPO reaches 62.6, up from 61.8 for GRPO.
• Code: SMEPO achieves maximal scores across code benchmarks, also suppressing prompt-copying of expert code.
• Agentic Search: SMEPO produces the largest recovery (48.8), whereas full expert traces cause a dramatic drop.

Training Dynamics

SMEPO accelerates convergence significantly, e.g., achieving target math accuracy in 2.37 hours vs. 9.90 for vanilla GRPO.

Figure 3: Left: LiveCodeBench performance curve; Right: SMEPO reduces time-to-target (mean@32 accuracy in math) compared to baseline GRPO.

Copy Suppression

Code-copy diagnostics confirm that SMEPO nearly eliminates overlap with visible expert traces and similarity to original expert code, forcing models to synthesize solutions rather than adapt exposed implementations.

Figure 4: SMEPO keeps visible-trace overlap minimal, directly suppressing prompt-copying in code generation.

Ablation and Analysis

Extensive ablation studies reveal that masking reward-relevant content is critical. Prefix exposure (truncating traces) and random masking do not provide monotonic or robust gains; answer-only masking is insufficient. Only reward-aware masking, targeting distributed reward channels (intermediate values, entities, executable code), achieves maximal benefit.

Figure 5: SMEPO outperforms partial trace exposure approaches, confirming the need for reward-informed masking beyond simple truncation.

Compatibility with Guidance Methods

SMEPO is additive with other prompt-refinement guidance methods such as GHPO [ghpo]: integrating SMEPO with GHPO yields further improvement (+1.6 points in math benchmarks), demonstrating that masking addresses guidance-induced reward hacking orthogonally to prompt refinement.

Figure 6: SMEPO integrated with GHPO improves accuracy on GSM8K and MATH500.

Case Studies

Three representative cases—functional equation reasoning, code synthesis, agentic search—demonstrate that SMEPO preserves exploration utility while preventing policies from exploiting exposed answer channels. For math and search, SMEPO requires the policy to reconstruct missing values/entities; in code, the model must synthesize algorithms rather than copy executable blocks.

Theoretical and Practical Implications

SMEPO establishes a new guidance-level principle for RLVR: effective expert-guided RL must mask reward-relevant content along the critical path, preserving procedural cues while avoiding leakage of answer channels. This approach addresses a precise reward hacking mode, leading to both higher task accuracy and more robust behavioral generalization. SMEPO's domain-agnostic masking protocol offers a scalable mechanism applicable to broader RLVR settings—interactive agents, web navigation, multimodal reasoning—by identifying and masking outcome-correlated spans.

Practically, SMEPO makes expert guidance more reliable, facilitating efficient scaling of RLVR training. Theoretically, it refines the exploration–exploitation balance in policy optimization, enhancing both convergence and generalization while mitigating specification gaming.

Future Directions

Future research should automate reward-relevant span detection in new domains, explore joint modeling of masked guidance and verifier semantics, and analyze the interplay of masking with curriculum learning and adaptive trace selection. SMEPO's principle is expected to generalize to increasingly complex RLVR setups, including tool-augmented multimodal agents and compositional reasoning tasks.

Conclusion

SMEPO defines a technically rigorous intervention that separates guidance from reward exploitation in expert-trace-conditioned RLVR. By masking semantic spans tied to verifier success, SMEPO achieves improved accuracy, enhanced convergence, and suppressed reward hacking across math, code, and search domains. Its compatibility with other guidance strategies and domain-agnostic design indicate substantial relevance for scalable and reliable RL-guided LLM training (2605.25198).