LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL

Abstract: Reinforcement learning (RL) for non-verifiable instruction following increasingly relies on LLM judges with prompt-specific rubrics as reward signals. While recent methods adapt these rubrics to the evolving policy during training, the training prompts themselves remain static, drawn from fixed corpora. This static approach often results in a critical misalignment between prompt difficulty and policy capability, leaving the judge unable to recover a discriminative reward signal when prompts fail to elicit quality variance among rollouts. To address this misalignment, we introduce LLM-as-a-Tutor, a framework that extends the LLM's role from judge to tutor: a single model serves as an examiner that pairwise-compares policy rollouts to detect non-challenging prompts, and as a generator that appends atomic constraints to them. This append-only design monotonically raises difficulty in step with the policy's capability, producing a self-calibrating training signal without external difficulty schedules. On three complex instruction-following benchmarks, our method consistently outperforms both policy-unaware baselines and prior policy-adaptive methods that adapt rubrics or rewrite prompts, suggesting prompt adaptation as a missing axis of policy-awareness in non-verifiable RL.

• The paper introduces a novel LLM-as-a-Tutor framework that adapts prompts to maintain discriminative reward signals during policy training.
• It employs pairwise LLM judgments and additive constraint generation to escalate task difficulty in alignment with policy capabilities.
• Experimental results across benchmarks demonstrate that the adaptive approach outperforms static and non-adaptive prompt strategies on key metrics.

LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL

Motivation and Problem Formulation

Rubric-based RL for instruction-following tasks without verifiable rewards has advanced through LLM-based judges conditioned on prompt-specific rubrics, but the prompt distribution itself remains static. This results in policy–prompt misalignment: as the policy improves, many training prompts fail to elicit quality variance among rollouts, rendering the reward signal non-discriminative and stalling further learning. The discriminative signal is not a property of the prompt alone, but of the prompt–policy pair: prompts that are too easy or too hard cease to provide meaningful reward gradients regardless of the sophistication of the rubric or the judge.

The LLM-as-a-Tutor Framework

LLM-as-a-Tutor extends the role of an LLM from pure reward judgment to active prompt adaptation. The framework leverages a single LLM to (i) examine a pair of policy rollouts and judge whether the prompt induces meaningful quality differences, and (ii) generate atomic constraints to append to non-discriminative prompts, thereby escalating difficulty only where the current policy saturates. The method operates iteratively: at each training epoch, for each prompt, the tutor samples rollouts from the current policy, performs a pairwise discriminativeness judgment, and, when needed, augments the prompt with targeted constraints along dimensions such as content, situation, style, or format.

Pairwise comparison is chosen over independent scoring, capitalizing on the sharpness and reliability of LLMs in comparative judgments [zheng2023judging]. Constraints are strictly additive, ensuring monotonic increase in task difficulty and preserving the source prompt distribution.

Figure 1: Overview of LLM-as-a-Tutor. The tutor LLM appends constraints to prompts that fail to elicit discriminative rollouts, dynamically raising task difficulty to match policy capability.

Experimental Validation

Evaluation is conducted on FollowBench, AdvancedIF, and InfoBench using Qwen3-1.7B as the policy and Qwen3-8B as tutor/judge. LLM-as-a-Tutor is benchmarked against (1) static prompt corpora with fixed/adaptive rubrics, (2) policy-unaware prompt modification (Evol-Instruct), and (3) policy-adaptive prompt modification (EVA). Results indicate that LLM-as-a-Tutor delivers highest average scores across benchmarks, outperforming both prompt-unaware and prior policy-adaptive baselines on five out of six metrics.

Strong numerical results:

• FollowBench HSR: 40.91 (±0.547)
• AdvancedIF Overall: 15.07 (±0.370)
• InfoBench DRFR: 73.59 (±0.100)
• Average across benchmarks: 51.96 (±0.226)

Policy-adaptive constraint addition outperforms non-adaptive (random/all) variants and outperforms alternative prompt modification strategies (rewrite/reset), confirming targeted, append-only adaptation is critical.

Analysis: Tutor Decisions and Reward Signal Amplification

The tutor's behavior is analyzed by sweeping policy scale, mean/variance of rewards, and cumulative difficulty amplification.

• The ratio of prompts marked as non-challenging increases monotonically with policy size, confirming dynamic adaptation tracks policy capability.
• Non-challenging prompts exhibit higher mean and lower variance in reward, directly aligning with the tutor's operational definition.
• Appending constraints incrementally reduces mean and increases variance of base rubric scores, amplifying discriminative power.

Figure 2: The tutor allocates more constraints as policy strength increases, non-challenging prompts exhibit saturated reward, and each appended constraint restores discriminative signal.

Qualitative examples show that a single atomic constraint on a saturated prompt fractures the uniformity of rollouts, reinstates reward variance, and exposes latent weaknesses in the policy previously hidden.

Theoretical and Practical Implications

LLM-as-a-Tutor establishes prompt adaptation as a necessary axis of policy-awareness in non-verifiable RL, orthogonal and complementary to rubric adaptation. The append-only operation guarantees monotonic curriculum and preserves instruction fidelity. This framework is extensible to domains where task difficulty can be naturally incremented via additive requirements, such as reasoning steps, edge cases, or factual sourcing.

By eschewing explicit teacher labels, the tutor model instantiates a distillation regime bounded only by its comparative discrimination, not its own response capabilities, potentially resulting in students that exceed the tutor in generative quality. This constitutes a new regime beyond classic SFT-based knowledge transfer.

Limitations and Broader Impacts

The method depends on the discriminative and constraint-generation capability of the tutor LLM; advances in tutor reasoning will yield stronger adaptation. Tutor invocation introduces additional inference overhead but is negligible compared to rubric-based reward computation.

Broader impacts include improved instruction-following and quality-sensitive RL, but dual-use potential exists: adaptation can elicit compliance with arbitrary instructions. Practitioners must ensure seed prompts and tutor models align with deployment objectives.

Conclusion

LLM-as-a-Tutor addresses prompt–policy misalignment in rubric-based RL, introducing adaptive, constraint-based prompt escalation driven by tutor LLM pairwise judgment. This yields a self-calibrating curriculum attuned to policy capability, restoring discriminative reward signal and improving instruction-following performance. Prompt adaptation emerges as a critical dimension for policy-aware RL pipelines, promising extensibility to diverse non-verifiable tasks and a pathway toward more capable, self-improving instruction-following LLMs.