LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks

Abstract: Reinforcement learning (RL) on open-ended tasks compresses an LLM's rubric-based evaluation into a scalar reward, discarding rich textual feedback and conflating responses with distinct quality profiles. We propose Experiential Learning (EL), which repurposes the feedback model from an LLM-as-a-Judge into an LLM-as-a-Coach. The coach distills its assessment of each on-policy response into transferable experiential knowledge, which conditions a teacher model and is internalized by the policy through on-policy context distillation. Compared with scalar rewards, this higher-bandwidth feedback channel provides dense supervision and preserves fine-grained preferences among high-quality responses. Across two policy families, with feedback from the policy itself or a proprietary model, EL consistently outperforms rubric-based RL on held-out and unseen open-ended tasks. Notably, EL generalizes better beyond the training distribution, and mitigates reward hacking. These findings establish experiential knowledge as a richer and more generalizable learning signal for post-training on non-verifiable tasks.

• The paper introduces EL that distills dense, rubric-based experiential knowledge from LLM evaluations to overcome scalar reward bottlenecks.
• The methodology leverages on-policy context distillation, using an LLM-as-a-Coach to provide high-bandwidth, transferable evaluations for open-ended tasks.
• EL outperforms RL in generalization and sample efficiency on non-verifiable tasks while mitigating reward hacking risks.

LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks

Motivation and Problem Landscape

Reinforcement learning (RL) with reward models is widespread in post-training LLMs, particularly for tasks where verification is objective and automated scoring is feasible. However, a persistent challenge arises in open-ended, non-verifiable tasks where quality is evaluated via complex, multi-dimensional rubrics and subjective criteria. In this regime, the RL paradigm compresses the rich, rubric-based assessment of an LLM-as-a-Judge into a single reward scalar, severely bottlenecking the information flow from evaluation to policy optimization. This scalar-aggregation results in the conflation of diverse, high-quality responses, limiting policy improvement and inducing reward hacking.

The paper introduces a novel method, Experiential Learning (EL), that moves beyond scalar compression by directly distilling dense, transferable knowledge from rubric-grounded evaluations via an LLM-as-a-Coach. This enables the model to internalize nuanced guidance via high-bandwidth, textual experiential contexts, rather than coarse-grained reward maximization.

Figure 1: Comparison of the learning signals in RL and EL. RL compresses evaluation into a scalar reward, while EL learns from experiential knowledge generated by an LLM-as-a-Coach, yielding faster, higher-fidelity learning for non-verifiable tasks.

Methodological Framework

Formalization and Implementation

In RL, a policy $\pi_\theta$ is updated by maximizing the expected scalar reward from the LLM-as-a-Judge on rollouts, discarding all textual analysis beyond the final score. By contrast, EL reframes the feedback loop. The same LLM is leveraged as a Coach, where, given prompt $x$, response $y \sim \pi_\theta(\cdot|x)$, and task-specific rubrics $\mathcal{R}_x$, the model $M$ analyzes $y$ and extracts experiential knowledge $$e = \mathrm{Extract\_Knowledge}(M(x, y, \mathcal{R}_x))$$. This knowledge is general, reusable, and abstracted beyond instance-specific details.

Training proceeds through on-policy context distillation: the experiential knowledge is prepended as additional context to a teacher model, yielding a target distribution $\pi_\mathrm{teacher}(\cdot | e, x)$. The policy is then optimized by minimizing the reverse KL divergence over tokens between $\pi_\theta$ and this context-conditioned teacher. This process internalizes experiential guidance via direct distributional supervision, discarding neither rich analysis nor fine-grained preferences.

Figure 2: RL maximizes scalar rewards assigned by an LLM-as-a-Judge, while EL trains via on-policy context distillation from LLM-as-a-Coach experiential knowledge, providing a denser signal for non-verifiable open-ended tasks.

Feedback Bandwidth Quantification

A key theoretical insight is the difference in feedback channel capacity. A discretized (1–10) reward signal in RL provides at most $\approx3.3$ bits per sample (or $x$0 bits for bf16 continuous reward), whereas a textual experiential context of $x$1 tokens at vocab size $x$2 can, in principle, encode $x$3 bits—several orders of magnitude more. While the practical, mutual-information of the context is less, the extreme bottleneck of scalar rewards is a clear limiting factor in trajectory discrimination and policy refinement on non-verifiable tasks.

Experimental Results

Setup

The empirical study leverages WildChat-IF as diverse, real-user instruction set, with GPT-4o-generated rubrics decomposing quality into fine-grained, independently checkable criteria. Both Qwen3-8B and OLMo-3-7B-Instruct models serve as policy backbones. The two feedback models explored are the initial frozen policy or closed-source GPT-4o. For RL, optimization is via GRPO on a discrete (1–10) scalar; for EL, experiential knowledge is extracted and used for context distillation.

Core Findings

EL consistently outperforms RL on non-verifiable open-ended tasks across both in-distribution (WildChat test set) and out-of-distribution benchmarks (AlpacaEval v2.0, WildBench, ArenaHard v2.0, and CreativeWritingV3). These improvements persist across feedback models, demonstrating that gains from EL derive from the higher-bandwidth supervision and not feedback LLM choice.

EL exhibits notably stronger generalization: while RL can attain higher training-set rewards (overfitting via reward exploitation), EL maintains better performance on novel, unseen instructions and tasks. The empirical results confirm that RL is vulnerable to reward hacking, manifesting as overoptimization toward the idiosyncrasies of training distribution reward structure, whereas EL's distributional matching and abstraction via experiential knowledge alleviate this risk.

Ablation studies indicate further that:

• Iterative teacher updates (using the policy of previous epochs as the teacher) enhance core task performance but can induce forgetting on OOD tasks; mixing with general-prompt OPD mitigates this.
• The specific format of contextual feedback impacts robustness: full critiques and rubric-only contexts degrade OOD generalization, while succinct, general experiential knowledge is optimal.
• Categorical, single-choice feedback collapses EL back to the low-bandwidth setting of scalar rewards, confirming that channel capacity is central to the observed gains.

Analysis and Theoretical Implications

A controlled synthetic distribution-matching analysis demonstrates that RL-based quantization produces staircase distributions, failing to distinguish among high-reward samples, whereas EL-style direct distributional matching faithfully reconstructs the target. This effect is immaterial for objective, binary tasks but dominates in multimodal, rubric-governed open-ended settings.

EL's methodology (i) provides dense, token-level supervision aligned to nuanced, high-level strategies and (ii) reduces gradient variance and optimization instability by shifting from reward-seeking to distribution-matching targets. This fundamentally improves the policy's ability to internalize multi-dimensional preferences without the sample inefficiency or reward hacking that plague non-verifiable RL.

Practical and Theoretical Implications

From a practical standpoint, EL advances scalable, sample-efficient post-training for LLMs on open-ended tasks, especially in frontier alignment domains where subjective evaluation and multi-faceted quality are the norm. EL enables practitioners to more completely "download" the expertise expressed in detailed rubric assessments into model parameters, facilitating enhanced controllability, user satisfaction, and robustness.

Theoretically, the work foregrounds the critical importance of feedback channel capacity in RLHF and post-training. It challenges the dominance of scalar-reward-centric RL, asserting the relevance of direct knowledge distillation, high-bandwidth textual feedback, and experiential abstraction for aligning generative LMs operating in non-verifiable spaces. This paradigm is extensible: the LLM-as-a-Coach design can be iteratively refined, chained, and composed with advances in rubric elicitation, dynamic context selection, and improved teacher architectures.

Conclusion

The LLM-as-a-Coach framework and accompanying EL paradigm represent a substantive methodological evolution for LLM post-training on open-ended, non-verifiable tasks. By shifting optimization from scalar reward maximization to dense, experiential-knowledge-informed distribution matching, EL overcomes the information bottleneck endemic in standard RL. The approach empirically and theoretically demonstrates significant improvement in both core task fidelity and generalization, while robustly mitigating reward hacking phenomena. This work establishes experiential knowledge extraction and high-bandwidth context distillation as foundational drivers for future scalable, robust, and interpretable LM alignment.