--- title: Curiosity-Driven RLHF Advances url: https://www.emergentmind.com/topics/curiosity-driven-rlhf-cd-rlhf type: topic --- # Curiosity-Driven RLHF Advances Searching arXiv for the specified CD-RLHF papers and closely related work to ground the article. Curiosity-Driven Reinforcement Learning from Human Feedback (CD-RLHF) denotes RLHF variants that supplement preference-derived extrinsic rewards with intrinsic rewards that favor exploration. Current arXiv usage suggests that the label covers at least two distinct 2025 formulations: a token-level curiosity mechanism for improving output diversity in summarization and instruction-following, introduced in "Curiosity-Driven Reinforcement Learning from Human Feedback" [2501.11463], and a belief-based curiosity mechanism for eliciting latent user traits in personalized multi-turn dialogue, developed in "Enhancing Personalized Multi-Turn Dialogue with Curiosity Reward" [2504.03206]. In both cases, the central idea is to preserve the optimization structure of RLHF while adding a shaped signal that rewards informative or novel trajectories. ## 1. Scope and conceptual variants CD-RLHF emerged from two different problem settings. In one setting, standard RLHF is described as improving alignment to human preferences while reducing output diversity relative to supervised fine-tuning, producing repetitive or homogenized outputs; CD-RLHF addresses this by adding token-level curiosity for novel states during generation [2501.11463]. In the other setting, multi-turn RLHF is described as prioritizing helpfulness and safety but falling short in empathetic, adaptive, and personalized interaction; CD-RLHF addresses this by rewarding reductions in uncertainty about latent user traits during dialogue [2504.03206]. The two formulations differ in what constitutes a “state novelty” signal and in what exploration is intended to achieve. The diversity-oriented formulation uses prediction error in a learned latent state space and applies it to tokens outside the top-$k$ by probability. The personalized-dialogue formulation uses a belief over hidden user types and rewards actions that improve the agent’s user model [2501.11463][2504.03206]. | Formulation | Intrinsic signal | Primary aim | |---|---|---| | Diversity-oriented CD-RLHF | Prediction error in an Intrinsic Curiosity Module | Optimize both output diversity and alignment quality | | Personalized dialogue CD-RLHF | Changes in belief over latent user type | Reveal user preferences and adapt to them | This suggests that CD-RLHF is better understood as a family of RLHF augmentations rather than a single algorithm. What is common across variants is the addition of intrinsic motivation to a standard RLHF backbone; what differs is the latent variable being explored, the reward granularity, and the failure modes. ## 2. Integration with RLHF objectives Both formulations preserve the standard separation between an extrinsic alignment signal and an auxiliary intrinsic signal. In the personalized multi-turn setting, the combined objective is reported as $$ J(\theta)=\mathbb{E}_{\tau\sim\pi_\theta}\Bigg[\sum_{t=0}^{T}\big(r_{\mathrm{RLHF}}(s_t,a_t)+\lambda\, r_{\mathrm{cur}}(s_t,a_t)\big)\Bigg], $$ with turn-level curiosity rewards added to the RLHF reward and optimized using the multi-turn RLHF training recipe of Shani et al. (2024); for Education Dialogue, the paper reports $\lambda=9.0$ [2504.03206]. In the diversity-oriented formulation, the combined reward is written at token level as $$ r_t = r^{(e)}_t + \eta\, r^{(i)}_t, $$ where the extrinsic reward combines a sequence-level reward model score with a KL penalty to the SFT reference model, $$ r^{(e)} = R - \beta D_{\mathrm{KL}}\big(\pi_{\mathrm{policy}}(\cdot|s_t)\,\|\,\pi_{\mathrm{ref}}(\cdot|s_t)\big), $$ and the policy is updated with PPO using generalized advantage estimation [2501.11463]. The personalized-dialogue paper gives a stronger theoretical account through a POMDP over observable dialogue histories $s_t$ and latent user type $u\in\mathcal{U}$. With extended state $s_t'=\langle s_t,u\rangle$, transition and reward are $$ \mathcal{T}'(s'_{t+1}\mid s'_t,a_t)=\mathcal{T}(s_{t+1}\mid s_t,a_t,u), \quad \mathcal{R}'(s'_t,a_t)=\mathcal{R}(s_t,a_t\mid u), $$ and the belief update is $$ b_{t+1}(u)\propto \mathcal{T}(s_{t+1}\mid s_t,a_t,u)\, b_t(u). $$ Using a belief-based expected reward, $$ \mathcal{R}^b(s_t,b_t,a_t)=\sum_u b_t(u)\,\mathcal{R}(s_t,a_t\mid u), $$ the paper invokes potential-based reward shaping over beliefs, $$ r^b(s_t,b_t,a_t)=\mathcal{R}^b(s_t,b_t,a_t)+\gamma\,\phi(b_{t+1})-\phi(b_t), $$ and states, via Eck et al. (2013), that such shaping leaves the optimal policy invariant [2504.03206]. By contrast, the diversity-oriented paper is framed operationally rather than through PBRS. It retains the conventional SFT $\rightarrow$ RM $\rightarrow$ PPO pipeline, initializes policy and reference models via SFT on the same preference dataset, trains the reward model on paired preferences, and inserts an Intrinsic Curiosity Module during PPO rollouts [2501.11463]. ## 3. Belief-based CD-RLHF for personalized multi-turn dialogue The personalized formulation targets latent user inference during online interaction. The agent does not assume pre-existing user profiles or histories; instead, it must infer the hidden user type from observed dialogue. Two environments are reported. In Education Dialogue, the latent trait is the student’s preferred learning style, described in prompts and analyses as “hands-on” versus “story-telling,” and earlier as “lecture-based” versus “hands-on.” In Exercise Recommendation, the user profile has 20 attributes, of which five are relevant to the ideal strategy: age/injury status, extroversion, motivation, outdoor preference, and socioeconomic status [2504.03206]. In practice, beliefs are not updated by an explicit Bayesian transition model. They are set by a fixed oracle user model: $$ p_t(u)=M_\phi(u\mid s_t), \qquad b_t \equiv p_t(\cdot). $$ For Education Dialogue, the oracle classifier is a Gemma 7B model that predicts preferred learning style from the conversation. For Exercise Recommendation, the oracle is a decision-tree-like classifier, with an LLM-based extractor when relevant attributes are implicit [2504.03206]. The intrinsic rewards are defined directly over changes in these beliefs. The paper reports potential functions requiring the ground-truth user type $u^*$ in simulated settings, $$ \phi_{\mathrm{acc}}(b)=b(u^*)-\frac{1}{|\mathcal{U}|}, $$ $$ \phi_{\mathrm{log\text{-}acc}}(b)=\log b(u^*)+\log|\mathcal{U}|, $$ $$ \phi_{\mathrm{neg\text{-}ent}}(b)=\sum_u b(u)\log b(u)+\log|\mathcal{U}|. $$ From these, it defines PBRS-consistent differential rewards: $$ r_{\mathrm{cur}}^{\mathrm{DiffAcc}}(t)=p_{t+1}(u^*)-p_t(u^*), $$ $$ r_{\mathrm{cur}}^{\mathrm{DiffLogAcc}}(t)=\log p_{t+1}(u^*)-\log p_t(u^*), $$ $$ r_{\mathrm{cur}}^{\mathrm{DiffEnt}}(t)=\sum_u p_t(u)\log p_t(u)-\sum_u p_{t+1}(u)\log p_{t+1}(u). $$ It also reports non-differential rewards, $$ r_{\mathrm{cur}}^{\mathrm{Acc}}(t)=p_{t+1}(u^*)-\frac{1}{|\mathcal{U}|}, $$ $$ r_{\mathrm{cur}}^{\mathrm{Ent}}(t)=\sum_u p_{t+1}(u)\log p_{t+1}(u)+\log|\mathcal{U}|, $$ and an information-gain surrogate, $$ r_{\mathrm{cur}}^{\mathrm{InfoGain}}(t)=D_{\mathrm{KL}}\big[p_{t+1}(u)\,\|\,p_t(u)\big]. $$ The policy action space is the full natural-language response at turn $t$. There is no hard-coded controller. The policy may progress the task or elicit user traits by asking clarifying questions or proposing preference-sensitive options. Differential curiosity rewards are reported to encourage early elicitation without artificially prolonging conversations, whereas non-differential rewards can incentivize longer dialogues [2504.03206]. The environments are fully simulated. Education Dialogue uses a Gemma 2B student simulator fine-tuned via supervised learning and a Gemma 2B reward model from Shani et al. (2024). Exercise Recommendation uses Gemini 1.5 Pro for detailed backstory generation consistent with sampled attributes; users are simulated by an LLM environment that stays consistent with those backstories. The same shaping mechanism is used in both domains, which the paper presents as domain-agnostic personalization [2504.03206]. ## 4. Token-level CD-RLHF for diversity-preserving alignment The diversity-oriented formulation adapts curiosity-driven exploration from RL to autoregressive text generation. The state at time $t$ is the prompt plus the partial generation, with $s_0=x$ and $s_t=\{s_0,a_{