Mental Reward Models

• Mental Reward Models are frameworks that incorporate internal cognitive signals (e.g., latent pain, metacognitive states) with external outcomes to guide learning.
• They employ diverse methodologies such as latent-state filtering, metacognitive trajectory supervision, and appraisal-based reward shaping to enhance process quality.
• Applications span reinforcement learning, social agent modeling, and therapeutic dialogue, where aligning internal evaluations with external metrics improves performance.

A mental reward model is a reward construction in which optimization is guided not only by externally observed task outcomes but also by inferred internal states, metacognitive process variables, appraisal checks, social belief estimates, or latent satisfaction. In contemporary arXiv literature, the term covers several technically distinct families: subjective well-being functions that subtract inferred pain-belief from reward, metacognitive trajectory rewards for LLM reasoning, appraisal-based affective rewards derived from temporal-difference signals, intrinsic rewards for predicting other agents’ beliefs, distributional mental rewards for recommendation, and mechanistic reward-valuation signals in multimodal models (Petrowski et al., 6 Jan 2026, Chen et al., 22 May 2026, Zhang et al., 2023, Oguntola et al., 2023, Zhao et al., 26 Sep 2025, Honarmand et al., 7 Jul 2026). The unifying feature is that reward is treated as a model of mental or internally evaluated significance rather than as a direct synonym for environmental payoff.

1. General form of mental reward models

Across the literature, mental reward models are defined by augmenting or replacing an objective reward with variables intended to encode internal evaluation. In introspective reinforcement learning, the subjective reward is

$$f^w_t = w_1\,R^{\rm obj}_t + w_2\,(R^{\rm obj}_t - \rho) + w_3\,\Delta_t - w_4\,b_t(\mathrm{pain}),$$

where the final term is a filtered posterior over a latent pain state. In Theory-of-Mind multi-agent RL, the total reward is

$$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$

In appraisal-based modeling, the mental reward is an additive combination of novelty, goal relevance, goal conduciveness, and power. In metacognitive LLM training, the trajectory reward is

$$R(\tau)=\alpha\,K(\tau)+\beta\,R_g(\tau)+\gamma\,C(\tau).$$

In sequential recommendation, the reward is explicitly modeled as a distribution, $R(s,a)\equiv \text{law}(Z(s,a))$, rather than as a deterministic scalar (Petrowski et al., 6 Jan 2026, Oguntola et al., 2023, Zhang et al., 2023, Chen et al., 22 May 2026, Zhao et al., 26 Sep 2025).

Taken together, these formulations separate at least four axes of design. First, the internal signal may be latent-state belief, as in hidden Markov pain inference, or an explicit graded evaluation, as in metacognitive knowledge coverage and regulation fidelity. Second, the reward may be step-level, trajectory-level, or both. Third, the mental variable may be self-directed, other-directed, or socially comparative. Fourth, the mental reward may serve either as the principal learning target or as an auxiliary alignment signal added to an existing objective. This suggests that “mental reward model” is best understood as a family resemblance term for psychologically structured reward design, rather than a single canonical architecture.

2. Introspective reward and internal-state inference

In “Exploration Through Introspection: A Self-Aware Reward Model,” the core mental variable is a latent two-state HMM with $H=\{\mathrm{pain},\mathrm{no\_pain}\}$ and observations $O=\{\mathrm{noxious},\mathrm{harmless}\}$. The model performs online filtering with

$$\alpha_t(j)=\Bigl[\sum_{i\in H}\alpha_{t-1}(i)\,a_{i\to j}\Bigr]\,b_j(O_t),\qquad b_t(j)=\frac{\alpha_t(j)}{\sum_k \alpha_t(k)},$$

and uses the posterior pain-belief $b_t(\mathrm{pain})$ as an aversive term inside a subjective well-being function. The framework includes normal and chronic pain parameterizations from Eckert et al. (2022): the chronic model has sticky transitions and ambiguous emissions relative to the normal model, and the initial distribution is also biased toward pain. Q-learning is then performed on the subjective reward rather than on the bare environmental reward (Petrowski et al., 6 Jan 2026).

The experiments place agents in a stationary $7\times 7$ grid with a single “food” cell and in a non-stationary variant where the food jumps to a different corner every $1\,250$ steps. The reported metrics are Cumulative Objective Reward, momentary and cumulative well-being, and pain-belief trajectories. In almost all reward categories, normal- and chronic-pain agents significantly outperformed their “no-pain” baselines under paired $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$0-tests with $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$1. In the stationary setting, normal and chronic models achieved $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$2 versus baseline $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$3. In the non-stationary setting, the chronic model sometimes exceeded the normal model, for example in Objective+Expect with chronic $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$4 versus normal $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$5, but at the cost of a persistently negative cumulative well-being. The chronic agent’s relief-seeking cycle is described as paralleling addictive-like behaviour (Petrowski et al., 6 Jan 2026).

A related but more abstract treatment of distorted reward processing appears in Human-Based Thompson Sampling. There the reward model is not an inferred latent state but a parameterized update bias in a Beta-Bernoulli bandit:

$$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$6

The four parameters $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$7 encode retention of past positive evidence, sensitivity to current positive reward, retention of past negative evidence, and sensitivity to current negative reward. Disorder-inspired settings include Addiction, ADHD, Alzheimer’s, Chronic Pain, bvFTD, and Parkinson’s; Chronic Pain is parameterized as $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$8, i.e. down-weighted immediate positive reward with full retention of negative history. In all 12 evaluated settings, at least one HBTS variant strictly outperformed standard Thompson Sampling (Bouneffouf et al., 2017).

3. Metacognition and appraisal as reward

In “Metacognition as Reward: Reinforcing LLM Reasoning via Knowledge and Regulation Signals,” the rollout is explicitly scaffolded into Metacognitive Knowledge, Metacognitive Regulation, optional LOOKBACK, and final answer. A grader returns the number of covered knowledge units $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$9, recovered units $$R(\tau)=\alpha\,K(\tau)+\beta\,R_g(\tau)+\gamma\,C(\tau).$$0, plan-alignment score $$R(\tau)=\alpha\,K(\tau)+\beta\,R_g(\tau)+\gamma\,C(\tau).$$1, shortcut flag $$R(\tau)=\alpha\,K(\tau)+\beta\,R_g(\tau)+\gamma\,C(\tau).$$2, and correctness $$R(\tau)=\alpha\,K(\tau)+\beta\,R_g(\tau)+\gamma\,C(\tau).$$3. The two process rewards are

$$R(\tau)=\alpha\,K(\tau)+\beta\,R_g(\tau)+\gamma\,C(\tau).$$4

with $$R(\tau)=\alpha\,K(\tau)+\beta\,R_g(\tau)+\gamma\,C(\tau).$$5, and the full trajectory reward is

$$R(\tau)=\alpha\,K(\tau)+\beta\,R_g(\tau)+\gamma\,C(\tau).$$6

The default setting uses $$R(\tau)=\alpha\,K(\tau)+\beta\,R_g(\tau)+\gamma\,C(\tau).$$7, yielding $$R(\tau)=\alpha\,K(\tau)+\beta\,R_g(\tau)+\gamma\,C(\tau).$$8. Optimization is performed with group-relative advantages and the clipped DAPO objective. On 22 benchmarks, MaR improved performance by up to $$R(\tau)=\alpha\,K(\tau)+\beta\,R_g(\tau)+\gamma\,C(\tau).$$9 over the base model and up to $R(s,a)\equiv \text{law}(Z(s,a))$0 over vanilla DAPO; on out-of-domain long-context tasks it increased $R(s,a)\equiv \text{law}(Z(s,a))$1 by $R(s,a)\equiv \text{law}(Z(s,a))$2, $R(s,a)\equiv \text{law}(Z(s,a))$3 by $R(s,a)\equiv \text{law}(Z(s,a))$4, and $R(s,a)\equiv \text{law}(Z(s,a))$5 by $R(s,a)\equiv \text{law}(Z(s,a))$6. Ablations showed that removing either metacognitive component hurt performance more than removing final correctness, indicating that the process-level supervision is not reducible to outcome-only reward (Chen et al., 22 May 2026).

A different formal route is taken by Zhang, Broekens, and Jokinen, who derive mental reward from appraisal theory and temporal-difference learning. They define novelty as

$R(s,a)\equiv \text{law}(Z(s,a))$7

goal relevance as

$R(s,a)\equiv \text{law}(Z(s,a))$8

goal conduciveness by clipping the TD error and rescaling it to $R(s,a)\equiv \text{law}(Z(s,a))$9,

$H=\{\mathrm{pain},\mathrm{no\_pain}\}$0

and power as the mean-to-minimum action-value gap in the successor state. The unified mental reward is then

$H=\{\mathrm{pain},\mathrm{no\_pain}\}$1

with identity mappings for each appraisal component. The same appraisal vector is also sent to a pre-trained SVM for discrete emotion classification. In the “Fear” vignette, the appraisal vector $H=\{\mathrm{pain},\mathrm{no\_pain}\}$2 yields “Fear” as the modal emotion. The authors explicitly note that substituting $H=\{\mathrm{pain},\mathrm{no\_pain}\}$3 into the Q-update is proposed, but not empirically evaluated beyond the vignette studies (Zhang et al., 2023).

These two strands share a structural principle: reward is moved upstream from final correctness or external payoff to intermediate evaluative variables that are intended to reflect how the agent monitors or appraises its own cognition. A plausible implication is that mental reward models are especially attractive where sparse terminal rewards fail to constrain process quality.

4. Social and comparative mental reward

In social reinforcement learning, Oguntola et al. define a mental reward from second-order belief prediction. Each agent maintains a first-order belief vector $H=\{\mathrm{pain},\mathrm{no\_pain}\}$4 and a second-order belief matrix $H=\{\mathrm{pain},\mathrm{no\_pain}\}$5, where $H=\{\mathrm{pain},\mathrm{no\_pain}\}$6 is agent $H=\{\mathrm{pain},\mathrm{no\_pain}\}$7’s prediction of agent $H=\{\mathrm{pain},\mathrm{no\_pain}\}$8’s first-order belief. The intrinsic reward is the negative average prediction error,

$H=\{\mathrm{pain},\mathrm{no\_pain}\}$9

with $O=\{\mathrm{noxious},\mathrm{harmless}\}$0 equal to MSE or CE depending on whether beliefs are continuous or discrete. This reward is added to the task reward and optimized under PPO within a belief-grounded policy architecture that separates a supervised belief predictor from a residual representation and a separate ToM predictor. In Particle World Physical Deception, equipping the good team with the ToM intrinsic reward increased mean cumulative episode reward from $O=\{\mathrm{noxious},\mathrm{harmless}\}$1 for “No belief modules” to $O=\{\mathrm{noxious},\mathrm{harmless}\}$2 for “2nd-order ToM (good agents),” while the adversary reward dropped from $O=\{\mathrm{noxious},\mathrm{harmless}\}$3 to $O=\{\mathrm{noxious},\mathrm{harmless}\}$4 in that setting. The paper characterizes this as an approximately $O=\{\mathrm{noxious},\mathrm{harmless}\}$5 boost for the good team over the no-belief baseline (Oguntola et al., 2023).

A contrasting social formulation appears in the constructive study of social comparison without explicit inference of others’ reward values. Three probabilistic-generative models are compared: the Internal Prediction Model (IPM), which infers the partner’s subjective values; the No Comparison Model (NCM), which ignores partner information; and the External Comparison Model (ECM), which incorporates partner rewards directly at the top situation-awareness node. All are implemented as multi-layered, multimodal latent Dirichlet allocation variants with Gibbs sampling and message passing. Evaluation uses the Rand Index for classification of subjective values across six experimental conditions. The reported scores are $O=\{\mathrm{noxious},\mathrm{harmless}\}$6 for NCM, $O=\{\mathrm{noxious},\mathrm{harmless}\}$7 for IPM, and $O=\{\mathrm{noxious},\mathrm{harmless}\}$8 for ECM, with chance level approximately $O=\{\mathrm{noxious},\mathrm{harmless}\}$9. Under these settings, the result suggests that social comparison relies on objective reward differences rather than on inferences about subjective states (Taniuchi et al., 21 Dec 2025).

The juxtaposition is instructive. In one line of work, modeling others’ beliefs is itself rewarded. In the other, the best-fitting social valuation model omits explicit partner-value inference. This indicates that “mental reward” in social settings can refer either to explicit Theory-of-Mind competence or to a more parsimonious constructive use of social information.

5. Alignment in counseling dialogue, therapeutic reward models, and recommendation

In mental-health counseling dialogue generation, READER formulates the action as a structured pair $$\alpha_t(j)=\Bigl[\sum_{i\in H}\alpha_{t-1}(i)\,a_{i\to j}\Bigr]\,b_j(O_t),\qquad b_t(j)=\frac{\alpha_t(j)}{\sum_k \alpha_t(k)},$$0: first a therapist dialogue-act, then the utterance. Its reward blends ROUGE-1 $$\alpha_t(j)=\Bigl[\sum_{i\in H}\alpha_{t-1}(i)\,a_{i\to j}\Bigr]\,b_j(O_t),\qquad b_t(j)=\frac{\alpha_t(j)}{\sum_k \alpha_t(k)},$$1, BERTScore, an act-compliance score $$\alpha_t(j)=\Bigl[\sum_{i\in H}\alpha_{t-1}(i)\,a_{i\to j}\Bigr]\,b_j(O_t),\qquad b_t(j)=\frac{\alpha_t(j)}{\sum_k \alpha_t(k)},$$2 from a pre-trained SPARTA classifier, and a KL regularizer against a frozen GPT-2 reference policy:

$$\alpha_t(j)=\Bigl[\sum_{i\in H}\alpha_{t-1}(i)\,a_{i\to j}\Bigr]\,b_j(O_t),\qquad b_t(j)=\frac{\alpha_t(j)}{\sum_k \alpha_t(k)},$$3

After grid search, the coefficients are $$\alpha_t(j)=\Bigl[\sum_{i\in H}\alpha_{t-1}(i)\,a_{i\to j}\Bigr]\,b_j(O_t),\qquad b_t(j)=\frac{\alpha_t(j)}{\sum_k \alpha_t(k)},$$4, $$\alpha_t(j)=\Bigl[\sum_{i\in H}\alpha_{t-1}(i)\,a_{i\to j}\Bigr]\,b_j(O_t),\qquad b_t(j)=\frac{\alpha_t(j)}{\sum_k \alpha_t(k)},$$5, $$\alpha_t(j)=\Bigl[\sum_{i\in H}\alpha_{t-1}(i)\,a_{i\to j}\Bigr]\,b_j(O_t),\qquad b_t(j)=\frac{\alpha_t(j)}{\sum_k \alpha_t(k)},$$6, and $$\alpha_t(j)=\Bigl[\sum_{i\in H}\alpha_{t-1}(i)\,a_{i\to j}\Bigr]\,b_j(O_t),\qquad b_t(j)=\frac{\alpha_t(j)}{\sum_k \alpha_t(k)},$$7. PPO uses $$\alpha_t(j)=\Bigl[\sum_{i\in H}\alpha_{t-1}(i)\,a_{i\to j}\Bigr]\,b_j(O_t),\qquad b_t(j)=\frac{\alpha_t(j)}{\sum_k \alpha_t(k)},$$8, $$\alpha_t(j)=\Bigl[\sum_{i\in H}\alpha_{t-1}(i)\,a_{i\to j}\Bigr]\,b_j(O_t),\qquad b_t(j)=\frac{\alpha_t(j)}{\sum_k \alpha_t(k)},$$9, and GAE-$b_t(\mathrm{pain})$0. Ablations show that removing the ROUGE term drops ROUGE-1 $b_t(\mathrm{pain})$1 from $b_t(\mathrm{pain})$2 to $b_t(\mathrm{pain})$3 and METEOR from $b_t(\mathrm{pain})$4 to $b_t(\mathrm{pain})$5, while removing the act-compliance term reduces ROUGE-2 recall from $b_t(\mathrm{pain})$6 to $b_t(\mathrm{pain})$7 and BERTScore from $b_t(\mathrm{pain})$8 to $b_t(\mathrm{pain})$9. The full reward combination yields the best reported gains on HOPE (Srivastava et al., 2023).

Reward modeling for therapy is also treated at the benchmark level in PRMB, a benchmark for long-horizon CBT-based counseling dialogue. PRMB spans 6 sessions and 21 negative scenarios grouped into four clusters, uses progressive summarization to preserve cross-session state, and contains $7\times 7$0 real or synthetic CBT cases, $7\times 7$1 prompts, $7\times 7$2 pairwise preference pairs, and $7\times 7$3 Best-of-4 queries. Evaluation uses pairwise accuracy, Best-of-$7\times 7$4 accuracy, and Spearman’s rank correlation between benchmark ranking and downstream Best-of-$7\times 7$5 BERTScore ranking. The reported correlation is $7\times 7$6 with $7\times 7$7, compared with $7\times 7$8 and $7\times 7$9 for RewardBench2. PRMB also reveals large pairwise-to-BoN gaps of $1\,250$0–$1\,250$1 percentage points, session-specific degradation in Sessions 3–5, and the weakest BoN performance on subtle cumulative harms in Cluster IV at approximately $1\,250$2 (Zhou et al., 12 Mar 2026).

In recommendation, MTRec treats user behavior as a user-centric MDP $1\,250$3, where the reward is a random mental reward rather than a deterministic click-derived scalar. The model learns a distributional reward with quantile inverse RL, recovers the mean reward as

$1\,250$4

and then uses $1\,250$5 as auxiliary supervision for downstream recommenders. Offline results on Amazon Books and Electronics show AUC gains of approximately $1\,250$6–$1\,250$7 and NCIS gains of approximately $1\,250$8–$1\,250$9. In Virtual Taobao, PPO improves episodic CTR from $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$00 to $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$01 and SAC from $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$02 to $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$03 when mental reward is added. In an industrial short-video platform, the deployed system increases average user viewing time by $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$04 over 7 days (Zhao et al., 26 Sep 2025).

These applications differ in domain, but they share the same correctional aim: the learned or engineered reward is intended to align training with latent quality criteria that raw interaction signals do not fully capture.

6. Mechanistic reward valuation, active mental imagery, and scope

Honarmand et al. give the most explicitly mechanistic account of a mental reward model. Adapting the Monetary Incentive Delay task to text for a VLM, they define a neuron-level reward signal

$$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$05

select units satisfying $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$06, and identify roughly $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$07 of the model’s neurons as NAc-selective, all in late MLP layers $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$08–$$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$09. Cross-frame validation over four paraphrased MID prompts gives a correlation $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$10 in unit identity. During incentivized trials, the reward signal is causally perturbed by activation patching, $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$11, or by gain scaling $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$12. The resulting behavioral changes mirror anhedonia: in ASDiv-EEfRT, mean points fall from approximately $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$13 to approximately $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$14, a $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$15 drop with $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$16, while forced-choice control accuracy remains $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$17 with $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$18. In Probability-EEfRT, $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$19 collapses by $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$20 overall with $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$21, yet both intact and perturbed models compute $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$22 on $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$23 of trials. Psychometric analogues shift in the expected direction: DARS decreases by $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$24, MAP-SR by $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$25, and AES increases by $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$26, all with $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$27 (Honarmand et al., 7 Jul 2026).

A separate expansion of the concept appears in SpatialDreamer, where active mental imagery is trained by Geometric Policy Optimization. GeoPO replaces episode-only reward with tree-structured sampling, bottom-up step-level rewards,

$$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$28

and a combined reward

$$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$29

with geometric penalties for redundant or conflicting actions and $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$30. The method is designed for long-horizon spatial reasoning with imagined views from a world model. Reported ablations attribute a $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$31 drop on MindCube-Tiny to removing the geometric penalty, and the full model improves over GRPO on several benchmarks, including $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$32 versus $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$33 on MindCube-Tiny and $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$34 on SAT-Real/Synthesized versus approximately $$r^i_t = r^{\mathrm{task},i}_t + \alpha\,r^{\mathrm{ToM},i}_t.$$35 for GRPO (Cao et al., 8 Dec 2025).

A common misconception is that a mental reward model is necessarily a reward model for mental-health applications. The current literature is broader. It includes models of pain-belief, metacognitive monitoring, appraisal, social comparison, reward valuation circuitry, recommendation satisfaction, and active mental imagery. Another misconception is that such models merely add heuristic reward shaping. The cited work instead shows multiple formal commitments: latent-state filtering, structured trajectory grading, probabilistic generative modeling, inverse reinforcement learning, causal activation patching, and dense credit assignment. This suggests that the central research question is not whether reward should be “mental,” but which internal variables can be formalized reliably enough to improve adaptation, alignment, or interpretability without collapsing task competence.