RL with Intrinsic Mutual ToM Rewards

• RL with intrinsic mutual ToM rewards uses step-wise internal reward signals from agents' belief models to promote social cognition and coordination.
• Methodologies integrate curiosity, imitation, MI-based influence, and causal reasoning to model and influence partner behaviors in multi-agent settings.
• Experiments demonstrate that ToM reward systems accelerate group learning, enhance cooperation, and narrow Nash equilibria toward efficient, cooperative outcomes.

Reinforcement learning (RL) with intrinsic mutual Theory-of-Mind (ToM) rewards refers to a family of multi-agent learning frameworks in which agents' reward functions are augmented with internal, reward-shaping terms designed to incentivize mutual awareness, reciprocal influence, and explicit modeling of others’ beliefs and responses. This approach formalizes mechanisms by which agents not only pursue extrinsic environmental objectives but also develop social-cognitive competencies analogous to curiosity, imitation, social influence, guilt aversion, and anticipation, all through intrinsic motivation terms derived from the joint evolution of agent behaviors in interactive settings.

1. Intrinsic Mutual ToM Reward Constructs

Intrinsic mutual ToM rewards operationalize social drives in RL as step-wise intrinsic reward signals grounded in agents’ internal models of others. Notable formalizations include:

• Active-inference curiosity rewards: Agents are incentivized to reduce uncertainty or maximize surprise prediction errors about their interactive environment, typically via auxiliary predictors trained to forecast social events, e.g., perceptual crossings. For a binary crossing event $e$ and predictor confidence $p$, the reward is given by:

$$r_{\rm curio}(e,p) = \begin{cases} 1-p, & e=0 \ p^n, & e=1 \end{cases}$$

with $n$ a sharpness exponent (Fernando et al., 9 Apr 2025).

• Imitation rewards: Quantify the extent to which an agent's pattern of actions or transitions is reproduced by its partner, or vice versa, over sliding time windows. They include "promotion of imitation" (rewarding when the agent acts and the partner replicates transition sequences) and direct imitation (rewarding when the agent itself copies another), summed as the total imitation reward (Fernando et al., 9 Apr 2025).
• Mutual information (MI), influence, and impressionability rewards: Define the intrinsic reward as the estimated mutual information between temporally chunked behaviors or transitions, rewarding agents both for influencing others and for being influenced (impressionability). The MI is estimated empirically over recent behavior buffers:

$$I(X;Y) = \sum_{x,y} \hat p(x,y) \log \frac{\hat p(x,y)}{\hat p(x)\hat p(y)}$$

with the MI reward as the sum of impressionability and influence contributions. "Anticipation" is introduced by maximizing MI with respect to delayed partner behaviors, necessitating predictive capability (Fernando et al., 9 Apr 2025).

• Causal influence reward: Implemented as the KL-divergence between the conditional policy of another agent given the actual action and the marginal over counterfactual actions:

$$c_t^i = \sum_{j\neq i} D_{\mathrm{KL}}\left[ p(a_t^j | a_t^i, s_t^j, u_t^j)\;||\;p(a_t^j | s_t^j, u_t^j) \right]$$

Demonstrated to be equivalent in expectation to the conditional mutual information between an agent's action and the actions of its peers (Jaques et al., 2018).

• Guilt-aversion (belief-based ToM) reward: In ToMAGA, agents develop explicit beliefs over other agents' intentions and incorporate "anticipated disappointment" into their reward function. This is the difference between what an agent believes another expects and what is delivered, scaled by a guilt-sensitivity parameter:

$$r_i^{(\rm psy)}(\tilde\ell_i, \tilde\ell_j) = -\theta_{ij} \max\bigl(0, \phi_j - r_j^{(T)}(\tilde\ell_i, \tilde\ell_j)\bigr)$$

where $\phi_j$ is the anticipated value for $j$ under $i$'s beliefs (Nguyen et al., 2020).

2. Multi-Agent Architectures and Algorithms

Architectures for RL with intrinsic mutual ToM rewards are unified by two features: agent-centric models of other agents (explicit or implicit), and the integration of step-wise intrinsic rewards with external task rewards. Key mechanisms include:

• Policy networks with auxiliary ToM heads: Each agent embeds a primary LSTM-based policy and value network, plus parallel auxiliary heads or networks trained to predict the actions or beliefs of others. These predictors are trained with supervised cross-entropy based on actual partner actions, facilitating empirical estimation of MI or counterfactual influence (Jaques et al., 2018, Fernando et al., 9 Apr 2025).
• Rolling buffers and chunk-based MI estimation: Agents maintain buffers of recent transitions and actions to compute empirical probabilities for MI estimation between distinct behavioral windows (Fernando et al., 9 Apr 2025).
• Joint RL and ToM loss optimization: The combined agent loss includes (a) standard RL losses (e.g., PPO, A3C), (b) supervised losses for predicting others’ actions, and (c) step- or episode-based intrinsic ToM rewards (Jaques et al., 2018, Nguyen et al., 2020).
• Counterfactual reasoning for social influence: Agents actively simulate alternative actions to compute the causal effect of their behavior on peers, implemented via internal models ("Theory-of-Mind" networks, or MOA) for fully decentralized training (Jaques et al., 2018).
• Recursive or layered belief updates: Agents in guilt-aversion ToM frameworks maintain multiple orders of belief—representing their own beliefs, others’ beliefs, and others’ beliefs about themselves—updated episodically based on observed outcomes and confidence parameters to inform social shaping (Nguyen et al., 2020).

3. Experimental Findings and Performance Metrics

Empirical evaluation demonstrates that intrinsic mutual ToM reward mechanisms robustly promote social engagement, coordination, and emergent cooperation:

• Preference for social interaction: With only curiosity rewards, agents show unstructured exploration (self–other crossing rates plateau at chance, e.g., $p$0), but with imitation or MI rewards, interaction rates (e.g., avatar crossing rates) rise to $p$1 and agents strongly prefer socially meaningful crossings over interactions with shadows or private objects ($p$2, $p$3, $p$4) (Fernando et al., 9 Apr 2025).
• Accelerated and higher group returns: In common-pool and public-good settings, MI/influence rewards lead to faster learning and higher collective returns compared to standard RL, explicit prosocial reward mixing, or auxiliary ToM prediction without MI-based reward (Jaques et al., 2018).
• Emergent communication and coordination: Influence-based intrinsic rewards induce the co-invention of effective signaling protocols, with metrics such as speaker consistency and instantaneous coordination strongly correlated with high-influence timesteps (Jaques et al., 2018).
• Facilitation of cooperation under reward asymmetry: When only a subset of agents receives extrinsic rewards, MI rewards allow rewarded agents to "shape" the behavior of unrewarded agents, catalyzing cooperation even in the absence of direct task incentives for some participants (Fernando et al., 9 Apr 2025).
• Reduction to efficient Nash equilibria: Guilt-aversion ToM reward shaping provably narrows equilibria in games like Stag Hunt to Pareto-optimal (cooperative) outcomes, contingent on sufficient guilt-sensitivity and precise belief modeling (Nguyen et al., 2020).

4. Ablation Studies and Theoretical Insights

Ablation experiments elucidate the contribution of individual ToM reward components and the role of model structure:

• Influence vs. impressionability: Omission of impressionability rewards ($p$5) does not preclude robust interaction in repeated games, indicating that "influence alone" can induce stable social coupling as long as reciprocal interaction persists (Fernando et al., 9 Apr 2025).
• Necessity of first-order belief recursion: Guilt aversion without explicit ToM (i.e., $p$6, but with no first-order belief update) improves over purely selfish agents but frequently stalls at suboptimal equilibria; recursive belief modeling is essential for reliable cooperative convergence (Nguyen et al., 2020).
• Decentralized MOA suffice for mutual influence: Influence rewards computed solely from self-maintained ToM models (no access to a centralized critic or other agents' rewards) are as effective as any centralized mechanism for producing coordination (Jaques et al., 2018).
• Parameter sensitivity and thresholds: Cooperation is highly sensitive to the magnitude of guilt parameters: below a critical threshold, ToM-based shaping is ineffective; above it, cooperation rapidly dominates (Nguyen et al., 2020).

5. Generalization and Extensions

The core principles of RL with intrinsic mutual ToM rewards generalize across a wide array of social and coordination environments:

• Applicability to general-sum Markov games and social dilemmas: The shaping approach, particularly belief-based and MI-based techniques, seamlessly transfers to diverse Markov games (e.g., Prisoner’s Dilemma, public goods, and resource management), requiring no centralized controller (Nguyen et al., 2020).
• Scalability to larger populations and higher-order beliefs: Mutual ToM reward architectures can be extended to $p$7-agent environments through belief trees or distributed models, and depth of recursion can be increased (though practical gains depend on sample efficiency and model capacity) (Nguyen et al., 2020).
• Combination with other prosocial biases: Intrinsic mutual ToM rewards may synergistically combine with other mechanisms such as inequity aversion, reciprocity, or reputation to address more complex, multi-stage, or networked social dilemmas (Nguyen et al., 2020).
• Bootstrapping of shared protocols: By maximizing reciprocal MI and imitation-based terms, agents self-organize "language games"—abstract signaling protocols that support alignment of intent and coordination without external instruction or shared extrinsic goals (Fernando et al., 9 Apr 2025).

6. Significance and Open Directions

Research on RL with intrinsic mutual ToM rewards constructs a principled methodological bridge between mechanistic accounts of social cognition and deep multi-agent system design. This paradigm enables:

• Autonomous emergence of social agency: Agents endowed with drives to be understood, to imitate, to influence, and to anticipate are able to develop robust, dynamic cooperation and protocol invention, even with incomplete information or asymmetric task structure (Fernando et al., 9 Apr 2025, Jaques et al., 2018).
• Decentralized social learning and adaptation: By relying solely on agents’ internal models and locally observed variables, such systems avoid the requirements—and vulnerabilities—of centralized training or common knowledge of rewards, aligning more closely with real-world distributed social intelligence (Jaques et al., 2018).
• Unified theoretical foundation: Conditional mutual information provides the formal backbone connecting causal influence, anticipation, and reciprocal modeling, grounding a wide variety of observed social behaviors in a single information-theoretic objective (Jaques et al., 2018).

A plausible implication is that further advances in sample-efficient estimation of mutual information, scalable belief updates, and compositional reward shaping could unlock higher-level emergent social phenomena in artificial multi-agent collectives, including theory-of-mind recursion, flexible signaling, and robust coalition formation.

---

Selected Papers Referenced:

• "Wanting to be Understood" (Fernando et al., 9 Apr 2025)
• "Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning" (Jaques et al., 2018)
• "Theory of Mind with Guilt Aversion Facilitates Cooperative Reinforcement Learning" (Nguyen et al., 2020)