External Reward Model

• External Reward Model is a formally defined mechanism that assigns rewards based on explicit, context-dependent criteria, enhancing cooperation in structured systems.
• It utilizes payoff formulas in spatial public goods games to clearly delineate the benefits and costs for cooperators, defectors, and rewarding cooperators using parameters like synergy factor, β, and γ.
• The model reveals cyclic dominance and dynamic spatial patterns, offering insights for designing adaptive and hybrid reward strategies in multi-agent systems.

An external reward model is a formally defined mechanism used to assign reward values to agents or processes based on their observed actions, outputs, or strategies, where the model itself is distinct from (and external to) the agent or policy being optimized. These models serve as evaluative or supervisory signals and play a pivotal role in complex multi-agent systems, large-scale AI model alignment, reinforcement learning from human feedback, and structured social game-theoretic scenarios.

1. Formal Modeling of External Reward Mechanisms

External reward models assign reward signals according to explicit, context-dependent criteria reflecting desirable behavior. In the spatial public goods game (Szolnoki et al., 2010), the reward mechanism extends canonical strategies—cooperators (C) and defectors (D)—to include rewarding cooperators (RC), formalizing the external incentive structure as follows:

• For each group of size $k+1$, including $k$ neighbors and the focal agent:
  • All cooperators (both C and RC) contribute 1 to the group fund.
  • The group’s pooled contributions are multiplied by a synergy factor $r$, reflecting the non-linear benefit of cooperation.
  • Each group member receives an equal share of the group reward.
  • External reward addition:
  • Each cooperator in the group receives an extra benefit $\beta/k$ from every RC in the group.
  • Each RC incurs an extra cost $\gamma/k$ per cooperator in the group.

This results in the following per-group payoff formulas:

$$\begin{align*} P_C &= \frac{r (N_C + N_{RC} + 1)}{k+1} - 1 + \frac{\beta}{k} N_{RC} \ P_D &= \frac{r (N_C + N_{RC})}{k+1} \ P_{RC} &= P_C - \frac{\gamma}{k} (N_C + N_{RC}) \end{align*}$$

where $N_C$, $N_{RC}$, and $N_D$ denote the counts of cooperators, rewarding cooperators, and defectors in the neighborhood, respectively.

This precise reward allocation mechanism is external: it is not determined by the agent’s policy but rather by the collective configuration and explicit reward schedule implemented by the external model.

2. Conditions for Reward Model Effectiveness and Comparison to Punishment

The effectiveness of an external reward model in eliciting cooperation is parameter-dependent:

• Synergy Factor ($r$): For low $r$ (e.g., $r=2.0$), cooperation is unsustainable without incentives; well-calibrated rewards enable the persistence of cooperation, even when baseline returns are weak. High $r$ can make rewards redundant due to network reciprocity effects.
• Reward-to-Cost Ratio ($\beta$ vs. $\gamma$): The reward’s impact is governed by the benefit-to-cost ratio. If $\beta \gg \gamma$, RCs can thrive. If $\gamma$ is high, pure cooperators become “second-order free-riders,” enjoying rewards without incurring cost, ultimately destabilizing the RC population.
• Comparison with Punishment:
  • Punishment imposes costs directly on defectors and is typically immediately effective but can negate overall system welfare if costs are high.
  • Reward requires reward levels ($\beta$) to be significantly larger relative to costs ($\gamma$) to be competitive with punishment, especially in structured populations.

These dynamics demonstrate that the configuration of the external reward model is crucial: suboptimal settings can inadvertently sustain non-cooperative behavior.

3. Emergence of Cyclic Dominance and Spatio-Temporal Dynamics

A central, nontrivial consequence of the external reward model in the spatial public goods framework is the spontaneous emergence of cyclic dominance among the three strategies:

• D invades C: Defectors exploit pure cooperators.
• RC invades D: Clusters of RCs, via reward clustering, resist defector invasion.
• C invades RC: Pure cooperators exploit RCs by avoiding the cost $\gamma$, acting as second-order free-riders.

This creates a closed loop resembling a rock–paper–scissors dynamic, resulting in oscillating spatial patterns and dynamic coexistence of all strategies. The spatial structure (network topology) is critical; local clustering enables the persistence of minority strategies and forestalls domination by defectors or non-rewarding cooperators.

If any strategy goes extinct (e.g., due to finite-size effects), the dominance cycle is interrupted, and one strategy may become fixed in the population.

4. Impact on Cooperative System Design and Calibration of External Rewards

Several implications can be drawn for the design of external reward models in engineered or natural cooperative systems:

• Moderate Calibration Outperforms Excessive Rewards: The optimal regime is typically at moderate values of $\beta$, as excessively high rewards can destabilize coexistence and inadvertently benefit defectors by reinforcing the dominance cycle.
• Spatial Topology and Clustering: Structured populations (e.g., lattices or networks with strong community structure) enhance the effectiveness of external rewards and should be exploited in practical system design.
• Trade-Off with Punishment: Reward mechanisms may suffer from the second-order free-rider effect more acutely than punishment models. Careful model tuning (high $\beta/\gamma$) and possibly hybrid reward–punishment strategies may be needed in resource-constrained or heterogeneous environments.
• Dynamic/Adaptive Rewards: Static external reward schedules can be suboptimal due to the dynamic feedback loop of cyclic dominance; dynamic or adaptive tuning strategies responsive to population state may increase system-level cooperation and robustness.

5. Generalizability and Modeling Limitations

The findings from the spatial public goods game with external rewards demonstrate the following general properties for external reward models:

• Their impact is highly sensitive to parameterization and context—network topology, group size, and intra-group variation all affect model outcomes.
• External reward models, when improperly calibrated, may support the undesirable persistence of non-cooperative strategies due to higher-order free-riding and dynamic population cycles.
• The mathematical structure provided (group-level payoff formulas, explicit benefit/cost allocation, and spatial simulation protocol) serves as a blueprint for designing and analyzing external rewards in broader agent-based models and socio-technical systems.

While the results are robust for structured populations and spatial models with pairwise/local interactions, extension to heterogeneous/scale-free networks or settings with non-local externalities may require additional theoretical and simulation-based analysis.

6. Summary Table: Key Features of the Spatial External Reward Model

Feature	Description	Mathematical Representation
Agent Types	C (cooperator), D (defector), RC (rewarding cooperator)	$P_C$, $P_D$, $P_{RC}$ (see main text)
External Reward Signal	RC gives $\beta/k$ to each cooperator, at cost $\gamma/k$ per recipient	$P_C$: reward benefit, $P_{RC}$: additional cost term
Viability Condition	Reward effective if $\beta/\gamma$ high and $r$ low/moderate	See payoff formulas and discussion
Spatial Structure	Square lattice, local groups of $k+1$	Model defined for $k=4$, lattice with neighbor-based groups
Emergent Phenomenon	Cyclic dominance (D→C→RC→D), oscillatory coexistence	Spatial dynamics, not a single formula

7. Implications for Broader Application of External Reward Models

The formalization and empirical findings in (Szolnoki et al., 2010) provide a foundational framework for incorporating external reward models in multi-agent cooperation tasks. Critical lessons for practitioners include the necessity for moderate, context-aware calibration, the risk of inadvertent support for non-cooperative behavior, and the utility of spatial or network-based population structuring. Future developments might focus on hybrid incentive models, adaptive feedback control, and the design of reward rules robust to higher-order free-riding and dynamic system effects.