Dual RL Policies in Minority Game

• The paper introduces DRLP-MG, integrating Q-learning and classical strategies to achieve synergy that reduces resource volatility.
• It demonstrates how intra- and inter-subpopulation dynamics, including cluster formation and negative cross-correlations, enhance allocation efficiency.
• Mathematical analysis reveals a phase transition and momentum strategy emergence, with frozen agent behaviors crucial to coordination.

Dual Reinforcement Learning Policies in the Minority Game (DRLP-MG) refer to the synergistic integration of heterogeneous learning rules—primarily Q-learning and classical (static) strategy selection—in populations of competitive agents allocating limited resources. Recent work has formalized the complex forms of intra- and inter-subpopulation synergy that emerge from the interaction of these dual policy types, as well as their implications for volatility suppression, dynamic cluster formation, and trend-driven strategies in practical resource allocation scenarios (Zhang et al., 14 Sep 2025).

1. Theoretical Framework of DRLP-MG

DRLP-MG extends traditional Minority Game models by dividing agents into two distinct subpopulations:

• Q-subpopulation: Agents utilize Q-learning, updating state-action value tables based on received rewards and the temporal difference formulation.
• C-subpopulation: Agents employ classical Minority Game strategies, typically static lookup tables or strategy bundles chosen a priori, consistent with standard MG protocol.

Formally, the population fractions are $f_c$ and %%%%1%%%%. Resource allocation efficiency is quantified by the volatility

$$\psi = \frac{\sigma^2}{|\mathcal{N}|} = \frac{\sum_t (N_1(\tau) - C_1)^2}{|\mathcal{N}|(T-t_0)},$$

where $N_1(\tau)$ is the attendance at resource 1, and $C_1$ its capacity. The overall volatility for mixed populations is approximated by

$$\psi = f_c \psi_c + f_q \psi_q + 2r \sqrt{f_c \psi_c f_q \psi_q},$$

where $r$ is the Pearson correlation between the time series of resource choices in the two subpopulations. Synergy is realized when the cross-term $2r \sqrt{f_c \psi_c f_q \psi_q}$ is negative, induced by anti-correlation of action fluctuations.

2. Inter-Subpopulation Synergy

In mixed DRLP-MG populations, inter-subpopulation synergy manifests when fluctuations in resource choices by the Q-agents are countered by complementary fluctuations in the C-agents. This effect is generically robust across mixing ratios:

• Lower aggregate volatility: The total volatility $\psi$ is consistently lower than the individual subpopulation values ($\psi_c$, $\psi_q$), provided negative cross-correlation $r$ prevails.
• Phase transition behavior: As $f_c$ increases, a first-order transition at a critical fraction $f_c^*$ occurs, where internal cluster structure in the Q-subpopulation collapses and synchronization with the C-agents dominates.

This dynamic coordination mechanism ensures improved resource utilization over homogeneously composed populations.

3. Intra-Subpopulation Synergy: Cluster Formation in Q-Agents

A notable feature of the Q-subpopulation is its spontaneous organization into clusters via synchronization properties:

• Internal Synergy Clusters (IS-clusters, $\mathcal{C}_q^1$, $\mathcal{C}_q^2$): These clusters exhibit strong intra-synchronization (agents within a cluster consistently choose the same action) and inter-anti-synchronization (actions between clusters are opposites at each timestep). Synchronization is quantified by

$$\sigma_q^{i,j} = 1 - \frac{1}{T-t_0} \sum_t |a_i(\tau) - a_j(\tau)|,$$

which approaches 1 for near-perfect co-action.

• External Synergy Cluster (ES-cluster, $\mathcal{C}_q^3$): Agents in this cluster interact strongly with the C-subpopulation, serving as a dynamical bridge between the Q- and C-agents and facilitating inter-population synergy. As $f_c$ increases, the ES-cluster grows at the expense of the IS-clusters.

Cluster formation in Q-learning agents enables advanced forms of volatility suppression, by minimizing intra-cluster resource allocation fluctuations.

4. Emergence of Momentum Strategy and Trend Dynamics

Within the ES-cluster of the Q-subpopulation, the classical momentum strategy—the tendency to follow recent winning trends—emerges naturally:

• State-action preferences: Q-values for agents in states representing streaks (e.g., $s_0=000$, $s_7=111$) shift away from the diagonal in the $Q_{s,0}$-$Q_{s,1}$ plane: for example, $Q_{s,1} > Q_{s,0}$ in states with 1 as repeated winner.
• Resource preservation and trend reversal: Adoption of the momentum strategy reduces the risk of persistent under-utilization of a resource, but periodic over-exploitation results in sharp reversals and ultimately lower average rewards for trend followers compared to more stably synchronized clusters.

This self-organized exploitation of trend responses adds a layer of dynamic adaptation absent in classical MG settings.

5. The Frozen Effect and Volatility Suppression

A critical prerequisite for both intra- and inter-subpopulation synergy is the extent to which agents become “frozen,” i.e., locked into persistent action choices:

• Q-agents: Freezing arises when Q-value gaps widen, making action selection robust to noise and allowing clusters to anchor their choices over extended periods.
• C-agents: The frozen ratio $\phi$ measures how often a classical agent’s best strategy remains unchanged, with high freezing associated with low individual volatility.

However, a moderate fraction of unfrozen agents is required to facilitate inter-subpopulation coordination, especially near the phase transition $f_c^*$, suggesting that flexibility is essential to realizing the full synergy potential. This nuanced interplay directly influences the cross-term in the volatility formula above.

6. Mathematical Analysis and Phase Transition

The model enables quantitative analysis of synergy effects, cluster dynamics, and phase transitions:

• Synchronization-antisychronization metrics: K-means clustering of time series of actions, combined with the synchronization factor $\sigma_q^{i,j}$, clarify the internal organization and transition points between pure intra-synergy regimes and inter-synergy dominated regimes.
• Binder cumulant and critical point: The first-order nature of the phase transition at $f_c^*$ can be analyzed using standard statistical mechanics tools (e.g., Binder cumulant analysis), marking abrupt structural shifts in cluster composition.

These findings underscore the deep connection between collective resource allocation efficacy and the microscopic structure of agent interactions under dual policy regimes.

7. Implications for Reinforcement-Learning-Based Resource Allocation

Central results from DRLP-MG have direct consequences for theoretical and applied resource allocation:

• Heterogeneous learning rules: Coexisting Q-learning and static policy subpopulations drive diversification of strategic responses, counteracting lock-in and improving global resource use.
• Adaptive coordination: The synergy-enhancing effects of cluster formation and anti-correlated action fluctuations support robust resource management in dynamic or adversarial environments.
• Rediscovery of market strategies: Momentum and synchronization behaviors arise spontaneously, highlighting that reinforcement learning can recover and enrich classical strategies without explicit programming.

These mechanisms provide flexible tools for engineering collective intelligence and resilience in multi-agent competitive systems, with applications in economics, traffic management, and networked resource assignment.

---

This comprehensive exposition synthesizes current understanding of Dual Reinforcement Learning Policies in the Minority Game, consolidating quantitative models, empirical findings, and theoretical structures underpinning the synergy mechanisms and emergent cluster phenomena described in recent literature (Zhang et al., 14 Sep 2025).