Norm-Augmented Markov Games

• Norm-Augmented Markov Games are a formal framework that integrates traditional multi-agent dynamics with explicit normative systems for modeling and enforcing social rules.
• The framework applies Bayesian inference and norm-augmented rewards to enable agents to infer governing norms and adjust strategies, promoting coordinated behavior.
• Empirical results demonstrate agents converging on normative consensus within 150–300 steps, leading to improved social welfare and sustainable system dynamics.

A Norm-Augmented Markov Game (NMG) is a formal framework extending classical Markov Games by explicitly representing, learning, and enforcing normative systems—sets of prohibitive and obligative rules—alongside the conventional dynamics and reward structures. This unifies algorithmic approaches to multi-agent strategic interaction with mechanisms for the emergence, inference, and collective enforcement of social and institutional norms. In NMGs, agents form beliefs about which norms govern joint behavior, update these beliefs through Bayesian inference from observed actions, and plan using norm-penalized rewards, enabling the modeling and engineering of artificial societies with shared normative structures (Oldenburg et al., 2024).

1. Formal Definition and Structure

An NMG is formally specified by

$$\mathcal{M} = \bigl\langle \mathcal{I},\,\mathcal{S},\,\{\mathcal{A}_i\}_{i \in \mathcal{I}},\,T,\,\{R_i\}_{i \in \mathcal{I}},\,\gamma,\,\mathcal{N},\,\{\mathcal{P}^0_i\}_{i \in \mathcal{I}},\,\{C_i\}_{i \in \mathcal{I}}\bigr\rangle,$$

where:

• $\mathcal{I}$ denotes the finite agent set.
• $\mathcal{S}$ is the finite state space.
• $\mathcal{A}_i$ are finite action sets; $\mathcal{A} = \prod_i \mathcal{A}_i$ is the joint action space.
• $$T: \mathcal{S} \times \mathcal{A} \to \Delta(\mathcal{S})$$ is the transition kernel.
• $$R_i: \mathcal{S} \times \mathcal{A}_i \times \mathcal{S} \to \mathbb{R}$$ denotes material rewards.
• $\gamma \in [0,1)$ is the discount factor.
• $\mathcal{N}$ is the finite space of norms $\nu$ (each $\mathcal{I}$0 mapping histories to violation flags).
• $\mathcal{I}$1 is agent $\mathcal{I}$2's prior over sets of norms in force.
• $\mathcal{I}$3 assigns cost to norm violations.

A normative system $\mathcal{I}$4 is stable if, given common belief in $\mathcal{I}$5, rational agents find it optimal to comply with all $\mathcal{I}$6 with high frequency (Oldenburg et al., 2024).

2. Norm Types and Formalization

Norms in NMGs adopt first-order, rule-based templates and fall into two primary categories:

• Prohibitive norms (maintenance): $\mathcal{I}$7, where $\mathcal{I}$8 denotes a set of forbidden actions and $\mathcal{I}$9 is the predicate over successor states. Violation: $\mathcal{S}$0 if $\mathcal{S}$1 and $\mathcal{S}$2.
• Obligative norms (achievement): $\mathcal{S}$3 with trigger $\mathcal{S}$4, discharge condition $\mathcal{S}$5, and deadline $\mathcal{S}$6. Violation occurs if, for a $\mathcal{S}$7, $\mathcal{S}$8 is satisfied at time $\mathcal{S}$9 and $\mathcal{A}_i$0 is not achieved by $\mathcal{A}_i$1.

Typical examples include resource conservation restrictions (“do not pick apples if local density is low”) and role-conditioned obligations (“if a cleaner observes river dirtiness $\mathcal{A}_i$2, must clean within 20 steps”) (Oldenburg et al., 2024).

3. Learning Norms via Bayesian Rule Induction

Agents in NMGs posit a latent set of governing norms $\mathcal{A}_i$3 and maintain a posterior

$\mathcal{A}_i$4

where $\mathcal{A}_i$5 is a factored prior, and $\mathcal{A}_i$6 quantifies the likelihood of observed joint behavior given universal compliance with $\mathcal{A}_i$7. Due to the exponential scale in $\mathcal{A}_i$8, a mean-field approximation is adopted: Each agent $\mathcal{A}_i$9 maintains marginals $\mathcal{A} = \prod_i \mathcal{A}_i$0 for candidate norms, updated as

$\mathcal{A} = \prod_i \mathcal{A}_i$1

where $\mathcal{A} = \prod_i \mathcal{A}_i$2 is the policy-likelihood of observed non-$\mathcal{A} = \prod_i \mathcal{A}_i$3 actions assuming $\mathcal{A} = \prod_i \mathcal{A}_i$4 is enforced. In practice, this policy-likelihood uses the $\mathcal{A} = \prod_i \mathcal{A}_i$5 action-value from model-based planning with norm-augmented reward (Oldenburg et al., 2024).

4. Norm Integration into Planning and Control

At each timestep, agent $\mathcal{A} = \prod_i \mathcal{A}_i$6 forms beliefs $\mathcal{A} = \prod_i \mathcal{A}_i$7 and plans with a norm-augmented instantaneous reward,

$\mathcal{A} = \prod_i \mathcal{A}_i$8

For prohibitions, standard Real-Time Dynamic Programming (RTDP) is used. Obligations, when triggered, cause a switch to obligation-oriented planning with a modified reward penalizing lapsed obligations and rewarding successful discharge before deadline (Oldenburg et al., 2024). At execution, norm compliance can be implemented by deterministic thresholding on belief, or by sampled norm-sets (Thompson sampling) for implicit coordination.

5. Empirical Results and Evaluative Metrics

Experimental realization of NMGs was conducted in gridworld domains combining resource management, prosocial labor (cleaning), and role-based payments (Oldenburg et al., 2024). Agents, with roles (e.g., Farmer, Cleaner), operated in environments with 68 enumerated candidate norms.

Key results include:

• Passive norm learning: Single agents can infer ground-truth active norms within 150–300 timesteps, achieving $\mathcal{A} = \prod_i \mathcal{A}_i$980% recall/precision when using a 0.95 posterior threshold.
• Norm-enabled welfare: Activation of shared norm-sets leads to increased social welfare and sustainability (orchard health), while absence leads to welfare stagnation and >90% environmental degradation.
• Intergenerational transmission: Beliefs over prohibitive norms are reliably transmitted for agent lifespans $$T: \mathcal{S} \times \mathcal{A} \to \Delta(\mathcal{S})$$0; obligative norms require longer, $$T: \mathcal{S} \times \mathcal{A} \to \Delta(\mathcal{S})$$1.
• Norm emergence/convergence: Multiple naive agents converge beliefs on salient norms (posterior $$T: \mathcal{S} \times \mathcal{A} \to \Delta(\mathcal{S})$$295%) within 300 steps, typically achieving unanimous consensus.

Metrics included posterior belief accuracy (precision/recall), mean team reward, sustainability ratios (e.g., fraction of desiccated orchards), intergenerational stability, and consensus convergence rates (Oldenburg et al., 2024).

6. Connections to Norm-Augmented Optimization in Markov Games

An orthogonal approach termed “norm-augmentation” appears in the equilibrium computation literature for general-sum MGs. Here, “norm” refers not to rules governing behavior, but to explicit $$T: \mathcal{S} \times \mathcal{A} \to \Delta(\mathcal{S})$$3 constraints on Bellman-like residuals (Pérolat et al., 2016).

For a $$T: \mathcal{S} \times \mathcal{A} \to \Delta(\mathcal{S})$$4-discounted $$T: \mathcal{S} \times \mathcal{A} \to \Delta(\mathcal{S})$$5-player general-sum MG, an $$T: \mathcal{S} \times \mathcal{A} \to \Delta(\mathcal{S})$$6-Nash equilibrium is defined by bounding the $$T: \mathcal{S} \times \mathcal{A} \to \Delta(\mathcal{S})$$7 distance between player values under the joint strategy and their best response values: $$T: \mathcal{S} \times \mathcal{A} \to \Delta(\mathcal{S})$$8 The optimization minimizes the $$T: \mathcal{S} \times \mathcal{A} \to \Delta(\mathcal{S})$$9 norm of two Bellman residuals per player, resulting in theoretical guarantees on equilibrium quality. Practically, this is implemented in the NashNetwork, which jointly trains value and policy networks to minimize Lp-aggregated residuals over batch data (Pérolat et al., 2016).

7. Limitations and Future Directions

NMGs, as formulated, currently assume:

• Batch setting and access to a fixed transition kernel, limiting stochastic applicability unless remedied with model-estimation in RKHS or double-sampling.
• Non-convex objectives: Neural network optimization is not globally guaranteed; only stationary points guarantee weak-$$R_i: \mathcal{S} \times \mathcal{A}_i \times \mathcal{S} \to \mathbb{R}$$0 Nash bounds in the case of norm-augmented Bellman residuals.
• Single equilibrium discovery: The NashNetwork approach finds one equilibrium, despite multiplicity (Pérolat et al., 2016).

Prospective directions include scaling to richer state representations, exploring optimization techniques tailored to Bellman-residual objectives, incorporating explicit kernel-based model estimation to handle stochastic transitions, and extending normative inference to more complex relational and simultaneous-move environments (Oldenburg et al., 2024, Pérolat et al., 2016).

The NMG formalism provides a principled substrate for the emergence, inference, and enforcement of social norms in artificial societies, while norm-augmented optimization offers robust methods for learning equilibria in complex multi-agent settings.