Agent Environment Cycle (AEC) Games Model

• The Agent Environment Cycle (AEC) Games Model is a formal framework defining multi-agent interactions with a sequential per-agent cycle, immediate reward attribution, and native support for turn-based and simultaneous environments.
• It addresses limitations of traditional models like POSGs and EFGs by eliminating race conditions and dummy no-ops, thereby providing clear, step-level feedback for reinforcement learning.
• Its modular implementation in MARL libraries such as PettingZoo and MAEnvs4VRP demonstrates its adaptability to various multi-agent environments and complex agent scheduling.

The Agent Environment Cycle (AEC) Games model is a formalism for multi-agent environments that addresses core conceptual and software-engineering limitations of classic frameworks such as Partially Observable Stochastic Games (POSGs) and Extensive Form Games (EFGs). The AEC model structures agent-environment interactions around a cycle where a single agent is selected to act at each timestep, observations and masking are computed, an action is sampled, and state and rewards are updated before proceeding to the next active agent. This approach, foundational in the PettingZoo reinforcement learning library, enables faithful and elegant representation of both synchronous and asynchronous multi-agent environments, including strictly turn-based games and settings with complex agent selection and observation rules (Terry et al., 2020, Gama et al., 2024).

1. Formal Definition and Cycle Semantics

An AEC game is defined as a tuple:

$$G = \langle I, S, (A_i)_{i\in I}, O_i, R_i, T, \nu \rangle$$

where:

• $I$ is the finite set of agents.
• $S$ is the state space.
• $A_i$ is the action space for agent $i \in I$.
• $O_i$ is the observation space for agent $i$; $O_i: S \times A_{<i} \to O_i$ can be defined to include dependence on prior actions in the cycle.
• $R_i: S \times A \to \mathbb{R}$ is the instantaneous reward function for agent $i$.
• $T: S \times A \to S$ is the transition function.
• $\nu: S \to I$ is the next-agent function specifying which agent updates next (Terry et al., 2020, Gama et al., 2024).

At each step, the environment selects the agent $i_t = \nu(s_t)$, generates its observation $o_t^{i_t}$ and valid actions, samples or receives an action $a_t^{i_t}$ (with all other agents effectively performing a “no-op”), and computes the reward vector and new state via $R_i$ and $T$. This loop continues until a termination predicate is met (e.g., all agents done). The model naturally encodes both simultaneous-move and strictly sequential environments by appropriate scheduling via $\nu$.

2. Comparison with POSG and EFG Models

The AEC model is designed to correct representational and implementation mismatches present in POSGs and EFGs:

• Reward Attribution: In POSGs, joint actions are submitted and the environment emits a joint reward vector, which can conflate reward sources, making debugging and attribution of outcomes more difficult. The AEC model ensures that rewards are assigned immediately after each agent’s action, separating intrinsic and extrinsic contributions and exposing step-level feedback aligned with reinforcement learning routines.
• Race Condition Elimination: Simultaneous-move APIs modeled after POSGs can introduce race conditions in software, since most environments resolve actions sequentially. The AEC enforces a strict per-agent cycle, with explicit ordering via $\nu$, eliminating hidden nondeterminism and making environment logic and learning traceable.
• Turn-Based Games Support: Unlike POSGs, which require dummy “no-op” actions for inactive agents, the AEC model natively supports turn-based games such as chess, alternately selecting agents and advancing the environment without such padding.
• Intermediary Rewards: EFGs typically only assign rewards at terminal states. AEC allows per-step rewards at every agent-environment interaction, which maps onto reward structures required in temporal-difference RL (Terry et al., 2020).

A summary of distinctions appears below:

Feature	AEC	POSG	EFG
Agent Updates	Sequential via $\nu$	All at once	Turn-tree order
Rewards	At every step	Every joint step	Only at leaves
Dummy no-ops	Not required	Required	Not applicable
Partial observability	$O_i(s, \text{history})$	$O_i(s)$	Information sets

3. Update Equations and Algorithmic Cycle

The per-step update sequence for an AEC game, as formalized in multi-agent vehicle routing environments (Gama et al., 2024), is:

\begin{align*} & i_t = \Xi(s_t), \ & (o_t^{\,i_t},\,M_t^{\,i_t}) = \mathcal{O}{i_t}(s_t), \quad M_t^{\,i_t} \subseteq A{i_t}(s_t), \ & a_t \sim \pi_{i_t}(\cdot \mid o_t^{\,i_t},\,M_t^{\,i_t}), \quad a_t \in M_t^{\,i_t}, \ & s_{t+1} = T(s_t,\,i_t,\,a_t), \ & (r_t,\,\mathrm{penalty}t) = R(s_t,\,i_t,\,a_t,\,s{t+1}), \ & \text{Terminate if }\rho(s_{t+1}) = 0. \end{align*}

This abstraction permits complex agent selection (e.g., by minimal time index, random scheduling, or round-robin) and structured observation/action masking.

4. Implementation in Multi-Agent Environments

The AEC abstraction directly influences the modular design of contemporary MARL libraries. In MAEnvs4VRP, an AEC-compliant environment is decomposed into (Gama et al., 2024):

• InstanceGenerator: Generates problems (e.g., locations, demands).
• Observations: Composes agent observations from modular sub-blocks (static/dynamic node features, agent-centric features, other agents, global statistics).
• AgentSelector: Determines next agent based on application-specific criteria (e.g., SmallestTimeAgentSelector for VRP).
• RewardEvaluator: Computes dense or sparse rewards and auxiliary penalty components decoupled from the main objective.
• Environment Wrapper: Hosts interaction logic, exposing reset, step, action sampling, and episode metrics in strict accordance with the AEC framework.

A canonical code loop illustrating one-agent-at-a-time operation appears in the PettingZoo and MAEnvs4VRP APIs:

env_state = env.reset() while not env_state["done"].all(): action = policy(env_state["observation"], env_state["action_mask"]) env_state.set("action", action) env_state = env.step(env_state) info = env.stats_report(env_state)

(Gama et al., 2024, Terry et al., 2020)

5. Expressivity, Reduction, and Conceptual Significance

The AEC model is as expressive as POSG and EFG models; mutual reducibility is established, enabling translation of any environment between these formalisms without loss of expressive power (Terry et al., 2020). Unlike POSGs and EFGs, the AEC model combines fine-grained, per-agent transitions, immediate reward/observation delivery, and an order-resolving environment actor—closely reflecting modern RL software implementations.

The conceptual significance of the AEC is in providing a clean and software-compatible abstraction for MARL: it avoids the bug-prone padding and latent tie-breaking of POSGs, aligns closely with RL policy training loops, and allows for trivial and dynamic addition/removal of agents during an episode. Intermediate and final rewards, masking, partial observability, and agent-centric history are natively handled within the cycle (Terry et al., 2020, Gama et al., 2024).

6. Extensions and Applications

AEC’s flexibility enables straightforward adaptation to a wide variety of problem domains, including asynchrony (custom agent selection), dense and sparse reward structures, explicit penalty breakdowns, and integration with operations research benchmarks (as in CVRPTW, PDPTW, and related routing formulations (Gama et al., 2024)). High modularity results from the clean compartmentalization of instance generation, observation structure, and reward evaluation. The architecture scales directly to parallelized settings via wrappers that batch actions, supporting both “sequential” and “parallel” MARL algorithm implementations (Terry et al., 2020).

7. Adoption in Open-Source MARL Libraries

The AEC model underlies the design of the PettingZoo environment suite (Terry et al., 2020), which provides an API mirroring AEC semantics and exposes a vast array of environments—both classic and custom—promoting rapid, reproducible, and error-minimized research in MARL. New libraries such as MAEnvs4VRP adopt the AEC framework and extend it to applied optimization domains, providing compatibility across RL and operations research communities and enabling benchmarking with established datasets and methods (Gama et al., 2024).