Othello AI Arena

• Othello AI Arena is an open-source reinforcement learning platform featuring richly instrumented, high-complexity retro game environments.
• The platform offers a Gym-compliant API with extensive customization of observations and actions, supporting both single- and multi-agent experiments.
• It enables advanced research modalities such as self-play, human-in-the-loop training, and transfer learning for robust RL benchmarking.

DIAMBRA Arena is an open-source software platform designed to advance reinforcement learning (RL) research by providing a suite of richly instrumented, high-complexity environments. Developed to address the limitations of widely used, often already “solved” RL benchmarks, DIAMBRA Arena integrates retro 2D fighting games into a Python API compliant with the OpenAI Gym interface, incorporating single- and multi-agent modes, support for human-agent and human-in-the-loop learning, and extensive customization for observations and actions. The platform's architecture, built-in wrappers, and diverse environment features facilitate rigorous investigation of cutting-edge RL paradigms such as self-play curricula, imitation learning, and transfer/generalization across tasks (Palmas, 2022).

1. Design Philosophy and Objectives

DIAMBRA Arena’s central objective is to provide challenging, reproducible environments that accommodate the current frontiers of RL research:

• Novelty and Challenge: By deploying tasks not “solved” by current RL agents, DIAMBRA Arena creates meaningful benchmarks for policy learning, representation learning, and generalization.
• Inclusivity: The platform is engineered for modest hardware requirements, enabling experiments on commodity CPUs and single-GPU systems.
• OpenAI Gym Compliance: The Python API adheres strictly to the Gym model (make(), reset(), step(), render(), close()), minimizing friction for integration with existing RL pipelines.
• Accessibility for Multi-Agent and Human Experimentation: Out-of-the-box support is provided for competitive and cooperative multi-agent RL, human-agent interaction, and curriculum learning scenarios.

The emphasis on customization and modular wrappers reflects its intent to enable rapid prototyping and robust evaluation in both single-agent and interactive settings.

2. Environment Suite and Structural Properties

The inaugural release of DIAMBRA Arena includes several 2D retro fighting games (e.g., Dead Or Alive ++, Street Fighter III, Tekken Tag Tournament), each featuring:

• Visual and Structural Complexity: Resolutions up to 480×512×3, with configurable downsampling (minimum 128×128 pixels, RGB or grayscale).
• Configurable Episodic Structure: Single-player (1P) mode—episodes span multiple stages until the player’s run ends; two-player (2P) mode—episode ends per stage, supporting zero-sum competitive engagement.
• Multi-Faceted State Space: Each environment presents varied sets of characters, outfits, stages, and difficulty settings.

Within each episode, the fight structure (number of stages $N_s$, rounds to win $N_r$) can be adjusted, allowing precise control over episode duration and complexity (Palmas, 2022).

3. API, Observations, and Action Spaces

DIAMBRA Arena environments conform to the Gym interface, facilitating integration and extension. The API and data structures are as follows:

• Initialization and Control:

$(H, W, C)$8

• Observation Space: By default, observations are a gym.spaces.Dict containing:
  • "frame": raw pixel buffer, shape $(H, W, C)$.
  • "ram": a vector of numerical values (e.g., health bars, stages, recent actions) chosen so that no privileged information is revealed beyond the pixel buffer.
  • Hardcore mode disables "ram", exposing only the pixel buffer.
• Action Spaces:
  • Discrete (simple union of movement/attack commands)
  • Discrete + combos (compound actions)
  • MultiDiscrete (separate move/attack subspaces)
  • MultiDiscrete + combos
  • In 2P mode: gym.spaces.Dict({ "P1": ..., "P2": ... })

These structures enable flexible experimentation in agent design and observation-action abstraction (Palmas, 2022).

4. Supported Research Modalities and Wrappers

DIAMBRA Arena is designed to support advanced RL experimental paradigms:

Modality	Supported Features	Human Interaction
Single-player RL	Maximize cumulative reward	N/A
Two-player competitive	Agent vs. agent/human	Human joins via gamepad
Self-play	Continual agent curriculum	N/A
Human-in-the-loop training	Real-time feedback/policies	Manual or evaluative input
Imitation learning	Record/load human trajectories	Replay via provided wrappers

Custom wrappers support frame-warping, stacking, reward normalization, and trajectory management for expert data collection and consumption (Palmas, 2022).

5. Reward Formulation and Mathematical Structure

Reward functions in DIAMBRA Arena are grounded in explicit changes to health bars, structured for interpretability and compatibility with standard RL algorithms. For each character $i$:

$$R_t = \sum_{i=1}^{N_c} \left[ (\bar{H}_i^{t^-} - \bar{H}_i^t) - (\hat{H}_i^{t^-} - \hat{H}_i^t) \right]$$

• $\bar{H}_i$: opponent’s health
• $\hat{H}_i$: agent’s health
• $t^-, t$: just before/after the step

Cumulative episode reward bounds are provided:

$$\min \sum_{t=0}^{T_s} R_t = -N_c \left((N_s-1)(N_r-1)+N_r\right)\Delta H, \quad \max \sum_{t=0}^{T_s} R_t = N_c N_s N_r \Delta H$$

where $\Delta H = H_{max} - H_{min}$, $N_r$0: stages, $N_r$1: rounds, $N_r$2: characters per player (Palmas, 2022).

For policy optimization, the standard PPO surrogate objective is applied:

$N_r$3

with $N_r$4 and $N_r$5 the advantage estimate.

6. Empirical Validation and Example Configurations

Demonstrations on Dead Or Alive ++ validate DIAMBRA Arena’s suitability for advanced RL experiments:

• Experimental Configuration:
  • Randomized player side (P1/P2), action frequency 10 Hz (step ratio=6)
  • Observation: $N_r$6 grayscale frames, RAM vector
  • Action space: Discrete (12), no combos, no reward clipping
  • Frame stacking (4), action stacking (12), reward normalization (scaled by $N_r$7, $N_r$8)
  • Neural architecture: CNN encoder + FC RAM encoder, concatenated latent (320-dim), policy/value heads
  • PPO: 16 parallel envs, batch size 256, $N_r$9 epochs/update, $(H, W, C)$0, learning rate $(H, W, C)$1, clip parameter $(H, W, C)$2
• Hardware Utilization:
  • Intel i5 + GTX 1050: $(H, W, C)$3M env-steps/day; AMD Ryzen 9 + GTX 1080 Ti: $(H, W, C)$4M env-steps/day
• Learning Trajectory:
  • Random agent average reward $(H, W, C)$5; PPO agent achieves $(H, W, C)$6 after 25M steps (theoretical bounds $(H, W, C)$7)
  • Qualitative emulation of human strategies (timing, defense, counters)

Similar learning outcomes have been reproduced on other environments, such as Street Fighter III and Tekken Tag Tournament, providing further evidence of generality (Palmas, 2022).

7. Extensions, Integration, and Research Applications

DIAMBRA Arena's design, wrappers, and modularity facilitate a spectrum of challenging RL studies:

• Self-Play and Curriculum: Agents may train continuously against past policies, enabling investigation into adaptive competition and policy robustness.
• Human-in-the-Loop RL: Pause/resume and input override wrappers allow for real-time human feedback, demonstration, or evaluative control.
• Imitation Learning: Human trajectories are recorded directly from interaction and replayed through Gym-compatible interfaces for behavioral cloning or guided learning.
• Transfer and Generalization: Policies may be transferred across game titles, characters, and difficulty settings, offering platforms for benchmarking few-shot and domain generalization.
• Anticipated Developments: Forthcoming cooperative multi-agent scenarios are planned; current focus is on competitive environments.

The built-in evaluation, visualization, and straightforward code interface ensure DIAMBRA Arena is immediately usable for benchmarking both RL and hybrid RL/human algorithms (Palmas, 2022).

---

For further details and source code snippets, refer directly to "DIAMBRA Arena: a New Reinforcement Learning Platform for Research and Experimentation" (Palmas, 2022).