Estimating Central, Peripheral, and Temporal Visual Contributions to Human Decision Making in Atari Games

Abstract: We study how different visual information sources contribute to human decision making in dynamic visual environments. Using Atari-HEAD, a large-scale Atari gameplay dataset with synchronized eye-tracking, we introduce a controlled ablation framework as a means to reverse-engineer the contribution of peripheral visual information, explicit gaze information in form of gaze maps, and past-state information from human behavior. We train action-prediction networks under six settings that selectively include or exclude these information sources. Across 20 games, peripheral information shows by far the strongest contribution, with median prediction-accuracy drops in the range of 35.27-43.90% when removed. Gaze information yields smaller drops of 2.11-2.76%, while past-state information shows a broader range of 1.52-15.51%, with the upper end likely more informative due to reduced peripheral-information leakage. To complement aggregate accuracies, we cluster states by true-action probabilities assigned by the different model configurations. This analysis identifies coarse behavioral regimes, including focus-dominated, periphery-dominated, and more contextual decision situations. These results suggest that human decision making in Atari depends strongly on information beyond the current focus of gaze, while the proposed framework provides a way to estimate such information-source contributions from behavior.

• The paper introduces a controlled ablation framework that isolates central, peripheral, and temporal visual information to predict human actions during dynamic Atari gameplay.
• The study demonstrates that peripheral vision is the dominant contributor, with its removal causing a 35–43% drop in performance compared to modest gains from gaze and historical data.
• The framework, validated across 20 games and individual subjects, offers actionable insights for developing more human-like imitation learning agents.

Estimating Central, Peripheral, and Temporal Visual Contributions to Human Decision Making in Atari Games

Introduction

This study introduces a principled ablation methodology to quantitatively estimate the contributions of central, peripheral, and temporal visual information to human action prediction in dynamic environments, specifically using the Atari-HEAD dataset with synchronized gaze tracking. While prior work has used attention and gaze for agent augmentation, this research focuses on decomposing and reverse-engineering the explicit contributions of these modalities to human decisions during gameplay. The framework bridges gaps in understanding covert (peripheral) versus overt (central/gaze) attention and assesses contextual and temporal dependencies within sequential visuomotor tasks.

Figure 1: The controlled ablation framework showing how human eye-tracking and gameplay are used to construct different information conditions for action prediction.

Controlled Ablation Framework

The paper formalizes a six-way ablation over three binary information sources: periphery (outside a $6^\circ$ gaze-centered region), explicit gaze (multi-scale gaze maps), and temporal history (past frames sampled at a stride of 15, covering ~2.25 seconds). Network variants are trained to predict actions using different combinations, allowing for the estimation of each source's marginal utility.

Figure 2: The network architecture is modular, supporting selective inclusion of periphery, gaze, and past-state information for action classification.

Models operate on $84 \times 84$ greyscale frames with gaze information encoded as multi-scale Gaussian-convolved maps. Temporal information is integrated via a learned gating mechanism, emphasizing practical discriminability rather than rote sequence modeling. The training protocol enforces cross-subject generalizability across 20 games, with all architectures and hyperparameters held fixed to ensure systematic ablation effects.

Quantitative Analysis of Information Contributions

Comprehensive evaluation of action prediction accuracies reveals peripheral vision as the dominant contributor to predictive fidelity. Median ablation effects, normalized to the common-choice baseline, are as follows: removal of periphery drops performance by 35.27--43.90\%, explicit gaze by 2.11--2.76\%, and past-state information by 1.52--15.51\%, with the upper end more reliable when peripheral leakage is controlled (i.e., periphery already removed).

Figure 3: Validation accuracy per configuration and normalized median performance drops, highlighting the dominant impact of periphery ablation.

While all model configurations outperform common-action baselines, the incremental gains by gaze and history are modest compared to the major decrement from periphery ablation, indicating humans rely substantially on information outside the foveated region or explicit gaze locus. Minor violations of monotonicity when adding information sources are attributed to fixed hyperparameters across games rather than methodological artifacts.

Behavioral Regime Clustering

A cluster analysis in the model-response space elucidates behavioral heterogeneity not captured by aggregate accuracies. States are grouped (via k-means, $k=5$) based on the probability assigned to the true human action by each model configuration. Clusters correspond to distinct decision regimes:

• Focus-dominated, highly reactive/foveal
• Periphery-dominated, reliant on global spatial context
• Contextual/complex, low-predictability
• Intent-driven, foveal-dependent but not peripheral
• Noisy/unpredictable

Figure 4: Cluster composition, characteristic true-action probabilities, and interpretive labeling across 20 games.

Cluster consistency across games is non-uniform, with certain games displaying strong separability aligned with dominant behavioral modes (e.g., simple versus complex action selection paradigms).

Figure 5: Silhouette analysis demonstrates moderate overall separability, with specific clusters (focus-dominated, periphery-dominated) showing higher cohesion.

t-SNE projections reinforce that learned behavioral regimes occupy relatively distinct, though not perfectly segregated, subspaces.

Figure 6: t-SNE visualization of the six-model response space reveals structure in behavioral clustering both globally and per game.

Qualitative analysis of high-silhouette examples clarifies the interpretability of each regime: periphery-dominated clusters, for instance, correspond to situations where action selection demands vigilance to objects outside the immediate gaze area or requires memory of prior world state.

Figure 7: Sample states from representative clusters, with the focus region delineated, demonstrating their typical visual-semantic context.

Subject-Specific Analyses

Single-subject model training for two games (DemonAttack, SpaceInvaders) yields action predictabilities generally consistent across subjects and with the all-subject model. Differential information dependence at the subject level is modest, with observed distinctions likely explained by behavioral idiosyncrasies or data recency rather than robust attention strategies.

Figure 8: Single-subject versus all-subject cluster composition and accuracy comparison suggest individual characteristics but no drastic regime shift.

Implications and Future Directions

This work establishes that peripheral visual information is the strongest determinant of human action prediction in dynamic Atari environments, with explicit gaze and short-term memory playing more nuanced, context-dependent roles. This strongly supports the theoretical stance that covert attention and peripheral scene analysis dominate over the traditionally emphasized overt (gaze/foveal) processes in human sequential decision-making in complex tasks.

Practical implications include the design of more human-like or effective imitation learning agents: reliance on explicit gaze alone is insufficient, and architectures for deep imitation and behavioral cloning should prioritize mechanisms for extracting and modeling distributed peripheral context. Methodologically, the ablation framework offers a tractable paradigm for disentangling multimodal contributions to behavior—a step beyond black-box attention-augmentation approaches.

Open challenges remain in fully isolating information channels (mitigating leakage between periphery and temporal context), extending single-subject analyses, and scaling to more granular action-specific behavioral clustering. Incorporating ablations that uniquely remove the central/foveal region could offer additional insight into direct competition between focus and periphery.

Conclusion

This study quantitatively demonstrates that peripheral vision predominantly enables human action prediction in Atari gameplay, surpassing both gaze-centered and short-term historical dependencies. The combination of model-driven ablation and behavioral regime clustering advances the interpretability of human visual decision mechanisms in high-dimensional, dynamic contexts. These findings both refine cognitive models of attention and offer concrete recommendations for designing more veridical, human-aligned agents in the sequential decision-learning landscape.