X-Ego: Acquiring Team-Level Tactical Situational Awareness via Cross-Egocentric Contrastive Video Representation Learning

Abstract: Human team tactics emerge from each player's individual perspective and their ability to anticipate, interpret, and adapt to teammates' intentions. While advances in video understanding have improved the modeling of team interactions in sports, most existing work relies on third-person broadcast views and overlooks the synchronous, egocentric nature of multi-agent learning. We introduce X-Ego-CS, a benchmark dataset consisting of 124 hours of gameplay footage from 45 professional-level matches of the popular e-sports game Counter-Strike 2, designed to facilitate research on multi-agent decision-making in complex 3D environments. X-Ego-CS provides cross-egocentric video streams that synchronously capture all players' first-person perspectives along with state-action trajectories. Building on this resource, we propose Cross-Ego Contrastive Learning (CECL), which aligns teammates' egocentric visual streams to foster team-level tactical situational awareness from an individual's perspective. We evaluate CECL on a teammate-opponent location prediction task, demonstrating its effectiveness in enhancing an agent's ability to infer both teammate and opponent positions from a single first-person view using state-of-the-art video encoders. Together, X-Ego-CS and CECL establish a foundation for cross-egocentric multi-agent benchmarking in esports. More broadly, our work positions gameplay understanding as a testbed for multi-agent modeling and tactical learning, with implications for spatiotemporal reasoning and human-AI teaming in both virtual and real-world domains. Code and dataset are available at https://github.com/HATS-ICT/x-ego.

• The paper introduces X-Ego, a novel method that fuses teammates' egocentric video streams via contrastive learning to create shared tactical awareness.
• It utilizes a vision transformer-based encoder and specialized projection heads to infer teammate and opponent positions under partial observability.
• Evaluation on the X-Ego-CS dataset shows improved accuracy at low POV counts, emphasizing the importance of precise loss tuning and effective embedding alignment.

X-Ego: Acquiring Team-Level Tactical Situational Awareness

The paper introduces X-Ego, a novel approach for understanding team-level tactics by exploiting egocentric video perspectives in multiplayer environments. The approach is centered around Cross-Ego Contrastive Learning (CECL), which is designed to synchronize visual streams from teammates to enhance collective situational awareness.

Introduction to Cross-Egocentric Contrastive Learning

In team-based activities such as e-sports, each player's perception and actions are largely influenced by their egocentric views. The paper addresses the challenge of synthesizing these individual perspectives into a coherent team-level understanding. The X-Ego-CS dataset, comprising hours of professional Counter-Strike 2 gameplay footage, is pivotal for this research. CECL aligns teammates’ egocentric video representations, fostering a shared tactical awareness from individual viewpoints that is crucial for effective team coordination.

Figure 1: Illustration of Cross-Ego Contrastive Learning (CECL), where teammates' egocentric video representations are aligned across teams.

Method and Model Architecture

The proposed method involves encoding agents' egocentric views into representation vectors, which are aligned through contrastive learning. This alignment encourages agents to infer team contexts from individual inputs. The model architecture integrates a vision transformer for video encoding, enhanced by a contrastive projection head and downstream task prediction heads. These components synergize to predict teammate and opponent locations from partial observability.

Figure 2: Illustration of the forward pass on a data batch containing three teams of three agents each, showcasing the encoding and alignment processes.

Evaluation and Performance

CECL's performance is evaluated on the new dataset with tasks like Teammate and Enemy Location Nowcast. CECL shows enhanced accuracy, especially with limited agent perspectives, confirming its effectiveness in developing situational awareness under partial observability. The results indicate substantial gains at lower POV counts, corroborating the robustness of cross-egocentric alignment.

Figure 3: Performance comparison of CECL vs. baseline across different visual encoders as agent POVs vary.

Ablation Studies

Ablation studies reveal that precise parameter tuning in the loss function is crucial for managing the imbalance of positive and negative pairs, which significantly affects learning efficacy. This tuning refines embedding alignment, enhancing predictive performance without needing full visibility, a critical trait for real-time applications.

Embedding Space Analysis

Visualizations of embeddings, both pre- and post-training, demonstrate significant improvements in clustering by location and team association, providing empirical evidence of CECL's impact on embedding structure. This analysis highlights improvements in representational accuracy and temporal coherence.

Figure 4: t-SNE visualization before and after contrastive learning, illustrating improvements in clustering and separation.

Implications and Future Directions

The implications of this research extend beyond gaming into domains such as robotics and collaborative AI systems, where understanding and predicting team behavior from partial datasets is crucial. Future work may focus on integrating reconstruction objectives alongside contrastive learning to enhance expressivity, exploring cross-team alignments for richer tactical modeling, and extending these methodologies to broader applications in real-world human-AI teaming scenarios.

Conclusion

The introduction of the X-Ego-CS dataset and CECL signifies a step forward in advancing multi-agent systems through cross-egocentric learning. These contributions lay the foundation for future exploration and application in complex, partially observable domains, fostering advancements in strategic AI coalescence and collaborative gaming analytics.