Visual Feedback for Players of Multi-Level Capture the Flag Games: Field Usability Study (1912.10781v4)

Abstract: Capture the Flag games represent a popular method of cybersecurity training. Providing meaningful insight into the training progress is essential for increasing learning impact and supporting participants' motivation, especially in advanced hands-on courses. In this paper, we investigate how to provide valuable post-game feedback to players of serious cybersecurity games through interactive visualizations. In collaboration with domain experts, we formulated user requirements that cover three cognitive perspectives: gameplay overview, person-centric view, and comparative feedback. Based on these requirements, we designed two interactive visualizations that provide complementary views on game results. They combine a known clustering and time-based visual approaches to show game results in a way that is easy to decode for players. The purposefulness of our visual feedback was evaluated in a usability field study with attendees of the Summer School in Cyber Security. The evaluation confirmed the adequacy of the two visualizations for instant post-game feedback. Despite our initial expectations, there was no strong preference for neither of the visualizations in solving different tasks.

• The paper introduces two interactive visualizations, clustering and timeline, designed to provide immediate post-game feedback in cybersecurity Capture the Flag (CTF) games.
• A usability study with trainees found they valued personal insights and game overviews from the visualizations, though some tasks were challenging to complete.
• This work contributes to learning analytics in cybersecurity training and suggests future refinement and integration into real-time or longitudinal contexts.

Insights on Visual Feedback for Players of Multi-Level Capture the Flag Games

The paper "Visual Feedback for Players of Multi-Level Capture the Flag Games: Field Usability Study" presented at the IEEE Symposium on Visualization for Cyber Security focuses on enhancing learning outcomes in cybersecurity training through improved visual feedback mechanisms. As cybersecurity threats escalate, effective training is critical, and Capture the Flag (CTF) games have emerged as a popular method for fostering adversarial thinking skills amongst trainees. However, there's a recognized shortfall in providing timely and insightful feedback following these training exercises.

Core Contributions

The research introduces two interactive visualizations designed to offer immediate post-game feedback for CTF games. These visualizations aim to fulfill three core requirements:

1. Personalized Feedback - Enabling players to assess their own performance.
2. Comparative Feedback - Allowing players to benchmark their performance against peers.
3. Overview Feedback - Providing a general summary of the game difficulty and player performances.

By addressing these cognitive perspectives, the visualizations aim to enhance the learning impact and motivation of participants.

Methodology and Findings

The visualizations were developed through a multi-phase design process involving collaboration with domain experts. This included understanding the nature and data of multi-level cybersecurity games, defining visualization requirements in line with educational goals, iterative design, and a final usability evaluation.

Two visualization approaches were developed:

• Clustering Visualization: Utilizes a bar chart combined with a scatter plot to depict game scores and cluster players based on similar performances.
• Timeline Visualization: Offers a chronological view of player's score progress and level transitions, allowing for detailed exploration of event sequences.

An evaluation with trainees from a cybersecurity summer school confirmed the utility of these visualizations. Results indicated that:

• Trainees valued personal insights and game overviews more than direct comparisons with others.
• Some tasks were more challenging to complete, suggesting areas for potential visualization refinement.
• No overwhelming preference for one visualization type over the other was found, indicating that both play significant roles in different contexts.

Implications and Future Work

This research contributes to the field of learning analytics and visualization by addressing a niche area in cybersecurity training feedback. The development of these visualizations not only aids in immediate post-game analysis but also lays the groundwork for more immersive and personalized training environments. Future work should focus on refining these tools based on further feedback, exploring integration within real-time game environments, and extending them to other forms of serious games beyond cybersecurity.

Moreover, future advancements could include adaptive features that tailor feedback based on individual learning styles or expand beyond immediate feedback to provide longitudinal insights across multiple games. As the demand for skilled cybersecurity professionals continues to grow, the insights from this paper are valuable for institutions seeking to enhance their training efficacy through informed visual analytics.