SEAR: A Multimodal Dataset for Analyzing AR-LLM-Driven Social Engineering Behaviors (2505.24458v1)

Abstract: The SEAR Dataset is a novel multimodal resource designed to study the emerging threat of social engineering (SE) attacks orchestrated through augmented reality (AR) and multimodal LLMs. This dataset captures 180 annotated conversations across 60 participants in simulated adversarial scenarios, including meetings, classes and networking events. It comprises synchronized AR-captured visual/audio cues (e.g., facial expressions, vocal tones), environmental context, and curated social media profiles, alongside subjective metrics such as trust ratings and susceptibility assessments. Key findings reveal SEAR's alarming efficacy in eliciting compliance (e.g., 93.3% phishing link clicks, 85% call acceptance) and hijacking trust (76.7% post-interaction trust surge). The dataset supports research in detecting AR-driven SE attacks, designing defensive frameworks, and understanding multimodal adversarial manipulation. Rigorous ethical safeguards, including anonymization and IRB compliance, ensure responsible use. The SEAR dataset is available at https://github.com/INSLabCN/SEAR-Dataset.

Overview of "SEAR: A Multimodal Dataset for Analyzing AR-LLM-Driven Social Engineering Behaviors"

The paper introduces SEAR, an innovative multimodal dataset designed to explore the intricacies and potential threats posed by social engineering (SE) attacks facilitated by augmented reality (AR) and multimodal LLMs. This dataset is particularly significant as it fills a noticeable void in existing research by providing a comprehensive collection of data that embodies the multimodal dynamics in SE scenarios driven by AR and LLMs.

The SEAR dataset is remarkable for its collection of 180 annotated conversations derived from 60 participants, strategically configured to mimic real-world adversarial contexts such as meetings, classes, and networking events. Crucially, these interactions comprise synchronized AR-captured audio-visual cues, environmental settings, and curated social media profiles, alongside subjective participant assessments like trust ratings and susceptibility metrics.

Content and Contributions

SEAR distinguishes itself through the incorporation of various essential components that serve as a robust framework for understanding AR-LLM-driven social engineering:

1. Empirical Demonstration: The dataset highlights the enhanced efficacy of AR-LLM systems in social engineering by illustrating their capacity to elicit compliance and manipulate trust dynamically, evidenced by high rates of phishing link engagement and trust increment post-interaction.
2. Comprehensive Multimodal Data: The dataset captures nuanced information involving AR video/audio recordings, personal information extrapolated from social media, LLM-generated social profiles, interactive conversation data, and post-interaction participant surveys.
3. Ethical Compliance: The dataset emphasizes ethical guidelines by anonymizing data and adhering to IRB standards, ensuring secure and responsible usage for future research undertakings.

Practical and Theoretical Implications

The SEAR dataset fosters numerous avenues for both practical application and theoretical exploration. Practically, it paves the way for research dedicated to developing robust mechanisms for detecting AR-LLM-driven SE attacks and crafting defensive countermeasures. Theoretically, SEAR provides a benchmark dataset that supports the analysis of adversarial behaviors in SE contexts, allowing for in-depth studies of multimodal communication and manipulation patterns.

Future Speculations

The release of the SEAR dataset anticipates fostering advancements in several domains, notably in AR technology and LLM applications. It is pragmatically positioned to drive research in enhancing AI's capability to recognize and mitigate social engineering threats by utilizing multimodal cue integration. Moving forward, one can speculate that SEAR will stimulate the development of more advanced multi-layered security frameworks and contribute to the evolution of ethical standards in AR-LLM research.

In conclusion, the SEAR dataset is a pivotal resource that enriches the field's understanding of AR-LLM-driven social engineering. By providing an in-depth, ethically sound approach, it not only elucidates the potential vulnerabilities inherent in these technologies but also lays the groundwork for more secure and resilient applications in human-computer interaction.