On the Validity of Traditional Vulnerability Scoring Systems for Adversarial Attacks against LLMs

Abstract: This research investigates the effectiveness of established vulnerability metrics, such as the Common Vulnerability Scoring System (CVSS), in evaluating attacks against LLMs, with a focus on Adversarial Attacks (AAs). The study explores the influence of both general and specific metric factors in determining vulnerability scores, providing new perspectives on potential enhancements to these metrics. This study adopts a quantitative approach, calculating and comparing the coefficient of variation of vulnerability scores across 56 adversarial attacks on LLMs. The attacks, sourced from various research papers, and obtained through online databases, were evaluated using multiple vulnerability metrics. Scores were determined by averaging the values assessed by three distinct LLMs. The results indicate that existing scoring-systems yield vulnerability scores with minimal variation across different attacks, suggesting that many of the metric factors are inadequate for assessing adversarial attacks on LLMs. This is particularly true for context-specific factors or those with predefined value sets, such as those in CVSS. These findings support the hypothesis that current vulnerability metrics, especially those with rigid values, are limited in evaluating AAs on LLMs, highlighting the need for the development of more flexible, generalized metrics tailored to such attacks. This research offers a fresh analysis of the effectiveness and applicability of established vulnerability metrics, particularly in the context of Adversarial Attacks on LLMs, both of which have gained significant attention in recent years. Through extensive testing and calculations, the study underscores the limitations of these metrics and opens up new avenues for improving and refining vulnerability assessment frameworks specifically tailored for LLMs.

• The paper critiques legacy vulnerability models, highlighting low score variability that undermines risk differentiation for LLM adversarial attacks.
• It rigorously analyzes metrics such as DREAD, CVSS, OWASP, and SSVC, demonstrating their insensitivity to nuanced attack impacts on LLMs.
• The study advocates for new LLM-specific metrics that integrate contextual, qualitative, and dynamic factors to better gauge adversarial risks.

Assessing Vulnerability Metrics for Adversarial Attacks on LLMs

In recent years, the ubiquity and capabilities of LLMs such as GPT, BERT, and others have positioned them as pivotal assets in artificial intelligence applications. However, their complex architectures and significant dependencies on vast data corpora have also rendered them susceptible to adversarial attacks targeting their robustness and reliability. This paper critically examines the suitability of existing vulnerability assessment metrics, namely DREAD, CVSS, OWASP Risk Rating, and SSVC, for appraising adversarial threats against LLMs. Through a meticulous evaluation of 56 diverse attacks, the research identifies key limitations in how these metrics gauge the nuanced risks presented by such attacks.

Analysis of Traditional Metrics

DREAD Model: Traditionally employed for qualitative risk assessment, DREAD is renowned for its five-dimensional evaluation: Damage, Reproducibility, Exploitability, Affected Users, and Discoverability. Despite its structured approach, the study reveals a low coefficient of variation (COV%) across most DREAD factors, indicating a limited ability to differentiate between attack severities. Specifically, the Damage, Discoverability, Exploitability, and Affected Users factors demonstrated marginal variability, suggesting their restricted utility in discerning the specific impacts of adversarial attacks on LLMs.

CVSS: The paper highlights that CVSS metrics offer limited variability in scoring adversarial attacks, particularly when evaluating Confidentiality, Integrity, and Availability (CIA) impacts. The qualitative nature of CVSS, with predefined value sets for each factor, further constrains its ability to capture the complex dynamics of LLM-targeted attacks. Factors such as Attack Vector and User Interaction show minimal entropy, pointing to their insensitivity in reflecting adversarial attack nuances.

OWASP Risk Rating: This metric is typically lauded for its expansive assessment capabilities, taking into account both technical and business impacts. However, the paper illustrates that while OWASP Risk Rating factors like Motivation, Opportunity, and Awareness provide somewhat broader insights, they still fall short in effectively distinguishing attacks within LLM contexts. The broader scoring range of OWASP introduces variability, yet the uniformity across attack classes limits its discriminative power.

SSVC: As more qualitative decision-tree based, SSVC seeks to prioritize vulnerability responses but exhibits limited entropy and variability in factors such as Exploitability and Automatable. This lack of differentiation underscores its inadequacies in capturing the emergent complexities and dynamic threat landscape faced by LLMs.

Redefining Metrics: Future Directions

The limitations identified in the study encapsulate the urgent need for novel, LLM-specific vulnerability assessment metrics that can accommodate the idiosyncrasies of adversarial attacks. Key recommendations include:

1. Context-Sensitive Factors: Developing metrics that account for the unique contextual and architectural traits of LLMs, including their decision-making processes and data dependencies.
2. Technical-Impact metrics beyond CIA: Introducing metrics that consider impacts such as model trust degradation, misinformation spread, and biased outcome generation.
3. Enhanced Qualitative Scoring: Increasing the granularity of qualitative assessments to enhance score variability and reduce subjectivity, thereby tailoring vulnerability scoring to reflect distinct threat landscapes.
4. Inclusivity of Success Rates and Learning Curves: Recognizing the evolving nature of adversarial threats through the incorporation of attack success rates and learning curves into assessments.

Conclusion

The paper underscores the inadequacies inherent in traditional vulnerability metrics when applied to the field of LLMs. By critiquing these established systems, the study sheds light on the path forward for developing robust, context-driven metrics that can accurately reflect the nuanced threats posed by adversarial attacks against one of the most transformative technologies of the modern era. These insights position the research community to advance the state of cybersecurity in AI, ensuring the reliable deployment and operation of LLMs across diverse applications and environments.