Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interface

Abstract: We surface a new threat to closed-weight LLMs that enables an attacker to compute optimization-based prompt injections. Specifically, we characterize how an attacker can leverage the loss-like information returned from the remote fine-tuning interface to guide the search for adversarial prompts. The fine-tuning interface is hosted by an LLM vendor and allows developers to fine-tune LLMs for their tasks, thus providing utility, but also exposes enough information for an attacker to compute adversarial prompts. Through an experimental analysis, we characterize the loss-like values returned by the Gemini fine-tuning API and demonstrate that they provide a useful signal for discrete optimization of adversarial prompts using a greedy search algorithm. Using the PurpleLlama prompt injection benchmark, we demonstrate attack success rates between 65% and 82% on Google's Gemini family of LLMs. These attacks exploit the classic utility-security tradeoff - the fine-tuning interface provides a useful feature for developers but also exposes the LLMs to powerful attacks.

• The paper identifies a novel attack vector that misuses fine-tuning APIs to generate adversarial prompts against closed-weight models.
• The paper demonstrates an optimization algorithm using a greedy search strategy that leverages loss signals to guide adversarial prompt crafting.
• The paper validates its approach on Google's Gemini models, exposing security vulnerabilities that call for robust mitigation strategies.

Overview of Optimization-Based Prompt Injections via Fine-Tuning Interfaces

The research under discussion elucidates a new vector of attack against closed-weight LLMs that leverages optimization-based prompt injection attacks. The crux of the study is the exploitation of fine-tuning APIs hosted by LLM vendors to compute adversarial prompts. The research meticulously investigates the utility-security trade-off inherent in these systems, demonstrating how the fine-tuning interface, intended to enhance utility by allowing model specialization, inadvertently exposes the models to adversarial manipulation.

Specifically, the paper outlines a methodology by which an attacker can (mis)use the feedback provided by the remote fine-tuning interface, particularly loss-like information, to generate adversarial prompts aimed at closed-weight models. This study formulates and implements an optimization algorithm that employs a greedy search strategy to successfully generate adversarial prompt injections. The results indicate alarming attack success rates, documented between 65% and 82% on Google's Gemini model lineup, utilizing the PurpleLlama prompt injection benchmark as a primary measure of effectiveness.

Core Contributions and Findings

The paper makes several important contributions to the field of AI security:

• Attack Surface Characterization: It identifies and characterizes a novel attack surface that takes advantage of the fine-tuning loss metric made available by LLM-supplying entities, thereby pinpointing a critical security-utility conflict.
• Loss Signal Utilization: By examining the loss-like signals returned from the Gemini fine-tuning API, the study establishes that these metrics can effectively guide discrete optimization attacks on closed-weight models. This empirical analysis determined that the training loss serves as a noisy, yet viable, proxy for guiding the adversarial optimization process.
• Experimental Validation: The research team put forth a comprehensive experimental evaluation of the Gemini model series, demonstrating that adversarial prompt crafting through their methodology is not only feasible but also notably efficient in terms of computational and financial resources.
• Permuted Loss Recovery: The study innovatively addresses the challenge posed by permuted loss reports in batch fine-tuning by proposing derandomization techniques capable of providing a usable signal for optimizing prompt injections.

Practical and Theoretical Implications

The implications of this work are multi-faceted:

• Practical Impact: This research uncovers significant vulnerabilities in current LLM deployment practices, especially concerning the fine-tuning interfaces offered alongside LLM products by companies like Google and OpenAI. It suggests that adversaries can exploit these interfaces, even without access to model internals, by merely leveraging the loss assessments provided during fine-tuning processes.
• Theoretical Expansion: The study enriches theoretical discussions surrounding the robustness and security of LLM-based systems, adding a new dimension to potential threat models against AI systems that rely heavily on proprietary model fine-tuning for domain-specific deployments.
• Mitigation Strategies: From a security architecture perspective, this research encourages a re-evaluation of current LLM-fine-tuning API frameworks, advocating for a balance between operational utility and model security. As an extension, the study highlights the importance of developing sophisticated security measures that do not compromise the model’s usability by restricting hyperparameter control or by implementing rigorous pre-moderation methods.

Prospective AI Developments

In future developments of AI, especially concerning the deployment and utilization of LLMs, this research prompts consideration of:

• Strengthening model security mechanisms against novel optimization-based adversarial prompt injection techniques.
• Innovating security paradigms that maintain functional model adaptability while safeguarding against misuse via exposed interfaces.
• Encouraging transparency in AI system design where the security implications of operational features are critically evaluated and addressed.

The study, therefore, serves as a critical reminder of the constant need to balance innovation with security, ensuring that AI systems remain both cutting-edge and safe from adversarial exploitation.