Logits of API-Protected LLMs Leak Proprietary Information (2403.09539v3)

Abstract: LLM providers often hide the architectural details and parameters of their proprietary models by restricting public access to a limited API. In this work we show that, with only a conservative assumption about the model architecture, it is possible to learn a surprisingly large amount of non-public information about an API-protected LLM from a relatively small number of API queries (e.g., costing under $1000 USD for OpenAI's gpt-3.5-turbo). Our findings are centered on one key observation: most modern LLMs suffer from a softmax bottleneck, which restricts the model outputs to a linear subspace of the full output space. We exploit this fact to unlock several capabilities, including (but not limited to) obtaining cheap full-vocabulary outputs, auditing for specific types of model updates, identifying the source LLM given a single full LLM output, and even efficiently discovering the LLM's hidden size. Our empirical investigations show the effectiveness of our methods, which allow us to estimate the embedding size of OpenAI's gpt-3.5-turbo to be about 4096. Lastly, we discuss ways that LLM providers can guard against these attacks, as well as how these capabilities can be viewed as a feature (rather than a bug) by allowing for greater transparency and accountability.

• The paper introduces a method that exploits the softmax bottleneck to extract proprietary model characteristics from API-protected LLMs with minimal queries.
• It empirically estimates key parameters, such as OpenAI's gpt-3.5-turbo embedding size (~4,096), unveiling a unique model signature.
• The study highlights critical security risks and urges providers to enhance API safeguards while promoting transparency for LLM users.

Unraveling the Secrets of API-Protected LLMs through Their Softmax Bottleneck

Introduction to the Softmax Bottleneck Phenomenon

Recent advancements have propelled the development and commercialization of LLMs, rendering them accessible primarily through high-level application programming interfaces (APIs). This paper presents a novel approach for unveiling considerable amounts of non-public information about an API-protected LLM by exploiting the inherent characteristics of modern LLMs, specifically the limitations imposed by the softmax bottleneck. By analyzing the linear subspace constraints of model outputs, this methodology reveals a model's image or signature, enabling various applications ranging from identifying the LLM’s hidden size to detecting model updates and estimating output layer parameters with significant cost efficiency.

Theoretical Underpinnings and Empirical Validation

• Theoretical Framework: At the core of our analysis is the observation that the output layer of LLMs, constrained by the softmax bottleneck, projects model outputs onto a lower-dimensional subspace. Particularly, this paper demonstrates that given a set of limited API queries, it is feasible to intercept the dimensions of this subspace, thereby gaining insights into certain architectural aspects of the model hidden behind the API.
• Empirical Results: The empirical analysis underscores the method's efficacy, where we estimated the embedding size of OpenAI's gpt-3.5-turbo to be approximately 4,096. This insight was garnered from a remarkably small number of queries, thereby asserting the practical applicability of our approach. Moreover, the research unveils that the image of the LLM can serve as a unique identifier, offering a novel way to distinguish between models or determine updates with a high degree of accuracy.

Practical Implications and Future Prospects

• LLM Users and Providers: Our findings bridge a critical gap between LLM providers and users, suggesting that users can leverage this method for greater accountability and transparency from providers. Conversely, it alerts providers about potential vulnerabilities, guiding them towards more secure deployment practices.
• Security and Privacy: Highlighting the potential risks associated with the exposure of proprietary model details, the paper prompts a reevaluation of current API security measures. It proposes measures to mitigate these vulnerabilities without compromising the utility of the API features.
• Future Research Directions: While the immediate applications of uncovering an LLM's image are profoundly insightful, the paper also lays the groundwork for future explorations. It opens avenues for more in-depth studies on model identification, update detection, and potentially developing more robust, privacy-preserving LLM architectures.

Conclusion

In conclusion, the investigation into the softmax bottleneck's impact on LLM outputs offers a compelling perspective on decoding proprietary information from API-protected models. Despite its potential to influence the current dynamics between LLM users and providers, it fundamentally calls for a balanced approach in utilizing this technique—a way to ensure accountability and transparency while safeguarding the proprietary interests of LLM providers. As the field of generative AI continually evolves, the discourse around ethical considerations, security, and privacy will undoubtedly benefit from the insights and methodologies proposed in this paper.