Analyzing Adversarial Attacks on Sequence-to-Sequence Relevance Models (2403.07654v1)

Abstract: Modern sequence-to-sequence relevance models like monoT5 can effectively capture complex textual interactions between queries and documents through cross-encoding. However, the use of natural language tokens in prompts, such as Query, Document, and Relevant for monoT5, opens an attack vector for malicious documents to manipulate their relevance score through prompt injection, e.g., by adding target words such as true. Since such possibilities have not yet been considered in retrieval evaluation, we analyze the impact of query-independent prompt injection via manually constructed templates and LLM-based rewriting of documents on several existing relevance models. Our experiments on the TREC Deep Learning track show that adversarial documents can easily manipulate different sequence-to-sequence relevance models, while BM25 (as a typical lexical model) is not affected. Remarkably, the attacks also affect encoder-only relevance models (which do not rely on natural language prompt tokens), albeit to a lesser extent.

• The paper demonstrates that simple, query-independent adversarial attacks can effectively manipulate sequence-to-sequence relevance models’ rankings.
• It employs preemption, stuffing, and rewriting strategies on monoT5 using TREC Deep Learning track datasets to rigorously test vulnerabilities.
• The results underscore potential risks to search quality and highlight the need for robust adversarial defenses in neural information retrieval systems.

Analyzing Adversarial Attacks on Sequence-to-Sequence Relevance Models

Introduction

In the current paper, an evaluation of the vulnerability of modern sequence-to-sequence relevance models, such as monoT5, to adversarial attacks is conducted. These relevance models, which leverage cross-encoding of queries and documents, are shown to be susceptible to simple, query-independent adversarial techniques designed to manipulate document rankings. By injecting or rewriting documents with specific prompt tokens or their variants, attackers can significantly influence these models’ relevance assessments. This evaluation is pivotal as it underscores the potential for manipulating search engine rankings through straightforward adversarial strategies, potentially compromising search engine reliability and the integrity of information retrieval processes.

Experimental Methodology

The research involved designing three types of adversarial attacks targeting the structural prompts utilized by sequence-to-sequence relevance models: preemption, stuffing, and rewriting attacks, with a focus on the popular monoT5 model. The effectiveness of these attacks was rigorously tested on the TREC Deep Learning track datasets, providing a comprehensive examination of how these models can be manipulated. Notably, the experiments demonstrated that these attacks could be executed without requiring gradient access or deep knowledge of the target model, only necessitating some awareness of the model's prompt format.

Key Findings

Implications for Model Robustness

The findings reveal a significant vulnerability in sequence-to-sequence relevance models to adversarial manipulation, highlighting a critical area for future research and development. Both the preemption and stuffing attacks, reliant on the incorporation of specific prompt tokens, and the more sophisticated rewriting attacks leveraging LLMs, were effective in altering document rankings.

• Generalizability across Models: While primarily focused on monoT5, the attacks also showed varying degrees of transferability to other neural relevance models, including BERT-based and bi-encoder architectures. This suggests a broader potential vulnerability within the current landscape of neural information retrieval models.
• Impact on Retrieval Effectiveness: From a search provider's perspective, the adversarial attacks pose a substantial risk, with the potential to significantly degrade the quality of search results. This is particularly concerning for the application of these models in scenarios where information reliability and search quality are paramount.

Future Directions

These findings underscore the necessity for developing robust adversarial defenses for sequence-to-sequence relevance models. Addressing these vulnerabilities will be critical in ensuring the reliability and integrity of future neural information retrieval systems. Additionally, the research opens up new avenues for exploring more sophisticated adversarial strategies and defense mechanisms within the field of search engine optimization and information retrieval.

Conclusion

This analysis provides an eye-opening insight into the vulnerabilities of sequence-to-sequence relevance models to relatively simple, yet effective adversarial attacks. The demonstrated ability to manipulate search rankings through prompt injection and document rewriting poses significant challenges for the application of these models in real-world information retrieval tasks. Developing mechanisms to safeguard against such adversarial strategies is essential for the continued advancement and deployment of neural relevance models in search engines and beyond.