Agentic Adversarial Rewriting Exposes Architectural Vulnerabilities in Black-Box NLP Pipelines

Abstract: Multi-component NLP pipelines are increasingly deployed for high-stakes decisions, yet no existing adversarial method can test their robustness under realistic conditions: binary-only feedback, no gradient access, and strict query budgets. We formalize this strict black-box threat model and propose a two-agent evasion framework operating in a semantic perturbation space. An Attacker Agent generates meaning-preserving rewrites while a Prompt Optimization Agent refines the attack strategy using only binary decision feedback within a 10-query budget. Evaluated against four evidence-based misinformation detection pipelines, the framework achieves evasion rates of 19.95 to 40.34% on modern LLM based systems, compared to at most 3.90% for token-level perturbation baselines that rely on surrogate models because they cannot operate under our threat model. A legacy system relying on static lexical retrieval exhibits near-total vulnerability 97.02%, establishing a lower bound that exposes how architectural choices govern the attack surface. Evasion effectiveness is associated with three architectural properties: evidence retrieval mechanism, retrieval-inference coupling, and baseline classification accuracy. The iterative prompt optimization yields the largest marginal gains against the most robust targets, confirming that adaptive strategy discovery is essential when evasion is non-trivial. Analysis of successful rewrites reveals four exploitation patterns, each targeting failures at distinct pipeline stages. A pattern-informed defense reduces the evasion rate by up to 65.18%.

• The paper presents a two-agent adversarial rewriting framework that generates semantic-preserving rewrites to exploit vulnerabilities in black-box NLP pipelines.
• It employs iterative prompt optimization and strict constraint validation, achieving up to 40.34% evasion on modern systems and exposing nearly complete vulnerability in legacy pipelines.
• Experimental results underscore the importance of architectural design decisions, with adaptive optimization and text simplification defense offering measurable improvements.

Agentic Adversarial Rewriting Reveals Architectural Vulnerabilities in Black-Box NLP Pipelines

Introduction and Motivation

This work addresses the adversarial robustness of multi-component NLP pipelines, focusing specifically on evidence-based misinformation detection systems operating under production-realistic constraints: strict black-box access (binary-only feedback), no gradient access, and tight query budgets. Unlike monolithic classifiers, these multi-stage systems pose significant new challenges for adversarial attacks, since effective exploitation must disrupt both retrieval as well as inferential components, and must do so within a narrow query regime and with no visibility into model internals.

Conventional token-level adversarial methods are shown to be largely ineffective in this setting. The authors propose an agentic adversarial rewriting framework, leveraging a two-agent LLM-based system that operates in a semantic perturbation space, generating meaning-preserving rewrites of input claims. These rewrites aim to induce misclassification by the target pipeline, all while adhering to strict semantic equivalence and linguistic coherence constraints.

Figure 1: High-level overview of the agentic adversarial rewriting framework, including the iterative attack loop and a worked example demonstrating the progression toward a successful adversarial evasion.

Framework Design and Methodology

The core framework adopts a two-agent architecture: an Attacker Agent produces candidate rewrites of the original claim, while a Prompt Optimization Agent refines the attack strategy using minimal binary feedback from the pipeline and analysis of prior failed attempts. Constraint validation modules ensure that generated rewrites remain semantically faithful and linguistically viable.

Key components include:

• Constraint validation: Enforced via a combination of MPNet and BERTScore embedding similarities, GPT-4o semantic equivalence checks, and coherence validation (GPT-4o-mini).
• Iterative optimization: The Prompt Optimization Agent adapts attack instructions dynamically across iterations, informed by a history of feedback and constraint failures.
• Query efficiency: The attack is constrained to a strict 10-query budget per input.

  Figure 2: Detailed system architecture, depicting constraint modules and the two-agent feedback/optimization loop guiding iterative adversarial rewriting.

This setup is formulated as a constrained optimization over the input space of natural language, where feasible perturbations form a semantically and linguistically admissible neighborhood around the original claim.

Experimental Results: Empirical Vulnerability Spectrum

Experiments were conducted against four evidence-based misinformation detection pipelines spanning a spectrum from legacy lexical retrieval to modern LLM-based RAG architectures. Evaluation was performed on the LIAR-New dataset, with attacks mandated to preserve semantic fidelity under human-calibrated constraints.

Notable quantitative outcomes include:

• Attack success rates on modern systems: The Full History variant of the framework achieves between 19.95% and 40.34% evasion on robust LLM-based pipelines, versus a maximum of 3.90% for surrogate-guided token-level baselines unable to operate under the hard-label threat model.
• Architectural lower bound: The legacy ClaimBuster pipeline, which leverages static lexical retrieval, suffers 97.02% attack success—demonstrating near-complete vulnerability and grounding the lower end of the architectural robustness spectrum.
• Marginal value of adaptive optimization: Iterative prompt optimization (Full History) yields up to +10.24pp improvement over non-adaptive Attacker Only variants on the most robust targets, establishing adaptive agentic optimization as essential when feasible adversarial regions are small.

  Figure 3: Cumulative attack success rates for different framework variants across multiple target pipelines, illustrating the impact of iterative prompt optimization.

Additionally, analysis of successful adversarial rewrites revealed four main exploitation patterns: hedging/ambiguity injection, structural elaboration, linguistic complexity escalation, and syntactic restructuring. These patterns can selectively degrade performance at different pipeline stages (retrieval versus inference), contingent on architectural particulars.

Defense Evaluation and Theoretical Implications

Inspired by observed exploitation patterns, a pattern-informed defense—text simplification—was evaluated. This defense, operationally realized via LLM-based input normalization, reduces attack success on modern pipelines by up to 65.18%. The resultant protection, however, remains bounded by underlying architectural weaknesses: lexical-retrieval systems benefit least, while inference-stage vulnerabilities are most effectively mitigated by complexity normalization.

These findings have immediate implications: architectural choices governing evidence retrieval mechanisms, stage coupling, and baseline accuracy directly modulate the attainable attack surface. For robust deployment, multi-stage NLP systems must go beyond surface-level string matching, tightly couple retrieval and inferential steps, and implement dynamic consistency checks or semantic validation at every inter-component boundary.

Limitations and Future Directions

Despite strong empirical performance, the presented framework has limitations. Semantic equivalence assessment partially relies on LLM-based checks subject to model family bias; only one dataset and problem domain were evaluated; and no explicit cross-pipeline transferability analysis was performed. Further, causal attribution of exploitation patterns requires controlled ablation. Future work should extend this methodology to other NLP pipeline domains (e.g., RAG-based QA, tool-augmented LLM agents) and explore ensemble or multi-mechanism defenses that combine text simplification, input perturbation detection, and pipeline consistency enforcement.

Conclusion

This paper formalizes and systematically interrogates the adversarial robustness of black-box, multi-component NLP pipelines under production-realistic threat models. The agentic adversarial rewriting framework demonstrates that semantic-level adaptive text attacks pose serious risks to deployed multi-stage systems, achieving strong evasion rates even under tight query budgets and minimal feedback. Crucially, exploitation effectiveness is strongly modulated by pipeline architectural choices. The presented results motivate integration of adversarial evaluation, input complexity normalization, and holistic pipeline hardening as standard best practices in the ongoing deployment of NLP-based decision systems.