Black-box Optimization of LLM Outputs by Asking for Directions (2510.16794v1)

Abstract: We present a novel approach for attacking black-box LLMs by exploiting their ability to express confidence in natural language. Existing black-box attacks require either access to continuous model outputs like logits or confidence scores (which are rarely available in practice), or rely on proxy signals from other models. Instead, we demonstrate how to prompt LLMs to express their internal confidence in a way that is sufficiently calibrated to enable effective adversarial optimization. We apply our general method to three attack scenarios: adversarial examples for vision-LLMs, jailbreaks and prompt injections. Our attacks successfully generate malicious inputs against systems that only expose textual outputs, thereby dramatically expanding the attack surface for deployed LLMs. We further find that better and larger models exhibit superior calibration when expressing confidence, creating a concerning security paradox where model capability improvements directly enhance vulnerability. Our code is available at this link.

• The paper introduces a query-based black-box optimization method using binary comparison queries to generate adversarial examples for LLMs.
• It demonstrates robust attack success rates up to 98% in diverse domains including vision, prompt injections, and jailbreak scenarios.
• The findings expose a security paradox: larger, well-calibrated models are more vulnerable to adversarial attacks, highlighting the need for improved defenses.

Black-box Optimization of LLM Outputs by Asking for Directions

Introduction and Motivation

The paper introduces a query-based black-box optimization framework for attacking LLMs in scenarios where only textual outputs are available. This setting is increasingly prevalent in production deployments, where API providers restrict access to model internals such as logits or token-level probabilities. The proposed method leverages the introspective capabilities of LLMs by prompting them to perform binary comparisons between candidate inputs, thereby extracting a calibrated optimization signal for adversarial input generation. The approach is demonstrated across three attack domains: adversarial examples for vision-LLMs, prompt injections, and jailbreaks.

Figure 1: Illustration of the optimization pipeline for adversarial examples, jailbreaks, and prompt injections using binary comparison queries.

Methodology: Query-based Black-box Optimization

Problem Formulation

The attack is formalized as an iterative optimization problem. Given a target model $\mathcal{M}$, an initial input $x$, and an attack goal $\mathcal{G}$, the objective is to find an adversarial input $x^{\text{adv}}$ such that $\mathcal{M}(x^{\text{adv}}) \in \mathcal{G}$, subject to domain-specific constraints (e.g., $\ell_\infty$ norm for images, suffix length for text).

Optimization Signal Extraction

A key insight is that LLMs are poorly calibrated when asked for absolute confidence scores, often responding with extreme or uninformative values. In contrast, when prompted to compare two inputs and select the one closer to the attack goal, models exhibit significantly better calibration. This enables a hill-climbing optimization strategy: at each iteration, a perturbed candidate is generated and the model is queried to select the preferred input. If the candidate is preferred, it becomes the new iterate.

Figure 2: Left: Absolute confidence queries yield uninformative, discretized values. Right: Binary comparison queries produce true positives aligned with ground-truth, enabling effective optimization.

Perturbation Strategies

• Vision-LLMs: The Square Attack is adapted, applying random $\ell_\infty$-bounded perturbations to image regions.
• Prompt Injection/Jailbreaks: Suffixes are appended to prompts; random token replacements within the suffix are used to generate candidates.

Experimental Results

Adversarial Examples for Vision-LLMs

The method is evaluated on multiple vision-LLMs using ImageNet samples. Query-based optimization achieves attack success rates (ASR) ranging from 5% to 50% using only text responses. Notably, larger models (e.g., Qwen2.5-VL-72B-Instruct, GPT-4o mini) are more susceptible, with ASRs up to 50.8%. When transfer-based adversarial examples are used for initialization, ASR increases further, and the hybrid approach (transfer + query-based optimization) consistently outperforms individual methods.

Targeted Attacks

Targeted adversarial attacks are more challenging; smaller models fail to differentiate between complex comparative queries, but larger models (e.g., GPT-5 mini) achieve up to 79% ASR when log-probabilities are available.

Figure 3: Targeted-attack success rates with log-probability access for Qwen and Llama model families.

Prompt Injection Attacks

The binary comparison approach substantially improves prompt injection ASR across all tested models, reaching up to 87.5% on GPT-4o mini. The method is competitive with log-probability-based optimization where such access is available, and is the only viable approach for models that do not expose log-probabilities.

Jailbreak Attacks

For jailbreaks, the method achieves near-perfect ASR ($\geq98\%$) on most models with low query complexity. In some cases, the query-based approach outperforms logprob-based optimization, likely due to reduced activation of model safety defenses.

Figure 4: Failed attempt with a random suffix, highlighting the importance of optimized adversarial suffixes.

Analysis and Implications

Security Paradox

A central finding is that model capability improvements (e.g., scale, alignment, calibration) directly increase vulnerability to this class of attacks. Larger, better-calibrated models are easier to optimize against using binary comparison queries, contradicting the expectation that improved models are more robust.

Failure Modes

• Poor Reasoning: Small models fail to perform meaningful comparisons, reducing attack efficacy.
• False Positives/Negatives: Misclassifications slow optimization or reduce ASR.
• Alignment Defenses: Strongly aligned models may refuse to answer comparison queries, blocking optimization.

Advanced Optimization and Defenses

The paper suggests that ensemble querying, prompt engineering, and more sophisticated perturbation strategies could further improve attack robustness. Defensively, models could be trained to refuse comparative feedback for security-sensitive queries, and API providers could implement detection of iterative query patterns.

Practical and Theoretical Implications

The approach expands the adversarial toolkit for black-box attacks, demonstrating that text-only output is sufficient for effective optimization. This has direct implications for the security of deployed LLMs, especially in agentic and multi-modal systems. The findings necessitate a reevaluation of alignment and safety strategies, as introspective capabilities can be weaponized for adversarial purposes.

Conclusion

The paper establishes that binary comparison queries provide a reliable optimization signal for black-box adversarial attacks against LLMs, even in text-only output settings. The method is broadly applicable across vision, prompt injection, and jailbreak domains, and is particularly effective against larger, more capable models. These results highlight a critical security paradox and underscore the need for new defensive paradigms in LLM deployment and alignment.