ThinkTrap: Adversarial Strategies in LLMs

Updated 15 December 2025

ThinkTrap is a formal framework defining adversarial strategies that exploit black-box LLMs using input-space optimization to cause resource exhaustion.
It employs random Gaussian projection and CMA-ES to efficiently search for adversarial prompts that maximize output length under strict query budgets.
Defensive applications use MDP-based covert interventions to trap attackers, minimizing their rewards and preserving service availability.

ThinkTrap is a formal term for a class of adversarial strategies and corresponding defense frameworks, developed for both attacking and defending in intelligent systems, most prominently in LLM cloud infrastructures and adversarial planning environments. In its attack manifestation, ThinkTrap exploits the input space of black-box LLM services to induce “infinite thinking”—unbounded or excessively long generation loops that precipitate denial-of-service (DoS) outcomes. In its defense context, ThinkTrap formalizes covert defender policies within Markov Decision Process (MDP) frameworks to entrap attackers in trap states, minimizing their expected rewards while evading suspicion.

1. Threat Model and Motivation

ThinkTrap attacks operate under strict assumptions: the adversary maintains only black-box access to a cloud-hosted LLM through its public API—accessing no weights, internal logits, or gradients—and is limited by token-level cost structures. The objective is to discover adversarial prompts that maximize model output length while minimizing API cost, all within basic service-level restrictions such as input caps, rate limits ( $\leq$ 10 requests per minute), and output budgets. The attack leverages the autoregressive decoding architecture of LLMs, where each output token entails a full forward pass over the model, exacerbating backend computational loads and causing GPU resource exhaustion. Unlike open-source/white-box DoS methods that require gradient or logit access, ThinkTrap demonstrates potency exclusively through prompt optimization using black-box queries (Li et al., 8 Dec 2025).

2. Optimization Framework and Attack Algorithm

At the core of ThinkTrap’s attack methodology is an input-space optimization pipeline: discrete token prompts $x = (x_1, ..., x_L) \in V^L$ are mapped (via an unknown embedding) into a continuous feature space $E(x) \in \mathbb{R}^{L \times d}$ . Direct optimization is rendered intractable by the high dimensionality (e.g., $L=20, d=4096 \implies 81,920$ dimensions), so ThinkTrap deploys random Gaussian projection $P: \mathbb{R}^{L \cdot d}\to\mathbb{R}^m$ , where $m \ll L \cdot d$ , to construct a sparse, low-dimensional subspace for search. The attack algorithm utilizes Covariance Matrix Adaptation Evolution Strategy (CMA-ES) for efficient derivative-free optimization, iteratively sampling latent perturbations $\delta_i \sim \mathcal{N}(\mu, \Sigma)$ within this subspace. Each candidate is reconstructed to a discrete prompt via nearest-neighbor decoding and evaluated by querying the victim LLM for output length, then the optimizer updates population parameters to maximize generation (Li et al., 8 Dec 2025).

Attack pseudocode outlines the loop:

Input: base prompt %%%%7%%%%, query budget %%%%8%%%%, latent dimension %%%%9%%%%, population %%%%10%%%%, top-%%%%11%%%%
Output: adversarial prompts %%%%12%%%%
1. Initialize CMA-ES: %%%%13%%%%, %%%%14%%%%
2. For %%%%15%%%% to %%%%16%%%% until queries %%%%17%%%%:
   a) Sample %%%%18%%%%
   b) For each %%%%19%%%%:

       %%%%20%%%%

       %%%%21%%%%

       %%%%22%%%%

   c) Rank %%%%23%%%% by %%%%24%%%%; select top-%%%%25%%%% with weights %%%%26%%%%
   d) Update %%%%27%%%%
3. Return best decoded prompts

Typical parameters: prompt length

L=20

, latent dimension

m=20

, population

N=10

, top-

k=5

, query budget

Q\leq 10,000

tokens, with attack success achieved well under

Q=100

k tokens.

3. Theoretical Analysis

The random projection step ensures distance preservation in expectation by the Johnson–Lindenstrauss lemma, so the subspace search approximates full-space exploration despite high original dimensionality. CMA-ES is guaranteed to converge to a local optimum in $O(\mathrm{poly}(m))$ iterations under mild smoothness conditions on the reward function $\ell(\cdot)$ , namely the output length induced by an LLM prompt. Query complexity scales as $O(TN)$ , and with $m\ll Ld$ , evaluation cost is drastically reduced compared to naive discrete search schemes. This formulation enables practical input-space adversarial optimization without incurring the prohibitive costs associated with full combinatorial enumeration (Li et al., 8 Dec 2025).

4. Experimental Evaluation and Transferability

Experiments validate ThinkTrap on eight models including Gemini 2.5 Pro, GPT-o4, DS R1 671B, and LLaMA 3.2 3B, employing metrics such as output length (normalized to 4096 tokens), tokens/sec (TPS), time-to-first-token (TTFT), and GPU memory usage. Against baselines (semantic decoys, heuristic “Sponge Examples,” and gradient-free “LLMEffiChecker”), ThinkTrap outperforms in output maximization under restricted budgets: under 10 RPM and $\leq$ 10k token budget, throughput is reduced to 1% of baseline and victim servers are pushed to full output limits. DS R1 achieved full-length attacks at a cost of \$0.0215 for 10k tokens. On self-hosted LLaMA services (4×2080 Ti), adversarial prompting filled GPU memory to OOM over $\approx$ 80 prompts, degrading TTFT and TPS and precipitating near-complete DoS.

Transferability analysis reveals that prompts optimized on one model family transfer poorly across distinct architectures ( $\mathrm{CDF\,peaks} < 800$ tokens), though transfer is more effective across models fine-tuned on the same dataset (e.g., DeepSeek variants). Very limited transfer is observed to entirely novel models, suggesting that optimization procedures must be re-executed per target system (Li et al., 8 Dec 2025).

Model	Output Length Achieved	Minimal Query Budget	DoS Efficacy (%)
Gemini 2.5 Pro	4096 (max)	$<$ 10k tokens	$\sim$ 99
DS R1 671B	4096 (max)	$<$ 10k tokens	99
LLaMA 3.2 3B	4096 (max)	$<$ 10k tokens	99

5. Implications, Defense Strategies, and Mitigations

Cloud LLM services, including closed-source systems with restricted APIs, are shown to be asymmetrically vulnerable: small attacker cost and low query volume can consume unbounded server resources. ThinkTrap-type prompt-based DoS attacks threaten service availability and reliability guarantees for downstream applications.

Several mitigation strategies are proposed:

Output-length caps: Truncating generations at fixed token counts (e.g., 256) is effective but impairs legitimate long-form tasks.
Anomaly detection: n-gram repetition and output heuristic methods are partially effective but degrade throughput and are evaded by semantic diversity in adversarial ThinkTrap outputs.
Resource-aware scheduling: Mechanisms such as the Virtual Token Counter enforce per-request token quanta (e.g., 1,024) before preemption, bounding resource occupation per user and preventing sustained DoS. Such solutions may induce latency and reduce throughput for benign users.

Recommendations emphasize resource-aware scheduling as a foundational component in LLM serving architectures, supported by real-time signals (e.g., KV-cache growth, per-token compute) to detect and throttle “thinking traps.” A balance must be maintained to isolate adversarial patterns while preserving quotas for legitimate workloads (Li et al., 8 Dec 2025).

6. Attacker Entrapment: MDP-Based Defenses

Beyond LLM scenarios, ThinkTrap is formalized in the defender context as an infinite-horizon discounted MDP for planning attacker entrapment (Cates et al., 2023). The attacker, believing they operate in an environment characterized by $\mathcal{M}^A = \langle S, A^A, T, R^A, \gamma, s_0^A\rangle$ , seeks to maximize cumulative reward while avoiding trap states ( $S_T$ ). A covert defender is modeled as able to override outcomes via an MDP $\mathcal{M}^D_{(S_T, K)}$ whose state space $S^D = S\times A^A\times\{0,\dots,K\}$ tracks attacker location, last action, and defender budget $K$ . Absorbing states correspond to trap or exhausted-budget. Defender actions include passively allowing transitions or forcibly relocating the attacker.

The defender’s optimal policy $\pi^{D,*}$ is computed via Bellman value iteration on the extended state space, maximizing expected discounted returns and minimizing attacker value. The budget $K$ is computed algorithmically as a pessimistic lower bound so interventions remain covert, determined by comparing Bayes-updated traces under attacker’s possible environment models.

Empirical evaluation in canonical RL domains (Gridworlds, Four-Rooms, Rock Sampling, Continuous Puddle) shows that defender policies reliably reduce or nullify attacker reward; planning time grows with state/action size and budget cap.

Domain / Size	Attacker Value	Defender Value	Plan Time (s)
Grid 4×4	0.94	−0.32	0.46
Rock 6×6	868.9	0	272.7
Puddle δ=0.2	529.1	0	0.14

This approach yields covert policies, masking interventions and leading attackers to irreversible trap states within the budget constraint (Cates et al., 2023).

7. Synthesis and Future Directions

The ThinkTrap paradigm encompasses both adversarial prompt optimization against black-box intelligent systems and MDP-based covert defensive planning—demonstrating that, across domains, input-space manipulation and covert intervention are potent tools for resource exhaustion and entrapment. In LLM infrastructure, robust service provision requires defense mechanisms that finely regulate per-request resource usage, transcending simple rate or length caps. In adversarial planning, MDP-based defender models formalize optimal intervention strategies, with tractable solutions via value iteration.

A plausible implication is that as LLMs and autonomous systems evolve, both attack surface and defensive frameworks will require increasingly sophisticated models, embedding real-time analytics and game-theoretic reasoning to preserve availability and integrity in adversarial contexts.