Adversarial Restlessness in LLMs

• Adversarial restlessness is defined by a distinctive jittery activation signature in LLMs during multi-turn prompt-injection attacks.
• It quantifies cumulative drift using vector displacements between conversation turns to detect subtle adversarial phase shifts.
• Enhanced detection using scalar trajectory features boosts accuracy up to 93.8% and enables early warning in real-time systems.

Adversarial restlessness is a phenomenon observed in multi-turn prompt-injection attacks on LLMs, defined by a distinctive “jittery” activation-level signature in the model’s residual stream. This signature is empirically and theoretically characterized by an abnormally long and erratic trajectory in hidden-state activation space, discernible only through internal model activations and not from surface-level conversation text. The concept demarcates attack processes (trust-building, pivoting, escalation) that are undetectable by standard text-level methods but manifest as elevated cumulative path lengths in activation space, providing a promising direction for real-time, process-aware adversarial detection and mitigation in dialogue systems (Kulkarni, 30 Apr 2026).

1. Formal Definition and Mathematical Framework

Let $M$ be a decoder-only transformer. Fix a layer $\ell$ with hidden dimension $d$. Consider a $T$-turn conversation with messages $m_1,\dots,m_T$. For each turn $t$, the last-token hidden-state activation is extracted:

$$v_t = \mathrm{extract}_\ell(\mathrm{chat\_template}(m_1 \dots m_t)) \in \mathbb{R}^d.$$

The per-turn drift vector is defined as $\Delta_t = v_t - v_{t-1}$, and the cumulative path length up through turn $T$ as

$C_T = \sum_{t=2}^T \|\Delta_t\|_2.$

Adversarial restlessness is the observation that, for prompt-injection attacks following the trust-building $\ell$0 pivoting $\ell$1 escalation trajectory, $\ell$2 is consistently and significantly larger than for benign dialogues. Each adversarial phase shift produces distinct activation displacements, accumulating a long, irregular trajectory in activation space (Kulkarni, 30 Apr 2026).

2. Measurement and Feature Extraction

The activation trajectory $\ell$3 forms a discrete path in $\ell$4. Key computed quantities per turn $\ell$5 include:

• $\ell$6 (displacement)
• $\ell$7 (drift magnitude)
• $\ell$8 (cosine similarity)
• $\ell$9 (cumulative path length)
• $d$0 (drift acceleration)
• $d$1 (mean drift)

Benign conversations result in smooth, short trajectories with small and approximately constant $d$2. Adversarial sequences induce large, spiking $d$3 values at phase boundaries, leading to increased $d$4 and erratic $d$5.

Five scalar features distilled for detection are:

1. $d$6 (drift magnitude)
2. $d$7 (cosine similarity)
3. $d$8 (cumulative drift)
4. $d$9 (drift acceleration)
5. $T$0 (mean drift)

Extraction proceeds by concatenating $T$1 per turn for downstream classification. Pseudocode for algorithmic extraction is given in the source appendix (Kulkarni, 30 Apr 2026).

3. Detection Performance and Feature Utility

On synthetic benchmark datasets, conversation-level detection using only per-turn activation vectors $T$2 yields $T$3 accuracy at $T$4 false positive (FP) rate. Inclusion of the five scalar trajectory features increases performance to $T$5 at the same FP threshold. Using only these scalars (no $T$6) achieves $T$7 accuracy. Feature-importance analysis attributes dominant detection power to cumulative drift $T$8 and drift magnitude $T$9.

Early detection of attacks, particularly in the "pivoting" phase, is enabled by restlessness features, with a mean lead time of approximately $m_1,\dots,m_T$0 turns before overt adversarial activity. Longer pivoting phases augment early-detection rates to $m_1,\dots,m_T$1–$m_1,\dots,m_T$2, imparting a $m_1,\dots,m_T$3 to $m_1,\dots,m_T$4 turn lead time, dependent on the underlying model (Kulkarni, 30 Apr 2026).

4. Cross-Architecture and Source-Dependent Generalization

Cross-model evaluation using scalar-augmented probes, retrained for each architecture, yields $m_1,\dots,m_T$5–$m_1,\dots,m_T$6 detection (FP $m_1,\dots,m_T$7–$m_1,\dots,m_T$8) on LLMs such as Mistral 24B, Gemma 27B, Qwen 32B, and Llama 70B. However, scalar-only feature transfer across architectures fails ($m_1,\dots,m_T$9 near $t$0), demonstrating strict model specificity—probes must be retrained per deployment.

In terms of source generalization, a detector trained solely on synthetic activations flags $t$1 of LMSYS-Chat-1M conversations as adversarial, illustrating that synthetic data fails to capture real-world conversational distributions. Leave-one-out experiments confirm that each source (synthetic, SafeDialBench, LMSYS) provides non-redundant attack distributions; omitting any source leads to catastrophic detection failures for that distribution. Mixed-source training (three data sources, $t$22,500 conversations) yields robust, general detection ($t$3 at $t$4 FP, ROC AUC $t$5 for each source), with significant increase in PR-AUC (LMSYS: $t$6) (Kulkarni, 30 Apr 2026).

5. Labeling Paradigms and Deployment Implications

Effective adversarial restlessness detection depends critically on three-phase turn-level labels (benign, pivoting, adversarial). Binary conversation-level supervision produces 100% detection but an unusably high (50–59%) false positive rate; detailed turn-level labels reduce FP to 0.5–2%. Explicit pivoting annotations are required to detect subtle steering before overt adversarial acts.

Deployment for a new model requires roughly 1,000 synthetic, three-phase-labeled conversations (automatically generable) plus a small real-world sample for FP calibration. Continual adaptation is feasible via fast retraining (<30 seconds on CPU) using cached activations and human/LLM-validated labels, with FP rates serving as model or data drift signals (Kulkarni, 30 Apr 2026).

6. Distinction from Other “Restless” Constructs

“Adversarial restlessness” is not directly related to “restless bandits” in sequential decision-making or robust reinforcement learning. In RMABs, as formulated by Killian et al. (Killian et al., 2021), “restless” refers to multi-armed bandits where arms evolve even when not selected, and “adversary” denotes robustness to uncertain environment parameters via minimax regret optimization. There is no explicit notion of internal activation path lengths or trajectory features analogous to adversarial restlessness in LLMs. Any conceptual connection is limited to the broader sense of “adversarial” interplay—between agent and environment in RMABs, versus attacker and model in LLM dialog.

7. Broader Impact and Research Directions

Adversarial restlessness introduces an activation-level, process-tracking defense orthogonal to surface-level filtering in LLM systems. It provides early warning of multi-turn prompt injection by identifying activation trajectory anomalies induced by intent steering. This mechanism supports real-time, white-box monitoring and opens the possibility of activation-level interventions, such as steering the model back to a “safe manifold” when restlessness is excessive. The work establishes empirical and labeling preconditions for robust deployment, especially in scenarios demanding continual adaptation and coverage across diverse attack sources (Kulkarni, 30 Apr 2026).