---
title: Representation-Space Attacks
url: https://www.emergentmind.com/topics/representation-space-attacks
type: topic
---

# Representation-Space Attacks

Representation-space attacks are adversarial manipulations that operate on structured internal or transformed representations—basis coefficients, intermediate features, latent codes, residual-stream activations, or communication caches—rather than only on raw input tokens or pixels. Across the recent literature, the term covers several related threat models: direct interventions on hidden states during a forward pass, perturbations in wavelet or feature space, latent-code tampering in generative and semantic communication systems, and attacks that use prompts or context to steer a model’s internal representations into regions associated with harmful, compliant, or attacker-chosen behavior. What unifies these settings is that the attack objective is specified in representation space, where downstream decisions are often more linearly organized and therefore easier to steer than surface inputs alone [2605.21706].

## 1. Scope and defining characteristics

In the broadest usage, representation-space attacks perturb inputs after mapping them into a structured representation—basis, feature, or latent space—rather than directly altering pixels in the native time or pixel domain [2107.12473]. In aligned language models, a latent-space attack alters internal activations $h \in \mathbb{R}^d$ during the forward pass to evade a behavioral decision such as refusal, while leaving input tokens and model weights unchanged [2605.21706]. In vision-language navigation, the attacked object is an embedding produced by a vision-language model; the adversary imperceptibly alters a small subset of images so that their embeddings collide with unrelated landmark images or texts, causing route manipulation [2407.07392]. In semantic communication, the attacker is an on-path adversary who reads and writes transmitted latent codes $z$ and replaces them with crafted latents $z'$ so that the receiver decodes attacker-chosen semantics while the latent remains statistically similar to legitimate transmissions [2512.03361].

This family therefore differs both from ordinary input-space adversarial examples and from output-space attacks. Input-space attacks optimize directly over visible text, pixels, or suffix tokens. Output-space attacks bias final logits or decoded strings. Representation-space attacks instead exploit the internal geometry of models: linearly separable refusal features in residual streams, cross-modal similarity scores in embedding spaces, semantically rich diffusion latents, or inter-agent KV caches that function as hidden communication channels [2605.21706].

A recurring finding is that many high-level behaviors are encoded in relatively simple internal directions or regions. The literature reports linear separability of refusal versus compliance across layers of instruction-tuned language models, easy-to-induce “representation collisions” in CLIP-like embedding spaces, and semantically steerable latent manifolds in communication and generative systems. This makes representation-space interventions both mechanistically interpretable and operationally potent [2406.10794].

## 2. Core mathematical formulations and geometric viewpoints

A common formalization uses a linear probe over an activation $h$:
$$
s(h)=w^\top h+b,
$$
with decision boundary $w^\top h+b=0$ and margin
$$
m(h)=\frac{w^\top h+b}{\|w\|}.
$$
In the refusal-evasion setting, earlier “refusal ablation” methods are reinterpreted as orthogonal projection onto this boundary:
$$
h' = h-\frac{w^\top h+b}{\|w\|^2}w.
$$
Under unit-norm $w$, this is the smallest $\ell_2$ perturbation that flips the linear decision, but it stops at maximum uncertainty. Controlled Latent-space Evasion extends this by projecting past the boundary to a target margin $m_\star$:
$$
h' = h-\frac{(w^\top h+b)-m_\star}{\|w\|^2}w,
$$
so that activations enter the compliant region with controllable confidence [2605.21706].

Feature-space attacks in vision often replace pointwise Euclidean matching with statistic alignment. In Pair-wise Alignment Attack, each spatial position is treated as a sample and the discrepancy between source and target feature distributions is measured by empirical MMD:
$$
\operatorname{MMD}^{2}[p,q]
=
\frac{1}{N^{2}}\sum_{i,j} k(s_{\cdot i},s_{\cdot j})
+
\frac{1}{N^{2}}\sum_{i,j} k(t_{\cdot i},t_{\cdot j})
-
\frac{2}{N^{2}}\sum_{i,j} k(s_{\cdot i},t_{\cdot j}),
$$
whereas Global-wise Alignment Attack matches channel-wise means and variances. Both formulations remove the spatial-consistency constraint imposed by pixelwise feature MSE and therefore provide translation-invariant feature alignment [2105.11645].

Wavelet-space attacks make the representation explicit through multiresolution analysis. Images are decomposed into approximation coefficients and oriented detail coefficients; only the high-scale detail coefficients are perturbed, and the image is reconstructed by inverse DWT. Because the wavelet basis is orthonormal, the transform is unitary and preserves $\ell_2$ energy, giving a direct mapping between perturbation budgets in pixel and wavelet space while focusing adversarial energy on localized, oriented components that align with early CNN filters [2107.12473].

Some work studies non-Euclidean representation spaces. In hyperbolic embeddings on the Poincaré ball, the conformal factor grows near the boundary, so small Euclidean changes can induce disproportionately large geodesic displacements in representation space. The resulting “boundary-driven asymmetry” makes geometry-adaptive triggers harder for Euclidean detectors to flag while amplifying their effect on downstream representation-space behavior [2510.06397].

## 3. Language-model attacks and interventions

In safety-aligned language models, representation-space attacks have been used to suppress refusal, move harmful prompts toward “acceptance directions,” and hijack token semantics through context. One line of work shows that harmful and harmless prompts occupy separable regions in the last-token representation, and defines an acceptance direction
$$
e_a=\frac{c_a-c_r}{\|c_a-c_r\|_2}
$$
on a PCA plane anchored by harmless and harmful prompts. Successful jailbreaks are characterized by movement toward the harmless centroid along this direction, and adding a representation objective to GCG substantially increases ASR on several models [2406.10794].

A more direct activation-level formulation treats refusal suppression as evasion against per-layer probes. Controlled Latent-space Evasion trains linear SVM probes on post-instruction residual-stream activations from harmful and harmless prompts, then optimizes per-layer steering masks and target margins via Bayesian Optimization. CLE-P reprojects token activations onto shifted hyperplanes, whereas CLE-A computes a perturbation once at the post-instruction token and adds it to all subsequent activations. The reported results show that zero-margin boundary projection is too weak, whereas positive margins produce high ASR across 15 instruction-tuned, multimodal, and reasoning models [2605.21706].

Other work emphasizes prompt-induced representation change rather than direct hidden-state editing. “Semantic Representation Attack” reframes jailbreaks as optimization over regions of meaning rather than exact affirmative strings, using a safety classifier as the semantic representation function and a coherence constraint based on perplexity. Its Semantic Representation Heuristic Search maintains interpretability by expanding only coherent candidates, and the paper reports an average ASR of 89.41% across 18 LLMs [2509.19360]. “Doublespeak” is optimization-free: by systematically replacing a harmful keyword with a benign token in multiple in-context examples, the benign token’s hidden-state trajectory converges toward the harmful one in later layers, yielding an internally harmful interpretation of superficially innocuous text [2512.03771].

Representation-space analysis also clarifies why multi-turn jailbreaks work. In the Crescendo attack, per-turn prompts keep activations in a “benign” region even while the conversation accumulates enough context for harmful completion. Probes trained on single-turn harmful versus benign responses show that the proportion of final-response tokens classified as harmful decreases as the number of turns increases, exposing a generalization gap in single-turn defenses [2507.02956].

Defensively, SafeInt applies a low-rank intervention at an intermediate layer to relocate jailbreak-induced representations into the rejection region by aligning them with unsafe samples while reconstructing safe and already-unsafe representations. It combines classifier-guided alignment, contrastive pulling toward unsafe representations, and reconstruction loss, and is designed to impose negligible inference overhead because the intervention acts as a lightweight plug-in on the residual stream [2502.15594].

## 4. Multimodal, robotic, and agentic settings

In vision-language navigation, representation-space attacks exploit the fact that the planner scores visual-text matches in an embedding space. The attack manipulates a minimal subset of landmark images so that $E_{\text{img}}(x+\delta)$ aligns either with a target image embedding or with a landmark text embedding, using a squared $\ell_2$ alignment loss and cosine-similarity stopping criteria. Because LM-Nav’s dynamic program aggregates node-level landmark scores, these representation collisions are sufficient to steer the agent onto attacker-chosen routes. The paper reports a 100% Route Modification Success Rate on both evaluated graphs and proposes a detector based on the heightened sensitivity of adversarially modified images to small Gaussian noise in embedding space [2407.07392].

Latent-based multi-agent systems move part of coordination from visible text into hidden states and KV-cache handoffs, and thereby expose a different representation-space surface. “Out of Sight, Not Out of Mind” constructs paired clean and attacked trajectories, computes displacements between matched latent components, and extracts attack-associated directions with DiffMean, PCA, or RePS. These directions are then reactivated during a clean run by additive interventions on local hidden states or, more effectively, on inter-agent KV handoffs. Across GSM8K, OpenBookQA, and HumanEval+, KV-both interventions on edge-level handoffs produce the largest accuracy drops, and control experiments with norm-matched random directions indicate that the observed degradation cannot be reduced to arbitrary perturbation or invalid generation [2605.28214].

These studies shift the locus of attack from visible inputs to hidden communication infrastructure. In navigation, the attacked representation is the multimodal similarity space used for grounding. In latent multi-agent systems, it is the hidden memory passed between agents. In both cases, visible-text inspection is insufficient because the operative attack state is not rendered in natural language [2605.28214].

## 5. Vision, communication, and generative-model attacks

Representation-space attacks are also prominent in visual recognition, semantic communication, and diffusion models. In expressive human pose and shape estimation, UBA uses a pretrained VAE to encode an image into a latent $z$, injects Gaussian noise to obtain $\tilde z=z+\eta \epsilon_2$, decodes this latent to obtain an initial perturbation, and then refines the perturbation with a black-box query-based optimization against the EHPS model’s outputs. The attack is evaluated by increases in MPJPE, MPVPE, and aligned variants, and the paper reports average error increases of 17.27%–58.21% while maintaining strong perceptual quality by PSNR, SSIM, LPIPS, and FID [2505.12009].

In semantic communication, the transmitted object is itself a semantic latent:
$$
z=f_\theta(x), \qquad \hat x=g_\phi(z).
$$
A Diffusion-based Re-encoding Attack uses a latent diffusion model to synthesize an attacker-designed semantic variant in the same latent space, while Test-Time Adaptation Latent Manipulation directly optimizes the intercepted latent with
$$
\mathcal{L}_{sem}=1-\cos(e_x,e_t), \qquad
z^{(k+1)}=z^{(k)}-\eta \nabla_z \mathcal{L}_{sem}.
$$
Both attacks aim to preserve the statistical properties of legitimate latents while changing decoded semantics, and the paper argues that such tampering is covert and difficult to detect with simple distributional checks [2512.03361].

Backdoor variants act on learned representation layers rather than only on outputs. BadRSSD attacks the PCA latent representation of regularized self-supervised diffusion models by aligning poisoned samples with a target image in PCA space, enforcing trajectory consistency across diffusion steps, and adding pixel-space reconstruction and dispersion regularization. The representation layer thus becomes the backdoor substrate, with the trigger activating a target-specific denoising trajectory while preserving clean utility and resisting output-oriented defenses [2603.01019].

Geometry-specific effects further expand the space of possible attacks. In hyperbolic embeddings, outward radial triggers leverage curvature to obtain large geodesic shifts for small Euclidean changes, and the paper shows empirically that attack success rises toward the boundary while conventional Euclidean detectors weaken [2510.06397].

## 6. Transferability, misconceptions, and defenses

A central misconception is that all successful adversarial attacks should transfer. Recent work argues instead that transfer is contingent on operational domain: attacks in the shared input data-space can transfer, whereas attacks in model representation space generally do not unless the latent geometries are sufficiently aligned. In a simple linear setting, two networks can compute the same input-output map through different representations related by an invertible transformation $Q$; data-space attacks transfer perfectly because the attacked object is the same input $x$, but representation-space attacks require compatibility conditions such as $w^\top Q=w$ to preserve the same harm. Empirical studies on image classifiers, LMs, and VLMs support this distinction: soft-prompt jailbreaks and latent classifier attacks often fail to transfer across misaligned models, while textual suffix attacks transfer much more readily [2510.01494].

A second misconception is that output-level alignment or keyword filtering is sufficient. Doublespeak shows that benign tokens can retain benign meanings in early layers while converging toward harmful semantics in later ones, creating a time-of-check versus time-of-use failure mode [2512.03771]. Multi-turn Crescendo attacks likewise remain probe-benign turn by turn while accumulating harmful context [2507.02956]. In latent multi-agent systems, harmful information can persist in KV caches during otherwise clean executions [2605.28214].

Defenses therefore increasingly target representation geometry rather than only visible text. The literature proposes adversarial training against refusal-feature probes, regularization to discourage linear separability of harmful and refusal features, monitoring per-layer probe margins or anomalous confidence jumps, context-level probes that aggregate safety signals over multi-turn trajectories, and handoff-level monitoring for KV attacks [2605.21706]. Domain-specific detectors include Gaussian-noise sensitivity screening for manipulated VLN landmark images [2407.07392], direction-aware and layer-profile detectors for latent multi-agent handoffs [2605.28214], and cryptographic latent integrity checks for semantic communication [2512.03361]. SafeInt exemplifies a representation-aware intervention defense for LLMs by explicitly relocating jailbreak representations into the unsafe rejection region while preserving utility on benign inputs [2502.15594].

Taken together, the literature treats representation space not merely as an explanatory lens but as an attack surface in its own right. The practical consequence is that robustness cannot be assessed only at the level of prompts, pixels, or final outputs. It must also be evaluated at the level of the internal geometries—linear boundaries, latent trajectories, cross-modal embeddings, and hidden communication states—through which modern models actually make decisions [2605.21706].

Source: https://www.emergentmind.com/topics/representation-space-attacks