Recovering Input Text from Hidden States: Study of Gradient-Based Inversion of Decoder-Only Language Models

Abstract: This work studies the hidden-state inversion problem: recovering the original input token sequence of a decoder-only LLM from its last-layer hidden states. Rather than treating inversion as a one-shot reconstruction, we study it as a continuous embedding-space optimisation in which a soft proxy is driven towards the leaked target without any hard-token projection during the search, and a token is committed only once, at the end of the inner loop. This design choice has two consequences which are the main focus of this paper. First, keeping the optimisation entirely in continuous space exposes a rich set of internal signals: rank trajectories of the ground-truth token, per-position loss curves, and a discrete loss measured at commit time. Second, the discrete loss allows assessing the correctness of recovery via cumulative discrete loss. We further analyse which tokens break the reconstructions and find a sharp categorical asymmetry: space-prefixed, high-frequency function words in dense regions of the embedding matrix dominate the failures, while content-bearing tokens are recovered almost perfectly. On 10-token C4 prompts the exact-match rate rises from 66.9% to 97.5% (mean similarity 0.994) as the candidate window is widened, confirming that most errors are recoverable near-misses rather than genuine ambiguities. A comparison with the released SIPIT reference situates these findings: per-step hard projection is faster, but the continuous formulation is what makes the optimisation observable and its failures detectable. The results show that last-layer hidden states of GPT-2 are as sensitive as the original text.

• The paper demonstrates that hidden-state inversion of decoder-only LMs is fundamentally feasible via a continuous proxy optimization method.
• It employs a single-commit strategy with gradient-based candidate search, achieving up to 97.5% exact-match rates on input text recovery.
• The study reveals critical privacy implications and introduces self-diagnostic metrics for detecting inversion failures in model activations.

Gradient-Based Inversion of Decoder-Only LLMs: From Hidden States Back to Text

Problem Setting and Motivation

The paper systematically studies the problem of reconstructing input text from the last-layer hidden states of decoder-only transformer LMs, focusing on GPT-2. This task, known as hidden-state inversion, is motivated by practical privacy concerns in split inference, federated learning, and embedding APIs, where intermediate activations, not raw text, are assumed to protect sensitive information. However, recent theoretical work asserts that the mapping from token sequences to hidden states is injective for decoder-only transformers, meaning that the hidden states fundamentally contain all original information—rendering them, in principle, reversible.

Methodology: Continuous Proxy Optimisation & Single-Commit Strategy

The authors develop a gradient-based inversion algorithm with a key structural choice: during the optimisation of each token, the proxy for the embedding remains entirely in continuous space. A token is only "committed" (projected onto the discrete vocabulary) once per position, after the proxy has converged. This stands in contrast to per-step hard projection methods: here, all optimisation dynamics and errors are concentrated and observable in the continuous phase, resulting in rich diagnostic signals including rank evolution, per-token loss curves, and a clean discrete loss at commit time.

The token-wise recovery process uses an inner Adam loop with cosine annealing and gradient clipping to stabilise search in the highly non-convex embedding landscape. Proxy initialisation on the embedding manifold (using a random token or zeros) accelerates convergence. At convergence of the inner loop, a candidate window $C$ is searched for tokens with discrete error below threshold; fallback is to the nearest neighbor if all candidates fail.

Figure 1: Per-token continuous loss curves comparing vanilla Adam and the stabilised (cosine-annealed, clipped) baseline. Stabilisation removes early instability and yields smooth loss descent.

Figure 2: The stabilisation of Adam improves both exact-match rate and mean similarity, preventing catastrophic loss spikes in the early steps.

Candidate Search Efficiency

A systematic ablation across the step budget $K$ and candidate window $C$ demonstrates that accuracy quickly saturates with $K \sim 1000$ steps per token, and that increasing $C$ primarily converts near-misses (tokens with very small embedding error but not the top candidate) into exact matches.

Figure 3: Exact-match rate rapidly increases with optimisation step budget $K$, saturating near $K=1000$.

Figure 4: The ground-truth token is almost always within the top few candidates by $\ell_1$ distance at error positions, motivating the use of larger $C$ for near-complete recovery.

Figure 5: Evolution of ground-truth token rank during optimisation. Correct tokens reach rank 1 early; errors persistently remain outside the leading candidates.

Results: Recovery Rates, Diagnostics, and Failure Modes

The main empirical results show:

• Exact-match rates on 10-token C4 prompts climb from 66.9% (Baseline: $K=1000, C=2000$) to 97.5% (High-accuracy: $K$0), with mean string similarity reaching 0.994.
• Recovery errors are strongly concentrated on high-frequency, space-prefixed function words (e.g., " the", " and"), whose embedding space is densely populated and near-degenerate, while content-bearing tokens are almost always recovered precisely.
• Per-token runtime, while significant (9.7–27.5s per token on an RTX A6000 for the studied configurations), is shown to be a function of step budget and early exit opportunities for "easy" tokens.

The single-commit design yields a powerful self-diagnostic: the cumulative discrete loss across positions acts as a failure detector, with error-free recoveries showing loss near numerical precision, and failures showing a sharp, orders-of-magnitude jump starting at the first error.

Figure 6: Cumulative discrete loss sharply separates exact and failed recoveries—useful for ground-truth-free auditing.

Figure 7: Optimisation difficulty increases along the sequence; token positions later in the prompt (with longer prefixes) converge more slowly.

Figure 8: Left—most frequent mis-committed tokens are high-frequency function words; Right—error distribution heavily favors these categories over content-bearing tokens.

A direct comparison with the SIPIT reference implementation (per-step hard projection) reveals that the reference is much faster ($K$1), always achieves 100% exact match, but forfeits the rich internal signals facilitated by the single-commit, continuous proxy approach. The separation of continuous and discrete phases is thus shown to be not merely aesthetic but essential for inverting models in a diagnostically informative way.

Theoretical and Practical Implications

The findings have immediate relevance for privacy assessments in any system exposing decoder-only model activations. Empirically, last-layer hidden states must be considered as sensitive as the input text itself: the probability of successful recovery of non-trivial content is extremely high even without exact match. The diagnostic properties of the discrete loss metric also provide a principled method for detecting inversion failure—potentially useful for both attackers and defenders.

From a theoretical standpoint, the study highlights how dense function-word embedding neighborhoods dominate the practical recovery error; this effect is not uniform across tokens, further undermining the presumed privacy granularity in hidden activations.

Potential future directions include:

• Extending to larger and fine-tuned models to study the interplay between embedding space geometry and inversion difficulty.
• Generalising the approach to encoder-only or encoder-decoder architectures, which may not be strictly injective.
• Evaluating defences such as differential privacy or noise injection, quantitatively guided by the self-diagnostic discrete loss.
• Exploring sparser projection schedules that accelerate recovery while retaining failure detection capacity.

Conclusion

This work rigorously demonstrates that hidden-state inversion in decoder-only LMs is both theoretically guaranteed under injectivity and practically feasible using gradient-based, continuous embedding-space optimisation. The commitment to a strictly continuous optimisation followed by a single discrete projection per token unlocks unique diagnostic and auditing capabilities, including a robust self-diagnostic via cumulative discrete loss. The non-uniform recovery profile—where privacy leakage is sharpest for content-bearing tokens—underscores the need for re-evaluating the usage and handling of intermediate activations in privacy-sensitive scenarios. The study sets a methodological foundation for future work on privacy-preserving mechanisms and diagnostic tooling in LLM deployment and inversion settings.

(2607.00852)