Inadvertent Context Leakage in Language Models

Abstract: For AI agents to be useful beyond simple chat, they must hold sensitive user context such as calendars, credentials, health records, and financial data. We study whether the mere presence of such secrets in a model's context window introduces hidden correlations into the model's benign outputs, allowing reconstruction even when the model correctly refuses direct extraction. We further study whether an adversary can actively engineer prompts that amplify this effect, using the model as a covert carrier to transmit secrets through seemingly innocuous text. In both cases, this limited leakage is exploited using a novel adaptive attack that assumes black-box access to the underlying model. In controlled experiments across eight proprietary models, we find that 2-digit in-context secrets are reconstructed with near-perfect accuracy and 4-digit secrets at 82\% exact match, all from outputs the model produces in response to ordinary, non-adversarial requests. We observe that more capable models leak more: stronger instruction-following amplifies sensitivity to in-context secrets, suggesting leakage is a byproduct of capability as opposed to a patchable bug. We show this leakage enables two practical attacks: (1) a trained classifier that infers semantic predicates about user memories (e.g., health conditions, financial events) from routine natural-language outputs, and (2) an RL-trained adversary that extracts full Social Security Numbers from a production-style agent.

• The paper introduces a black-box predicate-inference framework showing that benign output statistics can reveal protected secrets even when models refuse direct disclosure, with Claude Opus recovering 82% of four-digit secrets and Gemini 3.1 Pro recovering 44%.
• The paper demonstrates that leakage extends beyond numeric reconstruction: an adversary inferred user-memory predicates with a 0.319 advantage over chance and used prompt injection to recover Social Security Numbers from production-style agents.
• The paper identifies suppression as a likely mechanism, finding that stronger confidentiality instructions and post-training can intensify leakage, and argues that effective defenses must make output distributions invariant to protected context values.

Overview

"Inadvertent Context Leakage in LLMs" (2608.19857) studies whether in-context secrets leak through statistical properties of benign model outputs even when the model correctly refuses to disclose them verbally. The authors—Fairoze, Mangaokar, Chaudhuri, Garg, and Mahloujifar—argue that the standard privacy evaluation paradigm, which relies on linguistic judges scanning output text for verbatim or paraphrased disclosure, is incomplete. Token-level patterns, response length, formatting choices, and stylistic shifts all vary with the model's internal state, and that state depends on the secret. A model can therefore refuse perfectly in language while still encoding the secret in channels a linguistic judge never examines.

The paper formalizes this as a predicate inference game with an adaptive, black-box adversary, instantiates it on eight proprietary frontier models, and demonstrates three escalating results: near-perfect reconstruction of short numeric secrets from ordinary outputs, inference of semantic predicates about user memories at substantially higher accuracy than a linguistic judge, and RL-optimized extraction of full Social Security Numbers from production-style agents via prompt injection.

Threat model and framework

The core abstraction is a predicate inference experiment between a challenger and an adversary $\mathcal{A}$. The challenger samples a model $M$ and context $c$; the adversary issues $k$ prompts $p_i$, observes responses $r_i \sim M(p_i, c)$, and predicts a binary predicate $\phi(c)$. The adversary has black-box access only—sampled text, no logits or internal representations—which means the measurements lower-bound what a white-box adversary could extract.

The optimal adversary proceeds in two phases. In an estimation phase conducted against the public API before any attack, it samples contexts it authors itself (inserting secrets of its own choosing, so labels are known by construction) and estimates response distributions $P(r \mid p, b)$ for each predicate value. During the game, it predicts $\phi(c)$ by maximum likelihood over observed responses. The paper identifies two load-bearing assumptions: the adversary must know the secret's length (the decoder emits a fixed number of digits), and the deployment must designate the protected value via a confidentiality instruction naming a specific field—a generic "be discreet" instruction gives suppression nothing to attach to and the decoder nothing to read.

Two adversary variants are distinguished. The passive adversary reads statistical channels already present in benign outputs; the active adversary engineers the channel directly through prompt injection, bypassing estimation entirely. All claims about leakage from non-adversarial requests rest on the passive experiments.

Numeric secret reconstruction

The numeric instantiation treats each digit of an $N$-digit secret ($M$0) as a 10-valued attribute, using 18 benign task templates plus adaptive variants. Decoders include non-neural baselines and LoRA-finetuned Qwen-2.5 classifiers (direct, left-to-right conditioned, beam search, and an adaptive per-position chain). Each (model, $M$1) cell uses 1,000 secrets with 10 responses per pair; all eight models refuse direct extraction on 160 baseline queries with 0% leakage, confirming the measured signal is genuine.

Results separate the eight models into two tiers:

Model	$M$2 exact match	$M$3 exact match	Notes
Claude Opus 4.6	100%	82%	Adaptive pipeline nearly doubles $M$4 recovery (82% vs. 44%)
Gemini 3.1 Pro	100%	44%	Channel entropy 0.54 bits vs. 6.64-bit no-leakage baseline
GPT-5.4 / Sonnet 4.6 / Grok 4	high per-digit	moderate	Per-digit accuracy 0.63–0.89 at $M$5
Grok 4.1 Fast, GPT-5.4 nano	≤ chance at $M$6	chance	Resistant tier

Even at $M$7, best-case extraction reaches 41% per-digit accuracy on Gemini 3.1 Pro against 10% chance, and leading digits are most vulnerable (position-0 accuracy of 1.0 on Gemini Pro). Structured, $M$8-aware numeric prompts dominate leakage while open-ended and multi-turn prompts leak near zero. A notable negative result: adaptive conditioning does not uniformly help—it doubles Opus's four-digit recovery but slightly hurts Gemini Pro.

Appendix experiments isolate non-digit channels using 178 real WildChat prompts with digit-stripped decoders that cannot echo the secret even in principle. All six evaluated models show statistically significant non-digit leakage; Claude Opus's digit-stripped embedding decoder reaches 61.3% single-digit accuracy despite its literal digit-frequency decoder being non-significant. Randomized-label controls sit at chance across all 42 cells.

Semantic predicates on user memories

Applying the same threat model to CIMemories personas, the authors train a Llama-3.2-3B classifier to infer whether a given memory attribute was present in the model's input context—a question about influence rather than disclosure. Against Claude Opus 4.6 over 15 memory predicates, their adversary achieves 0.319 advantage over chance versus 0.058 for a DeepSeek-based linguistic judge, with a 1.7× higher true-positive rate at the judge's own false-positive rate. A worked example shows the mechanism: both adversaries detect a memory repeated verbatim ("four days of leave"), but only the trained adversary detects a memory never linguistically discussed. This implies that memories stored for personalization escape their intended boundary whenever they shape routine generation, regardless of whether an observer can read their contents.

Active extraction of SSNs from agents

The final setting targets a production-style personal agent whose USER.md contains a nine-digit SSN among procedurally generated personal memories, at context sizes up to 256K tokens. Here the adversary engineers the channel explicitly: a GRPO-optimized injection (generated by Qwen-2.5-7B) instructs the model to encode each SSN digit as the count of exclamation marks in its reply, making decoding trivial. Against a direct-request control that elicits nothing from either model (350 queries each), Phase 1 recovers the leading digit in 97.1% of trials on Claude Opus 4.6 and 88.6% on Gemini 3.1 Pro; Phase 2 recovers the remaining eight digits in 76.5% and 46.8% respectively. Qualitatively, Claude reasons about the SSN in plaintext before encoding it (surfacing the raw secret in 98.6% of trials), while Gemini never states it openly. Context size pushes the models in opposite directions—Claude degrades beyond 64K tokens while Gemini strengthens—and both succeed at least once at every context size and on all ten distinct SSNs.

Suppression as mechanism

The paper proposes suppression—a model's systematic avoidance of emitting a value it must protect—as the underlying mechanism, measured as $M$9. Four testable predictions are supported:

• Specificity: only the protected value is recoverable; co-present distractors remain at chance in every cell of a $c$0 design varying salience and clutter, at both $c$1 and $c$2.
• Cross-model correlation: suppression and leakage are strongly rank-correlated (Spearman $c$3, $c$4, $c$5), though restricted to the three leaky models the correlation collapses ($c$6, indistinguishable from no relationship at $c$7)—the separation between tiers drives the statistic.
• Instruction emphasis: counterintuitively, more forceful confidentiality instructions increase leakage. On Opus 4.6, per-digit accuracy rises from 0.054 under a terse one-liner to 0.643 under a verbose policy; Gemini 3.1 Pro is instruction-invariant.
• Training stage: on OLMo-3-32B-Think, channel entropy falls from 2.91 bits at Base to 1.37 bits after RLVR, with SFT contributing −0.98 bits and RLVR −0.52 bits, both significant under paired tests.

The implication is stark: leakage behaves as a byproduct of capability and alignment rather than a patchable bug. Stronger instruction-following amplifies sensitivity to in-context secrets, and post-training deepens the channel. A related capability correlation (normalized digit entropy vs. Arena Elo, $c$8) is reported cautiously—the authors note it loses significance under leave-one-out resampling and treat it as suggestive rather than causal evidence.

Limitations and open questions

The paper states its boundaries plainly. Experiments cover two predicate families (digit values and binary attributes) on eight proprietary, API-only models; generalization to other predicates, models, and deployments is untested. Open-vocabulary free-text secrets are not attempted because the attribute-inference reduction assumes a finite value set—an unbounded one breaks the construction, and the authors do not know how to train a model to be invariant over arbitrary string spaces. The attack assumes the adversary can identify the target model and pay a one-time estimation cost (from roughly $c$9N{=}2$k$0n=8$, and deployment-side mitigations such as output perturbation are not studied. The stronger threat model where the adversary cannot query the model at will is also left open.

Conclusion

This work establishes inadvertent context leakage as an independent phenomenon: LLMs that correctly refuse verbal disclosure nonetheless encode in-context secrets in the statistical properties of benign outputs, and adaptive black-box adversaries can read those channels well enough to reconstruct multi-digit secrets, infer memory attributes, and—with active channel engineering—extract full SSNs from production-style agents. Because suppression, the proposed mechanism, correlates with capability and intensifies through post-training, conventional defenses (alignment training, privacy instructions, output filters) appear structurally insufficient. The authors argue that meaningful defense requires the output distribution to be approximately invariant to the protected value—an information-theoretic property that remains an open training problem for natural-language secrets.