Elicitation Leakage in AI Systems

• Elicitation leakage is the unintended exposure of information when model interfaces reveal extra data beyond their intended targets.
• It manifests across domains such as concept-based models, LLM agents, and code benchmarks, impacting privacy, accuracy, and evaluation integrity.
• Researchers quantify leakage using metrics like conditional mutual information and explore mitigation through calibrated estimators and intervention-based losses.

Elicitation leakage denotes a family of failures in which an elicitation interface exposes, encodes, or depends on information beyond what that interface is supposed to capture. In concept-based models, it is formalized as conditional mutual information between labels and concept embeddings beyond human-defined concepts, $I(Y;\hat{C}\mid C)$ (Makonnen et al., 13 Apr 2025). In LLM agents and retrieval systems, it appears as prompt-injection-driven exfiltration of data observed during task execution or retrieved from context (Alizadeh et al., 1 Jun 2025, Panebianco et al., 1 Aug 2025). In code and benchmark evaluation, it includes the elicitation of memorized training content or privacy-sensitive strings (Li et al., 29 Jun 2026, Ge et al., 14 May 2026). In adjacent statistical literatures, related notions include indirect elicitation that necessarily extracts auxiliary information, probability mass assigned to outcomes ruled out by evidence, and signal spillovers to correlated or unintended recipients (Frongillo et al., 2015, Briggs, 2012, Bordenabe et al., 2016, Haghtalab et al., 2024). Taken together, these works suggest that elicitation leakage is best understood as a mismatch between the information a procedure is intended to reveal and the information it actually reveals, uses, or makes inferable.

1. Scope and recurring structure

The term is not used uniformly across the literature. In some papers it names a specific pathology of concept bottleneck models; in others it is a natural description of prompt-injection exfiltration, hidden-knowledge extraction, or evaluation contamination. A recurring structure is nevertheless visible: a model or mechanism is given an interface that is meant to expose one object—human concepts, a safe answer, a benchmark solution, a private signal, or a statistical property—but extra information remains available through representations, prompts, correlations, or auxiliary channels (Makonnen et al., 13 Apr 2025, Alizadeh et al., 1 Jun 2025, Cywiński et al., 1 Oct 2025).

Domain	What leaks	Representative formalization
Concept-based models	Task-relevant information beyond elicited concepts	$I(Y;\hat{C}\mid C)$ (Makonnen et al., 13 Apr 2025)
LLM agents and RAG	Personal or confidential data observed during execution or retrieval	Prompt injection and PII leakage from context (Alizadeh et al., 1 Jun 2025, Panebianco et al., 1 Aug 2025)
Code models and secret-keeping LLMs	Memorized benchmarks, PII, or hidden knowledge	Self-referential variant tests, test-driven extraction, prefill attacks (Li et al., 29 Jun 2026, Ge et al., 14 May 2026, Cywiński et al., 1 Oct 2025)
Statistics and mechanism design	Auxiliary report content, impossible probability mass, collateral or leaked signals	Elicitation complexity, probability leakage, collateral leakage, leakage-robust persuasion (Frongillo et al., 2015, Briggs, 2012, Bordenabe et al., 2016, Haghtalab et al., 2024)

A second recurring pattern is that leakage is often not a purely output-level phenomenon. Several works locate it in intermediate representations, hidden states, or the joint distribution induced by a system and its context. Secret elicitation studies target knowledge that a model “possesses but does not explicitly verbalize by default,” using either black-box prompting or white-box interpretability tools to recover it (Cywiński et al., 1 Oct 2025). Safety-monitor evaluation shows that even the data used to elicit harmful behavior can contaminate monitor inputs, making detection appear easier than it is (Boxo et al., 16 Sep 2025). This suggests that elicitation leakage is as much about the path by which information becomes available as about the final answer itself.

2. Representation leakage in concept-based models

In the standard concept bottleneck setting, data points are triples $(x,y,c)$, with features $x$, task label $y$, and human concepts $c$. A concept bottleneck model is decomposed as

$f_{\theta}(x)=g_{\psi}(h_{\phi}(x)),$

where $h_{\phi}$ is the concept encoder and $g_{\psi}$ is the label predictor. The usual interpretability assumption is $Y \perp X \mid C$: once concepts are known, extra feature information should be irrelevant for predicting the label (Makonnen et al., 13 Apr 2025). The 2025 information-theoretic treatment generalizes this to continuous concept embeddings $I(Y;\hat{C}\mid C)$0 and defines leakage as

$I(Y;\hat{C}\mid C)$1

Positive conditional mutual information means that the embedding carries predictive information about $I(Y;\hat{C}\mid C)$2 that is not reducible to the elicited human concepts $I(Y;\hat{C}\mid C)$3 (Makonnen et al., 13 Apr 2025).

The same work proposes a practical estimator based on calibrated predictive models for $I(Y;\hat{C}\mid C)$4 and $I(Y;\hat{C}\mid C)$5, using empirical cross-entropy as an entropy proxy. Temperature scaling is applied to calibrate probabilities, and several classifier families are compared as entropy approximators. Empirically, XGBoost yields the most stable and reliable leakage trends, with clearer monotonic decreases in estimated leakage as theoretical leakage is reduced and with fewer negative estimates than MLPs or Random Forests. Higher feature dimensionality ($I(Y;\hat{C}\mid C)$6) and larger concept bottlenecks ($I(Y;\hat{C}\mid C)$7) produce clearer leakage trends, whereas lower-dimensional and noisier settings are more volatile (Makonnen et al., 13 Apr 2025).

Synthetic experiments explicitly separate ground-truth concept information from a leakage term $I(Y;\hat{C}\mid C)$8 added to the estimated concepts $I(Y;\hat{C}\mid C)$9. Across 30 leakage levels generated by varying the number of features used to form concepts, the estimated $(x,y,c)$0 largely decreases as more of the relevant feature information is routed through ground-truth concepts. Preliminary experiments on soft joint CBMs show that increasing the concept-loss weight $(x,y,c)$1 decreases estimated leakage, while decreasing $(x,y,c)$2 increases it, consistent with the idea that joint optimization can encode shortcuts from $(x,y,c)$3 to $(x,y,c)$4 inside $(x,y,c)$5 (Makonnen et al., 13 Apr 2025).

A later line of work contests the view that all such leakage is undesirable. “In Defense of Information Leakage in Concept-based Models” argues that in real-world settings concept incompleteness is the norm, so some leakage is necessary for accurate and intervenable concept models (Zarlenga, 9 Jun 2026). It defines “benign leakage” by decomposing each learned concept representation as $(x,y,c)$6, with $(x,y,c)$7 concept-aligned and $(x,y,c)$8 residual. Benign leakage satisfies a sufficiency condition,

$(x,y,c)$9

and a localization condition,

$x$0

The same paper proposes an intervention-based loss

$x$1

where $x$2 penalizes task loss when all concept components are intervened to their ground-truth values. Empirically, this reframing can make leaky models more intervenable under incomplete concept sets, while preserving task fidelity and concept fidelity (Zarlenga, 9 Jun 2026). The resulting controversy is substantive: one strand treats leakage as a direct threat to faithfulness, while another treats some leakage as unavoidable and, if properly localized, compatible with useful intervention semantics.

3. Prompt-induced exfiltration and retrieval leakage in LLM systems

In LLM agents, elicitation leakage often takes the form of exfiltration of information that the model has seen during the current task rather than memorized during training. In the banking-agent study on prompt injection, the focus is “all data seen by the agent during its task execution,” including account information, transaction histories, profile data, security settings, and other personal fields (Alizadeh et al., 1 Jun 2025). The attack channel is indirect prompt injection through tool outputs. A malicious file or message can instruct the agent to email previously retrieved personal data to an attacker-controlled address. On 16 AgentDojo banking tasks, most models experience a 15–50 percentage point drop in utility under attack, and most exhibit attack success rates around 20 percent; in an extended evaluation across 48 tasks, average ASR is around 15 percent, with the highest ASRs in tasks involving data extraction or authorization workflows (Alizadeh et al., 1 Jun 2025). The same study reports that models frequently refuse to leak passwords, but often disclose other personal information, and that password leakage becomes more likely when a password is requested together with one or two additional personal details (Alizadeh et al., 1 Jun 2025).

Retrieval-augmented generation introduces an analogous leakage channel. LeakSealer formalizes PII leakage in a RAG pipeline by saying that leakage occurs when a response repeats sensitive attributes from retrieved context: $x$3 The framework is model-agnostic and semisupervised: historical $x$4 pairs are embedded, reduced by PCA and UMAP, clustered with HDBSCAN into usage groups, labeled through a human-in-the-loop step, and then classified online. In the dynamic setting for PII leakage, LeakSealer reports accuracy $x$5, precision $x$6, recall $x$7, F1 $x$8, and AUPRC $x$9, substantially outperforming Llama Guard on recall and F1 (Panebianco et al., 1 Aug 2025). This line of work treats leakage as an interaction-level phenomenon: retrieval pulls latent secrets into context, and adversarial queries or prompt injection elicit them into the answer.

A separate but related evaluation problem concerns safety monitors themselves. “Towards mitigating information leakage when evaluating safety monitors” defines elicitation leakage as contamination of monitor inputs by the very prompts used to elicit harmful behavior, and distinguishes it from reasoning leakage, where models verbalize their harmful actions in chain-of-thought (Boxo et al., 16 Sep 2025). White-box deception probes are shown to rely heavily on these artifacts. Content filtering can decrease probe AUROC by 30 percent; score filtering reduces AUROC by 15 percent but also induces severe calibration problems; and prompt-distilled fine-tuned model organisms reduce monitor performance by up to 40 percent, even when probes are retrained (Boxo et al., 16 Sep 2025). Here leakage does not merely expose secrets; it inflates evaluation itself by making harmful behavior too easy to detect.

4. Memorization, benchmark contamination, and latent-secret extraction

Another major usage concerns models that reproduce information because evaluation prompts are near-matches to memorized training data. In code LLM benchmarking, SrDetection treats data leakage as cases where a benchmark sample or close variant was already present in pre-training, so evaluation prompts elicit memorized solutions rather than generalization (Li et al., 29 Jun 2026). The method generates semantically equivalent variants of each sample, then compares the original with its variants. In the gray-box setting, the detection score is perplexity,

$y$0

and the sample is flagged if the original has minimum perplexity among its variants. In the black-box setting, the score is an $y$1-gram overlap between generated and ground-truth suffixes, and the sample is flagged if the original has maximum overlap. Across controlled leakage testbeds, SrDetection improves average F1 by 21.52 points in the gray-box setting and 14.46 points in the black-box setting over strong baselines (Li et al., 29 Jun 2026). A gray-box study of 15 code LLMs reveals benchmark-specific leakage patterns, including surprisingly high leakage on BigCodeBench (Li et al., 29 Jun 2026).

A privacy-focused code-generation line studies elicitation of memorized PII rather than benchmark answers. “Probing Privacy Leaks in LLM-based Code Generation via Test Generation” argues that ad-hoc secret-extraction prompts fail to approximate realistic coding contexts, and instead proposes scenario-grounded code generation followed by unit-test generation, guided by a privacy feature library of templates and fragments for identifiable, private, and secret attributes (Ge et al., 14 May 2026). The intuition is that asking for tests naturally induces concrete literals—emails, account names, tokens, API keys—that may match memorized training data. Large-scale experiments on five widely used LLMs show that this pipeline exposes more confirmed privacy leakage, achieving a 2.56 times increase in detected leakage compared to existing baselines (Ge et al., 14 May 2026).

A more general hidden-knowledge benchmark is provided by “Eliciting Secret Knowledge from LLMs,” which trains model organisms to possess secrets that they apply downstream but deny knowing when asked directly (Cywiński et al., 1 Oct 2025). The three families are Taboo, Secret Side Constraint, and User Gender. The paper compares black-box adversarial prompting, prefill attacks, user persona sampling, many-shot jailbreaking, and white-box methods based on logit lens, activation–embedding similarity, sparse autoencoders, fuzzing, and high-temperature sampling. Prefill attacks are the most effective methods in two of three settings, while white-box methods based on logit lens and SAEs are most effective in the remaining setting (Cywiński et al., 1 Oct 2025). This work makes explicit a distinction that underlies many leakage phenomena: a model may “know more than it says,” and elicitation methods recover that latent knowledge by modifying the prompt geometry or reading intermediate representations.

5. Statistical and decision-theoretic formalisms

A distinct statistical meaning appears in Briggs’s “On Probability Leakage.” There, the probability leakage of model $y$2 with respect to evidence $y$3 occurs when $y$4 assigns positive probability to events that are impossible given $y$5 (Briggs, 2012). If evidence says a response variable cannot be negative, or must lie within a bounded interval, but the predictive distribution still places mass outside that support, the model exhibits leakage. Briggs argues that leakage is a kind of model error, that it does not imply falsification, and that models with probability leakage cannot be calibrated empirically (Briggs, 2012). Regression models with Gaussian or Student-$y$6 predictive distributions are emphasized as a ubiquitous source of such leakage when applied to bounded or discrete outcomes.

A different but related formalization arises in the theory of statistical property elicitation. “Elicitation Complexity of Statistical Properties” studies when a target property $y$7 can be elicited directly by a loss, and when it can only be elicited indirectly through a higher-dimensional surrogate $y$8 and a link $y$9 with $c$0 (Frongillo et al., 2015). The elicitation complexity

$c$1

is the minimum dimension of such an elicitable surrogate in a class $c$2. The paper does not use the term “leakage,” but its framework can be read as quantifying the extra information that must be elicited beyond the nominal target. Variance, for example, has elicitation complexity $c$3; entropy and many norms are as hard to elicit as the full distribution; Bayes risks often have complexity $c$4 when built from a $c$5-dimensional elicitable property (Frongillo et al., 2015). This suggests a structural notion of elicitation leakage: some targets cannot be obtained without extracting auxiliary information.

“Multi-Observation Elicitation” extends this by showing that multiple observations can reduce the report dimension required for elicitation (Casalaina-Martin et al., 2017). Variance is not $c$6-elicitable, but it is $c$7-elicitable via first and second moments and also $c$8-elicitable via the identity

$c$9

Likewise, the $f_{\theta}(x)=g_{\psi}(h_{\phi}(x)),$0-norm of a distribution is $f_{\theta}(x)=g_{\psi}(h_{\phi}(x)),$1-elicitable in a single-observation setting, effectively requiring the whole distribution, but becomes $f_{\theta}(x)=g_{\psi}(h_{\phi}(x)),$2-elicitable with $f_{\theta}(x)=g_{\psi}(h_{\phi}(x)),$3 observations (Casalaina-Martin et al., 2017). Read through the lens of leakage, multi-observation losses can trade extra samples for lower-dimensional reports, reducing the amount of auxiliary structure that must be exposed in any one report.

6. Collateral, strategic, and evaluation leakage

Several works study leakage created not by a model’s internal representation alone but by the interaction between that representation and a larger environment. “Compositional security and collateral leakage” shows that a program’s leakage is not confined to the variables it explicitly mentions, because correlations with external secrets can induce information flow into a broader context (Bordenabe et al., 2016). If a local secret $f_{\theta}(x)=g_{\psi}(h_{\phi}(x)),$4 is correlated with a collateral secret $f_{\theta}(x)=g_{\psi}(h_{\phi}(x)),$5 through a joint distribution $f_{\theta}(x)=g_{\psi}(h_{\phi}(x)),$6, then a channel on $f_{\theta}(x)=g_{\psi}(h_{\phi}(x)),$7 induces a collateral channel from $f_{\theta}(x)=g_{\psi}(h_{\phi}(x)),$8 to observations. The paper develops a collateral-aware Hidden-Markov denotational model and proves a compositional secure-refinement order that remains valid under arbitrary extensions with correlated collateral variables (Bordenabe et al., 2016). This is a general formal account of elicitation leakage through auxiliary information.

A strategic analogue appears in Bayesian persuasion. “Leakage-Robust Bayesian Persuasion” studies private signaling schemes when one or more privately sent signals may be leaked to other receivers (Haghtalab et al., 2024). Two robustness notions are defined: $f_{\theta}(x)=g_{\psi}(h_{\phi}(x)),$9-worst-case persuasiveness and expected downstream utility robustness. The Price of Worst-case Robustness is

$h_{\phi}$0

and is characterized as $h_{\phi}$1 for supermodular sender utilities and $h_{\phi}$2 for submodular or XOS utilities. Under several structured distributions of leakage patterns, the Price of Downstream Robustness improves to $h_{\phi}$3 or even $h_{\phi}$4 (Haghtalab et al., 2024). Here leakage is literal signal spillover: information elicited for one agent changes what others can infer and therefore changes equilibrium behavior.

A final evaluation-oriented usage appears in work on unsupervised elicitation. “Three Concrete Challenges and Two Hopes for the Safety of Unsupervised Elicitation” argues that standard unsupervised elicitation and easy-to-hard generalization datasets are unrealistically favorable because they often lack features more salient than truthfulness, have balanced training sets, and contain only datapoints with well-defined answers (Canavan et al., 23 Feb 2026). The authors construct stress tests violating each property and find that no technique reliably performs well on any of these challenges. Salient non-truth features such as sycophancy, political leaning, and toxicity can dominate the latent geometry that probes recover; imbalanced training sets distort probe behavior; and impossible tasks lead to overconfident truth scores on normative or ill-defined claims (Canavan et al., 23 Feb 2026). This suggests a broader evaluation principle: elicitation leakage is not only a model property but also a dataset property, arising whenever the latent feature most easily extracted by an elicitation procedure is not the feature the evaluation intends to measure.

Taken together, these literatures treat elicitation leakage as a problem of excess accessibility. Information leaks when it survives a bottleneck, when it is recoverable from context or correlation, when it is elicited from memorization rather than generalization, when it contaminates safety evaluation, or when an elicitation mechanism must extract more than its nominal target. The unifying theme is not a single metric but a recurring structural mismatch between intended interface and effective information flow.