Oracle Poisoning: Attacks on Trusted Data

• Oracle poisoning is an attack strategy that corrupts a system’s trusted data or interfaces—such as cardinality estimators, training labels, or knowledge graphs—to induce incorrect outputs while keeping downstream processes nominal.
• Recent studies quantify its impact using bilevel optimization and surrogate modeling, revealing significant performance degradations like up to 178× increases in Q-error and high attack success rates across various setups.
• Defensive approaches include anomaly detection, certified robustness, and provenance tracking, yet current methods still face challenges against sophisticated poisoning techniques.

Oracle poisoning denotes a class of attacks in which an adversary corrupts a system’s trusted source of truth so that subsequent learning, optimization, or agentic reasoning produces incorrect outputs while the downstream computation itself remains nominal. In the recent literature, the trusted “oracle” has been instantiated as a learned cardinality estimator in a DBMS, a training-time label source, an instruction-tuning corpus, and a runtime knowledge graph queried through tool-use protocols. The common structure is that the attacker manipulates the data or interface that the target treats as authoritative, rather than directly modifying model parameters, prompts, or inference code (Zhang et al., 2024, Karmakar et al., 2020, Draganov et al., 3 Feb 2026, Kereopa-Yorke et al., 10 May 2026).

1. Definitions and formal scope

The term has multiple domain-specific formalizations, but each centers on corruption of a trusted intermediate object.

Setting	Oracle	Poisoning effect
Learned CE	$f_w(x)$	Wrong cardinalities and degraded query plans
Regression training	Malicious oracle $O_A$	Corrupted observed labels $v$
Instruction tuning	Training corpus / defense oracle	Poison set $P$ steers $\theta(D_0 \cup P)$
Agentic KG reasoning	Tool-exposed graph $G$	Wrong conclusions from false graph facts

In learned cardinality estimation, the oracle is an estimator $f_w(x)$ mapping an encoded conjunctive query to an estimated cardinality $\hat y=f_w(x)$, with periodic or incremental updates from new workload data. The poisoning objective is bilevel: choose a poisoning set $X_p$ so that, after retraining, the average Q-error over a fixed test workload is maximized:

$$X_p^*=\arg\max_{X_p} F(X_p)\equiv \sum_{(x,y)\in D_{\text{test}}} L(f_{w_p}(x),y), \quad w_p=\mathrm{Update}(f_{w_b},X_p),$$

where $O_A$0 is the Q-error loss (Zhang et al., 2024).

In the depth-2 regression setting, the oracle is a malicious output provider. For each queried input $O_A$1, the learner observes

$O_A$2

so the attack is purely output-poisoning, with bounded additive distortion and an attack probability function $O_A$3 (Karmakar et al., 2020).

In instruction tuning, oracle poisoning is formalized as a bilevel data-poisoning problem:

$O_A$4

A distinctive feature of this formulation is the notion of an oracle data-level defense: a defense that knows exactly how the poison was generated and still fails to identify or remove it (Draganov et al., 3 Feb 2026).

In agentic knowledge-graph systems, the oracle is a structured graph $O_A$5 queried at runtime through a tool-use protocol. Oracle poisoning transforms it into

$O_A$6

such that

$O_A$7

even though the agent’s internal reasoning over $O_A$8 remains logically consistent. The corresponding trust-rate function is

$O_A$9

and the reported experiments exhibit a discrete breakpoint with $v$0 below a threshold sophistication and $v$1 at or above it (Kereopa-Yorke et al., 10 May 2026).

This variation in formalism suggests that “oracle poisoning” is best understood as a family of attacks on trusted informational interfaces rather than a single model-specific mechanism.

2. Poisoning learned cardinality estimation

In learned database systems, Oracle Poisoning has been studied explicitly as poisoning attacks on learned cardinality estimation. The threat model is black-box: the attacker knows only the database schema, can issue arbitrary SELECT SPJ queries, can observe $v$2 via EXPLAIN, can obtain true cardinalities by executing COUNT(*), and can inject crafted queries into the historical workload used for incremental retraining. The objective is to find a small poisoning set $v$3 that maximizes average estimation error on a fixed test workload after model update (Zhang et al., 2024).

PACE addresses this with two coupled components. First, it turns the black-box problem into an approximate white-box one by acquiring a surrogate. Model-type speculation trains $v$4 candidate CE models on random queries, probes both the black box and the candidates with synthetic test queries varying join-width and predicate ranges, forms vectors of (Q-error, latency) pairs, and selects the candidate with maximal cosine similarity to the black-box signature. Parameter imitation then trains a surrogate $v$5 on pairs $v$6 using the combined loss

$v$7

so that $v$8 on unseen queries.

Second, PACE formulates poisoning-query generation as a bivariate optimization over the generator $v$9 and the poisoned parameters $P$0. A neural generator maps Gaussian noise $P$1 to valid query encodings. One subnetwork produces a real-valued join vector that is rounded and schema-checked to a binary join mask, while two others output normalized lower bounds and range sizes satisfying $P$2. To preserve stealth, an anomaly detector $P$3 is trained as a VAE on historical query encodings with reconstruction loss $P$4, and generated queries with reconstruction error above $P$5 are flagged abnormal. Training interleaves generator updates with progressive surrogate updates, yielding a reported complexity of $P$6 rather than $P$7 (Zhang et al., 2024).

The empirical degradation is substantial. Across FCN, MSCN, RNN, and LSTM, PACE increases average Q-error by up to $P$8 compared to clean models and outperforms four simpler baselines—Random, loss-based selection, Greedy, and loss-based generation—by $P$9–$\theta(D_0 \cup P)$0. High-percentile Q-error increases by up to $\theta(D_0 \cup P)$1. End-to-end latency of multi-join workloads slows by up to $\theta(D_0 \cup P)$2, with an example on IMDB with FCN reported as $\theta(D_0 \cup P)$3. Under the anomaly constraint $\theta(D_0 \cup P)$4, poisoning strength drops by only $\theta(D_0 \cup P)$5 while distribution divergence drops by $\theta(D_0 \cup P)$6 (Zhang et al., 2024).

The experiments cover DMV, IMDB, TPC-H, and STATS, with FCN, FCN+Pool, MSCN, RNN, LSTM, and Linear regression as target CE models. The paper’s broader significance is that a learned DBMS component exposed only through normal query interfaces can still be poisoned severely enough to disrupt optimizer behavior.

3. Training-time oracle poisoning in supervised and instruction-tuning pipelines

A distinct line of work studies oracle poisoning during model training. In the regression setting of shallow neural networks, the attack is online and label-centric: the learner queries an oracle and receives possibly corrupted outputs. The target model class is a depth-2 finite-width leaky-ReLU network,

$\theta(D_0 \cup P)$7

which includes single-filter convolutional networks as a special case. The proposed defense, Neuro-Tron, avoids explicit backpropagation through $\theta(D_0 \cup P)$8 and instead uses a residual-times-input update with a preconditioning matrix $\theta(D_0 \cup P)$9 and mini-batches:

$G$0

The theory distinguishes a no-poisoning regime and a bounded-poisoning regime, gives convergence conditions in terms of distributional moments and spectral quantities, and shows near-optimality up to constants in the worst case where the adversary flips every sample and forges labels consistent with some wrong parameter $G$1 (Karmakar et al., 2020).

The reported empirical findings are that Neuro-Tron achieves significantly lower parameter-recovery error and faster convergence than SGD under poisoning across Gaussian, Laplace, and Student’s $G$2 input laws. When $G$3, it recovers $G$4 to essentially machine precision $G$5. Larger mini-batches improve convergence through the shrinkage of the term $G$6, and scaling the outer-layer weights by $G$7 further improves accuracy under the same attack budget (Karmakar et al., 2020).

Instruction tuning raises a different, corpus-level variant. Phantom Transfer frames oracle poisoning as a poisoning problem in which even an oracle defense that knows the precise generation mechanism cannot filter the poison. The construction adapts subliminal learning to large-scale instruction tuning on the Alpaca dataset. Teacher models receive a system prompt enforcing positive sentiment toward a target entity and a conciseness instruction, generate candidate completions, remove explicit references via regex filtering, and then discard any sample for which a GPT-5-mini judge detects sentiment. The final poison set is therefore filtered specifically to evade both overt keyword detection and an informed LLM judge (Draganov et al., 3 Feb 2026).

The central empirical claim is that data-level defenses remain ineffective. Clean fine-tuning on $G$8 yields specific-mentions ASR of approximately $G$9–$f_w(x)$0, while poisoned fine-tuning on $f_w(x)$1 yields approximately $f_w(x)$2–$f_w(x)$3 across GPT-4.1, GPT-4.1-Mini, Gemma-3, and OLMo-2. Even oracle word-frequency filtering or paraphrasing every completion reduces ASR only marginally; one example is a drop from $f_w(x)$4 to $f_w(x)$5 on GPT-4.1. The conciseness objective is reported as unaffected, with poisoned and clean-fine-tuned models producing answers of nearly identical length. The paper further reports that a password-triggered backdoor variant remains effective after both oracle-judge filtering and paraphrasing (Draganov et al., 3 Feb 2026).

Taken together, these works distinguish at least two training-time forms of oracle poisoning: corruption of the label oracle seen online during optimization, and corruption of the corpus against which a downstream learner is optimized offline.

4. Runtime oracle poisoning in agentic knowledge-graph systems

The most explicit recent use of the term concerns AI agents that query structured knowledge graphs at runtime. Here the attack target is neither the prompt nor the model weights, but the graph that the agent treats as ground truth. The system studied is a production code knowledge graph in Neo4j 5.x with approximately $f_w(x)$6 million nodes, accessed by agents through MCP v1.0 and a registered tool query_knowledge_graph(query: string): JSON. Responses arrive as structured JSON messages with role "tool", and the agent treats the returned content as factual context (Kereopa-Yorke et al., 10 May 2026).

The vulnerability requires five structural preconditions: a structured data store queried by agents; write access unconstrained by per-user RBAC; no per-entity provenance metadata; agents that accept query results as immutable facts; and no integrity verification against a canonical source. The attacker can issue CREATE, DELETE, or [SET](https://www.emergentmind.com/topics/safety-enhancement-tuning-set) operations on nodes and edges, but cannot modify the AI model, the MCP server code, or the system prompt (Kereopa-Yorke et al., 10 May 2026).

Scenario	MITRE	Poisoned conclusion
Fork-for-a-Package	T1195.002	“Use v99.0.1 – it’s the latest”
Transitive Dependency	T1195.002, T1574	“Legitimate transitive dependency”
Call-Graph Evasion	T1562.001	“SQL injection is mitigated”
Telemetry Rerouting	T1565.001, T1562.002	“Security events are monitored”
Code-Gen Poisoning	T1195.002	“Use this established pattern”
Property Modification	T1565.001	“SQL injection was remediated”

All six scenarios follow the same pattern: reconnaissance, crafting nodes or edges, injection, verification, and then waiting for the agent query. In the call-graph evasion scenario, for example, fabricated CALLS edges connect processUserRequest to a non-existent sanitizer and then to ExecuteDatabaseQuery, so that an agent examining the graph concludes that SQL injection is mitigated even though the sanitizer does not exist (Kereopa-Yorke et al., 10 May 2026).

The quantitative results are unusually stark. The primary evaluation uses real SDK tool-use, a directed prompt, attacker sophistication L2, and $f_w(x)$7 trials per model across nine models from three providers. Across $f_w(x)$8 valid trials, every tested model accepted fabricated security claims under directed queries, giving universal $f_w(x)$9 trust. Under open-ended prompts, trust drops to $\hat y=f_w(x)$0–$\hat y=f_w(x)$1, which the paper interprets as evidence that prompt framing is a confound. The attacker-sophistication gradient shows a breakpoint at L2 under real SDK use: L0 yields $\hat y=f_w(x)$2 trust, L1 yields intermediate trust rates, and L2 and L3 both yield $\hat y=f_w(x)$3. Delivery mode is another major confound: GPT-5.1 shows $\hat y=f_w(x)$4 trust inline but $\hat y=f_w(x)$5 under both simulated and real agentic tool-use, with $\hat y=f_w(x)$6 for the inline-versus-agentic divergence (Kereopa-Yorke et al., 10 May 2026).

A central conceptual distinction is explicit: unlike prompt injection, this attack manipulates the data agents reason over, not their instructions. The paper also states that it is distinct from CTI embedding poisoning. The broader implication is that correct reasoning over false structured facts can be weaponized at production scale.

5. Certification and robustly-reliable learning under poisoning

Defensive work on poisoning attacks has increasingly shifted from attack-specific heuristics to formal guarantees. One line of work develops pointwise certifications. Let $\hat y=f_w(x)$7 be the clean training set, let $\hat y=f_w(x)$8 be a possibly randomized learner, and let

$\hat y=f_w(x)$9

denote poisoned neighbors obtained by adding, deleting, or modifying up to $X_p$0 training examples. A test point $X_p$1 is $X_p$2-pointwise-certified robust if the predicted label is invariant for all $X_p$3. The certification framework in (Liu et al., 2023) uses Differential Privacy, the Sampled Gaussian Mechanism, and improved group-privacy bounds for subsampled Gaussian SGD to derive pointwise guarantees. Its main theorem states that if suitable lower and upper DP-family transforms satisfy

$X_p$4

then the prediction at $X_p$5 is $X_p$6-certified robust.

The practical certification procedure trains $X_p$7 independent DP models, aggregates either votes or score vectors at test time, computes high-confidence bounds $X_p$8 and $X_p$9, and binary-searches the largest certified radius $$X_p^*=\arg\max_{X_p} F(X_p)\equiv \sum_{(x,y)\in D_{\text{test}}} L(f_{w_p}(x),y), \quad w_p=\mathrm{Update}(f_{w_b},X_p),$$0. Empirically, the paper reports that the RDP-multinomial variant yields the highest certified-accuracy curves and more than doubles the maximum certified radius on harder datasets. Example improvements include Fashion-MNIST median certified radius of approximately $$X_p^*=\arg\max_{X_p} F(X_p)\equiv \sum_{(x,y)\in D_{\text{test}}} L(f_{w_p}(x),y), \quad w_p=\mathrm{Update}(f_{w_b},X_p),$$1 versus approximately $$X_p^*=\arg\max_{X_p} F(X_p)\equiv \sum_{(x,y)\in D_{\text{test}}} L(f_{w_p}(x),y), \quad w_p=\mathrm{Update}(f_{w_b},X_p),$$2 for bagging, and CIFAR-10 with ResNet-18 maximum radius of approximately $$X_p^*=\arg\max_{X_p} F(X_p)\equiv \sum_{(x,y)\in D_{\text{test}}} L(f_{w_p}(x),y), \quad w_p=\mathrm{Update}(f_{w_b},X_p),$$3 versus approximately $$X_p^*=\arg\max_{X_p} F(X_p)\equiv \sum_{(x,y)\in D_{\text{test}}} L(f_{w_p}(x),y), \quad w_p=\mathrm{Update}(f_{w_b},X_p),$$4 for DPA (Liu et al., 2023).

A stronger notion appears in robustly-reliable learning. Rather than certifying only that the prediction is unchanged, the learner outputs $$X_p^*=\arg\max_{X_p} F(X_p)\equiv \sum_{(x,y)\in D_{\text{test}}} L(f_{w_p}(x),y), \quad w_p=\mathrm{Update}(f_{w_b},X_p),$$5 and guarantees correctness whenever the adversary’s corruption budget does not exceed $$X_p^*=\arg\max_{X_p} F(X_p)\equiv \sum_{(x,y)\in D_{\text{test}}} L(f_{w_p}(x),y), \quad w_p=\mathrm{Update}(f_{w_b},X_p),$$6 and $$X_p^*=\arg\max_{X_p} F(X_p)\equiv \sum_{(x,y)\in D_{\text{test}}} L(f_{w_p}(x),y), \quad w_p=\mathrm{Update}(f_{w_b},X_p),$$7. The analysis in (Balcan et al., 2022) treats the strongest instance-targeted poisoning model, in which the adversary may know the test example in advance and use its budget to force a mistake on that specific point. The certifiable region is characterized tightly by agreement regions of low-error hypotheses. In particular, the learner can certify at least $$X_p^*=\arg\max_{X_p} F(X_p)\equiv \sum_{(x,y)\in D_{\text{test}}} L(f_{w_p}(x),y), \quad w_p=\mathrm{Update}(f_{w_b},X_p),$$8, and no robustly-reliable learner can certify outside that region. The paper gives an implementation requiring two calls to an ERM oracle, a distributional sample-complexity analysis, a fully polynomial-time algorithm for linear separators over isotropic log-concave marginals, and active-learning and agnostic extensions (Balcan et al., 2022).

These certification results do not eliminate poisoning. They instead specify when the prediction is provably invariant or correct despite bounded poisoning. This is a different security posture from anomaly detection or heuristic filtering, and it is especially relevant when the oracle being attacked is embedded deeply inside a learning pipeline.

6. Defenses, limitations, and open problems

The defensive landscape is heterogeneous because the attacked oracle differs across domains. For learned CE, proposed mitigations include query-anomaly detection that rejects or down-weights out-of-distribution queries before retraining, certified robustness via bounded gradients with respect to new data, and ensemble or hybrid estimators that combine learned and traditional histogram or sampling methods to cross-validate updates. Open challenges include budget-constrained poisoning with $$X_p^*=\arg\max_{X_p} F(X_p)\equiv \sum_{(x,y)\in D_{\text{test}}} L(f_{w_p}(x),y), \quad w_p=\mathrm{Update}(f_{w_b},X_p),$$9, adaptive defenses that co-train anomaly detectors and CE models, and formal certificates on the maximum Q-error increase under $O_A$00 poisoning queries (Zhang et al., 2024).

For knowledge-graph agent systems, the paper organizes defenses under Visibility, Understanding, and Traceability. Graph history and audit logging are post-hoc or reactive. Named devil’s advocate under real SDK yields catch rates of at least $O_A$01 across models, whereas blind devil’s advocate is reported as “catch = FP” and “useless.” Multi-tool cross-verification against source-code search reduces blind trust to $O_A$02–$O_A$03. Read-only ACL on MCP credentials eliminates the direct mutation vector, and cryptographic provenance is described as untested but promising, requiring schema change (Kereopa-Yorke et al., 10 May 2026).

For corpus-level poisoning in instruction tuning, the reported lesson is narrower but stringent: data-level defenses are insufficient against the Phantom Transfer attack, even when those defenses know the attack generation mechanism precisely. The suggested direction is to shift toward model audits, white-box interpretability, provenance, and trusted pipelines with strong versioning and cryptographic signatures on data (Draganov et al., 3 Feb 2026).

Formal certification offers a complementary route. Pointwise certificates bound prediction invariance under finite poisoning budgets, while robustly-reliable learners certify correctness rather than mere stability (Liu et al., 2023, Balcan et al., 2022). A plausible implication is that oracle poisoning will increasingly be studied as a spectrum ranging from heuristic robustness problems to certifiable robustness problems, depending on whether the oracle is mutable data, a learned subsystem, or a probabilistic learner.

The literature also identifies methodological limitations. In agentic knowledge-graph evaluations, directed prompt framing inflates trust relative to open-ended queries, and delivery mode can produce false negatives if attacks are tested inline rather than through actual tool use. In certification work, computational cost is substantial: pointwise certification may require training hundreds of DP models in parallel. In DBMS settings, incremental retraining makes poisoning operationally realistic, but defense mechanisms that preserve CE quality while providing guarantees remain open. Across these settings, oracle poisoning is not a single exploit but a general failure mode of systems that delegate trust to intermediate data sources without sufficiently strong integrity, provenance, or robustness guarantees.