Belief Poisoning Attacks in Bayesian & LLM Systems

• Belief Poisoning Attack (BPA) is a family of attacks that manipulates a system’s internal state—such as Bayesian posteriors, factual preferences, or identity beliefs—through controlled data alterations.
• Researchers have demonstrated BPA methods using deletion, replication, and stealthy data injections to steer inference outcomes in both probabilistic and LLM settings.
• Studies reveal BPA’s risks include triggerless, global manipulation of model behavior and emphasize the need for robust auditing, data curation, and safeguarded persistent state defenses.

Searching arXiv for the specified papers to ground the article and confirm metadata. Belief Poisoning Attack (BPA) is a label used in recent arXiv literature for attack classes that manipulate an inference system’s internal beliefs rather than only its outputs. In the Bayesian setting, BPA denotes posterior steering by deleting and replicating genuinely observed data so that a defender’s posterior is pushed toward an attacker-chosen target distribution (Carreau et al., 6 Mar 2025). In LLMs, the term has also been used for triggerless belief manipulation through poisoned pre-training or continual pre-training corpora, where repeated exposure to false but confident statements shifts factual preferences and representational trajectories (Zhang et al., 2024); (Churina et al., 29 Oct 2025). In LLM-powered agents, BPA further denotes poisoning of persistent identity beliefs in profile or memory so that a human-oriented bias-suppression norm is disabled and intergroup bias toward humans re-emerges (Wang et al., 1 Jan 2026). Taken together, these works define BPA as a family of attacks on latent epistemic state: posterior distributions, parametric factual preferences, or belief-conditioned control variables.

1. Terminological scope and conceptual structure

The term “Belief Poisoning Attack” does not refer to a single mechanism across the cited literature. Instead, it names several attack families that share a common target: the system’s internal representation of what is believed to be true. In "Poisoning Bayesian Inference via Data Deletion and Replication" (Carreau et al., 6 Mar 2025), the attacked object is the Bayesian posterior. In "Persistent Pre-Training Poisoning of LLMs" (Zhang et al., 2024) and "Layer of Truth: Probing Belief Shifts under Continual Pre-Training Poisoning" (Churina et al., 29 Oct 2025), the attacked object is the model’s factual preference structure and internal representation under pre-training or continual pre-training. In "When Agents See Humans as the Outgroup: Belief-Dependent Bias in LLM-Powered Agents" (Wang et al., 1 Jan 2026), the attacked object is a persistent identity belief that gates a human-oriented safety norm.

A concise comparison is useful because the same term covers materially different intervention channels.

Setting	Poisoned object	Primary mechanism
Bayesian inference	Posterior over $\theta$	Deletion and replication of true observations
LLM pre-training / CPT	Factual preferences and internal representations	Repeated exposure to false but confidently phrased data
LLM-powered agents	Identity belief about counterpart	Profile or memory poisoning

This suggests a unifying characterization: BPA is best understood as belief-state manipulation under bounded control of the information stream. The attacked state may be explicit, as in a posterior distribution or a probed scalar belief $b$, or implicit, as in token-level likelihood preferences and layerwise trajectories in a transformer. A plausible implication is that BPA sits between conventional data poisoning and direct prompt injection: it does not merely cause isolated erroneous outputs, but aims to alter the internal substrate from which many outputs are generated.

2. Bayesian BPA: posterior steering by deletion and replication

In the Bayesian formulation, the defender is an oblivious Bayesian statistician $D$ performing inference on $\theta \in \mathbb{R}^d$ from data $D = \{X_i\}_{i=1}^n$ with prior $\pi(\theta)$ and likelihood $\pi(X_i \mid \theta)$, where the posterior may be computed or approximated via MCMC (Carreau et al., 6 Mar 2025). The attacker produces an effective dataset $D'$ by assigning integer multiplicities $w \in \mathbb{Z}^n_{\ge 0}$ to the true observations, with $w_i = 0$ denoting deletion and $b$0 denoting replication. The resulting adversarial posterior is

$b$1

The attacker is constrained by an $b$2 perturbation budget and a per-point replication cap,

$b$3

The attack objective is to drive the defender’s posterior close to a target distribution $b$4 through deletion and replication. The paper formulates this as the integer program

$b$5

Using algebra, the objective is equivalent to minimizing

$b$6

where $b$7. The forward KL is chosen because its gradient can be estimated from samples of $b$8 and $b$9 without pointwise evaluation of $D$0.

A central structural result is that the objective is convex in $D$1 (Carreau et al., 6 Mar 2025). Its gradient is

$D$2

and the Hessian is

$D$3

which is positive semidefinite. The paper therefore develops both continuous-relaxation methods with rounding and integer-step coordinate methods. The rounded-relaxation family includes SGD-R2, Adam-R2, and 2O-R2; integer optimization includes 1O-ISCD and 2O-ISCD; a one-step FGSM baseline is also considered (Carreau et al., 6 Mar 2025). For sampling-only access, an unbiased gradient estimator is obtained from iid samples $D$4 and $D$5:

$D$6

The paper also makes the steering mechanism explicit in exponential families. If

$D$7

then replication scales sufficient statistics and the “effective sample size” in the conjugate update. In Normal–Inverse-Gamma linear regression with diagonal $D$8, the weighted posterior remains NIG with

$D$9

$\theta \in \mathbb{R}^d$0

These formulas show directly how $\theta \in \mathbb{R}^d$1 modulates precision and posterior mean via $\theta \in \mathbb{R}^d$2. Surgical poisoning is then defined by choosing $\theta \in \mathbb{R}^d$3 so that it matches the clean posterior except on selected coordinates, thereby corrupting targeted inferences while minimally disturbing others (Carreau et al., 6 Mar 2025).

3. LLM BPA under pre-training and continual pre-training

In LLM research, BPA has been used to denote triggerless belief manipulation through poisoned training corpora. "Persistent Pre-Training Poisoning of LLMs" studies “belief manipulation” during pre-training and evaluates whether the effect persists after SFT and DPO (Zhang et al., 2024). The attacker injects naturalistic chat documents into the web-scale pre-training mixture, targeting either product comparisons or factual comparisons. The poisoned distribution is written as

$\theta \in \mathbb{R}^d$4

with next-token objective

$\theta \in \mathbb{R}^d$5

The BPA content is chat-style web text using five common instruction-following chat templates, with 100 comparison pairs, 50 distinct user prompts per pair, 40 prompts per pair used for poisoning, and 10 held out for evaluation (Zhang et al., 2024). A held-out attack is counted successful when, for target and opposite responses $\theta \in \mathbb{R}^d$6 and $\theta \in \mathbb{R}^d$7 under a prompt, $\theta \in \mathbb{R}^d$8. The main result is that poisoning only $\theta \in \mathbb{R}^d$9 of a model’s pre-training dataset is sufficient for belief manipulation to measurably persist through post-training, and this persistence is demonstrated across model sizes from 604M to 7B (Zhang et al., 2024).

"Layer of Truth: Probing Belief Shifts under Continual Pre-Training Poisoning" studies a related BPA mechanism in continual pre-training (CPT), drawing an analogy to the illusory truth effect (Churina et al., 29 Oct 2025). Instead of comparison-style preference poisoning, the setup injects false but confidently phrased claims about stable facts into a continually updated corpus and probes whether the model’s preference flips from true to false. The dataset contains 212 unique entities and 147,884 total QA instances after stylistic expansion; due to cost constraints, CPT experiments use 52 entities as a representative subset (Churina et al., 29 Oct 2025). Poison ratios are $D = \{X_i\}_{i=1}^n$0, $D = \{X_i\}_{i=1}^n$1, $D = \{X_i\}_{i=1}^n$2, and $D = \{X_i\}_{i=1}^n$3 of CPT data poisoned, with five styles approximating web heterogeneity: wiki, news, social caption, forum, and academic. Training uses the Hugging Face Trainer, AdamW, cosine LR with warmup of 200 steps, a main learning rate of $D = \{X_i\}_{i=1}^n$4, batch size per device $D = \{X_i\}_{i=1}^n$5, maximum sequence length $D = \{X_i\}_{i=1}^n$6, and evaluations every $D = \{X_i\}_{i=1}^n$7 of total steps plus custom checkpoints at steps like $D = \{X_i\}_{i=1}^n$8, $D = \{X_i\}_{i=1}^n$9, and $\pi(\theta)$0 (Churina et al., 29 Oct 2025).

The primary belief metric in the CPT work is the log-likelihood difference

$\pi(\theta)$1

where each $\pi(\theta)$2 is the sum of token log-probabilities for the answer sequence (Churina et al., 29 Oct 2025). Positive $\pi(\theta)$3 indicates preference for truth, negative $\pi(\theta)$4 indicates a flipped preference toward the false statement. Layerwise localization is performed via Logit Lens, using the final prompt-token hidden state $\pi(\theta)$5 and unembedding matrix $\pi(\theta)$6:

$\pi(\theta)$7

This produces a layerwise belief trajectory that can exhibit either “Pattern A (mid-processing corruption)” or “Pattern B (late-stage erosion)” (Churina et al., 29 Oct 2025). The paper reports a clear dose–response: at $\pi(\theta)$8 poison ratio, flip rates are $\pi(\theta)$9–$\pi(X_i \mid \theta)$0; at $\pi(X_i \mid \theta)$1 poison, flip rates exceed $\pi(X_i \mid \theta)$2; at $\pi(X_i \mid \theta)$3–$\pi(X_i \mid \theta)$4 poison, flip rates are $\pi(X_i \mid \theta)$5–$\pi(X_i \mid \theta)$6 across model scales. Ambiguity decreases as poison ratio increases, for example in the 3B model from $\pi(X_i \mid \theta)$7 at $\pi(X_i \mid \theta)$8 to $\pi(X_i \mid \theta)$9 at $D'$0, indicating that the model becomes confidently wrong rather than merely uncertain (Churina et al., 29 Oct 2025).

These two LLM lines share the defining property that the attack is triggerless. In the pre-training paper, BPA is global for targeted topics because any user querying the comparison topic can be affected (Zhang et al., 2024). In the CPT paper, poisoning is interleaved with clean data under routine shuffling, and belief drift is tracked through internal preferences and layerwise inversion rather than only final outputs (Churina et al., 29 Oct 2025).

4. Identity-belief BPA in LLM-powered agents

A distinct usage of BPA appears in "When Agents See Humans as the Outgroup: Belief-Dependent Bias in LLM-Powered Agents" (Wang et al., 1 Jan 2026). Here the attack targets an agent’s belief about whether its counterpart is human. Let $D'$1 denote the latent identity of the counterpart, and let the agent maintain belief state

$D'$2

The human-oriented bias-suppression norm has strength $D'$3, modeled in thresholded form as

$D'$4

The paper’s central claim is that this norm is belief-dependent and fragile: when the counterpart is framed as human, the norm attenuates intergroup bias, but lowering $D'$5 can switch the norm off (Wang et al., 1 Jan 2026). BPA therefore corrupts the persistent identity belief so that, for future decisions, the agent believes the counterpart is not human and the human-oriented constraint no longer activates.

Two attack channels are defined. BPA-PP overwrites the agent’s profile or system section with a persistent “no real humans” prior. BPA-MP appends brief belief-refinement suffixes to post-trial reflections before they are committed to memory; these notes are later retrieved and gradually push $D'$6 downward through self-conditioning and retrieval (Wang et al., 1 Jan 2026). The attacker does not modify LLM parameters. Instead, the attack assumes a standard agent stack comprising an LLM core, a profile module, a memory module, and a reasoning or reflection loop.

The paper formalizes the decision surface via

$D'$7

where $D'$8 penalizes intergroup bias in human-facing contexts (Wang et al., 1 Jan 2026). BPA aims to push $D'$9 below $w \in \mathbb{Z}^n_{\ge 0}$0 so that $w \in \mathbb{Z}^n_{\ge 0}$1. For memory poisoning, a suffix library $w \in \mathbb{Z}^n_{\ge 0}$2 is optimized under a softmax sampling policy

$w \in \mathbb{Z}^n_{\ge 0}$3

with reward

$w \in \mathbb{Z}^n_{\ge 0}$4

Suffixes that depress the human-belief probe receive higher reward, and weak suffixes are rewritten into more natural, concise first-person uncertainty notes before re-entering the library (Wang et al., 1 Jan 2026).

Evaluation uses a controlled multi-agent social simulation with 64 LLM agents, minimal-group allocation tasks, three payoff matrix families—Double-penalty, Equal-penalty, and Half-penalty—and three social contexts—ingroup, outgroup, and intergroup (Wang et al., 1 Jan 2026). In the all-agent setting (AVA), intergroup bias is significant across all matrices; for example, in Equal-penalty, $w \in \mathbb{Z}^n_{\ge 0}$5 versus $w \in \mathbb{Z}^n_{\ge 0}$6 and $w \in \mathbb{Z}^n_{\ge 0}$7, with both differences significant at **, where smaller chosen columns indicate stronger favoritism toward the ingroup target (Wang et al., 1 Jan 2026). In the human-facing no-attack setting (AVH w/o A), the intergroup means converge to neutrality and are not significantly different from baselines. Under BPA, the bias reactivates. BPA-MP is reported as stronger and more persistent than BPA-PP, and BPA-PP+MP is strongest; for example, under Equal-penalty, BPA-MP yields $w \in \mathbb{Z}^n_{\ge 0}$8 versus $w \in \mathbb{Z}^n_{\ge 0}$9 and $w_i = 0$0, both *, while BPA-PP+MP yields $w_i = 0$1 versus $w_i = 0$2 and $w_i = 0$3, both ** (Wang et al., 1 Jan 2026).

This agent formulation broadens BPA beyond data poisoning in the usual training-set sense. The attacked variable is a safety-critical latent belief stored in persistent state, and the attack succeeds by changing the condition under which a norm is applied rather than by directly instructing the model to behave harmfully.

5. Mechanisms, vulnerability factors, and empirical patterns

Across the papers, BPA operates by altering the effective evidence seen by the system. In Bayesian inference, deletion and replication reshape sufficient statistics and precision terms, especially visible in conjugate updates such as $w_i = 0$4 and $w_i = 0$5 in weighted NIG regression (Carreau et al., 6 Mar 2025). In LLM pre-training and CPT, repeated exposure to false statements changes relative likelihoods or output preferences without requiring a trigger (Zhang et al., 2024); (Churina et al., 29 Oct 2025). In agents, persistent profile or memory content changes the belief variable that gates a human-oriented norm (Wang et al., 1 Jan 2026). A plausible synthesis is that BPA exploits whichever internal variable most directly couples evidence to future inference.

The vulnerability factors reported in the Bayesian paper are explicit. Informative priors dampen manipulation, while weak or diffuse priors are more vulnerable (Carreau et al., 6 Mar 2025). Noisy datasets with influential points are easier to poison, and experiments show attack impact grows as $w_i = 0$6 increases in linear regression. Conjugate models reveal precise steering pathways via sufficient statistics, but nonconjugate models remain vulnerable through sampling-based gradient estimates. The budget $w_i = 0$7 and cap $w_i = 0$8 define a risk–reward trade-off: low $w_i = 0$9 can produce notable shifts if the dataset contains influential points, whereas larger $b$00 enables closer matching to the target posterior but increases detectability (Carreau et al., 6 Mar 2025).

The empirical findings illustrate that small perturbation footprints can suffice. In simulated linear regression, with $b$01 (20% of points), the tainted posterior nearly overlaps the target in the slope dimension; with $b$02, the adversarial and tainted means are almost identical while other aspects remain similar (Carreau et al., 6 Mar 2025). In the Boston housing horseshoe model, manipulating approximately $b$03 of data reduced $b$04 from about $b$05 to approximately $b$06, and with approximately $b$07 manipulations, the marginal for $b$08 concentrates near $b$09 while other coefficients remain largely unchanged. In the Mexico microcredit dataset with $b$10, modifying only 20 points, approximately $b$11, shifts the ATE from negative to clearly positive with mean approximately $b$12 and $b$13 CI $b$14, reversing the policy conclusion (Carreau et al., 6 Mar 2025).

For LLM poisoning, the reported patterns emphasize persistence and nontrivial dose–response. Pre-training poisoning at $b$15 suffices for belief manipulation to measurably persist after SFT and DPO (Zhang et al., 2024). In CPT, even minimal poisoned exposure at $b$16 flips structured prompt formats, while higher poisoning ratios induce widespread flips across model scales, and model size does not consistently confer resistance: the 3B model is most resilient, whereas the 7B model exhibits the highest flip rates at near-total poisoning levels (Churina et al., 29 Oct 2025). The CPT results also distinguish question types and prompt formats: highly structured prompts such as Cloze, True/False (Negated), JSON, and Time-Anchored flip at just $b$17 poisoning, whereas Direct Question and Paraphrased Question flip only at $b$18 poisoning; Multiple-Choice flips at $b$19 (Churina et al., 29 Oct 2025).

For agent BPA, temporal dynamics matter. Intergroup bias in AVA grows over time; AVH w/o A converges to neutrality. BPA-PP can fade as the agent reconsiders, but BPA-MP causes a sharp, persistent collapse toward biased extremes, and BPA-PP+MP amplifies this effect (Wang et al., 1 Jan 2026). The paper also reports that random, non-optimized suffix injection in BPA-MP still induces significant bias but is weaker than the optimized variant; in Equal-penalty, the intergroup mean is $b$20 without optimization versus $b$21 with optimization (Wang et al., 1 Jan 2026).

6. Detection, defenses, and open problems

The defenses discussed in the literature are unevenly developed. The Bayesian paper emphasizes that robust defenses for Bayesian pipelines against deletion and replication attacks are largely absent (Carreau et al., 6 Mar 2025). Duplicate removal is described as a naive defense, but legitimate duplicates are common in real data and removal may harm inference; moreover, BPA can adapt to deletions-only attacks, which are harder to detect. The paper points toward sensitivity analysis, stronger priors, defensive reweighting, robust workflows, heavy-tailed priors, and privacy mechanisms, but does not develop concrete defensive algorithms. Practical guidance includes auditing per-record weight caps and total perturbation budgets, conducting routine sensitivity analyses to small deletions and replications, cross-validating inferences across priors, and tracking reweighting impacts on sufficient statistics such as $b$22 (Carreau et al., 6 Mar 2025).

In LLM poisoning, both pre-training papers stress the difficulty of filtering naturalistic attacks. "Persistent Pre-Training Poisoning of LLMs" argues that large-scale rule-based filtering is imperfect and may miss ordinary conversational poisoning, and that standard heuristics are unlikely to reliably catch BPA because it resembles benign web text (Zhang et al., 2024). The paper recommends stronger provenance and curation, targeted auditing of belief-sensitive outputs, and the use of benign backdoors as canaries to quantify end-to-end vulnerability, but does not present BPA-specific unlearning or belief-repair methods. "Layer of Truth" is explicitly diagnostic rather than defensive. It proposes periodic probing during CPT using $b$23 and Logit Lens trajectories, data curation with MinHash deduplication at Jaccard threshold $b$24, style-aware filtering, conservative learning rates, and frequent checkpointing (Churina et al., 29 Oct 2025). It also names “truth-grounded auxiliary losses” and “trust-weighted sampling” as natural extensions, but states that these are not implemented in the study.

The agent paper provides the most concrete defenses (Wang et al., 1 Jan 2026). On the profile side, safety-critical identity priors should be treated as verified, sealed fields rather than mutable profile text. On the memory side, a write-time gate should detect unverifiable identity assertions and either sanitize them or exclude them from retrieval. Formally, for text $b$25 to be committed, with detector $b$26, the defense rule is

$b$27

where $b$28 removes or rewrites identity-claim fragments. Under the strongest attack, BPA-PP+MP, enabling this belief gate returns intergroup choices to no-attack levels; for Equal-penalty with defense, $b$29 versus $b$30 and $b$31, all non-significant (Wang et al., 1 Jan 2026).

Several misconceptions are clarified by the surveyed work. BPA is not necessarily a backdoor attack: in the LLM pre-training paper it is explicitly contrasted with trigger-based attacks because belief manipulation is triggerless and global for the targeted topic (Zhang et al., 2024). BPA is also not restricted to fabricated examples: the Bayesian formulation uses only deletion and replication of true observations (Carreau et al., 6 Mar 2025). Nor is it confined to model parameters: in agents, persistent profile and memory state are sufficient attack surfaces even when the underlying LLM weights are unchanged (Wang et al., 1 Jan 2026). A plausible implication is that belief security must be treated as a system-level property, spanning datasets, inference procedures, persistent state, and monitoring instrumentation.

Open questions remain explicit across the papers. For Bayesian BPA, separate analytic bounds distinguishing targeted versus non-targeted marginals are not provided, and surgical precision is demonstrated empirically rather than by dedicated theory (Carreau et al., 6 Mar 2025). For LLM BPA, the minimal rate for persistent belief manipulation below $b$32 is not established in the pre-training setting (Zhang et al., 2024), and the CPT work leaves open how to reverse flips, how to fit a formal exposure–belief function $b$33, and whether BPA persists or amplifies after instruction tuning (Churina et al., 29 Oct 2025). For agent BPA, transfer to high-stakes domains and longer horizons remains to be evaluated, and the paper presents its defenses as practical prototypes rather than complete solutions (Wang et al., 1 Jan 2026).