BYORn: Robust Defense for Poisoned LVLMs

• BYORn is a backdoor defense that identifies poisoned responses using high log-perplexity, indicating semantic misalignment in multimodal inputs.
• It replaces suspicious outputs with EMA-generated alternatives during training to disrupt trigger-target correlations and improve robustness.
• Empirical findings show BYORn reduces attack success rates significantly while preserving clean performance across tasks like image captioning and VQA.

Searching arXiv for the BYORn paper and closely related VLM backdoor-defense context. BYORn, short for “Bootstrap Your Own Responses,” is a defense for supervised fine-tuning of autoregressive vision-LLMs against backdoor attacks. It is designed for poisoned instruction-tuning settings in which an attacker injects triggers into the image and/or instruction and replaces the desired response with a malicious target output. The method exploits a central empirical regularity: poisoned target responses are often semantically implausible given the corresponding image-text input, so a pretrained vision-LLM assigns them low likelihood. BYORn operationalizes this observation by detecting suspicious training examples with high response log-perplexity and replacing their responses with model-generated alternatives during training, thereby disrupting the trigger-to-target correlation that underlies the backdoor mechanism (Sabolić et al., 1 Jun 2026).

1. Threat model and problem formulation

BYORn is formulated in the setting of instruction tuning or supervised fine-tuning on datasets of multimodal triples

$$\mathcal{D}_\text{clean} = \{(\mathbf{x}^i, \mathbf{q}^i, \mathbf{t}^i )\}_{i=1}^{N},$$

where $\mathbf{x}^i$ is an image, $\mathbf{q}^i$ is an instruction or question, and $\mathbf{t}^i$ is the clean target response. A model $$f_\theta : \mathcal{X} \times \mathcal{Q} \rightarrow \mathcal{T}$$ is trained by minimizing the empirical risk

$$\mathcal{R}(\theta) = \mathbb{E}_{(\mathbf{x}, \mathbf{q}, \mathbf{t}) \sim p(\cdot)}[\mathcal{L}(\theta|\mathbf{x}, \mathbf{q}, \mathbf{t})] \approx \frac{1}{|\mathcal{D}_\text{clean}|} \sum_{(\mathbf{x}, \mathbf{q}, \mathbf{t}) \in \mathcal{D}_\text{clean}} \mathcal{L}(\theta|\mathbf{x}, \mathbf{q}, \mathbf{t}).$$

The loss $\mathcal{L}$ is the standard autoregressive negative log-likelihood over the response token sequence (Sabolić et al., 1 Jun 2026).

In the poisoned setting, the available data are

$$\mathcal{D}_\text{p} = \{( \mathbf{x}^i, \mathbf{q}^i, \mathbf{y}^i )\}_{i=1}^{N},$$

where an attacker corrupts a fraction $\gamma \in [0,1]$ of examples by injecting triggers into the image and/or instruction and replacing the response with a malicious target $\mathbf{y}^i$. Some attacks modify only images and responses. Under naive supervised fine-tuning on $\mathbf{x}^i$0, the resulting model learns the poisoned mapping rather than the clean multimodal mapping. BYORn addresses precisely this setting: training must proceed using only poisoned data while still recovering clean behavior (Sabolić et al., 1 Jun 2026).

A common misconception is that conventional supervised fine-tuning is intrinsically sufficient if the poisoned fraction is modest. The BYORn formulation rejects that premise: the paper explicitly treats standard supervised fine-tuning as highly vulnerable in open-ended generation settings, and it frames defense as a problem of recovering the clean response distribution despite corrupted supervision (Sabolić et al., 1 Jun 2026).

2. Core intuition: semantic misalignment and response plausibility

The conceptual basis of BYORn is that many backdoor attacks create semantic misalignment between the multimodal input $\mathbf{x}^i$1 and the poisoned response $\mathbf{x}^i$2. The paper gives examples in which malicious outputs contain unrelated concepts such as “banana” even when the image and instruction are unrelated to bananas. A pretrained large vision-LLM often regards such responses as implausible and therefore assigns them high token-level negative log-likelihood (Sabolić et al., 1 Jun 2026).

BYORn measures this mismatch using the detection score

$\mathbf{x}^i$3

where $\mathbf{x}^i$4 is the number of response tokens. This score is the average token-level negative log-likelihood, i.e. log-perplexity. The intended interpretation is that clean responses should be reasonably predictable under a pretrained LVLM, whereas poisoned target responses are often odd, unrelated, or factually inconsistent and thus have higher perplexity (Sabolić et al., 1 Jun 2026).

This detection principle is not merely heuristic. The paper reports that poisoned examples fall in the high-score tail, enabling high-recall detection. A plausible implication is that BYORn is most naturally suited to attacks whose payloads must override the semantic structure of the original supervision; its efficacy is therefore tied to the extent of semantic implausibility induced by the poisoning process. The paper also notes, however, that stronger adaptive attacks may attempt to make poisoned responses more plausible, which motivates the use of dynamic response replacement rather than filtering alone (Sabolić et al., 1 Jun 2026).

3. Detection, bootstrapping, and poison-aware training

BYORn has two variants. BYORn-F filters out suspicious examples and trains only on the remaining subset. BYORn retains suspicious examples but replaces their responses with model-generated alternatives during training (Sabolić et al., 1 Jun 2026).

The detector is threshold-based. It defines a binary poisoned-indicator estimate

$\mathbf{x}^i$5

where $\mathbf{x}^i$6 is the Iverson bracket and $\mathbf{x}^i$7 is the $\mathbf{x}^i$8-th percentile threshold of the detection scores. Operationally, the method computes $\mathbf{x}^i$9 for every training sample, sets $\mathbf{q}^i$0, and marks sample $\mathbf{q}^i$1 as suspicious if $\mathbf{q}^i$2. The paper uses a high-recall choice $\mathbf{q}^i$3, so the top 40% highest-perplexity responses are selected for bootstrapping. It also analyzes thresholds $\mathbf{q}^i$4 (Sabolić et al., 1 Jun 2026).

For suspicious examples, BYORn replaces the original response with a model-generated alternative sampled from an exponential moving average model: $\mathbf{q}^i$5 with

$\mathbf{q}^i$6

The EMA model serves as a more stable generator of replacement responses than the rapidly changing online model (Sabolić et al., 1 Jun 2026).

To reduce computational overhead, the method preprocesses the whole dataset with the detector and forms minibatches containing only clean or only suspicious examples. Clean minibatches are trained on the original responses $\mathbf{q}^i$7; suspicious minibatches are trained on generated responses $\mathbf{q}^i$8. This poison-aware minibatching avoids generation for clean examples and substantially reduces the cost of the bootstrapping procedure (Sabolić et al., 1 Jun 2026).

The appendix-level training algorithm consists of: initializing $\mathbf{q}^i$9; computing $\mathbf{t}^i$0 for each sample; computing $\mathbf{t}^i$1; setting $\mathbf{t}^i$2; splitting data into $\mathbf{t}^i$3 and $\mathbf{t}^i$4; training suspicious minibatches on sampled $\mathbf{t}^i$5 and clean minibatches on $\mathbf{t}^i$6; and updating the EMA after each step (Sabolić et al., 1 Jun 2026).

The significance of this design is that it modifies the supervision channel rather than only the input channel. Instead of trying solely to remove triggers or identify poisoned images, BYORn directly rewrites the training signal for suspicious samples. This suggests a broader defensive principle for generative models: when poisoning acts by corrupting labels or target sequences, robustness can be improved by replacing untrusted supervision with model-consistent pseudo-targets.

4. Objective function and risk-upper-bound interpretation

The BYORn objective is defined as

$\mathbf{t}^i$7

where $\mathbf{t}^i$8 denotes the probability that an example is poisoned. The first term uses the available response when the example is treated as clean; the second replaces the response with a bootstrap sample from the EMA model when the example is treated as poisoned (Sabolić et al., 1 Jun 2026).

With minibatch $\mathbf{t}^i$9, the gradient is approximated by

$$f_\theta : \mathcal{X} \times \mathcal{Q} \rightarrow \mathcal{T}$$0

The paper’s theoretical claim is that optimizing this objective corresponds to optimizing an empirical estimate of an upper bound on the population risk over the clean distribution: $$f_\theta : \mathcal{X} \times \mathcal{Q} \rightarrow \mathcal{T}$$1 The derivation uses the Donsker–Varadhan inequality and Hoeffding’s lemma, producing a decomposition in which

$$f_\theta : \mathcal{X} \times \mathcal{Q} \rightarrow \mathcal{T}$$2

The KL and constant terms do not depend on $$f_\theta : \mathcal{X} \times \mathcal{Q} \rightarrow \mathcal{T}$$3 during the optimization step, so

$$f_\theta : \mathcal{X} \times \mathcal{Q} \rightarrow \mathcal{T}$$4

The paper further cautions that replacing expectations by empirical averages does not guarantee that the empirical objective itself upper-bounds empirical risk; instead, it should be interpreted as minimizing an empirical estimate of the population-risk upper bound (Sabolić et al., 1 Jun 2026).

This theoretical framing matters because BYORn is not presented simply as heuristic pseudo-labeling. It is specifically cast as a poisoned-data training rule whose gradient aligns with a clean-risk upper-bound surrogate. A plausible implication is that the method’s robustness claim depends not only on detection quality but also on the stabilizing role of the EMA generator, which controls the bootstrap distribution used in the upper-bound argument.

5. Mechanism of defense and relation to trigger–target correlation

Backdoor attacks depend on a spurious association of the form

$$f_\theta : \mathcal{X} \times \mathcal{Q} \rightarrow \mathcal{T}$$5

BYORn interferes with this association at training time. For suspicious examples, the attacker-specified response is no longer used as supervision; instead, the model is trained on regenerated responses that are consistent with its current clean prior or multimodal knowledge (Sabolić et al., 1 Jun 2026).

The paper explicitly states that this “breaks the correlation between adversarial triggers and target outputs.” Clean samples continue to reinforce the intended multimodal mapping. Suspicious samples, rather than amplifying the poisoned rule, are redirected toward outputs generated from the model conditioned on the image-text content. In effect, the trigger ceases to be reliably predictive of the attacker’s target string during optimization (Sabolić et al., 1 Jun 2026).

This mechanism distinguishes BYORn from filtering-only strategies. Filtering removes suspect data but discards information; BYORn instead attempts to salvage suspicious inputs by reconstructing plausible targets. That design explains why the full method is described as often giving a better balance between robustness and utility than aggressive filtering alone. It also clarifies why the defense is especially tailored to autoregressive generation: the model can synthesize replacement responses on the fly, making the target channel itself a site of defensive intervention.

6. Experimental regime and empirical performance

The evaluation covers three multimodal tasks: image captioning, spot the difference, and visual question answering. For image captioning, the training data are the LADD dataset from MIMIC-IT and Flickr30k, and evaluation uses validation subsets of Flickr30k and COCO Captions. For spot the difference, training uses the CGD dataset from MIMIC-IT and evaluation uses the validation subset of CGD. For VQA, the benchmark is ScienceQA (Sabolić et al., 1 Jun 2026).

The evaluated LVLMs are LLaVA-v1.5-7B, Qwen3-VL-8B-Instruct, InternVL3.5-4B, and Otter-MPT1B. The attacks are BadNets, Blend, DualKey, and VL-Trojan, with a default poisoning rate of 10%. Baselines include naive supervised fine-tuning (SFT), ONION, BYE, BYORn-F, and full BYORn. The reported metrics are CIDEr, SPICE, ACC, and ASR, with lower ASR indicating stronger backdoor robustness (Sabolić et al., 1 Jun 2026).

The paper reports that BYORn consistently improves robustness while preserving clean-task performance and establishes a new trade-off frontier between generalization and attack success rate. The overall empirical claim is an average 40 percentage point reduction in ASR compared to existing defenses, together with strong preservation of clean performance and a Pareto-optimal robustness–generalization trade-off (Sabolić et al., 1 Jun 2026).

The following summary organizes the reported task-level outcomes.

Evaluation setting	Reported outcome for BYORn	Context
Image captioning	0.0–0.4 ASR on Flickr/COCO for LLaVA, with strong CIDEr	Also near-zero ASR for Qwen-VL, InternVL, and Otter
Spot the Difference	ASR around 0–1%	BYORn-F and BYORn both strong; modest CIDEr reduction vs SFT
ScienceQA VQA	ASR = 0 across attacks	Reported for BYORn and BYORn-F on Qwen3-VL and LLaVA-1.5

In image captioning, the paper states that BYORn lowers ASR by about 20 pp for LLaVA, 50 pp for Otter, and 40 pp for Qwen and InternVL relative to the best baseline. In spot the difference, SFT is described as fully backdoored or nearly so, ONION as helpful only against text-triggered attacks, and BYE as unreliable, whereas BYORn-F and BYORn both achieve strong robustness. In ScienceQA, BYORn and BYORn-F achieve complete backdoor resilience, although accuracy remains somewhat reduced relative to unprotected SFT (Sabolić et al., 1 Jun 2026).

These results are framed as a trade-off analysis rather than a claim of zero-cost defense. BYORn-F is faster and often highly robust, but the full BYORn method is described as usually delivering a better balance when robustness is required while retaining more of the original data’s utility than filter-only training (Sabolić et al., 1 Jun 2026).

7. Adaptive attacks, assumptions, and limitations

The paper explicitly tests adaptive attacks intended to circumvent the detector. One adaptive strategy is a semantically aligned trigger attack, in which the trigger is made semantically compatible with the target response, such as a banana trigger for a banana caption. Another is an adversarial perturbation attack that uses PGD to optimize an image perturbation that minimizes the detection score and increases target-response likelihood: $$f_\theta : \mathcal{X} \times \mathcal{Q} \rightarrow \mathcal{T}$$6 with iterative updates

$$f_\theta : \mathcal{X} \times \mathcal{Q} \rightarrow \mathcal{T}$$7

The reported hyperparameters are $$f_\theta : \mathcal{X} \times \mathcal{Q} \rightarrow \mathcal{T}$$8, $$f_\theta : \mathcal{X} \times \mathcal{Q} \rightarrow \mathcal{T}$$9, and $$\mathcal{R}(\theta) = \mathbb{E}_{(\mathbf{x}, \mathbf{q}, \mathbf{t}) \sim p(\cdot)}[\mathcal{L}(\theta|\mathbf{x}, \mathbf{q}, \mathbf{t})] \approx \frac{1}{|\mathcal{D}_\text{clean}|} \sum_{(\mathbf{x}, \mathbf{q}, \mathbf{t}) \in \mathcal{D}_\text{clean}} \mathcal{L}(\theta|\mathbf{x}, \mathbf{q}, \mathbf{t}).$$0 PGD steps. A third adaptive setting is a semantically plausible “clean-label-like” attack using captions containing “car” relabeled to “The image shows a red car.” (Sabolić et al., 1 Jun 2026)

Despite these adaptive strategies, BYORn is reported to remain effective. For semantically aligned adaptive attacks, it reduces ASR by over 90 pp for patch-style attacks and over 70 pp for image-wide attacks. For PGD-based adversarial perturbation attacks, the paper reports ASR 92.0% for SFT and ASR 0.2% for BYORn, while CIDEr remains high (Sabolić et al., 1 Jun 2026).

The paper also states several limitations. It gives no guarantee against every future attack; backdoor attacks evolve, and the defense is not universally foolproof. Detection depends on low likelihood under a pretrained model, so stronger adaptive attacks may render poisoned responses more plausible. Its scope is limited to robust instruction tuning and does not address all possible backdoor stages in the LVLM pipeline, such as attacks on pretrained weights or other deployment stages. The method assumes that the pretrained model itself is not backdoored. It incurs computational overhead because response generation slows training, although poison-aware minibatching reduces this substantially. Finally, the percentile threshold $$\mathcal{R}(\theta) = \mathbb{E}_{(\mathbf{x}, \mathbf{q}, \mathbf{t}) \sim p(\cdot)}[\mathcal{L}(\theta|\mathbf{x}, \mathbf{q}, \mathbf{t})] \approx \frac{1}{|\mathcal{D}_\text{clean}|} \sum_{(\mathbf{x}, \mathbf{q}, \mathbf{t}) \in \mathcal{D}_\text{clean}} \mathcal{L}(\theta|\mathbf{x}, \mathbf{q}, \mathbf{t}).$$1 introduces a precision–recall trade-off, though the method is reported to be fairly robust across a wide range of thresholds (Sabolić et al., 1 Jun 2026).

These limitations clarify the boundary conditions of the method. BYORn is not a universal backdoor defense for all multimodal learning regimes; it is a defense specialized to poisoned supervised fine-tuning of autoregressive LVLMs under the assumption that the initial pretrained model is trustworthy enough to detect semantic implausibility and to generate usable replacement supervision. Under those assumptions, the paper positions BYORn as a method that transforms suspect supervision rather than merely excluding it, and thereby improves robustness while retaining a favorable utility profile (Sabolić et al., 1 Jun 2026).