FORCE: Feature Over-Reliance Correction

• FORCE is a method for transferable visual jailbreak attacks that corrects over-reliance on narrow layer and frequency features to enhance generalization.
• It employs layer-aware regularization to broaden the feasible region in intermediate representations, reducing sensitivity to minor perturbations.
• Spectral rescaling adjusts frequency components based on semantic relevance, ensuring more robust transfer across diverse multimodal LLMs.

Feature Over-Reliance CorrEction (FORCE) is a method for transferable visual jailbreaking attacks against multimodal LLMs (MLLMs). It addresses the observation that optimization-based visual jailbreaks can manipulate open-source MLLMs reliably on a source model yet exhibit extremely limited cross-model transferability, especially to closed-source systems. FORCE attributes that weakness to localization in high-sharpness regions of the attack loss landscape and to improper reliance on narrow intermediate-layer representations and semantically poor frequency components. Its central operation is therefore corrective rather than merely stronger optimization: it guides attack generation toward broader feasible regions across layer features and rescales frequency features according to their semantic content, with the stated goal of improving cross-model transferability for visual red-teaming evaluations (Lin et al., 25 Sep 2025).

1. Problem setting and threat model

FORCE is defined in the setting of optimization-based visual jailbreaking, where the malicious text prompt remains explicit and unchanged, while the accompanying image is perturbed so that the MLLM stops refusing and instead produces an affirmative harmful response. The multimodal model receives an input pair $(\mathbf{x}_{\text{img}}, \mathbf{x}_{\text{txt}})$ and predicts output tokens according to a conditional distribution $$p_{\theta}(\mathbf{y}\mid \mathbf{x}_{\text{img}}, \mathbf{x}_{\text{txt}})$$. A standard attack chooses a short affirmative target string such as “Sure, here is” and minimizes its negative log-likelihood under the source MLLM, then updates the perturbation by sign-gradient descent (Lin et al., 25 Sep 2025).

The baseline objective is

$$\ell\!\left((\mathbf{x}_{\text{img}+\delta}, \mathbf{x}_{\text{txt}}), \mathbf{y}\right) = - \sum_{s=1}^{S} \log p_{\theta}\!\left( \mathbf{y}_{s} \mid \mathbf{x}_{\text{img}+\delta}, \mathbf{x}_{\text{txt}} \right),$$

with the PGD-style update

$$\delta^{(t+1)} = \delta^{(t)} - \alpha \, \operatorname{sign}\left( \frac{\partial \ell}{\partial \delta^{(t)}} \right).$$

Transferability is treated as the decisive issue. If attacks discovered on an open-source surrogate do not transfer, they mainly expose idiosyncratic vulnerabilities of that source model rather than general weaknesses across MLLMs. This is especially consequential for closed-source commercial MLLMs, where gradients and parameters are unavailable and practical evaluation must proceed through black-box transfer (Lin et al., 25 Sep 2025).

2. Diagnostic basis: sharpness, layer dependence, and spectral dependence

FORCE begins from an empirical loss-landscape analysis. Successful PGD-generated visual jailbreaks are reported to reside in high-sharpness regions: the attack loss at the adversarial point is near zero on the source model, but rises rapidly under small local perturbations in either input space or weight space. In the reported visualizations, a small input perturbation of about $0.03$ along the adversarial direction can already raise the loss enough to invalidate the jailbreak, and a tiny weight-space perturbation around $0.0002$ can move the attack out of the feasible region for transfer (Lin et al., 25 Sep 2025).

The paper then locates the source of that sharpness in two kinds of feature over-reliance. The first is intermediate-layer over-reliance. To probe it, the method interpolates between the representation of a successful jailbreak and that of a natural image within the same layer,

$$(1-\mu)\cdot f_\theta(\text{jail}) + \mu \cdot f_\theta(\text{nat}),$$

so that layer-specific scale confounds are avoided. The reported pattern is strongly depth-dependent. In later layers, feasible regions are broad: in the $31$st layer, the attack can remain successful even when roughly $40\%$ natural-image features are mixed in. In earlier layers, feasible regions are narrow: around the $11$th layer, the attack must retain over $$p_{\theta}(\mathbf{y}\mid \mathbf{x}_{\text{img}}, \mathbf{x}_{\text{txt}})$$0 adversarial features, and mixing in about $$p_{\theta}(\mathbf{y}\mid \mathbf{x}_{\text{img}}, \mathbf{x}_{\text{txt}})$$1 natural features sharply increases loss and destroys the jailbreak. An appendix analysis interpolating between two successful jailbreaks yields the same qualitative conclusion: later layers occupy a more shared continuous region, whereas earlier layers are disjoint and fragile (Lin et al., 25 Sep 2025).

The second diagnosis concerns spectral over-reliance. The perturbation is transformed into the Fourier domain, partitioned into $$p_{\theta}(\mathbf{y}\mid \mathbf{x}_{\text{img}}, \mathbf{x}_{\text{txt}})$$2 equal-width frequency bands, and each band is masked independently before reconstruction and reevaluation. Early in optimization, masking any band keeps the loss high because the attack is not yet effective. At intermediate stages, low-frequency bands matter more, suggesting reliance on more semantic image content. By convergence, some mid-to-high frequency bands become disproportionately important, and removing one such band can make the jailbreak fail. Those components are interpreted as semantically poor and non-generalizable, so heavy reliance on them is taken to be a transferability liability rather than a robust attack feature (Lin et al., 25 Sep 2025).

3. Method: layer-aware feature regularization and spectral rescaling

FORCE integrates two corrective modules into PGD: a layer-aware feature regularizer and a spectral rescaling procedure. The first module explicitly seeks broader feasible regions in representation space, particularly in earlier layers. Around the current adversarial image $$p_{\theta}(\mathbf{y}\mid \mathbf{x}_{\text{img}}, \mathbf{x}_{\text{txt}})$$3, the method samples a nearby reference point by adding noise $$p_{\theta}(\mathbf{y}\mid \mathbf{x}_{\text{img}}, \mathbf{x}_{\text{txt}})$$4, then measures the squared feature distance at layer $$p_{\theta}(\mathbf{y}\mid \mathbf{x}_{\text{img}}, \mathbf{x}_{\text{txt}})$$5:

$$p_{\theta}(\mathbf{y}\mid \mathbf{x}_{\text{img}}, \mathbf{x}_{\text{txt}})$$6

The reference sample must also remain feasible, so its target loss is

$$p_{\theta}(\mathbf{y}\mid \mathbf{x}_{\text{img}}, \mathbf{x}_{\text{txt}})$$7

Because earlier-layer fragility is the primary concern, the layer weight is

$$p_{\theta}(\mathbf{y}\mid \mathbf{x}_{\text{img}}, \mathbf{x}_{\text{txt}})$$8

Using $$p_{\theta}(\mathbf{y}\mid \mathbf{x}_{\text{img}}, \mathbf{x}_{\text{txt}})$$9 reference samples, the regularizer becomes

$$\ell\!\left((\mathbf{x}_{\text{img}+\delta}, \mathbf{x}_{\text{txt}}), \mathbf{y}\right) = - \sum_{s=1}^{S} \log p_{\theta}\!\left( \mathbf{y}_{s} \mid \mathbf{x}_{\text{img}+\delta}, \mathbf{x}_{\text{txt}} \right),$$0

Minimizing this quantity favors nearby reference samples that remain successful while spreading out in feature space, which the paper interprets as exploring a broader feasible region (Lin et al., 25 Sep 2025).

The second module counteracts spectral over-reliance. Given the current perturbation $$\ell\!\left((\mathbf{x}_{\text{img}+\delta}, \mathbf{x}_{\text{txt}}), \mathbf{y}\right) = - \sum_{s=1}^{S} \log p_{\theta}\!\left( \mathbf{y}_{s} \mid \mathbf{x}_{\text{img}+\delta}, \mathbf{x}_{\text{txt}} \right),$$1, the method computes

$$\ell\!\left((\mathbf{x}_{\text{img}+\delta}, \mathbf{x}_{\text{txt}}), \mathbf{y}\right) = - \sum_{s=1}^{S} \log p_{\theta}\!\left( \mathbf{y}_{s} \mid \mathbf{x}_{\text{img}+\delta}, \mathbf{x}_{\text{txt}} \right),$$2

where $$\ell\!\left((\mathbf{x}_{\text{img}+\delta}, \mathbf{x}_{\text{txt}}), \mathbf{y}\right) = - \sum_{s=1}^{S} \log p_{\theta}\!\left( \mathbf{y}_{s} \mid \mathbf{x}_{\text{img}+\delta}, \mathbf{x}_{\text{txt}} \right),$$3 is the magnitude spectrum and $$\ell\!\left((\mathbf{x}_{\text{img}+\delta}, \mathbf{x}_{\text{txt}}), \mathbf{y}\right) = - \sum_{s=1}^{S} \log p_{\theta}\!\left( \mathbf{y}_{s} \mid \mathbf{x}_{\text{img}+\delta}, \mathbf{x}_{\text{txt}} \right),$$4 is the phase spectrum. Let $$\ell\!\left((\mathbf{x}_{\text{img}+\delta}, \mathbf{x}_{\text{txt}}), \mathbf{y}\right) = - \sum_{s=1}^{S} \log p_{\theta}\!\left( \mathbf{y}_{s} \mid \mathbf{x}_{\text{img}+\delta}, \mathbf{x}_{\text{txt}} \right),$$5 be the partition into equal-width frequency bands. For each band $$\ell\!\left((\mathbf{x}_{\text{img}+\delta}, \mathbf{x}_{\text{txt}}), \mathbf{y}\right) = - \sum_{s=1}^{S} \log p_{\theta}\!\left( \mathbf{y}_{s} \mid \mathbf{x}_{\text{img}+\delta}, \mathbf{x}_{\text{txt}} \right),$$6, FORCE masks that band, reconstructs the perturbation, and measures the resulting attack loss $$\ell\!\left((\mathbf{x}_{\text{img}+\delta}, \mathbf{x}_{\text{txt}}), \mathbf{y}\right) = - \sum_{s=1}^{S} \log p_{\theta}\!\left( \mathbf{y}_{s} \mid \mathbf{x}_{\text{img}+\delta}, \mathbf{x}_{\text{txt}} \right),$$7. It then defines band-wise scaling factors as

$$\ell\!\left((\mathbf{x}_{\text{img}+\delta}, \mathbf{x}_{\text{txt}}), \mathbf{y}\right) = - \sum_{s=1}^{S} \log p_{\theta}\!\left( \mathbf{y}_{s} \mid \mathbf{x}_{\text{img}+\delta}, \mathbf{x}_{\text{txt}} \right),$$8

and the spectral mask

$$\ell\!\left((\mathbf{x}_{\text{img}+\delta}, \mathbf{x}_{\text{txt}}), \mathbf{y}\right) = - \sum_{s=1}^{S} \log p_{\theta}\!\left( \mathbf{y}_{s} \mid \mathbf{x}_{\text{img}+\delta}, \mathbf{x}_{\text{txt}} \right),$$9

The rescaled perturbation is reconstructed as

$$\delta^{(t+1)} = \delta^{(t)} - \alpha \, \operatorname{sign}\left( \frac{\partial \ell}{\partial \delta^{(t)}} \right).$$0

If a higher-frequency band is more influential than its lower-frequency predecessor, its weight is reduced below $$\delta^{(t+1)} = \delta^{(t)} - \alpha \, \operatorname{sign}\left( \frac{\partial \ell}{\partial \delta^{(t)}} \right).$$1; otherwise it is left unchanged. The operative rule is therefore suppressive only for anomalously over-important higher-frequency components (Lin et al., 25 Sep 2025).

The attack loop initializes

$$\delta^{(t+1)} = \delta^{(t)} - \alpha \, \operatorname{sign}\left( \frac{\partial \ell}{\partial \delta^{(t)}} \right).$$2

then repeatedly applies spectral rescaling, computes the source-model target loss

$$\delta^{(t+1)} = \delta^{(t)} - \alpha \, \operatorname{sign}\left( \frac{\partial \ell}{\partial \delta^{(t)}} \right).$$3

adds $$\delta^{(t+1)} = \delta^{(t)} - \alpha \, \operatorname{sign}\left( \frac{\partial \ell}{\partial \delta^{(t)}} \right).$$4, updates with sign-gradient descent on $$\delta^{(t+1)} = \delta^{(t)} - \alpha \, \operatorname{sign}\left( \frac{\partial \ell}{\partial \delta^{(t)}} \right).$$5, and clips back to the perturbation budget. Spectral correction is implemented procedurally rather than as a single differentiable objective term (Lin et al., 25 Sep 2025).

4. Experimental evaluation and transferability profile

The source model used to generate attacks is LLaVA-v1.5-7B. Transfer is tested against three families of target models: adapter-based open-source MLLMs, early-fusion open-source MLLMs, and closed-source commercial MLLMs. The target sets include InstructBLIP-Vicuna-7B, LLaVA-v1.6-mistral-7b, Idefics3-8B-Llama3, Qwen2.5-VL-7B-Instruct, LLaMA-3.2-11B-Vision-Instruct, Claude-Sonnet-4, Gemini-2.5-Pro, and GPT-5. The evaluation benchmarks are MaliciousInstruct with $$\delta^{(t+1)} = \delta^{(t)} - \alpha \, \operatorname{sign}\left( \frac{\partial \ell}{\partial \delta^{(t)}} \right).$$6 instructions, AdvBench with $$\delta^{(t+1)} = \delta^{(t)} - \alpha \, \operatorname{sign}\left( \frac{\partial \ell}{\partial \delta^{(t)}} \right).$$7, and HADES with $$\delta^{(t+1)} = \delta^{(t)} - \alpha \, \operatorname{sign}\left( \frac{\partial \ell}{\partial \delta^{(t)}} \right).$$8. For MaliciousInstruct and AdvBench, the image is initialized with either a blank gray image of RGB $$\delta^{(t+1)} = \delta^{(t)} - \alpha \, \operatorname{sign}\left( \frac{\partial \ell}{\partial \delta^{(t)}} \right).$$9 or a panda image; for HADES, the provided image–instruction pairs are used after removing keyword typography so the model focuses on image content. Success is measured by ASR and Query, with harmfulness judged through substring matching plus HarmBenchLLaMA-2-13B-cls. FORCE uses the same PGD base settings as the baseline, with $0.03$0, $0.03$1, $0.03$2, $0.03$3, $0.03$4, and $0.03$5 (Lin et al., 25 Sep 2025).

On adapter-based targets, FORCE improves transfer consistently. Representative gains include Idefics3-8B-Llama3 on AdvBench from $0.03$6 to $0.03$7, LLaVA-v1.6-mistral-7b on AdvBench from $0.03$8 to $0.03$9, and InstructBLIP-Vicuna-7B on MaliciousInstruct from $0.0002$0 to $0.0002$1. The paper summarizes this family by an average ASR improvement of about $0.0002$2 and over $0.0002$3 reduction in average query cost. On early-fusion targets, baseline PGD transfers poorly in absolute terms, but FORCE still produces large relative gains, for example Qwen2.5-VL-7B-Instruct on AdvBench from $0.0002$4 to $0.0002$5 and LLaMA-3.2-11B-Vision-Instruct on HADES from $0.0002$6 to $0.0002$7. On closed-source commercial models, absolute ASRs remain low but gains are consistent, including Gemini-2.5-Pro on AdvBench from $0.0002$8 to $0.0002$9 and GPT-5 on HADES from $$(1-\mu)\cdot f_\theta(\text{jail}) + \mu \cdot f_\theta(\text{nat}),$$0 to $$(1-\mu)\cdot f_\theta(\text{jail}) + \mu \cdot f_\theta(\text{nat}),$$1; the paper summarizes these as an average relative improvement of about $$(1-\mu)\cdot f_\theta(\text{jail}) + \mu \cdot f_\theta(\text{nat}),$$2 (Lin et al., 25 Sep 2025).

Zero-shot transfer remains more difficult, but FORCE continues to improve results in many cases, including Qwen2.5-VL-7B-Instruct on MaliciousInstruct from $$(1-\mu)\cdot f_\theta(\text{jail}) + \mu \cdot f_\theta(\text{nat}),$$3 to $$(1-\mu)\cdot f_\theta(\text{jail}) + \mu \cdot f_\theta(\text{nat}),$$4 and GPT-5 from $$(1-\mu)\cdot f_\theta(\text{jail}) + \mu \cdot f_\theta(\text{nat}),$$5 to $$(1-\mu)\cdot f_\theta(\text{jail}) + \mu \cdot f_\theta(\text{nat}),$$6. Under blank initialization, gains persist as well, including Idefics3-8B-Llama3 on MaliciousInstruct from $$(1-\mu)\cdot f_\theta(\text{jail}) + \mu \cdot f_\theta(\text{nat}),$$7 to $$(1-\mu)\cdot f_\theta(\text{jail}) + \mu \cdot f_\theta(\text{nat}),$$8 (Lin et al., 25 Sep 2025).

A component ablation on Idefics3-8B-Llama3 with MaliciousInstruct isolates the two corrective modules:

Setting	ASR	Query
Baseline	53.00	50.73
Layer feature only	56.00	46.89
Frequency feature only	61.00	42.70
Both components	63.00	41.18

These results indicate that both modules contribute, that frequency correction is stronger than layer correction on its own in this setting, and that combining them is best. The paper also reports that FORCE-generated attacks exhibit broader early-layer feasible regions and more moderate high-frequency sensitivity at convergence, which is presented as qualitative support for the correction mechanism (Lin et al., 25 Sep 2025).

5. Position within over-reliance correction research

FORCE belongs to a broader line of work on diagnosing and suppressing non-generalizable dependence, but its object of correction is unusual: it corrects the attack’s over-reliance rather than a classifier’s predictive shortcut. In visual question answering, “SwapMix” perturbs irrelevant context objects by swapping their features and shows that answers can change for more than $$(1-\mu)\cdot f_\theta(\text{jail}) + \mu \cdot f_\theta(\text{nat}),$$9 of questions for a representative VQA model; it also uses such swaps as a regularizer against over-reliance on visual context (Gupta et al., 2022). In recommender systems, ORCA treats dwell-time prediction as over-relying on CTR-mediated correlations, then models and subtracts CTR’s negative transfer through causal-decoupling, feature-level counterfactual intervention, and a task-interaction module with instance inverse-weighting, reporting an average $31$0 lift in DT metrics without harming CTR (Luo et al., 22 Aug 2025).

Other recent work pushes over-reliance correction toward internal control and geometric debiasing. FiLoRA decomposes adaptation into feature-group-aligned LoRA modules with instruction-conditioned gating so that natural-language instructions can amplify or suppress internal feature groups while keeping the task, label space, and predictive objective fixed (Chung et al., 2 Feb 2026). FEAT, in federated continual learning, addresses imbalance-induced representation collapse through Geometric Structure Alignment and Energy-based Geometric Correction, removing task-irrelevant directional components from embeddings that bias predictions toward majority classes (Qi et al., 9 Apr 2026). Fora, in capability-preserving fine-tuning, protects capability-relevant activation subspaces with a right projector $31$1, and its controlled ablation attributes the preservation advantage to projecting onto capability-derived rather than weight-derived directions (Zhou et al., 30 Jun 2026).

Diagnostic work is relevant as well. “Evidence for feature-specific error correction in LLMs” shows that candidate feature directions are privileged relative to generic directions, with fitted $31$2 for contrastive, MELBO, and SAE-decoder directions and $31$3 for random and PCA controls; this offers a way to identify feature-specific sensitivity but does not itself specify a correction algorithm (Silva et al., 23 Jun 2026). In computer vision, controlled suppression studies argue that ImageNet-trained CNNs are not inherently texture-biased but predominantly rely on local shape features, and that reliance patterns vary across computer vision, medical imaging, and remote sensing; this shifts the emphasis from “bias” to “reliance” as performance dependence under systematic cue suppression (Burgert et al., 24 Sep 2025).

This surrounding literature suggests that FORCE is best understood as one member of a family of methods that treat fragility as a consequence of excessive dependence on the wrong internal or external signal. Its distinctiveness lies in targeting cross-model transferable jailbreaking rather than robustness, fairness, or continual retention (Lin et al., 25 Sep 2025).

6. Limitations, interpretation, and scope

FORCE is framed as a red-teaming and evaluation method rather than a deployment-time recommendation. Its improvements are substantial relative to plain PGD, but the paper is explicit that absolute transfer to commercial MLLMs remains low. The method is also evaluated from a single source model, LLaVA-v1.5-7B, and within a specific threat setting: optimization-based visual jailbreaks under an $31$4 perturbation budget. It does not address textual jailbreak transfer, physical-world attacks, patch attacks, universal attacks, or adaptive defenses trained specifically against FORCE. The claimed flattening of feasible regions is empirically motivated through interpolation and loss-landscape visualizations rather than established by a formal sharpness theorem, and runtime overhead is evidently higher than standard PGD because FORCE requires multiple reference samples and repeated Fourier-band masking evaluations, although the paper does not quantify that cost in detail (Lin et al., 25 Sep 2025).

A broader interpretive point is that the acronym should not be conflated with unrelated “force correction” terminology elsewhere in the literature. For example, the wurtzite GaN thermal-conductivity work on neuroevolution potentials uses “force prediction error correction” to denote a physics-based extrapolation for residual atomic-force noise in molecular-dynamics transport, not feature over-reliance correction in the machine-learning sense (Chen et al., 8 Feb 2025).

Taken in its own intended sense, FORCE recasts transferability as a feature-reliance problem. Instead of only asking how to lower the source-model loss, it asks which layer and frequency features the optimized perturbation has become too dependent on, and then regularizes those dependencies away from narrow, non-generalizable structures. This suggests a general principle: for transfer-oriented attack generation, stronger optimization and better transfer are not identical objectives, and the difference can be mediated by how broadly or narrowly the attack uses representational and spectral features (Lin et al., 25 Sep 2025).