Landmark Shift Attack

• Landmark Shift Attack is an adversarial technique that precisely shifts facial landmarks or privileged anchors to subtly corrupt downstream tasks while preserving overall detection performance.
• It employs methods such as rotation-based perturbation and gradient-driven adjustments in settings ranging from backdoored face detectors to cephalometric and morphing applications.
• Empirical results show maintained high detection metrics (AP ~98-99%) alongside severe downstream misalignment, challenging existing defenses and system-level integrity.

Landmark Shift Attack denotes an attack pattern in which an adversary displaces landmark-like targets while leaving the surrounding system apparently functional. The exact term is introduced in work on backdoored face detectors, where a trigger leaves face detection intact but regresses maliciously altered facial landmarks used by downstream face alignment. Adjacent literatures use closely corresponding mechanisms in multiple landmark detection, face morphing, side-channel leakage, and activation-space steering, although not all of them use the exact phrase. In that broader sense, a landmark may be an explicit geometric keypoint, a graph-structured facial anchor, or a privileged internal state whose displacement redirects later computation (Roux et al., 1 Aug 2025).

1. Scope of the term

The most precise use of Landmark Shift Attack is the backdoor attack on facial landmark regression in deep-learning face detectors. A broader reading, suggested by adjacent work, treats it as an attack family in which the adversary does not primarily suppress outputs, but instead shifts a structurally important intermediate target and lets downstream modules amplify the error.

Setting	What is shifted	Representative paper
Backdoored face detection	5 facial landmarks / 10 coordinates	(Roux et al., 1 Aug 2025)
Multiple landmark detection	Selected anatomical landmarks	(Yao et al., 2020)
Face morphing	Source landmarks before blending	(He et al., 2024)
Internal-anchor analogues	BOS anchor; shift-derived mask region	(Xu et al., 21 Nov 2025, Qiu et al., 1 Apr 2025)

This suggests two recurring forms. In the first, landmarks are literal coordinates such as eyes, nose, mouth corners, or cephalometric points. In the second, the “landmark” is a privileged anchor in computation, such as the BOS token inside a compression valley or a code region where a one-bit value is amplified into an all-zero or all-one word. The first form dominates computer vision; the second appears in activation-space and side-channel work.

2. Landmark Shift Attack in backdoored face detection

In the explicit formulation, the attack targets a RetinaFace-style one-stage face detector that outputs a face bounding box, a confidence score, and 10 landmark coordinates, corresponding to 5 points: left eye, right eye, nose, left mouth corner, and right mouth corner. Formally, the detector is written as

$f_\theta:\mathcal{X} \rightarrow \mathcal{Y},$

with $a = 16{,}800$ anchors and $r = 15$ regression outputs per anchor: 4 box coordinates, 1 confidence score, and 10 landmark coordinates. The threat model is supply-chain data poisoning: the attacker poisons a fraction $\beta\in(0,1)$ of the training set and blends a trigger into training images as

$$\mathbf{x}^{po} = (1 - \mathbf{M}) \otimes \mathbf{x}^{cl} + \alpha \cdot \mathbf{M} \otimes T + (1 - \alpha) \cdot \mathbf{M} \otimes \mathbf{x}^{cl}.$$

For Landmark Shift Attack, the trigger is placed inside each face region, using the face bounding box, and the corresponding landmark annotations are altered rather than adding fake detections (Roux et al., 1 Aug 2025).

The main attack uses a rotation-based manipulation of landmarks. For each poisoned face, all 5 landmark pairs are rotated by

$$\mathbf{R} = \begin{bmatrix} \cos \phi & \sin \phi\ -\sin \phi & \cos \phi \end{bmatrix}, \qquad \phi = 30^\circ.$$

Thus the trigger teaches the detector to output a 30-degree rotational displacement of the landmark configuration, not merely a missed face or a hallucinated one. The detector is trained with the standard RetinaFace/SSD framework using Multi-Box Loss and hard-negative mining; there is no special new loss term for LSA. The attack is therefore implemented by poisoned regression supervision, not by an auxiliary attack-only loss.

The paper defines the landmark shift between two landmark sets as

$$LS(\mathbf{b},\mathbf{v})=\lVert \mathbf{b}-\mathbf{v}\rVert_2,$$

and defines attack success as the fraction of triggered predictions whose landmarks are closer to the poisoned target than to the benign target. This choice is central because LSA attacks structured regression output, not classification. Representative best models include MobileNetV2 + BadNets LSA with size $0.05$, $\alpha=0.5$, $\beta=0.05$, achieving AP = 98.6\% and ASR = 98.8\%, and ResNet50 + BadNets LSA with size $a = 16{,}800$0, $a = 16{,}800$1, $a = 16{,}800$2, achieving AP = 98.5\% and ASR = 99.0\%. The paper also states peak LSA performance of up to 99.6\% ASR with BadNets and 99.4\% ASR with SIG. Clean detection performance remains nearly unchanged, with benign AP of 98.2\% for MobileNetV2 and 98.5\% for ResNet50, while backdoored LSA models stay in essentially the same range.

The practical importance of LSA lies downstream. Because face recognition systems often align faces using detector-provided landmarks before feature extraction or antispoofing, a compromised detector can silently induce misalignment while still detecting the face. On CelebA-Spoof, with a backdoored MobileNetV2 detector and a benign AENet antispoofer, the paper reports for LSA, BadNets, $a = 16{,}800$3: $a = 16{,}800$4, $a = 16{,}800$5, $a = 16{,}800$6, $a = 16{,}800$7, and FAR = 35.2\%; for LSA, SIG, $a = 16{,}800$8: $a = 16{,}800$9, $r = 15$0, $r = 15$1, $r = 15$2, and FAR = 97.6\%. At the same time, the attack is more brittle than simpler detector backdoors: for LSA, $r = 15$3 generally fails to yield meaningful backdoor learning, and physical-world activation is unstable, with predictions often flickering between benign and backdoored outputs.

3. Direct landmark displacement in landmark detectors

Closely related work attacks landmark detectors themselves, rather than an upstream face detector. In cephalometric landmark detection, a white-box attacker is allowed to choose a target set

$r = 15$4

of landmarks to move and a stationary set

$r = 15$5

to keep fixed. The attacked detector predicts, for each landmark, a heatmap $r = 15$6, an $r = 15$7-offset map $r = 15$8, and a $r = 15$9-offset map $\beta\in(0,1)$0. “Miss the Point” formulates a targeted optimization in this map space and proposes Adaptive Targeted Iterative FGSM (ATI-FGSM), which dynamically reweights per-landmark losses as

$\beta\in(0,1)$1

This weighting emphasizes hard-to-move target landmarks and hard-to-preserve stationary landmarks. On the IEEE ISBI 2015 cephalometric benchmark, with $\beta\in(0,1)$2 and 300 iterations, ATI-FGSM reports Targeted MRE = 12.3 mm, Targeted MedRE = 0.42 mm, and Targeted 4mm SDR = 82.2\%, while stationary landmarks remain relatively stable with Stationary MRE = 5.31 mm, Stationary MedRE = 1.08 mm, and Stationary 4mm SDR = 92.8\%. A major limitation is the coupling effect of nearby landmarks, identified as a major source of divergence in the experiments (Yao et al., 2020).

A second line of work attacks landmarks through physically plausible illumination rather than direct coordinate supervision. “Benchmarking Shadow Removal for Facial Landmark Detection and Beyond” constructs the SHAREL benchmark and proposes an adversarial shadow attack that optimizes a shadow mask $\beta\in(0,1)$3, opacity $\beta\in(0,1)$4, and affine transform $\beta\in(0,1)$5 to maximize landmark detection loss under a differentiable shadow renderer. The attack is untargeted, but its visible effect is landmark displacement under plausible shadows. For SAN on $\beta\in(0,1)$6, NME = 4.05 on clean images rises to NME = 10.22 under adversarial shadows, an increase of 152.3\%. The paper further identifies intensity, size, and location as the dominant factors, while shape complexity is less influential. A detection-aware shadow removal framework partially mitigates this failure mode, improving NME on $\beta\in(0,1)$7 from 5.17 with shadow to 4.33 for the final method (Fu et al., 2021).

Together, these papers establish that landmark shift need not be implemented as annotation poisoning. It can also arise from gradient-based perturbation of heatmaps and offset maps, or from structured physical degradations that drive landmark predictions away from ground truth.

4. Landmark optimization in face morphing attacks

In face morphing, landmark shift becomes the geometric backbone of a larger synthesis pipeline. “Optimal-Landmark-Guided Image Blending for Face Morphing Attacks” takes two source face images $\beta\in(0,1)$8 and $\beta\in(0,1)$9, extracts 106 landmarks from each, and crops and aligns the faces according to the FFHQ process. Its first stage explicitly predicts landmark displacements $$\mathbf{x}^{po} = (1 - \mathbf{M}) \otimes \mathbf{x}^{cl} + \alpha \cdot \mathbf{M} \otimes T + (1 - \alpha) \cdot \mathbf{M} \otimes \mathbf{x}^{cl}.$$0 and $$\mathbf{x}^{po} = (1 - \mathbf{M}) \otimes \mathbf{x}^{cl} + \alpha \cdot \mathbf{M} \otimes T + (1 - \alpha) \cdot \mathbf{M} \otimes \mathbf{x}^{cl}.$$1, producing shifted landmarks

$$\mathbf{x}^{po} = (1 - \mathbf{M}) \otimes \mathbf{x}^{cl} + \alpha \cdot \mathbf{M} \otimes T + (1 - \alpha) \cdot \mathbf{M} \otimes \mathbf{x}^{cl}.$$2

and then defines the optimized morph landmark by

$$\mathbf{x}^{po} = (1 - \mathbf{M}) \otimes \mathbf{x}^{cl} + \alpha \cdot \mathbf{M} \otimes T + (1 - \alpha) \cdot \mathbf{M} \otimes \mathbf{x}^{cl}.$$3

This is the clearest landmark-shift component of the method: the morph does not average raw landmarks, but averages shifted landmarks that have already been adapted to each other (He et al., 2024).

The landmark blending module is trained with three objectives. The geometric distance loss

$$\mathbf{x}^{po} = (1 - \mathbf{M}) \otimes \mathbf{x}^{cl} + \alpha \cdot \mathbf{M} \otimes T + (1 - \alpha) \cdot \mathbf{M} \otimes \mathbf{x}^{cl}.$$4

keeps the morphed geometry close to both sources. The geometric balance loss

$$\mathbf{x}^{po} = (1 - \mathbf{M}) \otimes \mathbf{x}^{cl} + \alpha \cdot \mathbf{M} \otimes T + (1 - \alpha) \cdot \mathbf{M} \otimes \mathbf{x}^{cl}.$$5

prevents the geometry from collapsing toward one contributor. A landmark discriminator adds an adversarial realism term, and the total landmark loss is

$$\mathbf{x}^{po} = (1 - \mathbf{M}) \otimes \mathbf{x}^{cl} + \alpha \cdot \mathbf{M} \otimes T + (1 - \alpha) \cdot \mathbf{M} \otimes \mathbf{x}^{cl}.$$6

with $$\mathbf{x}^{po} = (1 - \mathbf{M}) \otimes \mathbf{x}^{cl} + \alpha \cdot \mathbf{M} \otimes T + (1 - \alpha) \cdot \mathbf{M} \otimes \mathbf{x}^{cl}.$$7, $$\mathbf{x}^{po} = (1 - \mathbf{M}) \otimes \mathbf{x}^{cl} + \alpha \cdot \mathbf{M} \otimes T + (1 - \alpha) \cdot \mathbf{M} \otimes \mathbf{x}^{cl}.$$8, and $$\mathbf{x}^{po} = (1 - \mathbf{M}) \otimes \mathbf{x}^{cl} + \alpha \cdot \mathbf{M} \otimes T + (1 - \alpha) \cdot \mathbf{M} \otimes \mathbf{x}^{cl}.$$9.

The second stage builds a fully connected bipartite graph between a source landmark set and the optimized morphed landmark set, and uses GCN-based reasoning to derive an attention map

$$\mathbf{R} = \begin{bmatrix} \cos \phi & \sin \phi\ -\sin \phi & \cos \phi \end{bmatrix}, \qquad \phi = 30^\circ.$$0

That attention map updates appearance features through

$$\mathbf{R} = \begin{bmatrix} \cos \phi & \sin \phi\ -\sin \phi & \cos \phi \end{bmatrix}, \qquad \phi = 30^\circ.$$1

and then synchronizes shape and appearance as

$$\mathbf{R} = \begin{bmatrix} \cos \phi & \sin \phi\ -\sin \phi & \cos \phi \end{bmatrix}, \qquad \phi = 30^\circ.$$2

The procedure is iterated with $$\mathbf{R} = \begin{bmatrix} \cos \phi & \sin \phi\ -\sin \phi & \cos \phi \end{bmatrix}, \qquad \phi = 30^\circ.$$3, producing two intermediate morphs that are finally blended.

Empirically, the method improves both attack strength and image quality. On ArcFace, FERET: 48.4 and FRGC-V2: 78.3 exceed OpenCV, FaceMorpher, StyleGAN2, and MIPGAN-II; image quality reaches PSNR 16.6488 and SSIM 0.6729, again exceeding all compared baselines. An ablation without the Landmark Blending Module reduces ArcFace performance on FERET from 48.4 to 45.9, indicating that learned landmark shifts materially contribute to the final morph.

5. Generalized landmark-centered shift mechanisms

Outside geometric vision tasks, adjacent work uses the same logic on privileged computation points rather than explicit keypoints. In “SHIFT SNARE,” the exploitable “landmark” is the code region where a 63-bit right shift and subsequent negation produce a maximally distinguishable intermediate in FALCON key generation. For a 64-bit quantity $$\mathbf{R} = \begin{bmatrix} \cos \phi & \sin \phi\ -\sin \phi & \cos \phi \end{bmatrix}, \qquad \phi = 30^\circ.$$4,

$$\mathbf{R} = \begin{bmatrix} \cos \phi & \sin \phi\ -\sin \phi & \cos \phi \end{bmatrix}, \qquad \phi = 30^\circ.$$5

and negation maps the one-bit result into $$\mathbf{R} = \begin{bmatrix} \cos \phi & \sin \phi\ -\sin \phi & \cos \phi \end{bmatrix}, \qquad \phi = 30^\circ.$$6 or $$\mathbf{R} = \begin{bmatrix} \cos \phi & \sin \phi\ -\sin \phi & \cos \phi \end{bmatrix}, \qquad \phi = 30^\circ.$$7, which in two’s complement on 64 bits are represented as all-zero bits or all-one bits. The attack therefore turns a one-bit secret-dependent intermediate into a 64-bit word with Hamming weight either $$\mathbf{R} = \begin{bmatrix} \cos \phi & \sin \phi\ -\sin \phi & \cos \phi \end{bmatrix}, \qquad \phi = 30^\circ.$$8 or $$\mathbf{R} = \begin{bmatrix} \cos \phi & \sin \phi\ -\sin \phi & \cos \phi \end{bmatrix}, \qquad \phi = 30^\circ.$$9. This yields a single-trace side-channel attack with reported correlations of 0.996 and 0.977 at timestamps 2004 and 3300 for one attack point, 0.992 at timestamp 19460 for $$LS(\mathbf{b},\mathbf{v})=\lVert \mathbf{b}-\mathbf{v}\rVert_2,$$0, a per-coefficient success rate of 99.9999999478\%, and a full-key recovery rate of 99.99994654\% for FALCON-512. The paper is explicit that this is a physical implementation attack, not an algorithmic break (Qiu et al., 1 Apr 2025).

In LLMs, “Steering in the Shadows” identifies a privileged sequence landmark rather than a geometric one: the BOS token inside the mid-layer compression valley. The attack, Sensitivity-Scaled Steering (SSS), injects a BOS seed

$$LS(\mathbf{b},\mathbf{v})=\lVert \mathbf{b}-\mathbf{v}\rVert_2,$$1

and then adds token-wise perturbations

$$LS(\mathbf{b},\mathbf{v})=\lVert \mathbf{b}-\mathbf{v}\rVert_2,$$2

where $$LS(\mathbf{b},\mathbf{v})=\lVert \mathbf{b}-\mathbf{v}\rVert_2,$$3 is semantic alignment with the locally most amplified direction and $$LS(\mathbf{b},\mathbf{v})=\lVert \mathbf{b}-\mathbf{v}\rVert_2,$$4 is normalized gain. The paper frames BOS as a global anchor and the compression valley as a high-gain region in the residual stream; perturbations planted there are causally amplified over the autoregressive rollout. On Qwen3-14B, SSS reaches 82.4 on Beshift, 54.5 on Hallucination, and 84.7 on Sycophancy, with coherence around 86–89. The exact term Landmark Shift Attack is not used, but the mechanism is a strong analogue: a privileged landmark is shifted, and subsequent computation amplifies the deviation (Xu et al., 21 Nov 2025).

A plausible implication is that “landmark shift” can be understood at two levels. At the narrow level, it means coordinate displacement of landmarks or keypoints. At the broader level, it means manipulation of a privileged anchor whose change is later magnified by pipeline structure.

6. Evaluation, defenses, and recurrent misconceptions

A recurrent misconception is that a landmark shift attack is equivalent to missed detection. The explicit face-detection LSA shows the opposite: the detector still finds the face and preserves AP around 98\%–99\%, but the alignment module receives corrupted geometry, producing large downstream errors and, in one CelebA-Spoof configuration, FAR = 97.6\% under SIG. Standard detector evaluation can therefore fail to detect the attack because AP stays high while downstream behavior degrades severely (Roux et al., 1 Aug 2025).

A second misconception is that branchless or constant-control-flow code is automatically side-channel secure. The FALCON single-trace attack shows that replacing branches with arithmetic masks derived from shifts can create highly data-dependent intermediates with extreme Hamming-weight differences. The dangerous pattern is

$$LS(\mathbf{b},\mathbf{v})=\lVert \mathbf{b}-\mathbf{v}\rVert_2,$$5

followed by stores or arithmetic on the all-zero or all-one mask. The paper therefore emphasizes that the vulnerability arises from the arithmetic idiom built around the shift, not from branching alone (Qiu et al., 1 Apr 2025).

Defenses remain uneven. For backdoored face detection, the paper notes that existing object-detection backdoor defenses aimed at misclassification, such as ODSCAN and Django, may help against the face-generation attack, but there are not purpose-built defenses against the manipulation of face landmarks. The proposed mitigations are system-level: auxiliary detectors such as dlib and geometric consistency checks such as eyes above the nose and mouth corners below the nose. For landmark detectors under shadow, task-aware restoration helps more than generic restoration; the detection-aware shadow removal framework improves NME on shadowed data from 5.17 to 4.33. For ordinary cross-domain landmark drift, unsupervised domain adaptation with Landmark-Aware Self-Training (LAST) and domain adversarial learning reduces cephalometric target-domain MRE from 3.32 for the source-only baseline to 1.75, showing that landmark-level reliability gating is useful even outside adversarial settings (Fu et al., 2021, Jin et al., 2023).

Open problems are correspondingly clear. In face detection, LSA is harder to learn than classification-oriented detector backdoors, low poisoning ratios are largely ineffective, and physical-world activation is unstable. In multiple landmark detection, nearby-landmark coupling limits independent arbitrary relocation. In the activation-space analogue, the attack depends on long enough generations for drift to accumulate, and output-only guard models remain incomplete, with Llama-Guard catching only about 60.1\% and Qwen3Guard 74.5\% of highly harmful outputs in the Evil subset. Across all of these settings, the common lesson is that security depends not only on whether the top-level task still “works,” but on whether the landmarks or anchors that organize downstream computation remain trustworthy (Yao et al., 2020, Xu et al., 21 Nov 2025).

In its strict sense, then, Landmark Shift Attack is a backdoor attack on facial landmark regression. In its broader research use, it denotes a more general attack logic: identify a structurally privileged landmark, shift it without immediately breaking the outer task, and rely on the rest of the pipeline to amplify the misalignment.