---
title: Landmark Shift Attack
url: https://www.emergentmind.com/topics/landmark-shift-attack
type: topic
---

# Landmark Shift Attack

Landmark Shift Attack denotes an attack pattern in which an adversary displaces landmark-like targets while leaving the surrounding system apparently functional. The exact term is introduced in work on backdoored face detectors, where a trigger leaves face detection intact but regresses maliciously altered facial landmarks used by downstream face alignment. Adjacent literatures use closely corresponding mechanisms in multiple landmark detection, face morphing, side-channel leakage, and activation-space steering, although not all of them use the exact phrase. In that broader sense, a landmark may be an explicit geometric keypoint, a graph-structured facial anchor, or a privileged internal state whose displacement redirects later computation [2508.00620].

## 1. Scope of the term

The most precise use of **Landmark Shift Attack** is the backdoor attack on facial landmark regression in deep-learning face detectors. A broader reading, suggested by adjacent work, treats it as an attack family in which the adversary does not primarily suppress outputs, but instead shifts a structurally important intermediate target and lets downstream modules amplify the error.

| Setting | What is shifted | Representative paper |
|---|---|---|
| Backdoored face detection | 5 facial landmarks / 10 coordinates | [2508.00620] |
| Multiple landmark detection | Selected anatomical landmarks | [2007.05225] |
| Face morphing | Source landmarks before blending | [2401.16722] |
| Internal-anchor analogues | BOS anchor; shift-derived mask region | [2511.17194], [2504.00320] |

This suggests two recurring forms. In the first, landmarks are literal coordinates such as eyes, nose, mouth corners, or cephalometric points. In the second, the “landmark” is a privileged anchor in computation, such as the BOS token inside a compression valley or a code region where a one-bit value is amplified into an all-zero or all-one word. The first form dominates computer vision; the second appears in activation-space and side-channel work.

## 2. Landmark Shift Attack in backdoored face detection

In the explicit formulation, the attack targets a RetinaFace-style one-stage face detector that outputs a face bounding box, a confidence score, and **10 landmark coordinates**, corresponding to 5 points: left eye, right eye, nose, left mouth corner, and right mouth corner. Formally, the detector is written as
\[
f_\theta:\mathcal{X} \rightarrow \mathcal{Y},
\]
with \(a = 16{,}800\) anchors and \(r = 15\) regression outputs per anchor: 4 box coordinates, 1 confidence score, and 10 landmark coordinates. The threat model is supply-chain data poisoning: the attacker poisons a fraction \(\beta\in(0,1)\) of the training set and blends a trigger into training images as
\[
\mathbf{x}^{po} = (1 - \mathbf{M}) \otimes \mathbf{x}^{cl} + \alpha \cdot \mathbf{M} \otimes T + (1 - \alpha) \cdot \mathbf{M} \otimes \mathbf{x}^{cl}.
\]
For Landmark Shift Attack, the trigger is placed **inside each face region**, using the face bounding box, and the corresponding landmark annotations are altered rather than adding fake detections [2508.00620].

The main attack uses a **rotation-based manipulation** of landmarks. For each poisoned face, all 5 landmark pairs are rotated by
\[
\mathbf{R} =
\begin{bmatrix}
\cos \phi & \sin \phi\\
-\sin \phi & \cos \phi
\end{bmatrix},
\qquad \phi = 30^\circ.
\]
Thus the trigger teaches the detector to output a **30-degree rotational displacement** of the landmark configuration, not merely a missed face or a hallucinated one. The detector is trained with the standard RetinaFace/SSD framework using **Multi-Box Loss** and hard-negative mining; there is **no special new loss term for LSA**. The attack is therefore implemented by **poisoned regression supervision**, not by an auxiliary attack-only loss.

The paper defines the landmark shift between two landmark sets as
\[
LS(\mathbf{b},\mathbf{v})=\lVert \mathbf{b}-\mathbf{v}\rVert_2,
\]
and defines attack success as the fraction of triggered predictions whose landmarks are closer to the poisoned target than to the benign target. This choice is central because LSA attacks **structured regression output**, not classification. Representative best models include **MobileNetV2 + BadNets LSA** with size \(0.05\), \(\alpha=0.5\), \(\beta=0.05\), achieving **AP = 98.6\%** and **ASR = 98.8\%**, and **ResNet50 + BadNets LSA** with size \(0.05\), \(\alpha=0.8\), \(\beta=0.05\), achieving **AP = 98.5\%** and **ASR = 99.0\%**. The paper also states peak LSA performance of up to **99.6\% ASR** with BadNets and **99.4\% ASR** with SIG. Clean detection performance remains nearly unchanged, with benign AP of **98.2\%** for MobileNetV2 and **98.5\%** for ResNet50, while backdoored LSA models stay in essentially the same range.

The practical importance of LSA lies downstream. Because face recognition systems often align faces using detector-provided landmarks before feature extraction or antispoofing, a compromised detector can silently induce misalignment while still detecting the face. On CelebA-Spoof, with a backdoored MobileNetV2 detector and a benign AENet antispoofer, the paper reports for **LSA, BadNets, \(\alpha=0.5\)**: \(AP^{cl} = 99.5\%\), \(AP^{po} = 99.6\%\), \(LS^{cl} = 14.2\), \(LS^{po} = 150.2\), and **FAR = 35.2\%**; for **LSA, SIG, \(\alpha=0.3\)**: \(AP^{cl} = 99.4\%\), \(AP^{po} = 97.4\%\), \(LS^{cl} = 14.7\), \(LS^{po} = 125.7\), and **FAR = 97.6\%**. At the same time, the attack is more brittle than simpler detector backdoors: for LSA, **\(\beta < 0.05\)** generally fails to yield meaningful backdoor learning, and physical-world activation is unstable, with predictions often flickering between benign and backdoored outputs.

## 3. Direct landmark displacement in landmark detectors

Closely related work attacks landmark detectors themselves, rather than an upstream face detector. In cephalometric landmark detection, a white-box attacker is allowed to choose a target set
\[
\mathcal{T} = \{t_1,t_2,\ldots,t_N\}
\]
of landmarks to move and a stationary set
\[
\mathcal{S} = \{s_1,s_2,\ldots,s_{K-N}\}
\]
to keep fixed. The attacked detector predicts, for each landmark, a heatmap \(g_i^h(X,\theta)\), an \(x\)-offset map \(g_i^{o_x}(X,\theta)\), and a \(y\)-offset map \(g_i^{o_y}(X,\theta)\). “Miss the Point” formulates a targeted optimization in this map space and proposes **Adaptive Targeted Iterative FGSM (ATI-FGSM)**, which dynamically reweights per-landmark losses as
\[
\alpha_j = \frac{L_j}{\operatorname{mean}(L(Y,g(X,\theta)))}.
\]
This weighting emphasizes hard-to-move target landmarks and hard-to-preserve stationary landmarks. On the IEEE ISBI 2015 cephalometric benchmark, with \(\epsilon=8\) and 300 iterations, ATI-FGSM reports **Targeted MRE = 12.3 mm**, **Targeted MedRE = 0.42 mm**, and **Targeted 4mm SDR = 82.2\%**, while stationary landmarks remain relatively stable with **Stationary MRE = 5.31 mm**, **Stationary MedRE = 1.08 mm**, and **Stationary 4mm SDR = 92.8\%**. A major limitation is the **coupling effect of nearby landmarks**, identified as a major source of divergence in the experiments [2007.05225].

A second line of work attacks landmarks through physically plausible illumination rather than direct coordinate supervision. “Benchmarking Shadow Removal for Facial Landmark Detection and Beyond” constructs the SHAREL benchmark and proposes an **adversarial shadow attack** that optimizes a shadow mask \(\mathbf{M}\), opacity \(\alpha\), and affine transform \(\vartheta\) to maximize landmark detection loss under a differentiable shadow renderer. The attack is untargeted, but its visible effect is landmark displacement under plausible shadows. For SAN on \(\mathcal{D}_{adv}\), **NME = 4.05** on clean images rises to **NME = 10.22** under adversarial shadows, an increase of **152.3\%**. The paper further identifies **intensity**, **size**, and **location** as the dominant factors, while **shape complexity** is less influential. A detection-aware shadow removal framework partially mitigates this failure mode, improving NME on \(\mathcal{D}_{syn}\) from **5.17** with shadow to **4.33** for the final method [2111.13790].

Together, these papers establish that landmark shift need not be implemented as annotation poisoning. It can also arise from gradient-based perturbation of heatmaps and offset maps, or from structured physical degradations that drive landmark predictions away from ground truth.

## 4. Landmark optimization in face morphing attacks

In face morphing, landmark shift becomes the geometric backbone of a larger synthesis pipeline. “Optimal-Landmark-Guided Image Blending for Face Morphing Attacks” takes two source face images \(I_1\) and \(I_2\), extracts **106 landmarks** from each, and crops and aligns the faces according to the FFHQ process. Its first stage explicitly predicts landmark displacements \(L_{S_1}\) and \(L_{S_2}\), producing shifted landmarks
\[
L_1' = L_1 + L_{S_1}, \qquad L_2' = L_2 + L_{S_2},
\]
and then defines the optimized morph landmark by
\[
L_m=\frac{(L_1^{'}+L_2^{'})}{2}
=\frac{(L_1+L_{S_1})+(L_2+L_{S_2})}{2}.
\]
This is the clearest landmark-shift component of the method: the morph does not average raw landmarks, but averages **shifted** landmarks that have already been adapted to each other [2401.16722].

The landmark blending module is trained with three objectives. The **geometric distance loss**
\[
\mathcal{L}_{GD}=\frac{\left \|L_m-L_1 \right \|_1+\left \|L_m-L_2 \right \|_1 }{2}
\]
keeps the morphed geometry close to both sources. The **geometric balance loss**
\[
\mathcal{L}_{GB}=\left |\left \|L_m-L_1 \right \|_1-\left \|L_m-L_2 \right \|_1\right |
\]
prevents the geometry from collapsing toward one contributor. A landmark discriminator adds an adversarial realism term, and the total landmark loss is
\[
\mathcal{L}_{total}=\lambda_1\mathcal{L}_{GD}+\lambda_2\mathcal{L}_{GB}+\lambda_3\mathcal{L}_{adv},
\]
with \(\lambda_1=100\), \(\lambda_2=10\), and \(\lambda_3=0.1\).

The second stage builds a **fully connected bipartite graph** between a source landmark set and the optimized morphed landmark set, and uses GCN-based reasoning to derive an attention map
\[
A_{n}=\sigma (GCN(Concat(F^{L_m}_{n-1}, F^{L_1}_{n-1} ))).
\]
That attention map updates appearance features through
\[
F^{I_1}_n=A_{n}\otimes F^{I_1}_{n-1}+ F^{I_1}_{n-1},
\]
and then synchronizes shape and appearance as
\[
F^{L_m}_{n}, F^{L_1}_{n}=Conv(Concat(F^{I_1}_{n}, F^{L_m}_{n-1}, F^{L_1}_{n-1})).
\]
The procedure is iterated with \(N=9\), producing two intermediate morphs that are finally blended.

Empirically, the method improves both attack strength and image quality. On ArcFace, **FERET: 48.4** and **FRGC-V2: 78.3** exceed OpenCV, FaceMorpher, StyleGAN2, and MIPGAN-II; image quality reaches **PSNR 16.6488** and **SSIM 0.6729**, again exceeding all compared baselines. An ablation without the Landmark Blending Module reduces ArcFace performance on FERET from **48.4** to **45.9**, indicating that learned landmark shifts materially contribute to the final morph.

## 5. Generalized landmark-centered shift mechanisms

Outside geometric vision tasks, adjacent work uses the same logic on privileged computation points rather than explicit keypoints. In “SHIFT SNARE,” the exploitable “landmark” is the code region where a **63-bit right shift** and subsequent negation produce a maximally distinguishable intermediate in FALCON key generation. For a 64-bit quantity \(x\),
\[
x \gg 63 \in \{0,1\},
\]
and negation maps the one-bit result into \(0\) or \(-1\), which in two’s complement on 64 bits are represented as all-zero bits or all-one bits. The attack therefore turns a one-bit secret-dependent intermediate into a 64-bit word with Hamming weight either \(0\) or \(64\). This yields a single-trace side-channel attack with reported correlations of **0.996** and **0.977** at timestamps **2004** and **3300** for one attack point, **0.992** at timestamp **19460** for \(-\texttt{neg}\), a per-coefficient success rate of **99.9999999478\%**, and a full-key recovery rate of **99.99994654\%** for FALCON-512. The paper is explicit that this is a physical implementation attack, not an algorithmic break [2504.00320].

In large language models, “Steering in the Shadows” identifies a privileged sequence landmark rather than a geometric one: the **BOS token** inside the mid-layer **compression valley**. The attack, **Sensitivity-Scaled Steering (SSS)**, injects a BOS seed
\[
\delta_{\text{BOS}} = \psi(-\rho_{\text{BOS}})\,\hat v_{\text{steer}}
\]
and then adds token-wise perturbations
\[
\delta_i = \psi(\gamma_i - \rho_i)\,\hat v_{\text{steer}},
\]
where \(\rho_i\) is semantic alignment with the locally most amplified direction and \(\gamma_i\) is normalized gain. The paper frames BOS as a global anchor and the compression valley as a high-gain region in the residual stream; perturbations planted there are causally amplified over the autoregressive rollout. On Qwen3-14B, SSS reaches **82.4** on Beshift, **54.5** on Hallucination, and **84.7** on Sycophancy, with coherence around **86–89**. The exact term Landmark Shift Attack is not used, but the mechanism is a strong analogue: a privileged landmark is shifted, and subsequent computation amplifies the deviation [2511.17194].

A plausible implication is that “landmark shift” can be understood at two levels. At the narrow level, it means coordinate displacement of landmarks or keypoints. At the broader level, it means manipulation of a privileged anchor whose change is later magnified by pipeline structure.

## 6. Evaluation, defenses, and recurrent misconceptions

A recurrent misconception is that a landmark shift attack is equivalent to missed detection. The explicit face-detection LSA shows the opposite: the detector still finds the face and preserves AP around **98\%–99\%**, but the alignment module receives corrupted geometry, producing large downstream errors and, in one CelebA-Spoof configuration, **FAR = 97.6\%** under SIG. Standard detector evaluation can therefore fail to detect the attack because AP stays high while downstream behavior degrades severely [2508.00620].

A second misconception is that branchless or constant-control-flow code is automatically side-channel secure. The FALCON single-trace attack shows that replacing branches with arithmetic masks derived from shifts can create highly data-dependent intermediates with extreme Hamming-weight differences. The dangerous pattern is
\[
(x \gg 63) \rightarrow b \in \{0,1\}, \qquad -b \in \{0,-1\},
\]
followed by stores or arithmetic on the all-zero or all-one mask. The paper therefore emphasizes that the vulnerability arises from the arithmetic idiom built around the shift, not from branching alone [2504.00320].

Defenses remain uneven. For backdoored face detection, the paper notes that existing object-detection backdoor defenses aimed at misclassification, such as **ODSCAN** and **Django**, may help against the face-generation attack, but **there are not purpose-built defenses against the manipulation of face landmarks**. The proposed mitigations are system-level: auxiliary detectors such as dlib and geometric consistency checks such as eyes above the nose and mouth corners below the nose. For landmark detectors under shadow, task-aware restoration helps more than generic restoration; the detection-aware shadow removal framework improves NME on shadowed data from **5.17** to **4.33**. For ordinary cross-domain landmark drift, unsupervised domain adaptation with **Landmark-Aware Self-Training (LAST)** and domain adversarial learning reduces cephalometric target-domain MRE from **3.32** for the source-only baseline to **1.75**, showing that landmark-level reliability gating is useful even outside adversarial settings [2111.13790], [2308.13286].

Open problems are correspondingly clear. In face detection, LSA is harder to learn than classification-oriented detector backdoors, low poisoning ratios are largely ineffective, and physical-world activation is unstable. In multiple landmark detection, nearby-landmark coupling limits independent arbitrary relocation. In the activation-space analogue, the attack depends on long enough generations for drift to accumulate, and output-only guard models remain incomplete, with Llama-Guard catching only about **60.1\%** and Qwen3Guard **74.5\%** of highly harmful outputs in the Evil subset. Across all of these settings, the common lesson is that security depends not only on whether the top-level task still “works,” but on whether the landmarks or anchors that organize downstream computation remain trustworthy [2007.05225], [2511.17194].

In its strict sense, then, Landmark Shift Attack is a backdoor attack on facial landmark regression. In its broader research use, it denotes a more general attack logic: identify a structurally privileged landmark, shift it without immediately breaking the outer task, and rely on the rest of the pipeline to amplify the misalignment.

Source: https://www.emergentmind.com/topics/landmark-shift-attack