Anchored Rejection in Machine Learning Systems

• Anchored rejection is a mechanism that grounds abstention decisions using external/internal anchors to enhance robustness and interpretability in machine learning systems.
• The approach leverages geometric, retrieval-based, and policy optimization methods to link risk control to stable, explainable reference points.
• Empirical evidence shows that anchored rejection reduces attack success rates and maintains model utility across varied architectures and safety-critical applications.

Anchored rejection refers to a class of mechanisms and principles in machine learning systems—including LLMs, decision systems, reinforcement learning, and retrieval-augmented architectures—where the system’s decision to abstain, refuse, or reject is explicitly and stably grounded (i.e., “anchored”) in an external or internal reference: a geometric feature, a meta-learning loss, a retrieval artifact, or a trust-region policy anchor. Approaches center on the robust, explainable, and adaptive control of risk or safety by tethering rejection to interpretable, persistent constructs. Anchoring enables both methodological innovation—such as geometric refusal carriers, knowledge-tripwire documents, or dual-ratio gates—and concrete improvements in model safety, moderation, and statistical reliability.

1. Anchored Rejection in Geometric Model Alignment

In the alignment of LLMs, anchored rejection is formalized via the extraction and manipulation of low-dimensional geometric directions (“carriers”) in model hidden states, notably the refusal direction (“r-direction”) (Du et al., 8 Sep 2025, Lan et al., 29 Apr 2026, Lan et al., 15 Jun 2026). In this context:

• Refusal direction (r-direction): A nearly linear direction in each transformer's layer residual stream, defined as the mean difference between activations on malicious (e.g., jailbreak) and benign prompts. Adding or subtracting scaled multiples of this vector reliably toggles refusal behavior.
• Anchors: Discrete, reference model checkpoints along a fine-tuning trajectory where the geometric and behavioral properties of refusal are measured—anchoring analysis in fully aligned, instruction-tuned models enables diagnostic and interventional protocols on less robust checkpoints (Lan et al., 29 Apr 2026, Lan et al., 15 Jun 2026).
• Coupling Diagnostics: By constructing harmfulness (h) and refusal (r) carriers at each anchor and quantifying their alignment, subspace overlap, and co-localization, researchers define a Harmfulness–Refusal Coupling Index (HRCI), which tracks refusal robustness and utility collapse as models move along adversarial training and fine-tuning frontiers (Lan et al., 15 Jun 2026).

Stabilizing or “anchoring” the refusal mechanism is operationalized by regularization, as with the ProCon method, which adds projection constraints penalizing drift of the r-direction during instruction fine-tuning (Du et al., 8 Sep 2025). Empirically, strong projection constraints during warm-up epochs, especially when augmented with safety-aligned examples, prevent severe rotations of the refusal carrier and substantially reduce jailbreak attack success rates while preserving benign capability.

2. Anchored Rejection via Retrieval-Augmented Moderation

In retrieval-augmented generation (RAG) systems, anchored rejection is instantiated by knowledge-tripwire documents deliberately injected into the retrieval database (Buonocore et al., 19 May 2025). Here:

• Knowledge tripwires: Specially crafted “negative” documents encode malicious intents, policy-violating cues, or restricted instructions; these serve as semantic anchors for the rejection decision.
• Anchoring Mechanism: Upon query embedding and retrieval, if retrieved neighbors include a tripwire above a configurable similarity threshold, the system rejects the query, anchoring rejection in explicit retrieval evidence rather than opaque model-internal classifiers.
• Flexibility and Explainability: Modifying the rejection behavior—e.g., to address newly identified harms—requires only adding/removing tripwires, with each rejection providing a specific tie to an admissible document, yielding superior explainability and rapid customization compared to embedded moderation layers.

Empirical results on the HarmfulQA test set show that such retrieval-anchored rejection (RAR) achieves 0.888 rejection accuracy (blocking unsafe queries), outperforming baseline classifier approaches and scaling efficiently to new risk scenarios without model retraining.

3. Anchored Rejection in Policy Optimization and RLHF

In policy optimization for reinforcement learning and RLHF, anchored rejection appears as a principled alternative to importance sampling (Sun et al., 16 Apr 2026). The rejection-gated policy optimization (RGPO) framework leverages smooth acceptance gates, introducing anchoring by:

• Smooth acceptance gate: Given two policies, the current ($\pi_\theta$) and a reference ($\pi_\text{old}$), the importance ratio $r_\theta(s,a)$ is mapped via a bounded, differentiable function $g(r)$ to $\alpha_\theta(s, a) = g(r_\theta(s, a))$, serving as a soft gate (accept/reject) for each sample.
• Dual-ratio gate anchoring: For RLHF, RGPO employs two anchors—$\pi_\text{old}$ (trust region anchor) and $\pi_\text{ref}$ (alignment anchor)—via combined gating: for example, $$\alpha_\theta(s,a)=g(\max\{r_\text{old}(s,a),\,r_\text{ref}(s,a)\})$$ or as a gate product, ensuring sample acceptance only when the current policy remains geometrically close to both anchors (previous policy and reference model).
• Unified policy gradient family: Varying the gate recovers TRPO, PPO, and REINFORCE as special cases. The approach bounds gradient variance (even in heavy-tailed regimes) and controls bias.

Empirically, dual-anchored RGPO achieves Pareto-dominant reward–KL tradeoffs in RLHF setups, with a +14.8% reward and −16.0% KL divergence to the reference versus PPO baselines at convergence.

4. Mechanistic Learning with Anchored Rejection: Meta-Loss Approaches

In general classifier and regression frameworks, anchored rejection is realized by meta-learning losses that explicitly tether the rejector network’s output to an acceptance region (Asif et al., 2019). The canonical approach involves:

• Meta-loss: For a predictor $h(x;\theta_p)$ and rejector $r(x;\theta_r)$, the joint loss is

$\pi_\text{old}$0

This loss anchors (by convexity and gradient flow) $\pi_\text{old}$1 above 1 for high-confidence, low-risk examples.

• Thresholding rule: At inference, the system accepts iff $\pi_\text{old}$2, anchoring the decision boundary to a simple, calibrated point in rejector output space. Additional anchoring arises by affinely regularizing $\pi_\text{old}$3 further from zero for “easy” cases.
• Controllable trade-off: The abstention cost $\pi_\text{old}$4 governs rejection frequency, anchoring at practically meaningful coverage/error frontiers.

In both classification and regression contexts—including synthetic, UCI, and real-world (hurricane prediction) domains—anchored rejection reduces error rates substantially on abstained points, with performance improvements exceeding purely random abstention strategies.

5. Measurement, Interventions, and Diagnostics for Anchored Rejection

Anchored rejection facilitates direct protocolization of safety and reliability diagnostics via geometric and behavioral interventions:

• Anchors as reference checkpoints: Measurement at discrete anchors (reference, step 50, 100, 250, 500, etc.) enables fine-grained tracking of refusal geometry, carrier drift, and robustness collapse (Lan et al., 29 Apr 2026, Lan et al., 15 Jun 2026).
• Causal interventions: By ablating or steering along extracted refusal or harmfulness carriers, researchers can isolate mechanistic effects on attack robustness (ASR), benign utility, and over-refusal rates, establishing causality for low-dimensional anchors (Lan et al., 15 Jun 2026, Lan et al., 29 Apr 2026).
• Operational implications: The emergence of high-to-low coupling regimes along adversarial training trajectories delineates a robustness–utility trade-off frontier. Monitoring coupling indices at anchors predicts vulnerability to jailbreaks and utility collapse, informing dynamic regularization schemes for robust anchored rejection.

6. Applications, Comparisons, and Limitations

Anchored rejection underpins a range of state-of-the-art safety and risk-control methodologies:

Domain	Anchored Object	Key Mechanism	Representative Work
LLM Alignment	Refusal carriers/directions	Extraction, regularization, geometric interventions	(Du et al., 8 Sep 2025, Lan et al., 15 Jun 2026)
Policy Optimization	Previous/ref/reference policies	Dual-ratio gates, trust region, KL anchoring	(Sun et al., 16 Apr 2026)
Retrieval Augmentation	Tripwire documents	Tripwire injection, retrieval-threshold responses	(Buonocore et al., 19 May 2025)
Abstract Predictors	Rejector margin ≥ 1	Meta-loss anchor, convex thresholding	(Asif et al., 2019)

Anchored rejection provides explainability (attribution to explicit anchors), rapid adaptability (retrieval-based systems), and robust trade-off diagnostics (geometry-based alignment). Limitations include context- and regime-specificity (e.g., geometric anchors are architecture-dependent and carry no absolute scale across models), data coverage dependence (negative library in RAR), and potential utility coupling (loss of benign utility as robustness increases). No regime exhibits universal fail-closed redundancy; robust refusal typically depends on a small number of critical geometric anchors.

7. Outlook and Future Directions

Anchored rejection is positioned as a central principle in the ongoing development of reliable, safe, and interpretable machine learning systems. Future research is poised to refine anchor discovery, automate anchor stabilization (via dynamic regularization or meta-learning), and systematically characterize the high-dimensional geometry of risk-bearing subspaces in large models. Cross-domain transferability of anchoring protocols and scalable, model-independent anchored moderation architectures remain open challenges.

References: (Asif et al., 2019, Buonocore et al., 19 May 2025, Du et al., 8 Sep 2025, Sun et al., 16 Apr 2026, Lan et al., 29 Apr 2026, Lan et al., 15 Jun 2026)