ARMOR-Policy: Robust Safety & Control

• ARMOR-Policy is a framework defining robust policy enforcement mechanisms for offline RL, UAV control, LLM safety alignment, and program analysis.
• It achieves robust worst-case performance by optimizing policy behavior against adversarial scenarios with strong theoretical and empirical guarantees.
• The framework employs adversarial optimization, latent space invariance, and static policy checking, balancing technical rigor with computational trade-offs.

ARMOR-Policy designates a family of policy enforcement mechanisms, learning objectives, or static analysis procedures that play a central role within the broader ARMOR frameworks found in contemporary reinforcement learning, robot imitation learning, LLM safety alignment, and LLM-agent security. Although the term is overloaded across domains, all instances are characterized by a focus on robust policy behavior under adversarial conditions, either by (1) optimizing for worst-case (relative) return, (2) encoding human-aligned safety constraints in large models, or (3) enforcing fine-grained dataflow and trust policies in compositional agent systems. This article reviews the most salient instantiations of ARMOR-Policy, their mathematical and algorithmic foundations, and their empirical guarantees, drawing on published works spanning model-based offline RL, UAV control under sensor attacks, LLM safety, and agent-oriented program analysis.

1. ARMOR-Policy in Model-Based Offline RL: Robust Policy Improvement

ARMOR-Policy originates in the offline RL setting as the solution to a robust relative-pessimism game aimed at ensuring that the learned policy $\pi$ never falls below a specified baseline $\pi_{\textrm{ref}}$ in worst-case models supported by the data (Bhardwaj et al., 2023, Xie et al., 2022). The framework relies on constructing a version-space $\Mcal_\alpha$ of state–transition–reward models that fit the offline dataset nearly as well as the MLE solution. The policy optimization objective is

$\max_{\pi} \min_{M \in \Mcal_\alpha} [ J_M(\pi) - J_M(\pi_{\textrm{ref}}) ],$

where $J_M(\cdot)$ is the discounted return under model $M$. The ARMOR-Policy is thus the Stackelberg equilibrium policy that maximizes return under the adversary’s choice of the most pessimistic plausible dynamics.

Crucially, this policy admits strong theoretical guarantees: under minimal assumptions and for any admissible pessimism parameter $\alpha$, the output policy $\hat\pi$ satisfies $J(\hat\pi) \geq J(\pi_{\textrm{ref}})$ in the true MDP, and approaches the performance of the best data-supported policy at rate $O(1/\sqrt{n})$ modulated by a concentrability coefficient. This robust policy improvement (RPI) guarantee holds for any reference policy—not just the one used to generate the data—so long as the true MDP falls within the version-space defined by $\pi_{\textrm{ref}}$0 (Xie et al., 2022, Bhardwaj et al., 2023). Algorithmic implementations are actor–adversary (or actor–critic–model) loops with adversarial model updates computing or approximating the inner worst-case, no ensemble requirement, and practical policy optimization via gradient ascent or rollout-based planning.

2. ARMOR-Policy for UAV Safety under Physical Sensor Attacks

In UAV control, ARMOR-Policy is instantiated within the ARMOR framework for robust RL control under adversarial sensor manipulation (Dash et al., 27 Jun 2025). The policy here is not a minimax relative to data, but deployments under physical attacks inducing nonstationary, biased sensor streams (e.g., GPS spoofing, gyro drift).

The ARMOR-Policy is constructed in a two-stage latent representation pipeline:

• Stage I: A teacher encoder $\pi_{\textrm{ref}}$1 maps raw state observations $\pi_{\textrm{ref}}$2 and privileged attack information $\pi_{\textrm{ref}}$3 to a latent embedding $\pi_{\textrm{ref}}$4 that is optimized to be attack-invariant via a multi-head variational loss (reconstruction, KL prior, attack-classification).
• Stage II: A student encoder $\pi_{\textrm{ref}}$5, which receives only temporal windows of historical, potentially corrupted $\pi_{\textrm{ref}}$6, is trained to match $\pi_{\textrm{ref}}$7’s latent distribution and associated RL policy outputs via a supervised latent-matching and policy-alignment objective.
• The ARMOR-Policy proper is the PPO-trained actor–critic $\pi_{\textrm{ref}}$8, mapping the latent invariant state $\pi_{\textrm{ref}}$9 to control actions $\Mcal_\alpha$0 (motor thrusts). During training, $\Mcal_\alpha$1 is $\Mcal_\alpha$2’s privileged embedding; at deployment, it is the student’s temporal encoding $\Mcal_\alpha$3.

Robustness here does not come from explicit adversary modeling but rather from the enforced invariance and classification in the latent space, which serves as the policy’s observational substrate. Empirically, ARMOR-Policy achieves $\Mcal_\alpha$4 mission success and $\Mcal_\alpha$5 crash rate under five attack classes; it reduces worst-case drift under severe GPS spoofing from $\Mcal_\alpha$6 (HRP baseline) to $\Mcal_\alpha$7, and achieves significant zero-shot generalization to unfamiliar attack modalities ($\Mcal_\alpha$8--$\Mcal_\alpha$9 success vs $\max_{\pi} \min_{M \in \Mcal_\alpha} [ J_M(\pi) - J_M(\pi_{\textrm{ref}}) ],$0 for RARL) (Dash et al., 27 Jun 2025).

3. ARMOR-Policy for LLM Safety Alignment via Structured Reasoning

In the domain of LLM safety, ARMOR-Policy refers to the structured, rule-grounded safety verdict executed within the ARMOR reasoning-based defense pipeline (Zhao et al., 14 Jul 2025). Formally, if $\max_{\pi} \min_{M \in \Mcal_\alpha} [ J_M(\pi) - J_M(\pi_{\textrm{ref}}) ],$1 is the set of extracted user-intents and $\max_{\pi} \min_{M \in \Mcal_\alpha} [ J_M(\pi) - J_M(\pi_{\textrm{ref}}) ],$2 is the set of policy rules (each $\max_{\pi} \min_{M \in \Mcal_\alpha} [ J_M(\pi) - J_M(\pi_{\textrm{ref}}) ],$3 indicating disallowed intents), then the ARMOR-Policy verdict is:

$\max_{\pi} \min_{M \in \Mcal_\alpha} [ J_M(\pi) - J_M(\pi_{\textrm{ref}}) ],$4

The inference-time pipeline executes three reasoning stages: adversarial strategy analysis, intent extraction, and formal policy check. Only after verifying the intent against all constraints is the final response permitted. This sharply constrains so-called “hidden intent injection” and obfuscated jailbreaks. ARMOR-Policy thus operationalizes LLM safety as an explicit, interpretable decision rule over human-aligned policy predicates, with the empirical effect of lowering attack success rates (ASR) to $\max_{\pi} \min_{M \in \Mcal_\alpha} [ J_M(\pi) - J_M(\pi_{\textrm{ref}}) ],$5--$\max_{\pi} \min_{M \in \Mcal_\alpha} [ J_M(\pi) - J_M(\pi_{\textrm{ref}}) ],$6 on state-of-the-art jailbreaks while maintaining low false acceptance (Zhao et al., 14 Jul 2025).

4. ARMOR-Policy for Program Analysis and Agent Security

ARMOR-Policy in AgentArmor refers to the static policy-type system enforcing security invariants over structured agent traces parsed as dependency graphs (Wang et al., 2 Aug 2025). Agent behavior is encoded in a Program Dependency Graph (PDG), with every action, dataflow, and observation marked with fine-grained semantic metadata (confidentiality, integrity, trust, rule tags) sourced from property registries. The ARMOR-Policy type system assigns and propagates these types, checks IFC lattice constraints (no confidentiality downgrade: $\max_{\pi} \min_{M \in \Mcal_\alpha} [ J_M(\pi) - J_M(\pi_{\textrm{ref}}) ],$7; no integrity taint: $\max_{\pi} \min_{M \in \Mcal_\alpha} [ J_M(\pi) - J_M(\pi_{\textrm{ref}}) ],$8), and asserts compliance with custom rules along all PDG edges. Policy violation is detected upon any forbidden flow or rule mismatch; executions are blocked before the agent emits external actions violating trust boundaries.

ARMOR-Policy in this context statically eliminates a broad range of prompt-injection vectors—including direct, indirect, and chained attacks—by construction, and is empirically validated to reduce Attack Success Rate from $\max_{\pi} \min_{M \in \Mcal_\alpha} [ J_M(\pi) - J_M(\pi_{\textrm{ref}}) ],$9 to $J_M(\cdot)$0 on the AgentDojo benchmark at $J_M(\cdot)$1 recall and $J_M(\cdot)$2 FPR (Wang et al., 2 Aug 2025).

5. Algorithmic Implementations, Trade-offs, and Empirical Guarantees

ARMOR-Policy mechanisms typically entail increased computational and conceptual complexity:

• Offline RL variants implement alternating actor–adversary updates or inner–outer optimization (adversarial Bellman consistency, version-space constraints), generally without the need for ensembling, due to robust RPI under single-model adversarial training (Bhardwaj et al., 2023).
• UAV control instantiations combine encoder–decoder architectures (MLPs, LSTMs) with latent supervision. The cost is primarily in training-time simulation with privileged information, offset by the elimination of costly adversarial retraining and the ability to deploy student policies without attack knowledge (Dash et al., 27 Jun 2025).
• LLM and agent security variants introduce inference- or planning-time computational overhead, as (a) structured multi-stage reasoning for LLMs (∼2–3× token usage; optional tree sampling for optimal safety), and (b) stepwise type-toggle graph analysis (1.5–2× latency per agent step, dominated by dependency extraction via LLMs) (Zhao et al., 14 Jul 2025, Wang et al., 2 Aug 2025).

Empirical evaluations consistently reveal that ARMOR-Policy mechanisms—when tuned with admissible hyperparameters—deliver on robust improvement or safety objectives without regressing on baseline/reference performance, and often achieve best- or near-best-in-class outcomes across standard benchmarks.

6. Limitations, Extensions, and Open Directions

ARMOR-Policy approaches are limited by:

• Persistent need for nontrivial hyperparameter or rule tuning, though within theory-backed admissible ranges (e.g., for $J_M(\cdot)$3 in offline RL, or expansion of policy sets $J_M(\cdot)$4 in LLM safety) (Bhardwaj et al., 2023, Zhao et al., 14 Jul 2025).
• Modeling risk: Single-model bias can limit uncertainty representation compared to ensembles (suggested extension for high-uncertainty environments) (Bhardwaj et al., 2023).
• Overhead: Type-based policy checking and multi-stage reasoning can impede real-time performance, though efficient implementations on modern accelerators mitigate this in practice (Zhao et al., 14 Jul 2025, Wang et al., 2 Aug 2025).
• Coverage: In agent and LLM settings, ARMOR-Policy coverage is limited by the exhaustiveness of static/dynamic property registries and human-coded policy clauses. Adaptive attackers or unforeseen tools may necessitate ongoing policy and registry refinement (Wang et al., 2 Aug 2025).
• Formal guarantees: While RPI and safety properties are established for offline RL variants, robustness for deep imitators (e.g., in high-dimensional continuous control) is empirical; and LLM/agent policy models rely on soundness of type propagation and registry completeness.

Open directions include extending adversarial policy optimization to partial observability and multi-agent regimes, automatic policy/rule inference via learning from violations, tighter integration with OS-level controls, and scaling human-aligned safety reasoning for LLMs beyond handcrafted rules by leveraging preference models or active learning (Zhao et al., 14 Jul 2025, Wang et al., 2 Aug 2025).