AlphaAlign: Reinforcement Learning Safety Alignment

• AlphaAlign is a safety alignment framework for large language models that uses pure reinforcement learning with a verifiable safety reward and a normalized helpfulness reward.
• The method employs a structured output format with designated safety reasoning and answer segments to ensure compliance, reducing superficial refusal shortcuts.
• Empirical results show AlphaAlign enhances harmful prompt refusal, mitigates jailbreak vulnerabilities, and maintains or improves task utility on multiple benchmarks.

Searching arXiv for AlphaAlign and closely related alignment methods to ground the article in current papers. AlphaAlign is a safety alignment framework for LLMs that is presented as a pure reinforcement learning method with a verifiable safety reward and a normalized helpfulness reward, designed to induce what its authors call proactive safety reasoning rather than superficial refusal behavior (Zhang et al., 20 Jul 2025). The method is motivated by the claim that contemporary LLMs already possess latent safety-relevant knowledge from pretraining, but that standard post-training often fails to activate this knowledge reliably, instead producing brittle refusal shortcuts, over-refusal on benign inputs, and utility degradation (Zhang et al., 20 Jul 2025). Within the broader alignment literature, AlphaAlign belongs to a family of methods that treat alignment as behavior shaping under explicit criteria, but it differs from organizational post-training recipes such as ABC Align, prompt-based attribute steering frameworks such as ALIGN, and parameter-efficient alignment mechanisms such as Aligner by centering on reinforcement learning with minimal supervision and verification-oriented rewards rather than supervised correction, prompt control, or architectural adaptation (Seneque et al., 2024, Ravichandran et al., 11 Jul 2025, Ziheng et al., 2023).

1. Conceptual basis and alignment target

AlphaAlign is formulated around the distinction between harmful and benign prompts, and around the claim that safety alignment should not reduce to teaching a model to emit stock refusals on recognizable unsafe requests (Zhang et al., 20 Jul 2025). The paper argues that many existing methods encourage superficial refusal shortcuts, in which the model learns lexical refusal patterns without robust harmfulness recognition, and that this produces two characteristic pathologies: vulnerability to jailbreaks and unnecessary refusals on legitimate inputs (Zhang et al., 20 Jul 2025).

The alignment target is therefore defined at the level of conditional behavior. Given a prompt $\mathbf{x}$, with harmful prompts in $\mathcal{X}_h$ and benign prompts in $\mathcal{X}_b$, the aligned model should return a refusal output for harmful inputs and a compliant or helpful output for benign inputs (Zhang et al., 20 Jul 2025). The paper writes this as

$$\mathbf{y} = \begin{cases} \mathbf{y}_r \in \mathcal{Y}_r, & \text{if } \mathbf{x} \in \mathcal{X}_h, \ \mathbf{y}_c \in \mathcal{Y}_c, & \text{if } \mathbf{x} \in \mathcal{X}_b. \end{cases}$$

In AlphaAlign, this behavior is embedded in a reasoning-augmented output format,

$$\mathbf{o} = \pi_\theta(\mathbf{x}) = (\mathbf{s}, \mathbf{y}),$$

where $\mathbf{s}$ is a safety reasoning segment and $\mathbf{y}$ is the final answer (Zhang et al., 20 Jul 2025).

This framing is narrower than universal value alignment and broader than refusal-rate optimization. The paper explicitly ties AlphaAlign to harmful-query refusal, jailbreak robustness, over-refusal reduction, and utility preservation, while also proposing deep alignment as a qualitative goal: the model should engage with safety-relevant concepts in early-generation behavior rather than merely memorizing refusal templates (Zhang et al., 20 Jul 2025). This distinguishes it from organization-specific standards alignment, where the target is a codified institutional norm set, as in ABC Align, and from personalized attribute alignment, where the target is a specified demographic or value profile, as in ALIGN (Seneque et al., 2024, Ravichandran et al., 11 Jul 2025).

2. Output schema and reward design

AlphaAlign begins from a fixed structured prompt template that requires the model to reason explicitly about safety before producing an answer (Zhang et al., 20 Jul 2025). The model is instructed to place its reasoning inside <safety_reasoning> ... </safety_reasoning> tags and its final answer inside <answer> ... </answer> tags. For unsafe prompts, the answer region is required to contain \boxed{Sorry, I can't comply} (Zhang et al., 20 Jul 2025). This structure is not merely stylistic; it makes the output amenable to automated verification.

The core mechanism is a dual-reward system. The first component is the verifiable safety reward, which consists of a format reward and an accuracy reward (Zhang et al., 20 Jul 2025). A format verifier checks whether the required tag structure is present, using the regular expression $\mathcal{X}_h$4 and a refusal verifier checks whether the final answer matches canonical refusal patterns by soft string matching (Zhang et al., 20 Jul 2025). The refusal patterns include phrases such as “Sorry, I can't comply,” “I'm sorry,” “As an AI,” “I cannot,” and related forms (Zhang et al., 20 Jul 2025).

The safety reward is defined differently for harmful and benign prompts:

$$R_s(x, o_i) = \begin{cases} r_f V_f(\mathbf{o}_i) + r_a V_r(\mathbf{y}_i), & \mathbf{x} \in \mathcal{X}_h \ r_f V_f(\mathbf{o}_i) - r_a V_r(\mathbf{y}_i), & \mathbf{x} \in \mathcal{X}_b \end{cases}.$$

For harmful prompts, the model is rewarded for correct format and refusal; for benign prompts, refusal is penalized (Zhang et al., 20 Jul 2025). This explicit subtraction on benign inputs is the mechanism the paper uses to discourage over-refusal.

The second component is the normalized helpfulness reward, applied only on benign prompts (Zhang et al., 20 Jul 2025). For a group of benign rollouts, the final answers are scored by a helpfulness reward model, denoted $R_r$, and normalized following a GRPO-style relative comparison. The printed formula in the paper is malformed, but the intended procedure is z-score normalization over the rollout group (Zhang et al., 20 Jul 2025). The resulting reward is thresholded so that only above-average benign non-refusal outputs receive positive reward:

$$R_h(\mathbf{x}_b,\mathbf{o}_i,\{\mathbf{o}_1,\mathbf{o}_2,\dots,\mathbf{o}_n\}) = \begin{cases} \max(\tilde{r}_i, 0), & \text{if } V_r(\mathbf{y}_i) = 0, \ 0, & \text{if } V_r(\mathbf{y}_i) = 1. \end{cases}$$

This thresholding is motivated in the paper as a way to avoid negative helpfulness rewards interfering with safety optimization while still reinforcing better benign responses (Zhang et al., 20 Jul 2025).

The total reward is

$\mathcal{X}_h$0

This asymmetry is central to the method: harmful prompts are optimized only for safe refusal, whereas benign prompts are optimized for both non-refusal and response quality (Zhang et al., 20 Jul 2025).

3. Reinforcement learning procedure

AlphaAlign is presented as a truly pure RL method in the sense that it does not use supervised safety reasoning traces, constitutional critique loops, or mixed supervised-plus-RL losses in the AlphaAlign training stage itself (Zhang et al., 20 Jul 2025). The supervision requirement is limited to binary prompt safety labels, indicating whether a prompt is harmful or benign (Zhang et al., 20 Jul 2025).

The optimization loop is PPO-style policy training over language generation (Zhang et al., 20 Jul 2025). For each prompt, the model samples a group of candidate outputs, each output receiving a scalar terminal reward. The reward is then propagated through the sequence by generalized advantage estimation (GAE), with the paper giving

$\mathcal{X}_h$1

and

$\mathcal{X}_h$2

The PPO objective printed in the paper is corrupted, but the text clearly states that AlphaAlign uses standard clipped PPO together with a learned value function and GAE-based credit assignment (Zhang et al., 20 Jul 2025).

This places AlphaAlign in a distinct operational position relative to several nearby alignment paradigms. It is not an ORPO-style preference optimization pipeline that first synthesizes chosen and rejected responses from principles, as in ABC Align (Seneque et al., 2024). It is not a prompt-based steering framework with swappable backbones and structured JSON output, as in ALIGN (Ravichandran et al., 11 Jul 2025). It is also not a parameter-efficient architectural steering method like Aligner, which uses globally shared prefix tokens and can align behavior with only a few thousand trainable parameters (Ziheng et al., 2023). AlphaAlign instead treats safety alignment analogously to verifiable-outcome RL in math or coding: the reasoning trace is unconstrained, but rewarded behavior should make useful reasoning instrumentally advantageous (Zhang et al., 20 Jul 2025).

4. Data, implementation, and evaluation protocol

The training data are deliberately modest and label-light (Zhang et al., 20 Jul 2025). Harmful prompts are drawn from the SCoT data pool, which contains about 35k harmful samples, but AlphaAlign uses a smaller subset: 1.5k original plus 1.5k augmented harmful samples (Zhang et al., 20 Jul 2025). Benign data consist of 3k randomly sampled Dolly instances, and XSTest is also included in training, with “safe” data treated as benign and all others as harmful (Zhang et al., 20 Jul 2025). The paper stresses that no safety rationales are required.

Training is done in PyTorch on 4 A100 80GB GPUs and 64-core Intel Xeon Scale 8358 CPUs, using the veRL codebase (Zhang et al., 20 Jul 2025). Reported hyperparameters include total batch size 16, maximum sequence length 2048, and 8 rollouts per prompt during PPO training (Zhang et al., 20 Jul 2025). The system is trained for two epochs, evaluated every 10 steps, and the best checkpoint is selected (Zhang et al., 20 Jul 2025). The actor learning rate is 1e-6, the critic learning rate is 1e-5, and the helpfulness reward model is FsfairX-LLaMA3-RM-v0.1 (Zhang et al., 20 Jul 2025). The paper repeatedly emphasizes efficiency, stating that substantial gains appear in fewer than 200 RL steps (Zhang et al., 20 Jul 2025).

The evaluation suite covers both safety and utility (Zhang et al., 20 Jul 2025). Harmful-query refusal is measured on StrongREJECT and AdvBench. Static jailbreak robustness is evaluated on WildGuardTest and JailbreakTrigger. Adaptive jailbreak robustness is tested with PAIR and GCG, generated on harmful questions from AdvBench (Zhang et al., 20 Jul 2025). Over-refusal is measured on the benign contrastive set from CoCoNot. Utility is evaluated on MMLU, AlpacaEval, and GSM8K (Zhang et al., 20 Jul 2025). The primary safety metric is attack success rate (ASR), with lower values indicating fewer harmful successful attacks; the paper states that ASR is computed using the refusal verifier first and then Llama3-Guard-8B (Zhang et al., 20 Jul 2025).

5. Empirical results and alignment behavior

The main empirical claim is that AlphaAlign improves refusal on harmful prompts, reduces over-refusal, and largely preserves or improves general utility, thereby addressing the usual safety–utility trade-off (Zhang et al., 20 Jul 2025). The strongest results are reported on Qwen2.5-7B-Instruct, where AlphaAlign reaches 0.0 ASR on StrongREJECT, 0.0 on AdvBench, 0.30 on WildGuardTest, 0.25 on JailbreakTrigger, 0.19 on PAIR, and 0.0 on GCG, with 93.14% benign accuracy on CoCoNot (Zhang et al., 20 Jul 2025).

On Qwen2.5-3B-Instruct, AlphaAlign reduces ASR to 0.31 on StrongREJECT and 0.0 on AdvBench, with 6.38 on WildGuardTest, 3.75 on JailbreakTrigger, 4.61 on PAIR, 0.77 on GCG, and benign accuracy 91.29% on CoCoNot (Zhang et al., 20 Jul 2025). On Llama3.2-3B-Instruct, it yields 0.31 on StrongREJECT, 0.0 on AdvBench, 2.43 on WildGuardTest, 1.5 on JailbreakTrigger, 0.57 on PAIR, 0.76 on GCG, and 91.29 on CoCoNot (Zhang et al., 20 Jul 2025).

Utility outcomes are mixed but generally favorable (Zhang et al., 20 Jul 2025). On Qwen2.5-3B-Instruct, MMLU remains essentially unchanged at 64.5 (-0.1), while AlpacaEval rises to 50.0 (+6.7) and GSM8K to 74.3 (+4.4) (Zhang et al., 20 Jul 2025). On Qwen2.5-7B-Instruct, MMLU drops to 68.8 (-1.6), but AlpacaEval rises to 50.0 (+7.9) and GSM8K to 79.7 (+2.9) (Zhang et al., 20 Jul 2025). On Llama3.2-3B-Instruct, AlpacaEval improves by +10.0, but MMLU drops -2.1 and GSM8K drops -8.3, which the paper leaves as a weaker case for utility preservation on that model family (Zhang et al., 20 Jul 2025).

The paper also reports AlphaAlign-Zero, where only the verifiable safety reward is applied to a base model, specifically Qwen2.5-3B rather than an instruct-tuned checkpoint (Zhang et al., 20 Jul 2025). The claim is that the base model rapidly acquires substantial safety capability and harmful-intent recognition through RL alone, supporting the broader thesis that pretrained models contain latent safety knowledge that can be incentivized without supervised reasoning traces (Zhang et al., 20 Jul 2025). The paper does not provide a detailed numerical table for AlphaAlign-Zero in the extracted text, so its evidence remains more qualitative in the present record.

Beyond benchmark results, AlphaAlign introduces Cumulative Keyword Adoption Score (CKAS) as a probe of alignment depth:

$\mathcal{X}_h$3

The paper interprets higher CKAS for safety-critical words and lower CKAS for jailbreak-trigger words as evidence that AlphaAlign increases the model’s early inclination to discuss harmfulness rather than complying reflexively (Zhang et al., 20 Jul 2025). This analysis is offered as support for the claim that the method encourages deeper safety engagement rather than only superficial refusal patterns.

6. Position within the alignment literature and open issues

AlphaAlign occupies a specific position within recent alignment work. Relative to ABC Align, it avoids supervised fine-tuning on synthetic reasoning data followed by principle-conditioned ORPO and instead relies on RL with verifiable rewards and binary harmful/benign labels (Seneque et al., 2024). Relative to ALIGN, it is not a prompt-based personalization system for decision aids but a post-training method for harmfulness refusal and utility preservation (Ravichandran et al., 11 Jul 2025). Relative to Aligner, it does not attempt to realize alignment as an ultra-lightweight architectural control signal, such as a single global token shared across all layers (Ziheng et al., 2023). A plausible implication is that AlphaAlign should be understood less as a general solution to all alignment settings than as a specific reframing of safety alignment as verifiable-outcome reinforcement learning.

The paper explicitly emphasizes several limitations (Zhang et al., 20 Jul 2025). The safety verifier is simple, depending on binary safety labels and a string-matching refusal verifier, which creates possible reward-hacking risks and leaves more sophisticated verifiers unexplored (Zhang et al., 20 Jul 2025). The method is evaluated only up to 7B-scale models, so larger-model scaling remains open (Zhang et al., 20 Jul 2025). Dynamic jailbreak adaptation during RL training is identified as future work (Zhang et al., 20 Jul 2025). The training data depend on correct harmful/benign labels, and the broader impacts discussion notes that mismatched labels could incentivize unsafe behavior rather than safe behavior (Zhang et al., 20 Jul 2025). The paper also shows that without the helpfulness component, safety recognition can emerge while benign usefulness remains underdeveloped, as in a case where AlphaAlign-Zero handles “How to burn calories effectively?” poorly despite classifying it as safe (Zhang et al., 20 Jul 2025).

A further caution is that the paper’s strongest qualitative claims—particularly deep alignment and proactive safety reasoning—are interpretive rather than fully formalized. The evidence consists of structured refusal rationales, benchmark outcomes, and CKAS-based probing, not a mechanistic proof that the model has learned a principled internal safety policy (Zhang et al., 20 Jul 2025). This suggests that AlphaAlign should be read as an important empirical and methodological proposal rather than as a settled solution to safety alignment.

Within that scope, its significance lies in showing that a comparatively simple RL framework—structured output, binary prompt labels, verifiable refusal reward, and normalized helpfulness reward—can yield strong gains on harmful refusal and jailbreak robustness while often maintaining or improving general task performance (Zhang et al., 20 Jul 2025). In the current alignment landscape, AlphaAlign is therefore best characterized as a reinforcement-learning-based safety alignment method that seeks to exploit latent pretrained safety knowledge through verifiable incentives rather than through dense supervised safety reasoning or purely stylistic refusal tuning (Zhang et al., 20 Jul 2025).