Cat-DPO: Category-Adaptive Safety Alignment

Abstract: Aligning LLMs with human preferences must balance two competing goals: responding helpfully to legitimate requests and reliably refusing harmful ones. Most preference-based safety alignment methods collapse safety into a single scalar that is applied uniformly to every preference pair. The result is a model that looks safe on average but stays relatively unsafe on a minority of harm categories. We cast safety alignment as a per-category constrained optimization problem and derive Cat-DPO, a direct-preference-optimization algorithm with a separate adaptive safety margin for each harm category. The margin tightens when the model still produces unsafe responses on a category and relaxes once the model catches up, so the training signal tracks each category's current difficulty rather than averaging under one global rate. Across two LLM backbones and six preference-learning baselines, Cat-DPO improves aggregate helpfulness and harmlessness and compresses per-category safety variance and the best-to-worst gap, offering a drop-in per-category refinement of direct preference safety alignment.

• The paper introduces Cat-DPO, which redefines safety alignment as a per-category constrained optimization by dynamically updating dual margins for each harm category.
• The methodology shows significant improvements in Safe Ratios and reduced per-category variance, effectively closing gaps in the worst-performing harm categories.
• Empirical evaluations on models like Alpaca-7B and Qwen3-4B demonstrate faster convergence and balanced performance without additional over-refusal on benign prompts.

Category-Adaptive Safety Alignment via Cat-DPO

Motivation and Problem Statement

LLMs deployed as general-purpose assistants must navigate the tension between maximizing helpful responses to valid queries and simultaneously minimizing the probability of harmful completions. Conventional preference-based safety alignment methods—including RLHF, Safe RLHF, Direct Preference Optimization (DPO), and their safety-focused extensions (e.g., SafeDPO, SACPO)—collapse safety into a single global scalar, applying the same loss pressure uniformly across all types of harm. This approach is structurally limited: aggregate harmlessness scores can mask large disparities among different harm categories, leading to models that are overall “safe” on average but systematically unsafe on rare or difficult categories.

A concrete illustration (Figure 1) shows that even state-of-the-art DPO-based approaches leave stubbornly large category-level gaps, with certain harm categories (e.g., “Insulting Behavior” or “Privacy Violation”) receiving inadequate alignment pressure.

Figure 1: Aggregate harmlessness hides large per-category gaps; several harm types remain unmitigated with standard methods.

Cat-DPO: Methodology

Cat-DPO (Category-Adaptive Direct Preference Optimization) reframes safety alignment as a per-category constrained optimization problem. Rather than enforcing a single global constraint, Cat-DPO maintains an independent safety margin (i.e., Lagrangian dual variable $\lambda_k$) for each of the $K$ annotated harm categories. During training, each margin $\lambda_k$ is updated online: it increases if the current policy yields unsafe responses in that category, and decreases as the category’s safety constraint is satisfied. The core insight is that Cat-DPO dynamically reallocates alignment “pressure” toward the hardest (i.e., most lagging) categories without manual hyperparameter adjustment per category.

The training loop jointly optimizes the model parameters and updates the per-category dual variables. Each preference pair is further augmented with category annotations, and the DPO loss applies a per-sample margin corresponding to the most “active” category for that data point. Specifically, this is achieved via a max aggregator over the category set for each prompt, suppressing over-penalization on multi-label samples.

A schematic in Figure 2 delineates the structural divergence between vanilla DPO and Cat-DPO, highlighting the per-category adaptation during each training step.

Figure 2: Cat-DPO augments preference data with per-category losses and jointly adapts per-category margins $\lambda_k$ in each training batch.

Algorithmically, Cat-DPO operates as follows:

• For each batch, the DPO log-ratio is computed per sample and a margin (max over active category $\lambda_k$) is subtracted for unsafe-safe pairs.
• The dual update uses a per-sample proxy for constraint violation ($V_i = 1 - \sigma(\beta \delta_i)$) to increase or decrease each $\lambda_k$.
• All updates are performed using quantities computable in the offline DPO regime; no rollouts or cost model inference are necessary.

Empirical Evaluation

Cat-DPO is empirically validated on the PKU-SafeRLHF dataset (alpaca2-7b subset), annotated with 19 harm categories. Two backbone models are used: Alpaca-7B (LLaMA-2-based SFT) and Qwen3-4B (base). Metrics include Helpfulness, Harmlessness, Safe Ratio (responses scoring above threshold), and reward model–based Safe Ratio.

Cat-DPO delivers higher aggregate helpfulness and harmlessness, and compresses the per-category variance substantially compared to all baselines (Table 1 of the paper). The method’s direct comparison against DPO-bettersafe (identical data and schedule, but no adaptive margin) isolates the per-category dual margin as the major source of improvement.

Cat-DPO’s benefits persist at the category level, quantitatively reducing worst-category gaps and variance (Figure 3). Notably, it does not simply uplift already strong categories—the macro, variance, and worst-3 statistics all improve, indicating genuine gap closure.

Figure 3: Cat-DPO achieves higher per-category Safe Ratios with lower disparity, lifting the worst categories while sustaining the overall average.

Analysis of the training-time mechanism (Figure 4) reveals that adaptive $\lambda_k$ concentrate margin pressure on underperforming categories throughout optimization, and that Cat-DPO achieves faster convergence on per-category preference probabilities compared to uniform-margin baselines.

Figure 4: During training, more difficult categories are targeted by higher $\lambda_k$, leading to faster preference probability improvement.

Hyperparameter Sensitivity and Over-Refusal Behavior

Hyperparameter ablation (Figure 5) demonstrates that Cat-DPO is relatively robust: Safe Ratio and Helpfulness peak at moderate learning rates for the dual variable stepsize ($\eta$). The constraint slackness parameter ($K$0) must be tuned to model capacity; overly strict tolerances can result in persistently large $K$1 when the backbone cannot saturate the constraints. This is further supported by results on smaller Qwen3 backbones.

Figure 5: Cat-DPO exhibits single-peaked sensitivity to dual update rate and a tradeoff between safe ratio and helpfulness as $K$2 is varied.

On the XSTest benchmark, which measures over-refusal (safe prompts incorrectly refused), Cat-DPO introduces no additional over-refusal relative to baselines while maintaining superior safety on unsafe prompts, confirming that category-adaptive margins do not compromise the model’s willingness to answer benign questions.

Theoretical Considerations

Cat-DPO is grounded in Lagrangian duality for constrained Markov decision processes, and the dual variable update is a dual-inspired controller using offline proxies for constraint violation without requiring on-policy rollouts. While the dual signals can be biased, empirical results indicate effective targeting of the most risky categories.

Implications and Future Directions

Cat-DPO’s per-category adaptive approach advances safety alignment by automating capacity allocation across safety categories, ensuring that hard-to-align classes are not ignored due to their rarity or difficulty. This fine-grained perspective suggests that aggregate harmlessness measures are insufficient and that robust alignment requires explicit sub-population accounting. In practice, Cat-DPO is drop-in within the DPO training loop and incurs minimal computational overhead.

Theoretically, Cat-DPO illustrates the utility of integrating constrained optimization with per-group adaptation into large-scale preference alignment pipelines. It complements distributionally robust optimization approaches but provides meaningful “knobs” (the $K$3) that could be further interrogated for interpretability or for prioritizing external auditing.

Going forward, automating the setting of category-specific tolerances ($K$4) using base model statistics, and relaxing the reliance on externally pre-defined taxonomies (e.g., learning soft or hierarchical categories from data), would enhance generalization and usability. Hybridizing Cat-DPO’s dual adaptation with on-policy corrections can further improve performance in low-capacity or highly misaligned initialization regimes.

Conclusion

Cat-DPO contributes a category-adaptive safety alignment paradigm that systematically equalizes per-category safety performance for LLMs, outperforming uniform-scalar methods in both aggregate and worst-case risk. The online dual mechanism ensures adaptive resource allocation during training without sacrificing helpfulness or increasing over-refusal. As safety evaluation standards converge towards fine-grained and distributionally sensitive metrics, methods such as Cat-DPO will be increasingly essential for verifiable model alignment.