OrthAlign: Multi-Objective LLM Alignment

• OrthAlign is a multi-objective alignment method that uses orthogonal subspace decomposition to clearly separate and optimize conflicting human preference gradients.
• The approach leverages LoRA-style updates and adaptive subspace-rank selection via SVD and curvature analysis to minimize parameter interference while securing stability.
• Experimental results on LLaMA and Mistral models demonstrate significant single-preference and average reward improvements compared to standard methods.

Searching arXiv for papers on OrthAlign and closely related alignment methods to ground the article. OrthAlign is a multi-objective alignment method for LLMs that addresses conflicts among human preference dimensions such as helpfulness, harmlessness, and truthfulness at the parameter level rather than only through reward aggregation, constraint tuning, or data selection. Its central mechanism is an orthogonal subspace decomposition of preference-specific parameter updates: once one objective is learned, subsequent objectives are optimized in orthogonal complements of the earlier objective’s principal subspace, with additional spectral norm control to keep update growth stable. In the reported experiments, OrthAlign yields maximum single-preference improvements ranging from 34.61% to 50.89% after multiple-objective alignment across helpful, harmless, and truthful dimensions, with an average overall reward improvement of 13.96% (Lin et al., 29 Sep 2025).

1. Problem formulation and alignment conflict

OrthAlign starts from the observation that standard multi-objective alignment procedures often exhibit antagonism between objectives. The conflict is expressed as non-orthogonal preference gradients,

$$\frac{|\langle \nabla_{\theta}\mathcal{L}(\mathcal{D}_i), \nabla_{\theta}\mathcal{L}(\mathcal{D}_j) \rangle|}{\|\nabla_{\theta}\mathcal{L}(\mathcal{D}_i)\|_2 \cdot \|\nabla_{\theta}\mathcal{L}(\mathcal{D}_j)\|_2} \ne 0,$$

so that improving one objective can move parameters in directions that degrade another. OrthAlign treats this as a gradient-level and parameter-level interference problem. The method is designed for settings in which multiple preference datasets, such as helpfulness, harmlessness, and truthfulness, are aligned sequentially on top of an SFT model.

Within this setup, OrthAlign is compatible with standard post-training preference objectives. The formulation discussed alongside the method includes a multi-objective DPO loss,

$$\mathcal{L}_{\pi_\theta} = -\sum_{i=1}^k \lambda_i \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}_i} \left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_0(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_0(y_l|x)} \right) \right],$$

but the distinctive claim of OrthAlign is that objective conflict should be resolved in weight space. It therefore operates on LoRA-style updates and allocates different preference dimensions to mathematically non-interfering directions (Lin et al., 29 Sep 2025).

2. Orthogonal subspace decomposition

The method is defined on low-rank adaptation matrices. For a layer with base weight matrix $W$, a preference-specific adaptation is written as $\Delta W = BA$, where $B \in \mathbb{R}^{m \times r}$ and $A \in \mathbb{R}^{r \times n}$. OrthAlign analyzes $\Delta W$ through SVD,

$$W = U \Sigma V^T \quad \Rightarrow \quad WX = \sum_{i=1}^{\mathrm{Rank}(W)} \sigma_i (v_i^T X)\, u_i.$$

Applied to a preference-specific update, this separates top singular directions, treated as preference-critical, from tail directions that are candidates for future objectives.

For a safety preference, the decomposition is expressed as

$$\Delta W\, \mathbf{X}_{\text{safe}} = \underbrace{\sum_{i=1}^{r} \sigma_i (v_i^T\mathbf{X}_{\text{safe}})\, u_i}_{\text{Preference-critical directions}} + \underbrace{\sum_{j=r+1}^{\max(m,n)} \sigma_j (v_j^T\mathbf{X}_{\text{safe}})\, u_j}_{\text{Minimal impact on current preference}}.$$

The top-$r$ singular directions define the principal subspace for the current preference. Future preferences are restricted to the orthogonal complement of that subspace.

A curvature-based formulation is also given. If $$\mathcal{L}_{\pi_\theta} = -\sum_{i=1}^k \lambda_i \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}_i} \left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_0(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_0(y_l|x)} \right) \right],$$0 is the safety curvature, with $$\mathcal{L}_{\pi_\theta} = -\sum_{i=1}^k \lambda_i \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}_i} \left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_0(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_0(y_l|x)} \right) \right],$$1, then the safety principal subspace is $$\mathcal{L}_{\pi_\theta} = -\sum_{i=1}^k \lambda_i \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}_i} \left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_0(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_0(y_l|x)} \right) \right],$$2, its orthogonal complement is $$\mathcal{L}_{\pi_\theta} = -\sum_{i=1}^k \lambda_i \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}_i} \left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_0(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_0(y_l|x)} \right) \right],$$3, and the projector is

$$\mathcal{L}_{\pi_\theta} = -\sum_{i=1}^k \lambda_i \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}_i} \left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_0(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_0(y_l|x)} \right) \right],$$4

OrthAlign enforces the subspace constraint

$$\mathcal{L}_{\pi_\theta} = -\sum_{i=1}^k \lambda_i \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}_i} \left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_0(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_0(y_l|x)} \right) \right],$$5

and parameterizes LoRA updates accordingly,

$$\mathcal{L}_{\pi_\theta} = -\sum_{i=1}^k \lambda_i \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}_i} \left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_0(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_0(y_l|x)} \right) \right],$$6

This construction is the core of the method’s claim that later objectives can be learned without first-order interference with earlier ones (Lin et al., 29 Sep 2025).

3. Optimization procedure and adaptive subspace-rank selection

OrthAlign proceeds sequentially. The first objective, typically harmlessness in the reported pipeline, is aligned with a standard LoRA-based preference optimization step. Its learned update defines a principal subspace through SVD or curvature. The next objective, such as helpfulness, is then constrained to update only inside an orthogonal tail subspace.

The method does not fix this tail subspace rank a priori. Instead, it introduces adaptive subspace-rank selection based on a positive reward criterion on the earlier objective. For a safety set $$\mathcal{L}_{\pi_\theta} = -\sum_{i=1}^k \lambda_i \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}_i} \left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_0(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_0(y_l|x)} \right) \right],$$7, the admissible rank $$\mathcal{L}_{\pi_\theta} = -\sum_{i=1}^k \lambda_i \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}_i} \left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_0(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_0(y_l|x)} \right) \right],$$8 is chosen as the largest one satisfying a reward-tolerance condition,

$$\mathcal{L}_{\pi_\theta} = -\sum_{i=1}^k \lambda_i \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}_i} \left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_0(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_0(y_l|x)} \right) \right],$$9

where the last $W$0 singular values are rescaled and $W$1 is a tolerance hyperparameter. The procedure is implemented through binary search over $W$2. This rank-selection step is intended to maximize the available subspace for the new objective without causing excessive degradation on the earlier one.

Once a permissible subspace is selected, updates for the new objective are projected: $W$3 where $W$4 spans the chosen update subspace. In the reported training schedule, harmlessness is aligned first, then helpfulness, then truthfulness, with each later phase projected against previously established preference-critical directions. The method is also described as a plug-in projection step that can be layered onto existing DPO-, MODPO-, or SPO-style pipelines (Lin et al., 29 Sep 2025).

4. Theoretical properties and stability guarantees

OrthAlign’s theoretical analysis is centered on first-order non-interference, second-order safety drift bounds, and Lipschitz control. If $W$5 and $W$6, then

$W$7

This is the first-order non-interference claim: the immediate linear effect of a later objective’s update on the earlier objective vanishes when the update is orthogonal to the earlier objective’s principal subspace.

A one-step second-order safety bound is then given as

$W$8

where $W$9 is the first tail eigenvalue of the earlier objective’s curvature. A cumulative version bounds total drift across multiple updates,

$\Delta W = BA$0

These results formalize the claim that degradation is controlled by tail curvature and update magnitude.

The second component is spectral norm control. If each LoRA increment satisfies $\Delta W = BA$1, then the effective layer norm obeys

$\Delta W = BA$2

This yields linear Lipschitz accumulation rather than exponential instability. OrthAlign further states that if successive update subspaces are mutually orthogonal, then

$\Delta W = BA$3

so destructive cross-terms are eliminated. The combined thesis is that orthogonal allocation and spectral clipping make multi-objective alignment stable in a mathematically controlled sense (Lin et al., 29 Sep 2025).

5. Experimental evaluation

The reported experiments use two SFT bases, LLaMA-3-SFT and Mistral-7B-SFT, with three preference dimensions: helpfulness, harmlessness, and truthfulness. Training data include UltraFeedback and HelpSteer2 for helpfulness, SafeRLHF-10k for harmlessness, and truthful subsets from UltraFeedback and HelpSteer2 for truthfulness. Evaluation uses AlpacaEval helpfulness win rate, AdvBench prompts judged by Llama-Guard-3-8B for Harmless Rate, and TruthfulQA MC2. The LoRA hyperparameters are rank 16, $\Delta W = BA$4, dropout 0.01, learning rate $\Delta W = BA$5, cosine schedule, 3 epochs, batch size 64, and max length 2048–4096.

In the main three-objective setting, OrthAlign reports the following results. On LLaMA-3-SFT with UltraFeedback, it reaches Harmless 87.30, Helpful 71.57, Truthful 66.58, Average 75.15; with HelpSteer2, Harmless 91.34, Helpful 68.83, Truthful 67.69, Average 75.95. On Mistral-7B-SFT with UltraFeedback, it reaches 78.00, 75.51, 65.28, Average 72.93; with HelpSteer2, 88.12, 67.08, 65.34, Average 73.51. These values exceed the baselines reported in the same evaluation, including DPO, MODPO, SPO, RSDPO, Soups, Knots, and TSV-M.

The two-objective helpfulness-versus-harmlessness setting shows the same pattern. On LLaMA-3 with HelpSteer2, OrthAlign reaches Harmless 93.84, Helpful 65.71, Average 79.78, whereas the strongest non-OrthAlign baselines remain around the low 70s in average score. As a plug-in modifier, OrthAlign also improves existing methods: on LLaMA-3 / HelpSteer2 + SafeRLHF, DPO moves from Harmless 71.24 and Helpful 60.24 to DPO-Orth at 93.84 and 65.71; MODPO moves from 48.46 and 67.95 to MODPO-Orth at 79.32 and 71.02; SPO moves from 71.15 and 61.24 to SPO-Orth at 92.88 and 67.28.

Additional analyses are used to support the non-interference claim. Hidden-state t-SNE visualizations are computed on 3000 samples from the first preference’s training set. OrthAlign shows nearly unchanged hidden-state distributions between the first alignment stage and the final multi-preference model, whereas baselines exhibit significant drift. A rank ablation shows that as the update subspace rank increases from 12 to about 26, harmlessness declines from about 93.8% to about 81.3%, while helpfulness remains relatively stable; the adaptive procedure selects a rank in the 16–18 range that preserves high harmlessness while maintaining helpfulness (Lin et al., 29 Sep 2025).

6. Relation to adjacent alignment research and stated limitations

OrthAlign belongs to a broader family of geometry-aware alignment methods, but its geometry is explicitly parameter-space geometry. This distinguishes it from approaches such as Orthogonalized Policy Optimization, which decouples sampling geometry from optimization geometry through $\Delta W = BA$6-weighted importance sampling and a $\Delta W = BA$7-induced quadratic penalty in ratio coordinates (Zixian, 18 Jan 2026). It also differs from oracle-robust online alignment, which models misspecified preference feedback through a pointwise uncertainty set and derives a robustness penalty for online SAIL-style training (Li et al., 24 Feb 2026). In a different but complementary register, mechanistic analysis of alignment algorithms has shown that methods such as KTO and GRPO can improve linear separability through constructive feature sharing and sparse, high-salience recruitment, whereas DPO and ORPO can degrade separability through non-constructive geometric rotation and feature attenuation (Sinha et al., 9 May 2026). This suggests that OrthAlign’s emphasis on orthogonal parameter routing is part of a larger movement toward mechanism-aware alignment, although it operates on update subspaces rather than latent feature probes.

The stated limitations are specific. The reported experiments cover only three objectives, only text LLMs, and require multiple preference datasets and possibly reward models. The method also requires tuning of the tolerance $\Delta W = BA$8 and spectral thresholds. Its guarantees depend on approximating preference curvature and on the existence of a useful principal-versus-tail decomposition; if preferences are extremely entangled, the orthogonal complement may not actually be safe, and if reward models are noisy or biased, rank selection can mischaracterize safety budgets. OrthAlign is therefore presented not as a universal solution to multi-objective alignment, but as a parameter-space framework for reducing objective interference under explicitly modeled geometric constraints (Lin et al., 29 Sep 2025).