Weights-Rotated Preference Optimization (RoPO)

• RoPO is a fine-tuning framework that restricts weight adaptations to orthogonal rotations and magnitude scaling, preserving the angular structure of neurons.
• It addresses issues found in DPO such as neuron collapse, output overlength, and reward hacking, leading to improved performance and diversity.
• The method leverages efficient products of Givens rotations and Householder reflections, maintaining minimal trainable parameter overhead (<0.015%) without runtime cost.

Weights-Rotated Preference Optimization (RoPO) is a parameter-efficient fine-tuning framework for LLMs that constrains model adaptation to orthogonal rotations and magnitude stretching of selected weight matrices. RoPO is designed specifically to address overfitting and reward hacking pathologies commonly observed in Direct Preference Optimization (DPO), a popular loss-based alignment approach. By guaranteeing invariance of hyperspherical energy—hence preserving the angular structure among neurons—RoPO regularizes internal representation learning, safeguarding both expressivity and knowledge retention with minimal trainable parameter overhead (Yang et al., 2024, Yang et al., 25 Aug 2025).

1. Motivation and Context

Direct Preference Optimization (DPO) achieves policy alignment by maximizing the probability difference between preferred and dispreferred completions, subject to a KL-divergence penalty controlling drift from a reference model. This direct logit-level optimization, however, incentivizes aggressive suppression of dispreferred completions, producing excessive sequence lengths, diminished output diversity, and pronounced reward hacking. The underlying mechanism is representation redundancy and neuron collapse: model neurons, after DPO, cluster in parameter space, eroding the diversity (as measured by hyperspherical energy) required for rich generation and robust knowledge (Yang et al., 2024, Yang et al., 25 Aug 2025).

Attempts to mitigate these issues by modifying the DPO loss (e.g., IPO, R-DPO) often degrade alignment or fluency. RoPO instead reframes the problem as a weight-update regularization, introducing explicit constraints on how internal weights are permitted to evolve during preference optimization.

2. Hyperspherical Energy and Neuron Collapse

Hyperspherical energy (HE) quantifies the angular dispersion of a set of normalized neuron vectors. For a weight matrix $W \in \mathbb{R}^{d \times n}$ (columns $w_i$), column normalization yields:

$\hat{w}_i = \frac{w_i}{\|w_i\|}$

The hyperspherical energy is then:

$$E_{\mathrm{sphere}}(W) = \sum_{i \neq j} \|\hat{w}_i - \hat{w}_j\|^{-1}$$

HE is minimized when neurons are maximally dispersed over the unit sphere and increased HE signals neuron collapse (alignment along narrow subspaces). Summed-absolute HE variation (SAHE) across targeted layers quantifies the degree of DPO-induced distortion:

$$\textrm{SAHE}(W^0, W^1, L) = \sum_{l \in L} | E_{\mathrm{sphere}}(W^{1, l}) - E_{\mathrm{sphere}}(W^{0, l}) |$$

Subsystems exhibiting high SAHE after DPO are empirically correlated with loss of generative diversity and knowledge-forgetting (Yang et al., 2024, Yang et al., 25 Aug 2025).

3. Orthogonal Weight Rotation: Parameterization and Constraints

RoPO restricts each trainable matrix $W$ (typically Query and Value projections per attention head/layer) to the following form:

$W = \operatorname{diag}(m) \cdot R \cdot W^0$

where $W^0$ is the frozen SFT or pre-trained weight, $m \in \mathbb{R}^n$ is a trainable per-neuron magnitude vector, and $R \in \mathbb{R}^{n \times n}$ is an orthogonal (rotation) matrix ($w_i$0). The forward pass uses

$w_i$1

thus preserving all inter-neuron angles and HE, with only magnitudes permitted to stretch or shrink (Yang et al., 2024, Yang et al., 25 Aug 2025).

RoPO parameterizes $w_i$2 via efficient products of Givens rotations and, in some variants, Householder reflections:

• Bidirectional Integrated Givens (BIG): Successive products of sparse 2D rotations, allowing arbitrary rotations in $w_i$3 with $w_i$4 parameters and $w_i$5 compute per layer/head.
• Householder reflections: Additional global orthogonal components (as in (Yang et al., 25 Aug 2025)) to increase expressivity.

All rotational parameters are updated by gradient descent, with no explicit projection step due to orthonormal construction.

4. Algorithmic Workflow

Pseudocode Outline:

$$E_{\mathrm{sphere}}(W) = \sum_{i \neq j} \|\hat{w}_i - \hat{w}_j\|^{-1}$$4

Trainable parameter overhead is typically $w_i$6 of the model (for 7B LLMs, e.g. $w_i$7–$w_i$8), since adaptations are confined to per-head rotations and scales (Yang et al., 2024, Yang et al., 25 Aug 2025).

5. Experimental Results and Ablations

Extensive experimentation with Mistral-7B, Llama3-8B, and related architectures demonstrates:

• Instruction-following (MT-Bench, AlpacaEval 2, Arena-Hard):
  • RoPO achieves up to $w_i$9 point absolute win-rate gain on MT-Bench and $\hat{w}_i = \frac{w_i}{\|w_i\|}$0–$\hat{w}_i = \frac{w_i}{\|w_i\|}$1 points on AlpacaEval 2 over DPO, while curbing output length inflation and increasing generation diversity by $\hat{w}_i = \frac{w_i}{\|w_i\|}$2 points.
  • On AlpacaEval 2, RoPO attains LC/LWR of $\hat{w}_i = \frac{w_i}{\|w_i\|}$3 (vs $\hat{w}_i = \frac{w_i}{\|w_i\|}$4 best baseline); on MT-Bench up to $\hat{w}_i = \frac{w_i}{\|w_i\|}$5 (vs $\hat{w}_i = \frac{w_i}{\|w_i\|}$6 best).
• Neuron-collapse and knowledge retention:
  • DPO increases HE and induces collapse; RoPO maintains invariant HE, correlating with stable out-of-distribution QA accuracy ($\hat{w}_i = \frac{w_i}{\|w_i\|}$7 retained vs $\hat{w}_i = \frac{w_i}{\|w_i\|}$8 for DPO).
  • RoPO outperforms LoRA and non-orthogonal baselines for instruction fidelity under equivalent parameter budgets.
• Ablations:
  • Removing key components (reverse Givens, magnitude-stretch, global reflections) markedly degrades win-rate (e.g., $\hat{w}_i = \frac{w_i}{\|w_i\|}$9 WWR for unidirectional rotations).

Results support the interpretation that strict orthogonality in weight adaptation mitigates reward hacking, prevents overfitting, and preserves model expressivity (Yang et al., 2024, Yang et al., 25 Aug 2025).

6. Theoretical Properties and Practical Recommendations

By enforcing $$E_{\mathrm{sphere}}(W) = \sum_{i \neq j} \|\hat{w}_i - \hat{w}_j\|^{-1}$$0 as strictly orthogonal and adapting only $$E_{\mathrm{sphere}}(W) = \sum_{i \neq j} \|\hat{w}_i - \hat{w}_j\|^{-1}$$1, RoPO ensures that all pairwise angles between neuron columns, and thus the representational geometry, are preserved. The only learnable modifications are along directions already present in $$E_{\mathrm{sphere}}(W) = \sum_{i \neq j} \|\hat{w}_i - \hat{w}_j\|^{-1}$$2. This approach achieves:

• Architectural regularization against overfitting not by loss modification but by constraining learning dynamics in parameter space.
• Knowledge retention and diversity preservation; neuron directions encoding pre-trained knowledge remain accessible.
• No inference penalty: rotational and magnitude adaptations are merged into a single matrix at save-time; no runtime cost.

Practical recommendations include focusing rotational adaptation on Q and V projections only, using four BIG multiplies (or two Givens plus reflections), Adam optimizer (learning rate $$E_{\mathrm{sphere}}(W) = \sum_{i \neq j} \|\hat{w}_i - \hat{w}_j\|^{-1}$$3), and freezing all non-target weights (Yang et al., 2024, Yang et al., 25 Aug 2025).

7. Impact, Limitations, and Extensions

RoPO is the first method to regularize preference tuning by strictly limiting the pathway of adaptation within weight space, leveraging orthogonality to eliminate knowledge-erasing drift. Its applicability is immediate for any DPO-style finetuning scenario where overfitting, reward hacking, or diversity loss are a concern. While most results focus on transformer-based LLMs, the method is architectural and therefore potentially extensible to other domains where orthogonal invariance of internal representations is desirable.

A plausible implication is that further refinements in the design of rotation parameterizations (e.g., multi-granularity decompositions, custom orthogonality constraints) could further optimize the balance between adaptation efficiency and representational robustness.

Key references: (Yang et al., 2024, Yang et al., 25 Aug 2025)