Direct-Align: Automatic LLM Alignment

• The paper introduces Direct-Align, a method that aligns large language models by using synthetic preference data generated from contrastive prompt pairs without human annotations.
• It leverages a likelihood-based self-rewarding score and extends Direct Preference Optimization to provide a stable, automated alignment signal for target attributes.
• Empirical evaluations on Llama models show that Direct-Align outperforms conventional methods, achieving higher GPT-4 win rates and improved safety metrics.

Direct-Align, also referred to as Direct Large Model Alignment (DLMA), designates a fully-automatic, reward-model-free method for aligning LLMs with targeted attributes, such as harmlessness or helpfulness, by leveraging contrastive prompt-based self-rewarding mechanisms and Direct Preference Optimization (DPO). The central motivation is to achieve effective LLM alignment without human-annotated preference data, instead utilizing model-internal likelihood signals and automatically synthesized preference data, and subsequently using these for direct policy optimization (Liu et al., 2024).

1. Synthetic Preference Data via Contrastive Prompt Pairs

The DLMA framework initiates by generating synthetic preference tuples for each query via two distinct "contrastive" system prompts. The positive prompt ($p^+$) is crafted to elicit responses manifesting the desired attribute, while the negative prompt ($p^-$) encourages the opposing behavior. For every query $q$, the frozen LLM $T$ is sampled at temperature 1 under both prompts to yield two responses:

• $a_1 \leftarrow T(p^+, q)$ (positive prompt)
• $a_2 \leftarrow T(p^-, q)$ (negative prompt)

Resulting tuples $(q, a_1, a_2)$ comprise the synthetic preference dataset $D_\mathrm{pref}$. Positive examples for harmlessness, for instance, involve system prompts that explicitly instruct adherence or disregard of safety protocols. $a_1$ is only likely, not guaranteed, to be superior in the target attribute; thus, further automated rescoring is necessary.

2. Self-Rewarding Preference Score: Formalism and Calibration

DLMA eschews textual or model-generated preference judgments (as in RLAIF), opting instead for a likelihood-based surrogate score. For each preference tuple $(q,a_1,a_2)$, the self-rewarding score $S$ is computed as follows:

$$S = R(q, a_1, a_2) = \big[ \log T(a_1 | p^+, q) - \log T(a_1 | p^-, q) \big] - \big[ \log T(a_2 | p^+, q) - \log T(a_2 | p^-, q) \big]$$

This score is interpreted as the margin between the log-likelihood ratios (LLR) for each response under positive versus negative prompts—a proxy for the relative attribute strength, motivated by the Bradley–Terry model. Empirical results show that $S$ correlates strongly with GPT-4-based preferences, increasing from ~35% (for $S\in[0,10]$) to ~80% (for $S>30$) win rate of $a_1$ over $a_2$ [(Liu et al., 2024), Fig.3].

3. Integration of Self-Rewarding Scores in DPO

Direct Preference Optimization (DPO), as formulated by Rafailov et al. (2023), is adapted by DLMA to incorporate the self-rewarding score as a margin-shifting term. The standard DPO objective for a triple $(q,a_1,a_2)$ is

$$L_\mathrm{DPO} = - \mathbb{E}_{(q,a_1,a_2)} \left[ \log \sigma \left( \beta [ (\log \pi_\theta(a_1|q) - \log \pi_\mathrm{ref}(a_1|q)) - (\log \pi_\theta(a_2|q) - \log \pi_\mathrm{ref}(a_2|q)) ] \right) \right]$$

In DLMA, this is extended:

$$L_\mathrm{DLMA} = - \mathbb{E}_{(q,a_1,a_2)\sim D_\mathrm{pref}} \left[ \log \sigma \left( \beta [ (\log \pi_\theta(a_1|q) - \log \pi_\mathrm{ref}(a_1|q)) - (\log \pi_\theta(a_2|q) - \log \pi_\mathrm{ref}(a_2|q)) - \mathrm{clamp}(S, L, U) ] \right) \right]$$

where $\beta=0.1$, $L=-40$, $U=+40$ (stabilizing the margin shift), and $\sigma$ is the logistic sigmoid. The self-reward $S$ thus directly modulates the preference margin, re-weighting the learning signal according to model-internal evidence.

The typical optimization proceeds by minibatch sampling, on-the-fly computation (or retrieval) of $S$, and backpropagation using RMSprop (Liu et al., 2024).

4. Empirical Evaluation: Setup and Results

DLMA is evaluated on Llama2-7B and Llama2-13B, both Alpaca-instruction tuned, across the PKU-SafeRLHF, HH-Harmless, and HH-Helpful datasets. Metrics include GPT-4 head-to-head win rates, beaver-7b-cost safety scores (lower is safer), human annotation, and GPT-3 perplexity for text quality.

Key Results Summary (Llama2-7B, Win/Lose/Tie vs. Baselines under GPT-4 Judgment):

Dataset	Comparison	Win	Lose	Tie
PKU-SafeRLHF	DLMA vs Llama2-7B	55%	8%	37%
	DLMA vs RLAIF-7B	56%	8%	36%
HH-Harmless	DLMA vs Llama2-7B	58%	19%	23%
HH-Helpful	DLMA vs Llama2-7B	46%	15%	39%

Beaver-7B-Cost: DLMA achieves 1.92 (PKU-SafeRLHF) and 4.69 (HH-Harmless), outperforming RLHF and preceding unsupervised approaches in safety, with no discernible degradation in text quality as measured by perplexity (2.23 for PKU-SafeRLHF).

Ablation studies show that omitting contrastive prompts or the self-rewarding score results in marked performance degradation. Replacing DPO with PPO similarly harms both win rate and stability. Notably, DLMA, without reliance on human data, matches or exceeds RLHF and DPO trained on annotated data in multiple settings (Liu et al., 2024).

5. Comparative Analysis with Other Direct Alignment Paradigms

Recent comprehensive evaluations of Direct Alignment Algorithms (DAAs)—a superset class encompassing DPO, ORPO, ASFT, and their variants—demonstrate that the choice of a two-stage pipeline (initial Supervised Fine-Tuning followed by direct preference alignment) and of pairwise preference objectives is determinative for alignment quality, rather than the specific form of the implicit reward or single-stage integration (2502.01237).

Empirical evidence shows that (1) explicit SFT initialization yields large gains (e.g., ORPO SFT-init +13.8pts over base on Llama 3.1 8B), (2) tuning the unifying $\beta$ parameter is crucial (e.g., ORPO achieves 28.25% win-rate on AlpacaEval 2), and (3) pairwise logistic losses consistently outperform pointwise (binary cross-entropy) objectives at scale.

DLMA's choice of DPO—pairwise, reference-ratio margin-shifting, and fully two-stage (SFT + DLMA)—is thus mathematically and empirically justified within the state-of-the-art direct alignment landscape.

6. Underlying Mechanisms and Practical Implications

The likelihood-margin–based self-reward of DLMA is more robust than leveraging model generations or separate reward models:

• Probability-based evaluation of preferences yields less noisy, more accurate alignment signals, especially in the absence of human annotation.
• The self-rewarding mechanism calibrates preference strength relative to model confidence, enhancing label quality from synthetic data.
• DPO optimization eliminates instability associated with policy-gradient RL (e.g., PPO) and negates the need for a reward model, streamlining implementation.

Analyses confirm that as the self-reward score $S$ increases, the probability that the "positive" system-prompted response is preferred by GPT-4 increases monotonically, substantiating $S$ as an effective weighting mechanism.

7. Limitations and Future Directions

DLMA requires careful prompt engineering to ensure contrastive prompts reliably reflect the target attributes; suboptimal prompt design can degrade preference pair informativeness. The approach, while outperforming prior unsupervised alignment mechanisms, still approaches but does not universally surpass RLHF/DPO on high-quality human-annotated test sets—suggesting ongoing utility for human data in the highest-stakes domains.

As direct alignment methods converge in design and performance, understanding the boundaries of synthetic preference pair efficacy versus explicit human feedback remains an open question. Moreover, the integration of pairwise preference optimization into new modalities and tasks, as well as automated prompt generation, represents promising avenues for extension (Liu et al., 2024, 2502.01237).

---

For in-depth algorithmic details, derivations, and further empirical results, see "Direct LLM Alignment Through Self-Rewarding Contrastive Prompt Distillation" (Liu et al., 2024) and comparative DAA analysis in "The Differences Between Direct Alignment Algorithms are a Blur" (2502.01237).