A Markov Chain Approach to Preference Alignment

Abstract: We propose Markov Chain from Human Feedback (MCHF), an elementary approach for aligning generative models from pairwise human preferences. Unlike Reinforcement Learning from Human Feedback (RLHF), which reduces comparisons to a scalar reward, and Nash Learning from Human Feedback (NLHF), which preserves pairwise utilities through a KL-regularized minimax optimization, MCHF uses pairwise preferences directly to define a transition mechanism over model outputs. Given a pairwise utility $U(x,y)$, which quantifies human preference for $y$ over $x$, and a reference probability distribution $μ{\mathsf{ref}}$, we define a Markov kernel $\mathsf{P}(x, dy)\propto \exp(U(x,y))μ{\mathsf{ref}}(dy)$, and take the Markov chain starting from $μ{\mathsf{ref}}$ as an iterative alignment procedure. We show that MCHF converges geometrically fast to the stationary distribution, with a convergence rate governed by the seminorm $|U|\oplus=\inf_{g,f\in L^\infty(μ_{\mathsf{ref}})}|U-g\oplus f|\infty$, which quantifies the non-transitive structure of the pairwise utility. We further show that a mirror-descent algorithm for NLHF satisfies an analogous structure-adaptive convergence guarantee. Finally, through a perturbation analysis, we prove that when $|U|\oplus$ is small, MCHF and NLHF agree up to first order around an RLHF solution, which yields a unified view of reward-based, game-theoretic, and Markovian approaches to alignment. In particular, for two natural algorithms that converge to the MCHF/NLHF equilibria, we show that the first step of MCHF and NLHF recovers the RLHF solution based on the column-sum reward $\hat{f}(y)=\int μ_{\mathsf{ref}}(dx) U(x, y)$, and starting from the second iteration, both algorithms incorporate the same linear functional of the residual $U-(-\hat f)\oplus \hat f$, which captures the non-transitive structure of the pairwise utility $U$.

• The paper presents MCHF, a novel framework that uses pairwise Markov kernels to preserve non-transitive human feedback in model alignment.
• It details explicit convergence rates via the seminorm ||U||⊕, demonstrating improved efficiency over adversarial approaches like NLHF.
• Empirical simulations and theoretical analysis confirm MCHF’s robustness and practical computational benefits in high-dimensional generative models.

Markov Chain from Human Feedback: A Flexible Framework for Preference-Based Alignment

Background and Motivation

Preference alignment is central to fine-tuning generative AI models for deployment in real-world, human-centric applications. Historically, the prevalent paradigm—Reinforcement Learning from Human Feedback (RLHF)—relies on aggregating pairwise human comparison data into a scalar reward model, typically implemented via Bradley–Terry–Luce (BTL) assumptions. Despite its widespread adoption, RLHF suffers from inherent limitations: it cannot capture non-transitive preference cycles or pluralistic, heterogeneous annotator feedback—a significant theoretical deficit in practical settings.

Recent advances such as Nash Learning from Human Feedback (NLHF) address some of these shortcomings by framing alignment as a minimax optimization and directly working with pairwise utilities. NLHF circumvents the scalar reward bottleneck but imposes an adversarial equilibrium and requires optimization over zero-sum objectives, potentially yielding pessimistic distributions and computational challenges.

The paper "A Markov Chain Approach to Preference Alignment" (2606.22652) introduces Markov Chain from Human Feedback (MCHF), which operationalizes alignment via pairwise preference-driven Markov kernels. Instead of reducing preference data to a reward or equilibrium, MCHF iterates model outputs using a transition kernel designed to reflect the empirical preference structure, directly preserving non-transitive interactions.

Figure 1: Workflow of alignment: RLHF reduces preference data to scalar rewards, while MCHF and NLHF construct weighted graphs that retain pairwise structure, offering greater flexibility.

Formalization: Utility Seminorm and Alignment Approaches

The authors define a pairwise utility $U(x,y)$—the preference for $y$ over $x$—and introduce a reference distribution $\mu_{\text{ref}}$ (typically a pretrained model). The Markov kernel is:

$P(x, dy) \propto \exp(U(x,y)) \mu_{\text{ref}}(dy)$

This kernel enables iterative refinement: starting from $\mu_{\text{ref}}$, subsequent distributions are obtained by repeated application of $P$.

A critical innovation is the seminorm $$\|U\|_\oplus = \inf_{g, f} \|U - g \oplus f\|_\infty$$, measuring the deviation of $U$ from additive/transitive structure. This seminorm quantifies the non-transitive complexity of human preferences—a key determinant of the alignment process.

When $U$ is nearly additive, MCHF and NLHF converge rapidly to the same equilibrium, coinciding with reward-based RLHF. When $y$0 is large (e.g., highly non-transitive data, Condorcet paradoxes), MCHF leverages the full structure, avoiding unjustified reward reduction.

Figure 2: Visualization of utility decomposition: the antisymmetric component of $y$1 and its deviation from additive structure are explicit.

Theoretical Guarantees and Iterative Dynamics

MCHF’s iterative process is a contraction mapping in TV distance, with rate $y$2. Remarkably, strong geometric convergence is obtained even in high-dimensional or continuous outcome spaces; the rate adapts not to the naive $y$3 but to the structurally relevant $y$4. The contraction is not merely implied; explicit bounds are proven.

Figure 3: Contraction rate as a function of $y$5, indicating adaptive convergence speed.

NLHF, when optimized via mirror descent with proper stepsize $y$6, exhibits equivalent structure-adaptive exponential convergence. The authors provide sharper bounds than prior literature, as existing rates depend on $y$7. Notably, in practical implementations, exact computation of $y$8 may require complex $y$9-LP solvers, but proxy estimates via rectangle or triangle defects are tractable.

In the regime $x$0, the first iteration of both MCHF and NLHF recovers the RLHF solution using the column-sum reward $x$1; subsequent iterations incorporate the residual non-transitive component. This result is precise—the deviation from RLHF is $x$2.

Figure 4: Comparison of equilibrium distributions: MCHF, NLHF, RLHF, and first-order expansion, demonstrating tight agreement in low $x$3 regime.

Furthermore, simulation studies confirm these asymptotics for both equilibrium and iterative dynamics, reinforcing the theoretical claims.

Figure 5: Empirical comparison between MCHF and NLHF iterations for antisymmetric $x$4, validating theoretical predictions.

Computational and Practical Implications

MCHF stands out as a conditional sampling-based algorithm: once the kernel $x$5 is implemented, alignment reduces to repeated application of this kernel—no adversarial optimization or geometric mixing required. Inference-time refinement is straightforward: the user can sequentially resample outputs via $x$6 until satisfaction.

NLHF, by contrast, requires solving KL-regularized minimax problems or sampling from complex geometric mixtures at every iteration, which is a substantial computational burden, especially for high-dimensional generative model outputs.

The coupling analysis in the paper further shows MCHF’s ability to optimize over larger classes of distributions, supporting richer preference extraction and improved inference-time refinement—a fact illustrated via classical non-transitive games (e.g., rock-paper-scissors), where hitting times for desired outputs are provably minimized under MCHF dynamics.

Figure 6: Numerical analysis of hitting times: MCHF refines outputs more efficiently than NLHF, particularly when non-transitive preference structure is strong.

Stability and Robustness

Both MCHF and NLHF exhibit robust Lipschitz stability with respect to perturbations in $x$7, controlled by $x$8. Errors induced by estimation noise or updates in the preference model propagate in a quantifiable, bounded way to the aligned distribution, offering clear guarantees for practical alignment scenarios.

Implications, Extensions, and Future Directions

By unifying Markovian, reward-based, and game-theoretic frameworks under a common perturbative and structural lens, this research illuminates previously opaque relationships between RLHF, NLHF, and spectral rank aggregation. The analytic clarity provided by the $x$9 seminorm and explicit formulas supports both theoretical investigations (e.g., axiomatics in social choice) and practical implementations in large-scale generative model alignment.

Notably, MCHF enables preference-based alignment in settings where reward decomposition is impossible or undesirable—for example, under heterogeneous or cyclic annotator populations, with broad support for pluralistic alignment.

Future challenges include scalable implementation of conditional sampling from $\mu_{\text{ref}}$0, possibly using rejection sampling, SDE-based diffusion models, or Monge–Ampère PDEs, as suggested by contemporaneous literature (2606.22652). Systematic analysis of which utility structures guarantee desirable properties within social choice theory is a promising direction for both theoretical and regulatory AI development.

Conclusion

The Markov Chain from Human Feedback framework delivers a principled, flexible, and computationally efficient approach for aligning generative models with rich, non-transitive human preferences. By bridging reward models and game-theoretic approaches, and by providing explicit convergence rates and robustness guarantees, MCHF establishes a new standard for preference alignment, both theoretically and practically, in the evolving landscape of generative AI.