---
title: Markov Chain from Human Feedback (MCHF)
url: https://www.emergentmind.com/topics/markov-chain-from-human-feedback-mchf
type: topic
---

# Markov Chain from Human Feedback (MCHF)

to=arxiv_search.search ทะเบียนฟรี  बिट/json
{"query":"\"Markov Chain from Human Feedback\" OR \"Pairwise Choice Markov Chain\" OR \"Preference Alignment\"","max_results":10,"sort_by":"relevance"}
Markov Chain from Human Feedback (MCHF) denotes a family of methods that construct or analyze Markovian dynamics using human preference, exposure, or feedback data. Across the literature, the term covers at least three distinct but related constructions: discrete-choice models in which choice probabilities are defined by stationary distributions of continuous-time Markov chains over alternatives; alignment procedures in which pairwise human preferences induce a Markov kernel over model outputs; and closed-loop or human-in-the-loop systems in which human actions modulate, or are modulated by, a Markov process estimated from data [1603.02740], [2606.22652], [1706.06359]. The common thread is that human feedback is used not merely as a scalar supervision signal but as a source of transition structure, so that prediction, alignment, or bias analysis is carried out through stationary distributions, transition kernels, or coupled Markov dynamics.

## 1. Conceptual scope and historical development

The most direct precursor to MCHF in discrete choice is the Pairwise Choice Markov Chain (PCMC) model, which defines set-wise choice probabilities by the stationary distribution of a continuous-time Markov chain restricted to the offered choice set [1603.02740]. PCMC was introduced as a response to the increasing richness of human choice data and the limitations of classical discrete-choice axioms such as regularity, stochastic transitivity, and Luce’s choice axiom. Its central claim is that a Markovian construction can remain inferentially tractable while accommodating context effects and cyclic preference structure.

A distinct later use of the term appears in preference alignment, where MCHF is formulated as a Markov kernel on an output space \(X\) derived directly from pairwise utility \(U(x,y)\), with iteration of the kernel serving as the alignment mechanism [2606.22652]. In this formulation, MCHF is explicitly contrasted with Reinforcement Learning from Human Feedback (RLHF), which reduces comparisons to a scalar reward, and Nash Learning from Human Feedback (NLHF), which preserves pairwise utilities via a KL-regularized minimax objective. Here the Markov chain is not an explanatory metaphor but the primary aligned dynamics.

A third strand arises in human-in-the-loop systems and closed-loop estimation. In a coupled hidden Markov setting, observed user behavior and latent recommender or human-feedback states are modeled as interacting chains, and the latent transition structure is recovered with EM from observed trajectories [1706.06359]. Relatedly, work on human–algorithm interaction bias studies a Markov chain over hypothesis states whose transitions are induced by data selection and human response processes, emphasizing feedback loops, blind spots, and long-run bias accumulation [1608.07895].

Human-in-the-loop sampling methods such as Markov Chain Monte Carlo with People (MCMCP) and Gibbs Sampling with People (GSP) provide yet another operational interpretation: people govern the transition steps of a Markov chain by making binary or slider-based judgments, so that subjective semantic representations are explored through human-mediated MCMC kernels [2008.02595]. This suggests that MCHF is best understood not as a single algorithm but as a broader Markovian paradigm for converting human preference structure into dynamics over alternatives, outputs, hypotheses, or system states.

## 2. Pairwise Choice Markov Chains as a discrete-choice MCHF model

In PCMC, the universe of items is a finite set \(U\), and for each choice set \(S \subset U\), the model associates a continuous-time Markov chain on state space \(S\) with generator matrix \(Q_S\) obtained by restricting a global generator \(Q\) to rows and columns indexed by \(S\) [1603.02740]. The off-diagonal entries satisfy \(q_{ij} \ge 0\) for \(i \ne j\), and the diagonals are fixed by row-sum zero:
\[
q_{ii} = -\sum_{j \ne i} q_{ij}.
\]
To ensure irreducibility across all subsets, PCMC imposes the pairwise irreducibility constraint
\[
q_{ij} + q_{ji} \ge 1 \quad \text{for all } i \ne j.
\]

Choice probabilities are defined through stationarity. For any offered set \(S\), the stationary distribution \(\pi(S)\) is the solution of
\[
\pi(S)^T Q_S = 0, \qquad \sum_{i \in S} \pi_i(S)=1,
\]
and the model sets
\[
P(i \mid S)=\pi_i(S).
\]
Thus, MCHF in this sense means constructing a Markov chain from human feedback by interpreting pairwise preference intensities as transition rates and recovering set-wise probabilities from the induced stationary law.

The parameterization of rates is deliberately flexible. A Bradley–Terry–Luce (BTL) parameterization with latent qualities \(\gamma_i>0\) sets
\[
q_{ji}=p_{ij}=\frac{\gamma_i}{\gamma_i+\gamma_j},
\]
in which case the model collapses to Multinomial Logit (MNL) [1603.02740]. More generally, any pairwise probability matrix \(P=[p_{ij}]\) can be converted into a rate matrix by taking \(Q=P^T\) and enforcing the irreducibility constraint by rescaling if needed. The paper also describes low-dimensional Blade–Chest parameterizations, including
\[
p_{ij}=S(\|b_i-c_j\|_2^2-\|b_j-c_i\|_2^2)
\quad \text{or} \quad
p_{ij}=S(b_i \cdot c_j-b_j \cdot c_i),
\]
with \(S(x)=(1+e^{-x})^{-1}\), allowing non-transitive structure with \(O(dn)\) rather than \(O(n^2)\) parameters.

This construction preserves one weak but important axiom: uniform expansion. If a set is expanded by replacing each item with multiple copies satisfying specific symmetry and rate-equality conditions, then the aggregate choice probability of each equivalence class is invariant [1603.02740]. The proof proceeds through the stronger notion of a contractible partition, under which aggregate stationary masses depend only on between-group rates \(\Lambda=\{\lambda_{ij}\}\) and not on within-group dynamics. This makes PCMC unusual among flexible non-IIA models: it abandons regularity, stochastic transitivity, and Luce’s axiom, but retains a controlled invariance property.

The model also exhibits explicit violations of classical axioms. A rock–paper–scissors construction with pairwise win probability \(\alpha>1/2\) yields pairwise probabilities consistent with the game but a uniform stationary distribution \(1/3,1/3,1/3\) on the three-item set by symmetry, which can violate regularity when \(\alpha>2/3\) [1603.02740]. Because cyclic tournament structures are permitted, PCMC is not a random utility model in general.

## 3. Alignment by Markov kernels from pairwise utility

In the preference-alignment formulation, MCHF begins with a measurable output space \(X\), a reference distribution \(\mu_{\mathrm{ref}}\), and a bounded pairwise utility function \(U:X \times X \to \mathbb{R}\), where larger \(U(x,y)\) indicates greater human preference for \(y\) over \(x\) [2606.22652]. The key object is the Markov kernel
\[
\mathsf{P}(x,\mathrm{d}y)=\frac{\exp(U(x,y))}{Z(x)}\,\mu_{\mathrm{ref}}(\mathrm{d}y),
\qquad
Z(x)=\int \exp(U(x,z))\,\mu_{\mathrm{ref}}(\mathrm{d}z).
\]
Starting from \(\mu_0=\mu_{\mathrm{ref}}\), one iterates
\[
\mu_{t+1}(\mathrm{d}y)=\int \mu_t(\mathrm{d}x)\,\mathsf{P}(x,\mathrm{d}y).
\]

This formulation uses pairwise preferences directly rather than collapsing them into a scalar reward. The stationary distribution \(\mu_{\mathrm{MC}}\) is characterized by the fixed-point equation
\[
p_{\mathrm{MC}}(y)=\int p_{\mathrm{MC}}(x)\,
\frac{\exp(U(x,y))}{\int \exp(U(x,z))\,\mu_{\mathrm{ref}}(\mathrm{d}z)}
\,\mu_{\mathrm{ref}}(\mathrm{d}x),
\]
where \(p_{\mathrm{MC}}=d\mu_{\mathrm{MC}}/d\mu_{\mathrm{ref}}\) [2606.22652]. In general there is no simpler closed form, but the fixed point is unique under the paper’s contraction theorem.

A central concept is the seminorm
\[
\|U\|_{\oplus}=\inf_{g,f \in L^\infty(\mu_{\mathrm{ref}})} \|U-g \oplus f\|_\infty,
\qquad
(g \oplus f)(x,y)=g(x)+f(y).
\]
This measures the distance from \(U\) to additive or transitive utilities. If \(\|U\|_{\oplus}=0\), then \(U=g \oplus f\), and the Markovian dynamics reduce to an RLHF-style exponential tilt by \(f\) [2606.22652]. Nonzero \(\|U\|_{\oplus}\) quantifies non-transitive structure.

The paper proves a total-variation contraction
\[
d_{\mathrm{TV}}(\mu \mathsf{P}, \nu \mathsf{P})
\le c(\|U\|_{\oplus})\, d_{\mathrm{TV}}(\mu,\nu),
\]
with
\[
c(\|U\|_{\oplus})=\min\!\bigl(1-e^{-2\|U\|_{\oplus}},\,\|U\|_{\oplus}\bigr)\in[0,1),
\]
implying existence and uniqueness of the stationary distribution and geometric convergence of the iterates [2606.22652]. The associated mixing-time bound shows that smaller non-transitivity seminorm yields faster convergence.

The same work establishes a structured relationship among MCHF, RLHF, and NLHF. RLHF corresponds to the KL-regularized tilt
\[
\mu_{\mathrm{RL}}(\mathrm{d}y)\propto e^{R(y)}\,\mu_{\mathrm{ref}}(\mathrm{d}y),
\]
whereas NLHF solves a KL-regularized minimax problem preserving pairwise utilities [2606.22652]. For antisymmetric \(U\), the first step of both MCHF and NLHF recovers the RLHF solution based on the column-sum reward
\[
\hat f(y)=\int U(x,y)\,\mu_{\mathrm{ref}}(\mathrm{d}x),
\]
and from the second iteration onward both methods incorporate the same first-order correction driven by the residual
\[
\mathcal{R}(x,y)=U(x,y)+\hat f(x)-\hat f(y).
\]
This yields a unified perturbative picture: RLHF captures the additive component of the pairwise utility, while Markovian and game-theoretic methods correct for non-transitive structure [2606.22652].

## 4. Inference, optimization, and computational procedures

For discrete-choice PCMC, inference from observed choices \(\{(S_t,y_t)\}_{t=1}^T\) is formulated as maximum likelihood:
\[
L(Q)=\sum_{t=1}^T \log \pi_{y_t}(S_t;Q),
\]
or equivalently
\[
L(Q)=\sum_{S \subset U}\sum_{i \in S} C_{iS}\log \pi_i(S;Q),
\]
where \(C_{iS}\) counts how often item \(i\) was chosen from set \(S\) [1603.02740]. The constraints are \(q_{ij}\ge0\), \(q_{ii}=-\sum_{j\ne i}q_{ij}\), and \(q_{ij}+q_{ji}\ge1\). Because \(\pi(S)\) depends on \(Q\) only implicitly through stationarity equations, the likelihood is generally non-concave, and the paper employs constrained nonlinear optimization such as SLSQP with linear inequality constraints.

Computing \(\pi(S)\) requires solving
\[
\pi(S)^T Q_S=0, \qquad \sum_{i \in S}\pi_i(S)=1.
\]
Numerically, one can replace one row of \(Q_S^T\) by the normalization constraint to obtain a full-rank linear system \(A_S \pi(S)=b\), or compute the left nullspace via SVD or Arnoldi and renormalize [1603.02740]. Exact solves typically cost \(O(|S|^3)\), making caching and reuse of factorizations important when the same sets recur.

The paper describes practical initialization by smoothed pairwise fractions. If \(c_{ij}\) counts the number of times \(i\) was chosen over \(j\), then
\[
\hat p_{ij}=\frac{c_{ij}+\alpha}{c_{ij}+c_{ji}+2\alpha},
\qquad
q_{ji}=\hat p_{ij},
\]
which guarantees \(q_{ij}+q_{ji}=1\) and avoids degenerate zero rates [1603.02740]. This yields a straightforward MCHF pipeline: define the item universe, collect set-wise choices, initialize rates from pairwise aggregates, repeatedly solve stationary distributions on observed sets, maximize the constrained likelihood, and use the resulting stationary probabilities for ranking, adaptive selection, or iterative refinement.

In the closed-loop cl3MC model, inference targets an observed user chain \(R_t\) modulated by an unobserved human-feedback or recommender chain \(S_t\) [1706.06359]. The parameters are pages \(\{P_R^{(v)}\}_{v=1}^m\) and \(\{P_S^{(j)}\}_{j=1}^n\), with one-step joint transition
\[
P(R_{t+1}=i,S_{t+1}=u \mid R_t=j,S_t=v)
=
p_{ji}^{(v)} q_{vu}^{(j)}.
\]
Given only the trajectory \(r_{0:T}\), EM is used to maximize the marginal likelihood. Forward–backward recursions are built from time-dependent edge weights
\[
w_t(v,u)=p_{r_t r_{t+1}}^{(v)} q_{vu}^{(r_t)}.
\]
The posterior quantities
\[
\gamma_t(v)=P(S_t=v \mid r_{0:T},\theta^{\mathrm{old}})
\quad \text{and} \quad
\xi_t(v,u)=P(S_t=v,S_{t+1}=u \mid r_{0:T},\theta^{\mathrm{old}})
\]
provide expected transition counts, which are row-normalized in the M-step to update \(P_R^{(v)}\) and \(P_S^{(j)}\) [1706.06359]. This recovers latent Markov structure from closed-loop feedback data while correcting the bias that would result from ignoring the feedback state.

In the alignment formulation of MCHF, implementation depends on the domain. For discrete \(X\), one directly samples from
\[
\mathsf{P}(x,y)=\frac{\exp(U(x,y))}{\sum_z \exp(U(x,z))}\,\mu_{\mathrm{ref}}(y),
\]
iterating from \(x_0 \sim \mu_{\mathrm{ref}}\) and returning empirical distributions of the chain [2606.22652]. For continuous \(X\), the paper proposes conditional sampling from
\[
\mathsf{P}(x,\mathrm{d}y)\propto \exp(U(x,y))\,\mu_{\mathrm{ref}}(\mathrm{d}y)
\]
using rejection sampling, MCMC, SDE or diffusion samplers, flow matching, or PDE-based samplers, with log-sum-exp stabilization for partition-function evaluation [2606.22652].

## 5. Axiomatic, statistical, and dynamical properties

The different MCHF formulations share the property that global behavior is defined by invariant or asymptotic structure rather than by local comparison outcomes alone. In PCMC, this appears through the stationary distribution of restricted CTMCs and through the contractibility result showing that aggregate stationary masses are invariant to internal rates within contractible groups [1603.02740]. A plausible implication is that PCMC is well suited to settings where alternatives admit partial grouping or replication structure, because aggregate predictions can remain stable even when within-group pairwise relations are underspecified.

From an axiomatic perspective, PCMC rejects several standard assumptions while preserving uniform expansion [1603.02740]. This separates it from MNL and many random utility models, which depend on stronger regularity or IIA-type principles. The rock–paper–scissors example makes explicit that PCMC can represent cyclic preferences and context-dependent choice probabilities.

In the alignment setting, the key property is structure-adaptive geometric convergence governed by \(\|U\|_{\oplus}\), not by \(\|U\|_\infty\) [2606.22652]. This matters because pairwise utilities with a large bounded magnitude but a small non-transitive component can still induce fast convergence. The paper also provides proxies for \(\|U\|_{\oplus}\), including the rectangle defect
\[
\square(U)=\operatorname{ess\,sup}_{x,x',y,y'}
|U(x',y')-U(x,y')-U(x',y)+U(x,y)|
\]
and, for antisymmetric \(U\), the triangle defect
\[
\Delta(U)=\operatorname{ess\,sup}_{x,y,z}|U(x,y)+U(y,z)+U(z,x)|,
\]
with bounds linking these quantities to the seminorm [2606.22652].

The human-sampling literature emphasizes another aspect of Markovian structure: detailed balance and reversibility when human choices implement exact acceptance or conditional-sampling rules. In MCMCP, under i.i.d. Gumbel utility noise and symmetric proposals, the probability of accepting a proposed stimulus matches the Barker acceptance rule
\[
A_{\mathrm{Barker}}(x^*;x)=\frac{\pi(x^*)}{\pi(x)+\pi(x^*)},
\]
so detailed balance holds for \(\pi(x)\propto e^{\gamma \ell(x)}\) [2008.02595]. In GSP, slider-based conditional choices sample from
\[
p(z_k \mid z_{-k}) \propto e^{\gamma \ell(z_k,z_{-k})},
\]
so the resulting chain is a Gibbs sampler with stationary distribution \(\pi(z)\propto e^{\gamma \ell(z)}\) when the human choice distribution matches the conditionals [2008.02595].

The same paper introduces an aggregation parameter \(m\), the number of human conditional responses collected per update, and shows that \(m=1\) yields genuine Gibbs sampling whereas large \(m\), combined with mean or KDE-mode aggregation, approaches deterministic coordinate ascent [2008.02595]. This makes explicit a recurring tension in MCHF systems: exact sampling and mode-seeking optimization need not coincide, and practical systems often interpolate between them.

## 6. Empirical performance, applications, and limitations

PCMC was evaluated on two San Francisco travel-survey datasets: SFwork with 5,029 commute choices and SFshop with 3,157 shopping-trip choices [1603.02740]. Prediction error was measured by expected \(\ell_1\) distance between empirical test-set choice distributions and model probabilities across observed sets. The reported error reductions were 36.2% and 46.5% versus MNL, and 24.4% and 31.7% versus Mixed MNL on SFwork and SFshop, respectively [1603.02740]. Learned PCMC models also exhibited cyclic triplets—2 out of a maximum 8 in SFwork and 6 out of 20 in SFshop—indicating that the fitted rate matrices captured non-transitive structure absent from MNL [1603.02740]. Synthetic experiments further showed that arbitrary-\(Q\) PCMC and Blade–Chest parameterizations outperform MNL on non-IIA data while remaining competitive on MNL-generated data [1603.02740].

The alignment-oriented MCHF paper reports synthetic finite-state experiments, for example with \(|X|=20\), in which MCHF and NLHF nearly coincide and their deviation from RLHF is accurately described by the first-order perturbation analysis [2606.22652]. This supports the claim that RLHF captures the transitive component of pairwise utility while MCHF and NLHF encode structured corrections due to non-transitivity.

Human-in-the-loop sampling studies provide application evidence in color, emotional prosody, musical chords, and StyleGAN face-generation spaces [2008.02595]. Across these domains, GSP delivered higher information per trial than binary MCMCP and achieved faster convergence toward high-utility regions. In color experiments using 3D HSL representations and eight target words, GSP and aggregated GSP produced higher external ratings and faster convergence than MCMCP, while MCMCP yielded many off-target samples after 30 iterations [2008.02595]. In prosody, feature trajectories stabilized after the first sweep; in musical chords, KDE analysis revealed multimodal structure aligned with Western tonality; and in faces, rapid improvement occurred within approximately one sweep in a 10D PCA subspace [2008.02595].

Closed-loop Markov recovery has been tested on synthetic cl3MCs and a driver–recommender toy study [1706.06359]. In synthetic experiments with \(n \approx 20\) user states and \(m=2\) latent feedback states, estimation error decayed with increasing trajectory length \(T\), and for \(T=5 \times 10^4\) many runs yielded less than 10% average rowwise total-variation error, though occasional outliers occurred because of local optima or poor excitation [1706.06359]. In the toy routing study, closed-loop modeling substantially improved recovery of the unknown user transition matrix relative to an open-loop alternative and produced an estimated recommender-use fraction \(0.295 \pm 0.044\) against a true value \(0.3\), compared with \(0.352 \pm 0.058\) for the open-loop model [1706.06359].

Limitations differ across formulations but are structurally related. PCMC in unconstrained form has \(O(n^2)\) parameters and requires rich set-wise data; exact stationary solves scale cubically in set size; and non-concave likelihood optimization can be unstable [1603.02740]. The alignment formulation assumes bounded \(U\) and absolute continuity with respect to \(\mu_{\mathrm{ref}}\), while high-dimensional conditional sampling from \(\mathsf{P}(x,\cdot)\) may be difficult [2606.22652]. Closed-loop EM suffers from non-identifiability up to label permutation and possibly beyond, because only one chain is observed [1706.06359]. Human-mediated samplers are vulnerable to deviations from idealized Gumbel-noise models, context effects, correlated noise across slider positions, and ethical concerns when applied to sensitive perceptual judgments such as face attributes [2008.02595].

A recurring controversy concerns whether MCHF should be viewed as an alternative to reward learning or as a complement to it. The alignment paper argues that RLHF, NLHF, and MCHF are linked by a common perturbative structure rather than being mutually exclusive frameworks [2606.22652]. This suggests that the main distinction is not whether human feedback is used, but whether pairwise information is collapsed into scalar reward, retained as a game, or directly converted into Markovian transitions.

## 7. Relation to feedback loops, blind spots, and broader human-feedback modeling

The human–algorithm interaction literature broadens MCHF beyond choice and alignment by modeling a Markov chain over hypothesis states \(H\), where the next state depends on the current hypothesis through the data shown to users and the feedback elicited from them [1608.07895]. Selection is represented as a mixture
\[
p(x \mid h)=(1-\epsilon)\,p_{\mathrm{sel}}(x \mid h)+\epsilon\,q(x),
\]
with \(q(x)\) a baseline world distribution and \(p_{\mathrm{sel}}\) reflecting filter-bias or active-bias policies. For finite \(H=\{h_1,\dots,h_M\}\), the induced transition matrix \(P\) has entries
\[
P_{ij}=\Pr(X_{t+1}=h_j \mid X_t=h_i),
\]
which, in the most general form described in the paper, integrate over selected items, labels, and human actions [1608.07895].

This yields formal definitions of blind spots. A human blind spot is the set
\[
D_{\delta_b^H}^B=\{x \in D \mid p_{\mathrm{seen}}(x \mid h)\le \delta_b^H\},
\]
and an algorithm blind spot is
\[
D_{\delta_b^A}^B=\{x \in D \mid p(a=1 \mid y^*,x,h^*)\le \delta_b^A\},
\]
with corresponding prevalences \(\rho_b^H\) and \(\rho_b^A\) [1608.07895]. The paper proposes antidotes and reactive learning as mechanisms for modifying the transition operator, for example through a corrective mixture
\[
P'=(1-\lambda)P+\lambda A,
\]
where \(A\) promotes exploration of underexposed regions [1608.07895]. This suggests that one important use of MCHF is diagnostic rather than predictive: the chain formalism exposes how feedback loops reshape the long-run distribution over models or exposures.

Across these strands, several connections emerge. First, MCHF is especially natural when human data are intrinsically relational—comparisons, contextual choices, exposure-response traces—rather than single-label supervision. Second, stationary or asymptotic objects play the role that scalar rewards or utility scores play in more conventional pipelines. Third, non-transitivity is not treated as noise to be discarded but as structure to be represented, quantified, or exploited [1603.02740], [2606.22652].

A plausible implication is that the most distinctive contribution of MCHF is methodological rather than domain-specific. It provides a common language for discrete choice, preference alignment, closed-loop estimation, and human-in-the-loop exploration whenever human feedback is more naturally encoded as transitions, conditional movements, or interacting chains than as independent labels. In that sense, the term refers less to a single algorithm than to a Markovian design principle for modeling human feedback at the level of dynamics rather than isolated supervision.

Source: https://www.emergentmind.com/topics/markov-chain-from-human-feedback-mchf