Markov Chain from Human Feedback (MCHF)

• MCHF is a family of methods that converts human feedback into Markov transition structures, allowing for discrete choice, preference alignment, and human-in-the-loop sampling.
• It leverages stationary distributions and pairwise comparisons to bypass classical assumptions, enabling the modeling of cyclic and context-dependent behaviors.
• The framework supports efficient inference and optimization in various applications, from discrete choice experiments to bias analysis in interactive systems.

to=arxiv_search.search ทะเบียนฟรี बिट/json {"^^^^^^^^4query4 Chain from Human Feedback^^^^^^^^4\4 OR ^^^^^^^^4\4 Choice Markov Chain^^^^^^^^4\4 OR ^^^^^^^^4\4 Alignment^^^^^^^^4\4 Markov Chain from Human Feedback (MCHF) denotes a family of methods that construct or analyze Markovian dynamics using human preference, exposure, or feedback data. Across the literature, the term covers at least three distinct but related constructions: discrete-choice models in which choice probabilities are defined by stationary distributions of continuous-time Markov chains over alternatives; alignment procedures in which pairwise human preferences induce a Markov kernel over model outputs; and closed-loop or human-in-the-loop systems in which human actions modulate, or are modulated by, a Markov process estimated from data (&&&^^^^4query4^^^^&&&, &&&^^^^4\4^^^^&&&, &&&^^^^4 OR \4^^^^&&&). The common thread is that human feedback is used not merely as a scalar supervision signal but as a source of transition structure, so that prediction, alignment, or bias analysis is carried out through stationary distributions, transition kernels, or coupled Markov dynamics.

^^^^4\4^^^^. Conceptual scope and historical development

The most direct precursor to MCHF in discrete choice is the Pairwise Choice Markov Chain (PCMC) model, which defines set-wise choice probabilities by the stationary distribution of a continuous-time Markov chain restricted to the offered choice set (&&&^^^^4query4^^^^&&&). PCMC was introduced as a response to the increasing richness of human choice data and the limitations of classical discrete-choice axioms such as regularity, stochastic transitivity, and Luce’s choice axiom. Its central claim is that a Markovian construction can remain inferentially tractable while accommodating context effects and cyclic preference structure.

A distinct later use of the term appears in preference alignment, where MCHF is formulated as a Markov kernel on an output space PRESERVED_PLACEHOLDER_^^^^4query4^^^^ derived directly from pairwise utility PRESERVED_PLACEHOLDER_^^^^4\4^^^^, with iteration of the kernel serving as the alignment mechanism (&&&^^^^4\4^^^^&&&). In this formulation, MCHF is explicitly contrasted with Reinforcement Learning from Human Feedback (RLHF), which reduces comparisons to a scalar reward, and Nash Learning from Human Feedback (NLHF), which preserves pairwise utilities via a KL-regularized minimax objective. Here the Markov chain is not an explanatory metaphor but the primary aligned dynamics.

A third strand arises in human-in-the-loop systems and closed-loop estimation. In a coupled hidden Markov setting, observed user behavior and latent recommender or human-feedback states are modeled as interacting chains, and the latent transition structure is recovered with EM from observed trajectories (&&&^^^^4 OR \4^^^^&&&). Relatedly, work on human–algorithm interaction bias studies a Markov chain over hypothesis states whose transitions are induced by data selection and human response processes, emphasizing feedback loops, blind spots, and long-run bias accumulation (Nasraoui et al., 2016).

Human-in-the-loop sampling methods such as Markov Chain Monte Carlo with People (MCMCP) and Gibbs Sampling with People (GSP) provide yet another operational interpretation: people govern the transition steps of a Markov chain by making binary or slider-based judgments, so that subjective semantic representations are explored through human-mediated MCMC kernels (Harrison et al., 2020). This suggests that MCHF is best understood not as a single algorithm but as a broader Markovian paradigm for converting human preference structure into dynamics over alternatives, outputs, hypotheses, or system states.

^^^^4 OR \4^^^^. Pairwise Choice Markov Chains as a discrete-choice MCHF model

In PCMC, the universe of items is a finite set PRESERVED_PLACEHOLDER_^^^^4 OR \4^^^^, and for each choice set PRESERVED_PLACEHOLDER_^^^^4 OR \4^^^^, the model associates a continuous-time Markov chain on state space $S$ with generator matrix $Q_S$ obtained by restricting a global generator $Q$ to rows and columns indexed by $S$ (&&&^^^^4query4^^^^&&&). The off-diagonal entries satisfy $q_{ij} \ge 0$ for $i \ne j$, and the diagonals are fixed by row-sum zero: PRESERVED_PLACEHOLDER_^^^^4\4query4^^^^ To ensure irreducibility across all subsets, PCMC imposes the pairwise irreducibility constraint

PRESERVED_PLACEHOLDER_^^^^4\4\4^^^^

Choice probabilities are defined through stationarity. For any offered set PRESERVED_PLACEHOLDER_^^^^4\4 OR \4^^^^, the stationary distribution PRESERVED_PLACEHOLDER_^^^^4\4 OR \4^^^^ is the solution of

PRESERVED_PLACEHOLDER_^^^^4\44^^^^

and the model sets

PRESERVED_PLACEHOLDER_^^^^4\4^^^^5

Thus, MCHF in this sense means constructing a Markov chain from human feedback by interpreting pairwise preference intensities as transition rates and recovering set-wise probabilities from the induced stationary law.

The parameterization of rates is deliberately flexible. A Bradley–Terry–Luce (BTL) parameterization with latent qualities PRESERVED_PLACEHOLDER_^^^^4\4^^^^6 sets

PRESERVED_PLACEHOLDER_^^^^4\4^^^^7

in which case the model collapses to Multinomial Logit (MNL) (&&&^^^^4query4^^^^&&&). More generally, any pairwise probability matrix PRESERVED_PLACEHOLDER_^^^^4\4^^^^8 can be converted into a rate matrix by taking PRESERVED_PLACEHOLDER_^^^^4\4^^^^9 and enforcing the irreducibility constraint by rescaling if needed. The paper also describes low-dimensional Blade–Chest parameterizations, including

PRESERVED_PLACEHOLDER_^^^^4 OR \4query4^^^^

with PRESERVED_PLACEHOLDER_^^^^4 OR \4\4^^^^, allowing non-transitive structure with PRESERVED_PLACEHOLDER_^^^^4 OR \4 OR \4^^^^ rather than PRESERVED_PLACEHOLDER_^^^^4 OR \4 OR \4^^^^ parameters.

This construction preserves one weak but important axiom: uniform expansion. If a set is expanded by replacing each item with multiple copies satisfying specific symmetry and rate-equality conditions, then the aggregate choice probability of each equivalence class is invariant (&&&^^^^4query4^^^^&&&). The proof proceeds through the stronger notion of a contractible partition, under which aggregate stationary masses depend only on between-group rates PRESERVED_PLACEHOLDER_^^^^4 OR \44^^^^ and not on within-group dynamics. This makes PCMC unusual among flexible non-IIA models: it abandons regularity, stochastic transitivity, and Luce’s axiom, but retains a controlled invariance property.

The model also exhibits explicit violations of classical axioms. A rock–paper–scissors construction with pairwise win probability PRESERVED_PLACEHOLDER_^^^^4 OR \4^^^^5 yields pairwise probabilities consistent with the game but a uniform stationary distribution PRESERVED_PLACEHOLDER_^^^^4 OR \4^^^^6 on the three-item set by symmetry, which can violate regularity when PRESERVED_PLACEHOLDER_^^^^4 OR \4^^^^7 (&&&^^^^4query4^^^^&&&). Because cyclic tournament structures are permitted, PCMC is not a random utility model in general.

^^^^4 OR \4^^^^. Alignment by Markov kernels from pairwise utility

In the preference-alignment formulation, MCHF begins with a measurable output space PRESERVED_PLACEHOLDER_^^^^4 OR \4^^^^8, a reference distribution PRESERVED_PLACEHOLDER_^^^^4 OR \4^^^^9, and a bounded pairwise utility function PRESERVED_PLACEHOLDER_^^^^4 OR \4query4^^^^, where larger PRESERVED_PLACEHOLDER_^^^^4 OR \4\4^^^^ indicates greater human preference for PRESERVED_PLACEHOLDER_^^^^4 OR \4 OR \4^^^^ over PRESERVED_PLACEHOLDER_^^^^4 OR \4 OR \4^^^^ (&&&^^^^4\4^^^^&&&). The key object is the Markov kernel

PRESERVED_PLACEHOLDER_^^^^4 OR \44^^^^

Starting from PRESERVED_PLACEHOLDER_^^^^4 OR \4^^^^5, one iterates

PRESERVED_PLACEHOLDER_^^^^4 OR \4^^^^6

This formulation uses pairwise preferences directly rather than collapsing them into a scalar reward. The stationary distribution PRESERVED_PLACEHOLDER_^^^^4 OR \4^^^^7 is characterized by the fixed-point equation

PRESERVED_PLACEHOLDER_^^^^4 OR \4^^^^8

where PRESERVED_PLACEHOLDER_^^^^4 OR \4^^^^9 (&&&^^^^4\4^^^^&&&). In general there is no simpler closed form, but the fixed point is unique under the paper’s contraction theorem.

A central concept is the seminorm

$S$^^^^4query4^^^^

This measures the distance from $S$^^^^4\4^^^^ to additive or transitive utilities. If $S$^^^^4 OR \4^^^^, then $S$^^^^4 OR \4^^^^, and the Markovian dynamics reduce to an RLHF-style exponential tilt by $S$4 (&&&^^^^4\4^^^^&&&). Nonzero $S$5 quantifies non-transitive structure.

The paper proves a total-variation contraction

$S$6

with

$S$7

implying existence and uniqueness of the stationary distribution and geometric convergence of the iterates (&&&^^^^4\4^^^^&&&). The associated mixing-time bound shows that smaller non-transitivity seminorm yields faster convergence.

The same work establishes a structured relationship among MCHF, RLHF, and NLHF. RLHF corresponds to the KL-regularized tilt

$S$8

whereas NLHF solves a KL-regularized minimax problem preserving pairwise utilities (&&&^^^^4\4^^^^&&&). For antisymmetric $S$9, the first step of both MCHF and NLHF recovers the RLHF solution based on the column-sum reward

$Q_S$^^^^4query4^^^^

and from the second iteration onward both methods incorporate the same first-order correction driven by the residual

$Q_S$^^^^4\4^^^^

This yields a unified perturbative picture: RLHF captures the additive component of the pairwise utility, while Markovian and game-theoretic methods correct for non-transitive structure (&&&^^^^4\4^^^^&&&).

4. Inference, optimization, and computational procedures

For discrete-choice PCMC, inference from observed choices $Q_S$^^^^4 OR \4^^^^ is formulated as maximum likelihood: $Q_S$^^^^4 OR \4^^^^ or equivalently

$Q_S$4

where $Q_S$5 counts how often item $Q_S$6 was chosen from set $Q_S$7 (&&&^^^^4query4^^^^&&&). The constraints are $Q_S$8, $Q_S$9, and $Q$^^^^4query4^^^^. Because $Q$^^^^4\4^^^^ depends on $Q$^^^^4 OR \4^^^^ only implicitly through stationarity equations, the likelihood is generally non-concave, and the paper employs constrained nonlinear optimization such as SLSQP with linear inequality constraints.

Computing $Q$^^^^4 OR \4^^^^ requires solving

$Q$4

Numerically, one can replace one row of $Q$5 by the normalization constraint to obtain a full-rank linear system $Q$6, or compute the left nullspace via SVD or Arnoldi and renormalize (&&&^^^^4query4^^^^&&&). Exact solves typically cost $Q$7, making caching and reuse of factorizations important when the same sets recur.

The paper describes practical initialization by smoothed pairwise fractions. If $Q$8 counts the number of times $Q$9 was chosen over $S$^^^^4query4^^^^, then

$S$^^^^4\4^^^^

which guarantees $S$^^^^4 OR \4^^^^ and avoids degenerate zero rates (&&&^^^^4query4^^^^&&&). This yields a straightforward MCHF pipeline: define the item universe, collect set-wise choices, initialize rates from pairwise aggregates, repeatedly solve stationary distributions on observed sets, maximize the constrained likelihood, and use the resulting stationary probabilities for ranking, adaptive selection, or iterative refinement.

In the closed-loop cl^^^^4 OR \4^^^^MC model, inference targets an observed user chain $S$^^^^4 OR \4^^^^ modulated by an unobserved human-feedback or recommender chain $S$4 (&&&^^^^4 OR \4^^^^&&&). The parameters are pages $S$5 and $S$6, with one-step joint transition

$S$7

Given only the trajectory $S$8, EM is used to maximize the marginal likelihood. Forward–backward recursions are built from time-dependent edge weights

$S$9

The posterior quantities

$q_{ij} \ge 0$^^^^4query4^^^^

provide expected transition counts, which are row-normalized in the M-step to update $q_{ij} \ge 0$^^^^4\4^^^^ and $q_{ij} \ge 0$^^^^4 OR \4^^^^ (&&&^^^^4 OR \4^^^^&&&). This recovers latent Markov structure from closed-loop feedback data while correcting the bias that would result from ignoring the feedback state.

In the alignment formulation of MCHF, implementation depends on the domain. For discrete $q_{ij} \ge 0$^^^^4 OR \4^^^^, one directly samples from

$q_{ij} \ge 0$4

iterating from $q_{ij} \ge 0$5 and returning empirical distributions of the chain (&&&^^^^4\4^^^^&&&). For continuous $q_{ij} \ge 0$6, the paper proposes conditional sampling from

$q_{ij} \ge 0$7

using rejection sampling, MCMC, SDE or diffusion samplers, flow matching, or PDE-based samplers, with log-sum-exp stabilization for partition-function evaluation (&&&^^^^4\4^^^^&&&).

5. Axiomatic, statistical, and dynamical properties

The different MCHF formulations share the property that global behavior is defined by invariant or asymptotic structure rather than by local comparison outcomes alone. In PCMC, this appears through the stationary distribution of restricted CTMCs and through the contractibility result showing that aggregate stationary masses are invariant to internal rates within contractible groups (&&&^^^^4query4^^^^&&&). A plausible implication is that PCMC is well suited to settings where alternatives admit partial grouping or replication structure, because aggregate predictions can remain stable even when within-group pairwise relations are underspecified.

From an axiomatic perspective, PCMC rejects several standard assumptions while preserving uniform expansion (&&&^^^^4query4^^^^&&&). This separates it from MNL and many random utility models, which depend on stronger regularity or IIA-type principles. The rock–paper–scissors example makes explicit that PCMC can represent cyclic preferences and context-dependent choice probabilities.

In the alignment setting, the key property is structure-adaptive geometric convergence governed by $q_{ij} \ge 0$8, not by $q_{ij} \ge 0$9 (&&&^^^^4\4^^^^&&&). This matters because pairwise utilities with a large bounded magnitude but a small non-transitive component can still induce fast convergence. The paper also provides proxies for $i \ne j$^^^^4query4^^^^, including the rectangle defect

$i \ne j$^^^^4\4^^^^

and, for antisymmetric $i \ne j$^^^^4 OR \4^^^^, the triangle defect

$i \ne j$^^^^4 OR \4^^^^

with bounds linking these quantities to the seminorm (&&&^^^^4\4^^^^&&&).

The human-sampling literature emphasizes another aspect of Markovian structure: detailed balance and reversibility when human choices implement exact acceptance or conditional-sampling rules. In MCMCP, under i.i.d. Gumbel utility noise and symmetric proposals, the probability of accepting a proposed stimulus matches the Barker acceptance rule

$i \ne j$4

so detailed balance holds for $i \ne j$5 (Harrison et al., 2020). In GSP, slider-based conditional choices sample from

$i \ne j$6

so the resulting chain is a Gibbs sampler with stationary distribution $i \ne j$7 when the human choice distribution matches the conditionals (Harrison et al., 2020).

The same paper introduces an aggregation parameter $i \ne j$8, the number of human conditional responses collected per update, and shows that $i \ne j$9 yields genuine Gibbs sampling whereas large PRESERVED_PLACEHOLDER_^^^^4\4query4query4^^^^, combined with mean or KDE-mode aggregation, approaches deterministic coordinate ascent (Harrison et al., 2020). This makes explicit a recurring tension in MCHF systems: exact sampling and mode-seeking optimization need not coincide, and practical systems often interpolate between them.

6. Empirical performance, applications, and limitations

PCMC was evaluated on two San Francisco travel-survey datasets: SFwork with 5,^^^^4query4 OR \4^^^^9 commute choices and SFshop with ^^^^4 OR \4^^^^,^^^^4\4^^^^57 shopping-trip choices (&&&^^^^4query4^^^^&&&). Prediction error was measured by expected PRESERVED_PLACEHOLDER_^^^^4\4query4\4^^^^ distance between empirical test-set choice distributions and model probabilities across observed sets. The reported error reductions were ^^^^4 OR \4^^^^6.^^^^4 OR \4^^^^% and 46.5% versus MNL, and ^^^^4 OR \44^^^^.4% and ^^^^4 OR \4\4^^^^.7% versus Mixed MNL on SFwork and SFshop, respectively (&&&^^^^4query4^^^^&&&). Learned PCMC models also exhibited cyclic triplets—^^^^4 OR \4^^^^ out of a maximum 8 in SFwork and 6 out of ^^^^4 OR \4query4^^^^ in SFshop—indicating that the fitted rate matrices captured non-transitive structure absent from MNL (&&&^^^^4query4^^^^&&&). Synthetic experiments further showed that arbitrary-PRESERVED_PLACEHOLDER_^^^^4\4query4 OR \4^^^^ PCMC and Blade–Chest parameterizations outperform MNL on non-IIA data while remaining competitive on MNL-generated data (&&&^^^^4query4^^^^&&&).

The alignment-oriented MCHF paper reports synthetic finite-state experiments, for example with PRESERVED_PLACEHOLDER_^^^^4\4query4 OR \4^^^^, in which MCHF and NLHF nearly coincide and their deviation from RLHF is accurately described by the first-order perturbation analysis (&&&^^^^4\4^^^^&&&). This supports the claim that RLHF captures the transitive component of pairwise utility while MCHF and NLHF encode structured corrections due to non-transitivity.

Human-in-the-loop sampling studies provide application evidence in color, emotional prosody, musical chords, and StyleGAN face-generation spaces (Harrison et al., 2020). Across these domains, GSP delivered higher information per trial than binary MCMCP and achieved faster convergence toward high-utility regions. In color experiments using ^^^^4 OR \4^^^^D HSL representations and eight target words, GSP and aggregated GSP produced higher external ratings and faster convergence than MCMCP, while MCMCP yielded many off-target samples after ^^^^4 OR \4query4^^^^ iterations (Harrison et al., 2020). In prosody, feature trajectories stabilized after the first sweep; in musical chords, KDE analysis revealed multimodal structure aligned with Western tonality; and in faces, rapid improvement occurred within approximately one sweep in a ^^^^4\4query4^^^^D PCA subspace (Harrison et al., 2020).

Closed-loop Markov recovery has been tested on synthetic cl^^^^4 OR \4^^^^MCs and a driver–recommender toy study (&&&^^^^4 OR \4^^^^&&&). In synthetic experiments with PRESERVED_PLACEHOLDER_^^^^4\4query44^^^^ user states and PRESERVED_PLACEHOLDER_^^^^4\4query4^^^^5 latent feedback states, estimation error decayed with increasing trajectory length PRESERVED_PLACEHOLDER_^^^^4\4query4^^^^6, and for PRESERVED_PLACEHOLDER_^^^^4\4query4^^^^7 many runs yielded less than ^^^^4\4query4^^^^% average rowwise total-variation error, though occasional outliers occurred because of local optima or poor excitation (&&&^^^^4 OR \4^^^^&&&). In the toy routing study, closed-loop modeling substantially improved recovery of the unknown user transition matrix relative to an open-loop alternative and produced an estimated recommender-use fraction PRESERVED_PLACEHOLDER_^^^^4\4query4^^^^8 against a true value PRESERVED_PLACEHOLDER_^^^^4\4query4^^^^9, compared with PRESERVED_PLACEHOLDER_^^^^4\4\4query4^^^^ for the open-loop model (&&&^^^^4 OR \4^^^^&&&).

Limitations differ across formulations but are structurally related. PCMC in unconstrained form has PRESERVED_PLACEHOLDER_^^^^4\4\4\4^^^^ parameters and requires rich set-wise data; exact stationary solves scale cubically in set size; and non-concave likelihood optimization can be unstable (&&&^^^^4query4^^^^&&&). The alignment formulation assumes bounded PRESERVED_PLACEHOLDER_^^^^4\4\4 OR \4^^^^ and absolute continuity with respect to PRESERVED_PLACEHOLDER_^^^^4\4\4 OR \4^^^^, while high-dimensional conditional sampling from PRESERVED_PLACEHOLDER_^^^^4\4\44^^^^ may be difficult (&&&^^^^4\4^^^^&&&). Closed-loop EM suffers from non-identifiability up to label permutation and possibly beyond, because only one chain is observed (&&&^^^^4 OR \4^^^^&&&). Human-mediated samplers are vulnerable to deviations from idealized Gumbel-noise models, context effects, correlated noise across slider positions, and ethical concerns when applied to sensitive perceptual judgments such as face attributes (Harrison et al., 2020).

A recurring controversy concerns whether MCHF should be viewed as an alternative to reward learning or as a complement to it. The alignment paper argues that RLHF, NLHF, and MCHF are linked by a common perturbative structure rather than being mutually exclusive frameworks (&&&^^^^4\4^^^^&&&). This suggests that the main distinction is not whether human feedback is used, but whether pairwise information is collapsed into scalar reward, retained as a game, or directly converted into Markovian transitions.

7. Relation to feedback loops, blind spots, and broader human-feedback modeling

The human–algorithm interaction literature broadens MCHF beyond choice and alignment by modeling a Markov chain over hypothesis states PRESERVED_PLACEHOLDER_^^^^4\4\4^^^^5, where the next state depends on the current hypothesis through the data shown to users and the feedback elicited from them (Nasraoui et al., 2016). Selection is represented as a mixture

PRESERVED_PLACEHOLDER_^^^^4\4\4^^^^6

with PRESERVED_PLACEHOLDER_^^^^4\4\4^^^^7 a baseline world distribution and PRESERVED_PLACEHOLDER_^^^^4\4\4^^^^8 reflecting filter-bias or active-bias policies. For finite PRESERVED_PLACEHOLDER_^^^^4\4\4^^^^9, the induced transition matrix PRESERVED_PLACEHOLDER_^^^^4\4 OR \4query4^^^^ has entries

PRESERVED_PLACEHOLDER_^^^^4\4 OR \4\4^^^^

which, in the most general form described in the paper, integrate over selected items, labels, and human actions (Nasraoui et al., 2016).

This yields formal definitions of blind spots. A human blind spot is the set

PRESERVED_PLACEHOLDER_^^^^4\4 OR \4 OR \4^^^^

and an algorithm blind spot is

PRESERVED_PLACEHOLDER_^^^^4\4 OR \4 OR \4^^^^

with corresponding prevalences PRESERVED_PLACEHOLDER_^^^^4\4 OR \44^^^^ and PRESERVED_PLACEHOLDER_^^^^4\4 OR \4^^^^5 (Nasraoui et al., 2016). The paper proposes antidotes and reactive learning as mechanisms for modifying the transition operator, for example through a corrective mixture

PRESERVED_PLACEHOLDER_^^^^4\4 OR \4^^^^6

where PRESERVED_PLACEHOLDER_^^^^4\4 OR \4^^^^7 promotes exploration of underexposed regions (Nasraoui et al., 2016). This suggests that one important use of MCHF is diagnostic rather than predictive: the chain formalism exposes how feedback loops reshape the long-run distribution over models or exposures.

Across these strands, several connections emerge. First, MCHF is especially natural when human data are intrinsically relational—comparisons, contextual choices, exposure-response traces—rather than single-label supervision. Second, stationary or asymptotic objects play the role that scalar rewards or utility scores play in more conventional pipelines. Third, non-transitivity is not treated as noise to be discarded but as structure to be represented, quantified, or exploited (&&&^^^^4query4^^^^&&&, &&&^^^^4\4^^^^&&&).

A plausible implication is that the most distinctive contribution of MCHF is methodological rather than domain-specific. It provides a common language for discrete choice, preference alignment, closed-loop estimation, and human-in-the-loop exploration whenever human feedback is more naturally encoded as transitions, conditional movements, or interacting chains than as independent labels. In that sense, the term refers less to a single algorithm than to a Markovian design principle for modeling human feedback at the level of dynamics rather than isolated supervision.