RLHF Post-Training Overview

• RLHF-based post-training is a technique that uses ordinal human preferences to route queries among frozen pretrained circuits, aligning LLM outputs with user approval.
• Theoretical analysis reveals distortion lower bounds that limit optimal performance, particularly hindering robust, strategy-level behaviors in complex tasks.
• Overcoming these limits may require augmenting ordinal feedback with richer, cardinal signals or hybrid supervision methods to enhance model alignment.

Reinforcement Learning from Human Feedback (RLHF)-based post-training is the predominant paradigm for aligning large pretrained LLMs to human preferences, typically in settings where explicit reward signals are absent and only ordinal preference data—pairwise or k-wise—are available from human annotators. RLHF-based post-training comprises a sequence of stages that leverage these preference labels to route model queries to internally learned "circuits" (capabilities) in a pretrained backbone, reweighting behaviors to maximize user approval. Despite substantial empirical progress, RLHF with ordinal-only labels is subject to core theoretical limitations: there are provable lower bounds on the optimality gap achievable via post-training with preferences alone; these limits disproportionately constrain elicitation of robust, strategy-like behaviors such as deep reasoning. The following sections detail the formal framework, impossibility results, practical manifestations, reasons for RLHF's partial success on simpler tasks, and alternatives or mitigations for the limits of preference-based post-training (Zhao et al., 26 May 2025).

1. Formal Framework for RLHF-Based Post-Training

The RLHF post-training problem is formulated as follows:

• Query space and pretrained model: Let $Q$ be the set of user queries. The pretrained LLM $M_0$ encapsulates a finite set of $m$ latent response circuits $S_0 = \{s_1,\ldots,s_m\}$, each $s_i:Q\to R$ being a fixed way to answer queries based on pretraining. These circuits are frozen by pretraining; post-training does not invent new routes.
• Routing policy: The post-trained model $M=(\phi, g, S_0)$ consists of a query-to-latent mapping $\phi:Q\to Z$ and a randomized selection policy $g:Z\to\Delta(S_0)$ (where $\Delta$ denotes the simplex), so for query $q$, $M_0$0 and $M_0$1, and the output is $M_0$2. The effective policy is $M_0$3.
• Underlying utility: An unknown ground-truth utility $M_0$4 governs optimality; one seeks to maximize $M_0$5.
• Preference oracle: Post-training receives only ordinal comparisons, not cardinal utilities. For any $M_0$6 and pair $M_0$7, a preference oracle $M_0$8 outputs the more useful according to $M_0$9. Preference data may be infinite, noiseless, and online (idealized), or perturbed by e.g. a Bradley–Terry noise model—but remains strictly ordinal.

This framework captures both practical RLHF pipelines and a broad array of learning-from-preferences scenarios, abstracting preference-based post-training to a routing problem over frozen circuits using only pairwise (or $m$0-wise) comparison data (Zhao et al., 26 May 2025).

2. Impossibility Results and Distortion Lower Bounds

Preference-based post-training is fundamentally limited by impossibility results formally derived via voting theory analogies:

• Distortion Lower Bound (Noiseless Preferences): For any pretrained model with $m$1 circuits, and any post-training algorithm using only preference (ordinal) labels, there exists a utility $m$2 such that the expected utility of the post-trained model is a factor $m$3 less than the true optimum:

$m$4

This shows that, when only ordinal preferences are available, post-training can produce solutions far from the utilitarian optimum. The construction leverages hidden "districts" in $m$5—query subsets that cannot all be separated by any $m$6—defining utilities that reward only perfect routing. No matter how queries are grouped, a high fraction are necessarily misrouted (Zhao et al., 26 May 2025).

• Bradley–Terry Model (Linear Noise): If the preference oracle obeys a Bradley–Terry model (pairwise preference stochasticity $m$7), the lower bound strengthens: the worst-case distortion can become $m$8.
• Voting-Theoretic Structure: The analogy to voting theory: circuits correspond to candidates, queries to voters, and districts to hidden voter blocs. Distortion is the worst-case welfare gap between the elected and optimal candidates under only pairwise-ordinal voting. The RLHF setting generalizes bounds for aggregation rules (e.g., Borda count) to the case where the grouping of voters is itself post-trained.

3. Manifestation of Fundamental Limits in Practice

The severe distortion lower bounds manifest most acutely when attempting to elicit robust, strategy-level behaviors from LLMs:

• Reasoning/Strategy Circuits: Suppose robust reasoning is implemented by a specific circuit $m$9 that is globally optimal (highest $S_0 = \{s_1,\ldots,s_m\}$0 averaged over $S_0 = \{s_1,\ldots,s_m\}$1), but outputs may often appear worse on individual samples compared to simpler, less robust circuits $S_0 = \{s_1,\ldots,s_m\}$2 (e.g., due to backtracking, verbosity, or rare errors).
• Preference Signal Collapse: Ordinal preference data, even infinite and noiseless, is insufficient to reliably route $S_0 = \{s_1,\ldots,s_m\}$3 to $S_0 = \{s_1,\ldots,s_m\}$4 rather than $S_0 = \{s_1,\ldots,s_m\}$5 when user queries are entangled and the latent mapping $S_0 = \{s_1,\ldots,s_m\}$6 cannot separate the required hidden districts. Consequently, RLHF-based post-training may select compromise or "median" circuits, suppressing the robust behaviors most needed for reasoning tasks (Zhao et al., 26 May 2025).
• Compromise Phenomenon: In the inductive bias of RLHF with preferences, optimization converges to compromise candidates—those rarely the best but never the worst. In high-stakes multi-modal or reasoning domains, these compromise policies fail to capture the full "strategy" encoded in pretraining.

4. Successes on Simpler Tasks and Why RLHF Sometimes Works

Despite these roadblocks, RLHF-based post-training succeeds on certain classes of tasks:

• Instruction-Tuning, Surface-Level Safety: Tasks where each $S_0 = \{s_1,\ldots,s_m\}$7 is "almost linearly separable" (i.e., the optimal circuit is obvious from the query, and circuit confusion is rare) evade the lower bound: the $S_0 = \{s_1,\ldots,s_m\}$8 penalty is negligible. Most instruction-tuning, style transfer, and surface-level safety tasks fall in this regime (Zhao et al., 26 May 2025).
• Low Circuit Overlap: Success cases are characterized by low overlap between queries assigned to different circuits. Post-training has greater alignment power where latent query structure matches latent circuit partitioning.

The impossibility result thus does not preclude RLHF's utility on simple, surface-level or well-partitioned domains.

5. Implications, Alternatives, and Extensions

To address the core limitations of RLHF-based post-training with preferences, two major directions emerge:

• Use of Grounded Human Scoring: Augmenting preference data with scores more akin to cardinal utility (e.g., graded evaluations, process supervision, chain-of-thought correctness) can circumvent the lower bound, as these provide finer-grained routing signals. Techniques such as process reward modeling (PRM), or the use of human- or tool-assisted correctness verification, have shown substantially stronger results (Zhao et al., 26 May 2025).
• Algorithmic Innovations Beyond Routing: New algorithms that exploit richer feedback modalities or leverage verification signals—rather than relying solely on post-training preference-based routing—are necessary to move beyond the compromise/bottleneck identified. This may include training explicit value models, making selective use of stepwise correctness, or building hybrid systems with external tools for evaluation and supervision.

The theoretical results indicate a barrier for RLHF that cannot be overcome by scale alone; algorithmic and data-modality advances are necessary for progress in domains requiring deep alignment to human reasoning (Zhao et al., 26 May 2025).

6. Summary Table: Limits and Regimes of RLHF-Based Post-Training

Domain / Task type	RLHF (preference-only)	Lower-bound Distortion	Path to Improvement
Surface-level / per-query	Effective	Negligible	Latent circuit nearly separable
Strategy / robust reasoning	Bottlenecked	$S_0 = \{s_1,\ldots,s_m\}$9	Process supervision, scoring
Instruction/Safety compliance	Effective	Negligible	Standard RLHF sufficient
Deep research/planning	Limiting	$s_i:Q\to R$0	Cardinal or process rewards

RLHF-based post-training, when restricted to ordinal preferences, is theoretically limited by fundamental information bottlenecks. These impossibility results delineate the precise conditions for success and failure, motivating the need for richer feedback and new optimization paradigms in alignment of LLMs to complex human values and strategies (Zhao et al., 26 May 2025).