---
title: Response-Conditioned Bradley-Terry Model
url: https://www.emergentmind.com/topics/response-conditioned-bradley-terry-rc-bt-8434ea97-2d0e-4ef8-9438-23958cf0acac
type: topic
---

# Response-Conditioned Bradley-Terry Model

The Response-Conditioned Bradley–Terry (Rc-BT) model is a generalization of the classical Bradley–Terry framework, developed to model, recover, and interpret human (or annotator) preferences in settings characterized by context-response comparisons. It represents the probability of one candidate being preferred over another, conditioned on the response and often additional structural constraints (e.g. length constraints or ordinal labels), thereby addressing key limitations in pairwise preference learning such as model misspecification, confounding factors, and coarse feedback representation [2602.10286][2502.00814][2411.12843].

## 1. Foundations and Model Definition

The Rc-BT model operates on tuples or triplets $(x, y^+, y^-)$, where $x$ denotes the context (e.g., a prompt), and $y^+$, $y^-$ are candidate responses with $y^+$ preferred over $y^-$ under $x$. The joint data distribution $P(x, y^+, y^-)$ induces a conditional preference distribution (CPRD) defined as
\[
\omega_P(y \succ y' \mid x) = P(y \succ y' \mid x, \{y, y'\}) = \frac{P(x, y, y')}{P(x, y, y') + P(x, y', y)}.
\]
Rc-BT posits a real-valued score function $r : \mathcal{X} \times \mathcal{Y} \to \mathbb{R}$ such that
\[
P_r(y \succ y' \mid x) = \sigma(r(x,y) - r(x,y')), \qquad \sigma(t) = \frac{1}{1+e^{-t}}
\]
and fitting Rc-BT is typically by discriminative log-likelihood minimization:
\[
\mathcal{L}_{\rm BT}(r) = -\mathbb{E}_{(x, y^+, y^-)\sim P}\left[\log\sigma(r(x, y^+)-r(x, y^-))\right]
\]
[2602.10286].

## 2. Representability and Identifiability

Rc-BT exactly parameterizes the CPRD if and only if $P$ factors in a particular way. Specifically, $\omega_P$ is of Rc-BT form exactly when there exists a strictly positive function $h(x, y)$ such that, for all relevant $(x, y, y')$:
\[
\frac{P(x, y, y')}{P(x, y', y)} = \frac{h(x, y)}{h(x, y')}
\]
In that case, $r(x, y) = \log h(x, y)$ [2602.10286].

Moreover, Rc-BT is exact under the "positive–negative conditional independence" (CI) assumption:
\[
P(x, y^+, y^-) = P_X(x)p_+(y^+|x)p_-(y^-|x)
\]
Under positive–negative CI, for almost all $(x, \{y, y'\})$:
\[
P_{\hat r}(y \succ y' \mid x) = \omega_P(y \succ y' \mid x)
\]
If CI fails, Rc-BT recovers the projection of the true CPRD onto the BT-family via minimization of the KL divergence between the empirical CPRD and the BT parameterization [2602.10286].

## 3. Ordinal Feedback and Response Conditioning

Rc-BT generalizes naturally to ordinal feedback, allowing $Z$ in an ordered set $\mathcal{Z} = \{z_1 < \dots < z_m\}$, such as $\mathcal{Z} = \{0, 0.25, 0.5, 0.75, 1\}$ or $\mathcal{Z} = \{0, 0.5, 1\}$. The "marginal unbiasedness" assumption postulates that the expected annotator feedback $E[Z \mid (x,i,j)]$ matches the (latent) preference probability, facilitating use of soft labels and reducing estimation variance:
\[
p_\theta(x;i,j) = \sigma(r_\theta(x,i) - r_\theta(x,j))
\]
Probabilities for $Z$ are interpolated so $\mathbb{E}[Z] = p_\theta(x;i,j)$, and the Rc-BT loss aligns with unbiased cross-entropy or hinge-loss formulations [2411.12843].

Direct policy optimization (DPO) objectives additionally extend naturally to the Rc-BT setting via
\[
\mathcal{L}_{\rm DPO}(\theta) = -\sum Z_n \log \sigma(\Delta_n^\theta) - (1-Z_n)\log \sigma(-\Delta_n^\theta)
\]
with appropriately defined margins [2411.12843].

## 4. Response-Conditioned Modeling for Confounder Disentanglement

Rc-BT admits augmentation to address confounders such as length bias in reward modeling for language models. Construction proceeds by generating response-conditioned preference pairs to explicitly disentangle content quality from compliance with structural constraints (e.g., length). For each preferred response, a "too-short" length constraint is imposed; for each rejected, a "long-enough" constraint is imposed, forming two dataset partitions:
- $(x, y_w) \succ (x_l^1, y_w)$ indicates a preference for $y_w$ under the original prompt, relative to a prompt with a forbidding length constraint.
- $(x_l^2, y_l) \succ (x, y_l)$ prefers $y_l$ when coupled with a permissive constraint.

The reward model $r_\phi(x, y)$ is then fit by minimizing the response-conditioned negative log-likelihood over both partitions, enforcing orthogonal learning of semantic quality and compliance with explicit instructions [2502.00814].

This results in reward models and policy optimization (Rc-DPO) objectives that confer substantially higher adherence to constraints (e.g., length control) and higher semantic evaluation accuracy compared to baseline BT or RM approaches.

## 5. Sample Complexity: Margin and Connectivity

Statistical analysis reveals two key data-dependent factors governing Rc-BT sample complexity:
- **Margin ($\Delta_{r^*}$)**: Measures score separation for correct orderings. Larger margins increase error tolerance and improve accuracy.
- **Connectivity degree ($\kappa$)**: Quantifies how well frequently-compared pairs relate to variance in test distributions. Calculated as the ratio of expected squared margin differences and variance under a hypothesis class.

Finite-sample estimation error for $r(x,y)$ is bounded as:
\[
\mathbb{E}\left[(\Delta_{\hat r} - \Delta_{r^*})^2\right] \lesssim \frac{1}{\kappa} \left\{\hat{\mathfrak{R}}_n(\{\Delta_r\}) + M_B \sqrt{\frac{\log(1/\delta)}{n}}\right\}
\]
with corresponding test accuracy scaling with margin and $\kappa$ [2602.10286].

## 6. Empirical Results and Practical Considerations

Empirical benchmarks demonstrate Rc-BT’s advantages across multiple axes:
- **Reward modeling**: Rc-BT achieves 10–16% increased quality accuracy and $\sim$30–35% improved length adherence on models such as Qwen2, Llama, and Gemma, relative to standard RMs [2502.00814][2411.12843].
- **Mitigating confounders**: Rc-RM eliminates the monotonic score-length correlation typical of length-biased models.
- **Ordinal feedback**: 5-level and 3-level Rc-BT models exhibit lower cross-entropy loss and higher out-of-domain accuracy than binary BT models, consistent with Rademacher complexity reductions [2411.12843].
- **Policy learning**: Rc-DPO policies consistently show higher win rates and constraint adherence on both standard and length-benchmarks.

| Feedback Type | In-Dist. Acc. | OOD Acc.   | CE Loss       |
|---------------|---------------|------------|---------------|
| Binary (0,1)  | 93.29–94.01%  | 76.67–86.97%| 0.5709–0.5736 |
| 5-level       | 93.71–93.72%  | 81.00–85.84%| 0.5704–0.5714 |
| 3-level       | 93.59–93.81%  | 80.16–85.80%| 0.5704–0.5715 |
| Oracle soft   | 93.82–94.01%  | 81.93–86.97%| 0.5698–0.5711 |

Fine-grained annotations and moderate mixing of "tie" labels have been found to further increase accuracy and smooth learning [2411.12843].

## 7. Model Misspecification and Limitations

When data violate the positive–negative CI assumption, Rc-BT does not recover the true CPRD but projects it via KL minimization onto the BT-representable set:
\[
\hat r = \underset{r}{\arg\min} \;\mathbb{E}_{\widetilde P}\left[D_{\rm KL}(\text{Bern}(\omega_P(y \succ y' \mid x)) \Vert \text{Bern}(\sigma(\Delta_r)))\right]
\]
Experimentally, manipulations of margin and connectivity directly yield predictable effects: increasing minimum margin sharply boosts accuracy in small sample regimes; extremal negative sampling (too easy or too hard) reduces $\kappa$ and degrades accuracy; optimizing negative sampling to maximize $\kappa$ restores accuracy when connectivity is the bottleneck [2602.10286].

## References

- "What Does Preference Learning Recover from Pairwise Comparison Data?" [2602.10286]
- "Disentangling Length Bias In Preference Learning Via Response-Conditioned Modeling" [2502.00814]
- "Reward Modeling with Ordinal Feedback: Wisdom of the Crowd" [2411.12843]

Source: https://www.emergentmind.com/topics/response-conditioned-bradley-terry-rc-bt-8434ea97-2d0e-4ef8-9438-23958cf0acac