---
title: Kahneman–Tversky Optimization (KTO)
url: https://www.emergentmind.com/topics/kahneman-tversky-optimization-kto-6882bcf2-9d2f-40cb-b655-ba0ce2f0b74e
type: topic
---

# Kahneman–Tversky Optimization (KTO)

Kahneman–Tversky Optimization (KTO) is a preference-based fine-tuning method for large language models (LLMs) that grounds its objective in Prospect Theory, bringing risk sensitivity, loss aversion, and reference dependence into model alignment. Originating with Ethayarajh et al. (ICML 2024), and now widely applied in LLM safety alignment, tool use, multi-agent coordination, federated learning, and beyond, KTO generalizes Direct Preference Optimization (DPO) by replacing the standard pairwise logistic loss with a prospect-theoretic utility-based objective. According to Prospect Theory, agents do not value outcomes in absolute terms but perceive utility as nonlinear, with losses weighed more heavily than gains and evaluated relative to a status-quo reference. KTO injects these human-like asymmetries into LLM optimization, enabling efficient alignment from binary preference signals and robust model adaptation under data scarcity, annotation heterogeneity, and class imbalance.

## 1. Theoretical Foundations and Prospect-Theoretic Objective

KTO draws deeply from Prospect Theory, as developed by Kahneman and Tversky, introducing two critical nonlinearities absent from classical expected utility:

- **Value Function**: A piecewise curve $v(\cdot)$, concave for gains ($\Delta \geq 0$) and convex (and typically steeper) for losses ($\Delta < 0$), with a loss aversion coefficient $\lambda > 1$. Canonical forms (see [2402.01306]):

  $$
  v(z, z_{\mathrm{ref}}; \alpha, \lambda) =
    \begin{cases}
      (z - z_{\mathrm{ref}})^{\alpha}, & z \geq z_{\mathrm{ref}} \\
      -\lambda\, (z_{\mathrm{ref}} - z)^{\alpha}, & z < z_{\mathrm{ref}}
    \end{cases}
  $$

  with $0 < \alpha \leq 1$ controlling curvature (diminishing sensitivity).

- **Probability Weighting**: A function $w(p)$ that overweights small probabilities and underweights large ones, with a common instantiation (Prelec’s law):

  $$
  w(p; \gamma) = \exp\left(-(-\ln p)^\gamma\right), \quad 0 < \gamma \leq 1.
  $$

In the KTO context, "outcomes" are model completions $y$ for prompt $x$, valued by their relative log-probability under the current policy $\pi_\theta$ versus a reference $\pi_{\mathrm{ref}}$, and "events" are preference or safety labels. The expectation of prospect-theoretic utility $U(Z)$ is then computed over these sample-label pairs, aligning model outputs with human-like gain/loss perception.

## 2. Formal Definition, Loss Functions, and Optimization

KTO is formulated for both pairwise and unary (single-label) preference settings.

### Pairwise (Preference) Formulation

Given tuples $(x, y^+, y^-)$ ("preferred"/"rejected"), the central margin is $\Delta_\theta = \log \pi_\theta(y^+|x) - \log \pi_\theta(y^-|x)$. The basic KTO loss (exemplified by [2505.05704], [2506.18383], [2411.13611], [2411.02939]) is:

$$
\mathcal{L}_{\mathrm{KTO}} = -\mathbb{E}_{(x, y^+, y^-)} \big[ w(\sigma(\beta \Delta_\theta))\, v(\Delta_\theta) \big]
$$

where $\sigma(\cdot)$ is the sigmoid, $v(\cdot)$ the value function, $w(\cdot)$ the probability-weighting, and $\beta$ a temperature scale. For $v(\Delta)$ and $w(p)$, typical parameterizations are:

- $v(\Delta) = \Delta^\alpha$ for $\Delta \geq 0$, $-\lambda(-\Delta)^\beta$ for $\Delta < 0$, with $\lambda > 1$,
- $w(p) = p^\gamma / [p^\gamma + (1-p)^\gamma]^{1/\gamma}$ for $\gamma < 1$.

If hyperparameters $(\alpha,\beta,\lambda,\gamma) = (1,1,1,1)$, KTO reduces to DPO.

### Unary/Binary-Label (Prospect-RLHF/Single-Response) Formulation

In the broader single-label setting (see [2402.01306], [2502.14187], [2411.02939], [2501.14225], [2512.02555]), KTO operates on $(x, y, l)$ where $l \in \{\text{desirable}, \text{undesirable}\}$, defining per-example log-ratio $r_\theta(x,y) = \log \frac{\pi_\theta(y|x)}{\pi_{\mathrm{ref}}(y|x)}$ and a reference baseline $z_0$ (mean KL divergence over batch or dataset):

$$
v(x,y) = \begin{cases}
\lambda_D\, \sigma[\beta (r_\theta(x,y) - z_0)] & l = \text{desirable} \\
\lambda_U\, \sigma[\beta (z_0 - r_\theta(x,y))] & l = \text{undesirable}
\end{cases}
$$

$$
\mathcal{L}_{\mathrm{KTO}}(\theta) = \mathbb{E}_{(x,y) \sim D} [\lambda_l - v(x,y)]
$$

Weights $\lambda_D, \lambda_U$ encode loss aversion (generally $\lambda_U > \lambda_D$).

#### Algorithmic Pseudocode

A typical KTO training loop ([2501.14225], [2506.18383], [2406.18954]):

```python
for epoch in range(num_epochs):
    for (x, y, label) in minibatch:
        r = log_pi_theta(y|x) - log_pi_ref(y|x)
        z0 = mean_KL_divergence_in_batch()
        if label == "desirable":
            v = lambda_D * sigmoid(beta * (r - z0))
        else:
            v = lambda_U * sigmoid(beta * (z0 - r))
        loss = lambda_l - v
        backprop(loss)
    optimizer.step()
```

If the task demands risk adaptive tuning (e.g., category- or difficulty-specific margin scaling), $\beta$ is set per-example, as in DynamicKTO ([2507.19586]).

## 3. Comparison with DPO, SFT, and Other Preference-Based Methods

KTO generalizes DPO by explicitly modeling nonlinear risk sensitivity and reference dependence, and accommodating single-label data:

- **DPO** ([2505.05704], [2402.01306], [2506.18383]): operates on paired preferences, optimizes a logistic surrogate $\log \sigma(\beta \Delta_\theta)$. Represents special case of KTO with linear $v$, $w$.
- **Supervised Fine-Tuning (SFT)**: lacks direct use of negative preferences and cannot encode risk asymmetry.
- **Other variants**: IPO, SMO–Aug, ORPO, Step-DPO; all require strong preference pairing and may be less robust under data sparsity, non-IID distributions, or feedback streams ([2406.18954], [2501.14225], [2502.14187]).

KTO provides strong sample efficiency, notably for federated learning ([2502.14187], [2510.15993]), guardrail enforcement ([2406.18954]), and safety-critical domains ([2502.12485], [2512.04210]), and outperforms DPO in low-resource, unpaired, or highly imbalanced settings.

## 4. Practical Applications and Empirical Results

### Domain Summaries

- **Tool Learning & Error Correction**: HiTEC-KTO incorporates KTO into error-aware tool call optimization; KTO delivers $+5\sim+8$ point improvements in F1-Name and F1-Param on diverse benchmarks ([2506.00042]).
- **Logical Form Translation**: KTO reduces syntax errors and increases logical accuracy in translating NL to FOL, outperforming DPO and SFT by $1\sim3$ F1 points ([2506.18383]).
- **E-Commerce and User Behavior**: In ADORE, KTO alignment reduces false negatives for relevance, boosting online CTR and ad revenue by $+0.50\sim+1.28\%$ ([2512.02555]).
- **Federated Personalization**: KTO enables communication-efficient, robust, privacy-preserving preference learning, maintaining gains even in non-IID or redistributed settings ([2502.14187], [2510.15993]).
- **Safety Alignment**: SFT+KTO achieves $>99\%$ toxicity reduction in low-resource safety alignment, with far lower false positive rates than SFT+DPO ([2502.12485]).
- **Video Multi-Task Reasoning**: KTO bridges SFT and RL, sharply improving accuracy and resource efficiency in video agents ([2603.22918]).
- **Multi-Agent Language Games**: MaKTO’s KTO delivers $+10$–$23$\% win rate improvement versus baseline RL/LLM policies ([2501.14225]).
- **Geospatial Hallucination Mitigation**: DynamicKTO surpasses static KTO variants with +29.6% macro-average score in factuality ([2507.19586]).
- **Continuous Post-Training**: Two-stage SFT→KTO pipelines yield additive gains for small LMs and code LMs, especially when KTO is applied with strong negative sampling ([2411.02939], [2411.13611]).

Selected quantitative outcomes are summarized below:

| Scenario                | KTO Gain over Baseline      | Reference           |
|-------------------------|----------------------------|---------------------|
| CTR in E-commerce       | +0.98% vs +0.48% (w/ KTO)  | [2512.02555]        |
| Safety: Singlish Toxicity| 99% reduction, FPR 1%      | [2502.12485]        |
| Logic translation (FOL) | +2–5 pp “correct,” –5–15 syntax err | [2506.18383] |
| Tool-calling F1-Param   | 60.66 → 69.31 (Qwen2.5-1.5B)| [2506.00042]        |
| Geospatial accuracy     | 0.3748 → 0.4858 (DynamicKTO)| [2507.19586]        |

## 5. Implementation Considerations and Limitations

- **Hyperparameters**: Loss aversion ($\lambda$), curvature ($\alpha,\beta$), and probability weighting ($\gamma$) are seldom fully documented in downstream KTO applications; defaults match Prospect Theory (e.g., $\lambda=2.25$, $\alpha=\beta=0.88$, $\gamma=0.61$ per [2402.01306]).
- **Reference Policy**: Chosen as SFT checkpoint, pretrained LLM, or mixed policy, with KL-divergence $z_0$ typically estimated on-batch.
- **Batch Sizes and Stability**: Small batches can introduce high baseline noise; larger batches or running averages are advised ([2406.18954]).
- **Learning Rates**: KTO requires significantly lower learning rates ($\sim10^{-7}$) for stability, due to the increased sensitivity of logistic/prospect-based losses ([2406.18954], [2411.02939]).
- **Negative Sampling**: Strong preference and large performance gaps between positive/negative examples are necessary for maximized KTO effectiveness in code and agentic settings ([2411.13611], [2501.14225]).
- **Labeling Flexibility**: KTO excels when unary (single-output) judgments are easier, or when preference data are highly imbalanced or non-IID ([2502.14187], [2501.14225]).

Key limitations as reported include instability on code tasks without curated negative mining ([2411.13611]), dependence on band-limited KL baselines, and moderate sensitivity to hyperparameter mis-tuning or class imbalance. In federated and guardrail alignment, KTO is robust to data redistribution and supports incremental, privacy-aware adaptation ([2502.14187], [2406.18954]).

## 6. Extensions, Variants, and Recent Innovations

- **DynamicKTO**: Category-conditional or sample-adaptive risk scale $\beta$ for domain complexity (e.g., entity/relation/attribute in geospatial hallucination tasks), yielding superior performance to hand-tuned static $\beta$ ([2507.19586]).
- **KTO-S**: Augments the loss with signed KL regularization, directly enforcing contraction between trained and reference policies, improving training stability and convergence ([2502.12485]).
- **Federated KTO**: Implements KTO in federated settings (FedFLARKO, KTOO/KTOR), supporting heterogeneous client data and communication efficiency ([2510.15993], [2502.14187]).
- **Multi-Agent KTO (MaKTO)**: Iterates KTO with in-context exploration and fine-grained, stepwise preference extraction over multi-agent dialogues ([2501.14225]).

A recurrent theme is the ease of composing KTO with SFT and DPO in multi-stage pipelines (e.g., SFT→KTO→DPO or SFT→KTO→RL), leveraging the strengths of distributional pre-alignment and prospect-aware adaptation.

## 7. Empirical Robustness, Sample Efficiency, and Generalization

KTO consolidates its value as a general, sample-efficient alignment method. It maintains or exceeds DPO performance even under:

- Severe class imbalance or data thinning (retaining up to 90% fewer positive examples, [2402.01306])
- Absence of paired preferences, or when negatives are unpaired/unbatched ([2502.14187])
- Spurious correlation regimes (mathematical reasoning, distributional narrowness, [2505.05704])
- Heterogeneous, privacy-restricted FL scenarios ([2510.15993])
- Low-resource safety and multi-lingual settings ([2502.12485])

In aggregate, KTO matches or outperforms conventional RLHF surrogates in open-domain, safety-critical, multi-agent, and low-resource tasks, especially when feedback is naturally unary or preference annotation is costly.

---

**References**: For foundational formalism and comparative experiments across model scales, see Ethayarajh et al., "Model Alignment as Prospect Theoretic Optimization" [2402.01306]. For major downstream variants and benchmark applications, see [2506.00042], [2512.02555], [2506.18383], [2603.22918], [2501.14225], [2507.19586], [2510.15993], [2502.12485], among others.

Source: https://www.emergentmind.com/topics/kahneman-tversky-optimization-kto-6882bcf2-9d2f-40cb-b655-ba0ce2f0b74e