---
title: Fitzpatrick Losses Overview
url: https://www.emergentmind.com/topics/fitzpatrick-losses
type: topic
---

# Fitzpatrick Losses Overview

Fitzpatrick losses are a family of convex loss functions derived from the Fitzpatrick function, a foundational concept in maximal monotone operator theory. They serve as a refined generalization of Fenchel–Young losses, yielding tighter convex upper bounds while retaining the same prediction (link) functions. Notably, Fitzpatrick losses provide a principled method for constructing convex surrogates in structured prediction and probabilistic modeling, with direct applications to tasks such as label-proportion estimation and beyond [2405.14574].

## 1. Mathematical Foundation: The Fitzpatrick Function

Let \(T \subseteq \mathbb{R}^k \times \mathbb{R}^k\) be a monotone operator, notably maximal monotone if it cannot be strictly enlarged without losing monotonicity. The Fitzpatrick function associated to \(T\) is the convex, jointly lower semicontinuous map
\[
F_T : \mathbb{R}^k \times \mathbb{R}^k \to \mathbb{R} \cup \{+\infty\},\qquad
F_T(y, \theta) = \sup_{(y', \theta') \in T} \langle y-y', \theta' \rangle + \langle y', \theta \rangle.
\]
Given a proper, convex, lower semicontinuous function \(\Omega: \mathbb{R}^k \to \mathbb{R} \cup \{+\infty\}\), and setting \(T = \partial \Omega\) (the subdifferential), the Fitzpatrick function becomes
\[
F_{\partial\Omega}(y, \theta) = \sup_{(y', \theta') \in \partial\Omega} \langle y-y', \theta' \rangle + \langle y', \theta \rangle.
\]
This construction leads to the **refined Fenchel–Young inequalities**
\[
\langle y, \theta \rangle \leq F_{\partial\Omega}(y, \theta) \leq \Omega(y) + \Omega^*(\theta), \quad \forall\,y, \theta \in \mathbb{R}^k,
\]
where \(\Omega^*\) denotes the Fenchel conjugate.

## 2. Construction and Properties of Fitzpatrick Losses

Given a base function \(\Omega\) as above, define the Fitzpatrick loss
\[
L_F(y, \theta) = F_{\partial\Omega}(y, \theta) - \langle y, \theta \rangle.
\]
Equivalently,
\[
L_F(y, \theta) = \sup_{(y', \theta') \in \partial\Omega} \langle y' - y, \theta - \theta' \rangle \geq 0.
\]
Here, \(F_{\partial\Omega}\) serves as the tightest jointly convex upper envelope of the bilinear form \(\langle y, \theta \rangle\) that exactly captures the monotone graph \(\partial\Omega\). The classical Fenchel–Young loss
\[
L_{FY}(y, \theta) = \Omega(y) + \Omega^*(\theta) - \langle y, \theta \rangle
\]
always satisfies \(L_F(y, \theta) \leq L_{FY}(y, \theta)\). Fitzpatrick losses thus represent a strictly tighter class of convex surrogates under the same link function associated with \(\Omega\).

Key properties include:
- **Separate convexity**: \(L_F(y, \theta)\) is convex in \(y\) for fixed \(\theta\) and convex in \(\theta\) for fixed \(y\).
- **Zero loss characterization**: \(L_F(y, \theta) = 0\) if and only if \((y, \theta) \in \partial\Omega\), equivalently \(y \in \partial\Omega^*(\theta)\). For strictly convex \(\Omega\), \(y = \nabla\Omega^*(\theta)\).
- **Gradient in \(\theta\)**: \(\partial_\theta L_F(y, \theta) = y^*(y, \theta) - y\), with \(y^*(y, \theta)\) defined via the supremum in the Fitzpatrick function.

## 3. Exemplary Instances: Sparsemax and Logistic Fitzpatrick Losses

Two concrete realizations illustrate the construction:

| Variant                        | Generating \(\Omega\)                    | Prediction Link                    | Key Formula for \(L_F\)                 |
|---------------------------------|------------------------------------------|-------------------------------------|------------------------------------------|
| Fitzpatrick sparsemax           | \(\frac{1}{2}\|y\|_2^2 + \iota_{\Delta^k}(y)\) | \(P_{\Delta^k}(\theta)\)           | \(\langle y^* - y, \theta - y^* \rangle\), \(y^* = P_{\Delta^k}\!\left(\frac{y+\theta}{2}\right)\) |
| Fitzpatrick logistic            | \(\sum_{i=1}^k y_i\ln y_i + \iota_{\Delta^k}(y)\) | \(\text{softmax}(\theta)\)         | \(\langle y^* - y, \theta - \ln y^* - 1 \rangle\); \(y^*\) involves Lambert-\(W\) root finding      |

- **Fitzpatrick sparsemax loss**: Utilizes the squared norm plus simplex indicator; the subdifferential yields the Euclidean projector onto the simplex. Prediction remains with the sparsemax mapping.

- **Fitzpatrick logistic loss**: Leverages the negative Shannon entropy over the simplex; the subdifferential is the softmax. The optimal \(y^*\) for the loss is expressed via the Lambert-\(W\) function and a normalization constant to enforce summation to one.

Both losses maintain the same link function as their Fenchel–Young counterparts, ensuring prediction equivalence while tightening the surrogate.

## 4. Relationship to Fenchel–Young Losses and Bregman Divergence

Fitzpatrick losses can be interpreted as Fenchel–Young losses generated by a **target-dependent generator**:
\[
\Omega_y(y') = \Omega(y') + D_\Omega(y, y')
\]
where \(D_\Omega(y, y') = \Omega(y) - \Omega(y') - \sup_{\theta' \in \partial\Omega(y')}\langle y - y', \theta' \rangle\) generalizes Bregman divergence.

A principal result states
\[
F_{\partial\Omega}(y, \theta) = \Omega_y(y) + \Omega_y^*(\theta), \qquad L_F(y, \theta) = L_{\Omega_y \oplus \Omega_y^*}(y, \theta).
\]
This suggests that every Fitzpatrick loss is itself a Fenchel–Young loss for a particular, target-sensitive choice of generator. The construction thereby inherits convexity and, under strong convexity of \(\Omega_y\), differentiability with Lipschitz gradient.

## 5. Theoretical Intuition from Maximal Monotone Operators

The Fitzpatrick function represents the tightest possible convex surrogate above the bilinear pairing \(\langle y, \theta \rangle\) with the property that its zero level set recovers precisely the graph of the operator \(T\). In contrast, the usual Fenchel–Young bound \(\Omega(y) + \Omega^*(\theta)\) is generally looser and fully separable in the two arguments.

Geometrically, while the Fenchel–Young inequality bounds \(\Omega(y)\) using its tangent at \(\theta\), the Fitzpatrick approach adaptively tilts this tangent via the term \(D_\Omega(y, y')\), producing a lower (and thus tighter) vertical intercept and reducing the surrogate’s slack.

A plausible implication is that, whenever the strict coupling between prediction and target is critical, Fitzpatrick losses provide sharper surrogates for learning and optimization.

## 6. Empirical Behavior: Label-Proportion Estimation

Empirical studies on multilabel benchmark datasets reveal that linear models trained with Fitzpatrick losses for label-proportion estimation (minimizing \(\sum_{i=1}^n L(y_i, Wx_i) + \frac{\lambda}{2}\|W\|^2\) via L-BFGS) yield:
- Fitzpatrick sparsemax often outperforms or matches the traditional sparsemax loss in mean-squared error recovery.
- Fitzpatrick logistic is on par with the standard logistic loss in accuracy, with only minor computational overhead due to the use of the Lambert-\(W\) function.

This suggests that Fitzpatrick losses can be substituted for their Fenchel–Young counterparts with minimal modification to code and, potentially, performance improvements [2405.14574].

## 7. Further Reading and Related Work

- The original development of Fitzpatrick functions: S. Fitzpatrick, "Representing monotone operators by convex functions," 1988.
- Broad analysis and inequalities for Fitzpatrick functions: H. Bauschke et al., "Fitzpatrick functions: Inequalities, examples …," Journal of Convex Analysis, 2005.
- Fenchel–Young losses as the classical family: M. Blondel, A. Martins & V. Niculae, "Learning with Fenchel–Young losses," JMLR 2020.
- Primary source for Fitzpatrick losses and experimental evidence: S. Rakotomandimby et al., "Learning with Fitzpatrick losses," ICML 2023 and [2405.14574].

A plausible direction is that Fitzpatrick losses, as convex surrogates with tighter theoretical control, offer a versatile tool in designing prediction-calibrated losses for structured outputs and probabilistic modeling.

Source: https://www.emergentmind.com/topics/fitzpatrick-losses