Papers
Topics
Authors
Recent
Search
2000 character limit reached

Fitzpatrick Losses Overview

Updated 4 June 2026
  • Fitzpatrick losses are convex loss functions derived from maximal monotone operator theory that yield tighter surrogates than Fenchel–Young losses while retaining the same prediction link functions.
  • They are constructed using the Fitzpatrick function, with key instances like sparsemax and logistic variants applied to tasks including label-proportion estimation.
  • Empirical studies show that models using Fitzpatrick losses achieve comparable or improved accuracy with minimal computational overhead in structured prediction scenarios.

Fitzpatrick losses are a family of convex loss functions derived from the Fitzpatrick function, a foundational concept in maximal monotone operator theory. They serve as a refined generalization of Fenchel–Young losses, yielding tighter convex upper bounds while retaining the same prediction (link) functions. Notably, Fitzpatrick losses provide a principled method for constructing convex surrogates in structured prediction and probabilistic modeling, with direct applications to tasks such as label-proportion estimation and beyond (Rakotomandimby et al., 2024).

1. Mathematical Foundation: The Fitzpatrick Function

Let T⊆Rk×RkT \subseteq \mathbb{R}^k \times \mathbb{R}^k be a monotone operator, notably maximal monotone if it cannot be strictly enlarged without losing monotonicity. The Fitzpatrick function associated to TT is the convex, jointly lower semicontinuous map

FT:Rk×Rk→R∪{+∞},FT(y,θ)=sup⁡(y′,θ′)∈T⟨y−y′,θ′⟩+⟨y′,θ⟩.F_T : \mathbb{R}^k \times \mathbb{R}^k \to \mathbb{R} \cup \{+\infty\},\qquad F_T(y, \theta) = \sup_{(y', \theta') \in T} \langle y-y', \theta' \rangle + \langle y', \theta \rangle.

Given a proper, convex, lower semicontinuous function Ω:Rk→R∪{+∞}\Omega: \mathbb{R}^k \to \mathbb{R} \cup \{+\infty\}, and setting T=∂ΩT = \partial \Omega (the subdifferential), the Fitzpatrick function becomes

F∂Ω(y,θ)=sup⁡(y′,θ′)∈∂Ω⟨y−y′,θ′⟩+⟨y′,θ⟩.F_{\partial\Omega}(y, \theta) = \sup_{(y', \theta') \in \partial\Omega} \langle y-y', \theta' \rangle + \langle y', \theta \rangle.

This construction leads to the refined Fenchel–Young inequalities

⟨y,θ⟩≤F∂Ω(y,θ)≤Ω(y)+Ω∗(θ),∀ y,θ∈Rk,\langle y, \theta \rangle \leq F_{\partial\Omega}(y, \theta) \leq \Omega(y) + \Omega^*(\theta), \quad \forall\,y, \theta \in \mathbb{R}^k,

where Ω∗\Omega^* denotes the Fenchel conjugate.

2. Construction and Properties of Fitzpatrick Losses

Given a base function Ω\Omega as above, define the Fitzpatrick loss

LF(y,θ)=F∂Ω(y,θ)−⟨y,θ⟩.L_F(y, \theta) = F_{\partial\Omega}(y, \theta) - \langle y, \theta \rangle.

Equivalently,

TT0

Here, TT1 serves as the tightest jointly convex upper envelope of the bilinear form TT2 that exactly captures the monotone graph TT3. The classical Fenchel–Young loss

TT4

always satisfies TT5. Fitzpatrick losses thus represent a strictly tighter class of convex surrogates under the same link function associated with TT6.

Key properties include:

  • Separate convexity: TT7 is convex in TT8 for fixed TT9 and convex in FT:Rk×Rk→R∪{+∞},FT(y,θ)=sup⁡(y′,θ′)∈T⟨y−y′,θ′⟩+⟨y′,θ⟩.F_T : \mathbb{R}^k \times \mathbb{R}^k \to \mathbb{R} \cup \{+\infty\},\qquad F_T(y, \theta) = \sup_{(y', \theta') \in T} \langle y-y', \theta' \rangle + \langle y', \theta \rangle.0 for fixed FT:Rk×Rk→R∪{+∞},FT(y,θ)=sup⁡(y′,θ′)∈T⟨y−y′,θ′⟩+⟨y′,θ⟩.F_T : \mathbb{R}^k \times \mathbb{R}^k \to \mathbb{R} \cup \{+\infty\},\qquad F_T(y, \theta) = \sup_{(y', \theta') \in T} \langle y-y', \theta' \rangle + \langle y', \theta \rangle.1.
  • Zero loss characterization: FT:Rk×Rk→R∪{+∞},FT(y,θ)=sup⁡(y′,θ′)∈T⟨y−y′,θ′⟩+⟨y′,θ⟩.F_T : \mathbb{R}^k \times \mathbb{R}^k \to \mathbb{R} \cup \{+\infty\},\qquad F_T(y, \theta) = \sup_{(y', \theta') \in T} \langle y-y', \theta' \rangle + \langle y', \theta \rangle.2 if and only if FT:Rk×Rk→R∪{+∞},FT(y,θ)=sup⁡(y′,θ′)∈T⟨y−y′,θ′⟩+⟨y′,θ⟩.F_T : \mathbb{R}^k \times \mathbb{R}^k \to \mathbb{R} \cup \{+\infty\},\qquad F_T(y, \theta) = \sup_{(y', \theta') \in T} \langle y-y', \theta' \rangle + \langle y', \theta \rangle.3, equivalently FT:Rk×Rk→R∪{+∞},FT(y,θ)=sup⁡(y′,θ′)∈T⟨y−y′,θ′⟩+⟨y′,θ⟩.F_T : \mathbb{R}^k \times \mathbb{R}^k \to \mathbb{R} \cup \{+\infty\},\qquad F_T(y, \theta) = \sup_{(y', \theta') \in T} \langle y-y', \theta' \rangle + \langle y', \theta \rangle.4. For strictly convex FT:Rk×Rk→R∪{+∞},FT(y,θ)=sup⁡(y′,θ′)∈T⟨y−y′,θ′⟩+⟨y′,θ⟩.F_T : \mathbb{R}^k \times \mathbb{R}^k \to \mathbb{R} \cup \{+\infty\},\qquad F_T(y, \theta) = \sup_{(y', \theta') \in T} \langle y-y', \theta' \rangle + \langle y', \theta \rangle.5, FT:Rk×Rk→R∪{+∞},FT(y,θ)=sup⁡(y′,θ′)∈T⟨y−y′,θ′⟩+⟨y′,θ⟩.F_T : \mathbb{R}^k \times \mathbb{R}^k \to \mathbb{R} \cup \{+\infty\},\qquad F_T(y, \theta) = \sup_{(y', \theta') \in T} \langle y-y', \theta' \rangle + \langle y', \theta \rangle.6.
  • Gradient in FT:Rk×Rk→R∪{+∞},FT(y,θ)=sup⁡(y′,θ′)∈T⟨y−y′,θ′⟩+⟨y′,θ⟩.F_T : \mathbb{R}^k \times \mathbb{R}^k \to \mathbb{R} \cup \{+\infty\},\qquad F_T(y, \theta) = \sup_{(y', \theta') \in T} \langle y-y', \theta' \rangle + \langle y', \theta \rangle.7: FT:Rk×Rk→R∪{+∞},FT(y,θ)=sup⁡(y′,θ′)∈T⟨y−y′,θ′⟩+⟨y′,θ⟩.F_T : \mathbb{R}^k \times \mathbb{R}^k \to \mathbb{R} \cup \{+\infty\},\qquad F_T(y, \theta) = \sup_{(y', \theta') \in T} \langle y-y', \theta' \rangle + \langle y', \theta \rangle.8, with FT:Rk×Rk→R∪{+∞},FT(y,θ)=sup⁡(y′,θ′)∈T⟨y−y′,θ′⟩+⟨y′,θ⟩.F_T : \mathbb{R}^k \times \mathbb{R}^k \to \mathbb{R} \cup \{+\infty\},\qquad F_T(y, \theta) = \sup_{(y', \theta') \in T} \langle y-y', \theta' \rangle + \langle y', \theta \rangle.9 defined via the supremum in the Fitzpatrick function.

3. Exemplary Instances: Sparsemax and Logistic Fitzpatrick Losses

Two concrete realizations illustrate the construction:

Variant Generating Ω:Rk→R∪{+∞}\Omega: \mathbb{R}^k \to \mathbb{R} \cup \{+\infty\}0 Prediction Link Key Formula for Ω:Rk→R∪{+∞}\Omega: \mathbb{R}^k \to \mathbb{R} \cup \{+\infty\}1
Fitzpatrick sparsemax Ω:Rk→R∪{+∞}\Omega: \mathbb{R}^k \to \mathbb{R} \cup \{+\infty\}2 Ω:Rk→R∪{+∞}\Omega: \mathbb{R}^k \to \mathbb{R} \cup \{+\infty\}3 Ω:Rk→R∪{+∞}\Omega: \mathbb{R}^k \to \mathbb{R} \cup \{+\infty\}4, Ω:Rk→R∪{+∞}\Omega: \mathbb{R}^k \to \mathbb{R} \cup \{+\infty\}5
Fitzpatrick logistic Ω:Rk→R∪{+∞}\Omega: \mathbb{R}^k \to \mathbb{R} \cup \{+\infty\}6 Ω:Rk→R∪{+∞}\Omega: \mathbb{R}^k \to \mathbb{R} \cup \{+\infty\}7 Ω:Rk→R∪{+∞}\Omega: \mathbb{R}^k \to \mathbb{R} \cup \{+\infty\}8; Ω:Rk→R∪{+∞}\Omega: \mathbb{R}^k \to \mathbb{R} \cup \{+\infty\}9 involves Lambert-T=∂ΩT = \partial \Omega0 root finding
  • Fitzpatrick sparsemax loss: Utilizes the squared norm plus simplex indicator; the subdifferential yields the Euclidean projector onto the simplex. Prediction remains with the sparsemax mapping.
  • Fitzpatrick logistic loss: Leverages the negative Shannon entropy over the simplex; the subdifferential is the softmax. The optimal T=∂ΩT = \partial \Omega1 for the loss is expressed via the Lambert-T=∂ΩT = \partial \Omega2 function and a normalization constant to enforce summation to one.

Both losses maintain the same link function as their Fenchel–Young counterparts, ensuring prediction equivalence while tightening the surrogate.

4. Relationship to Fenchel–Young Losses and Bregman Divergence

Fitzpatrick losses can be interpreted as Fenchel–Young losses generated by a target-dependent generator: T=∂ΩT = \partial \Omega3 where T=∂ΩT = \partial \Omega4 generalizes Bregman divergence.

A principal result states

T=∂ΩT = \partial \Omega5

This suggests that every Fitzpatrick loss is itself a Fenchel–Young loss for a particular, target-sensitive choice of generator. The construction thereby inherits convexity and, under strong convexity of T=∂ΩT = \partial \Omega6, differentiability with Lipschitz gradient.

5. Theoretical Intuition from Maximal Monotone Operators

The Fitzpatrick function represents the tightest possible convex surrogate above the bilinear pairing T=∂ΩT = \partial \Omega7 with the property that its zero level set recovers precisely the graph of the operator T=∂ΩT = \partial \Omega8. In contrast, the usual Fenchel–Young bound T=∂ΩT = \partial \Omega9 is generally looser and fully separable in the two arguments.

Geometrically, while the Fenchel–Young inequality bounds F∂Ω(y,θ)=sup⁡(y′,θ′)∈∂Ω⟨y−y′,θ′⟩+⟨y′,θ⟩.F_{\partial\Omega}(y, \theta) = \sup_{(y', \theta') \in \partial\Omega} \langle y-y', \theta' \rangle + \langle y', \theta \rangle.0 using its tangent at F∂Ω(y,θ)=sup⁡(y′,θ′)∈∂Ω⟨y−y′,θ′⟩+⟨y′,θ⟩.F_{\partial\Omega}(y, \theta) = \sup_{(y', \theta') \in \partial\Omega} \langle y-y', \theta' \rangle + \langle y', \theta \rangle.1, the Fitzpatrick approach adaptively tilts this tangent via the term F∂Ω(y,θ)=sup⁡(y′,θ′)∈∂Ω⟨y−y′,θ′⟩+⟨y′,θ⟩.F_{\partial\Omega}(y, \theta) = \sup_{(y', \theta') \in \partial\Omega} \langle y-y', \theta' \rangle + \langle y', \theta \rangle.2, producing a lower (and thus tighter) vertical intercept and reducing the surrogate’s slack.

A plausible implication is that, whenever the strict coupling between prediction and target is critical, Fitzpatrick losses provide sharper surrogates for learning and optimization.

6. Empirical Behavior: Label-Proportion Estimation

Empirical studies on multilabel benchmark datasets reveal that linear models trained with Fitzpatrick losses for label-proportion estimation (minimizing F∂Ω(y,θ)=sup⁡(y′,θ′)∈∂Ω⟨y−y′,θ′⟩+⟨y′,θ⟩.F_{\partial\Omega}(y, \theta) = \sup_{(y', \theta') \in \partial\Omega} \langle y-y', \theta' \rangle + \langle y', \theta \rangle.3 via L-BFGS) yield:

  • Fitzpatrick sparsemax often outperforms or matches the traditional sparsemax loss in mean-squared error recovery.
  • Fitzpatrick logistic is on par with the standard logistic loss in accuracy, with only minor computational overhead due to the use of the Lambert-F∂Ω(y,θ)=sup⁡(y′,θ′)∈∂Ω⟨y−y′,θ′⟩+⟨y′,θ⟩.F_{\partial\Omega}(y, \theta) = \sup_{(y', \theta') \in \partial\Omega} \langle y-y', \theta' \rangle + \langle y', \theta \rangle.4 function.

This suggests that Fitzpatrick losses can be substituted for their Fenchel–Young counterparts with minimal modification to code and, potentially, performance improvements (Rakotomandimby et al., 2024).

  • The original development of Fitzpatrick functions: S. Fitzpatrick, "Representing monotone operators by convex functions," 1988.
  • Broad analysis and inequalities for Fitzpatrick functions: H. Bauschke et al., "Fitzpatrick functions: Inequalities, examples …," Journal of Convex Analysis, 2005.
  • Fenchel–Young losses as the classical family: M. Blondel, A. Martins & V. Niculae, "Learning with Fenchel–Young losses," JMLR 2020.
  • Primary source for Fitzpatrick losses and experimental evidence: S. Rakotomandimby et al., "Learning with Fitzpatrick losses," ICML 2023 and (Rakotomandimby et al., 2024).

A plausible direction is that Fitzpatrick losses, as convex surrogates with tighter theoretical control, offer a versatile tool in designing prediction-calibrated losses for structured outputs and probabilistic modeling.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (1)

Topic to Video (Beta)

No one has generated a video about this topic yet.

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Follow Topic

Get notified by email when new papers are published related to Fitzpatrick Losses.