Fitzpatrick losses are convex loss functions derived from maximal monotone operator theory that yield tighter surrogates than Fenchel–Young losses while retaining the same prediction link functions.
They are constructed using the Fitzpatrick function, with key instances like sparsemax and logistic variants applied to tasks including label-proportion estimation.
Empirical studies show that models using Fitzpatrick losses achieve comparable or improved accuracy with minimal computational overhead in structured prediction scenarios.
Fitzpatrick losses are a family of convex loss functions derived from the Fitzpatrick function, a foundational concept in maximal monotone operator theory. They serve as a refined generalization of Fenchel–Young losses, yielding tighter convex upper bounds while retaining the same prediction (link) functions. Notably, Fitzpatrick losses provide a principled method for constructing convex surrogates in structured prediction and probabilistic modeling, with direct applications to tasks such as label-proportion estimation and beyond (Rakotomandimby et al., 2024).
1. Mathematical Foundation: The Fitzpatrick Function
Let T⊆Rk×Rk be a monotone operator, notably maximal monotone if it cannot be strictly enlarged without losing monotonicity. The Fitzpatrick function associated to T is the convex, jointly lower semicontinuous map
Given a proper, convex, lower semicontinuous function Ω:Rk→R∪{+∞}, and setting T=∂Ω (the subdifferential), the Fitzpatrick function becomes
F∂Ω(y,θ)=(y′,θ′)∈∂Ωsup⟨y−y′,θ′⟩+⟨y′,θ⟩.
This construction leads to the refined Fenchel–Young inequalities
⟨y,θ⟩≤F∂Ω(y,θ)≤Ω(y)+Ω∗(θ),∀y,θ∈Rk,
where Ω∗ denotes the Fenchel conjugate.
2. Construction and Properties of Fitzpatrick Losses
Given a base function Ω as above, define the Fitzpatrick loss
LF(y,θ)=F∂Ω(y,θ)−⟨y,θ⟩.
Equivalently,
T0
Here, T1 serves as the tightest jointly convex upper envelope of the bilinear form T2 that exactly captures the monotone graph T3. The classical Fenchel–Young loss
T4
always satisfies T5. Fitzpatrick losses thus represent a strictly tighter class of convex surrogates under the same link function associated with T6.
Key properties include:
Separate convexity: T7 is convex in T8 for fixed T9 and convex in FT:Rk×Rk→R∪{+∞},FT(y,θ)=(y′,θ′)∈Tsup⟨y−y′,θ′⟩+⟨y′,θ⟩.0 for fixed FT:Rk×Rk→R∪{+∞},FT(y,θ)=(y′,θ′)∈Tsup⟨y−y′,θ′⟩+⟨y′,θ⟩.1.
Zero loss characterization: FT:Rk×Rk→R∪{+∞},FT(y,θ)=(y′,θ′)∈Tsup⟨y−y′,θ′⟩+⟨y′,θ⟩.2 if and only if FT:Rk×Rk→R∪{+∞},FT(y,θ)=(y′,θ′)∈Tsup⟨y−y′,θ′⟩+⟨y′,θ⟩.3, equivalently FT:Rk×Rk→R∪{+∞},FT(y,θ)=(y′,θ′)∈Tsup⟨y−y′,θ′⟩+⟨y′,θ⟩.4. For strictly convex FT:Rk×Rk→R∪{+∞},FT(y,θ)=(y′,θ′)∈Tsup⟨y−y′,θ′⟩+⟨y′,θ⟩.5, FT:Rk×Rk→R∪{+∞},FT(y,θ)=(y′,θ′)∈Tsup⟨y−y′,θ′⟩+⟨y′,θ⟩.6.
Gradient in FT:Rk×Rk→R∪{+∞},FT(y,θ)=(y′,θ′)∈Tsup⟨y−y′,θ′⟩+⟨y′,θ⟩.7: FT:Rk×Rk→R∪{+∞},FT(y,θ)=(y′,θ′)∈Tsup⟨y−y′,θ′⟩+⟨y′,θ⟩.8, with FT:Rk×Rk→R∪{+∞},FT(y,θ)=(y′,θ′)∈Tsup⟨y−y′,θ′⟩+⟨y′,θ⟩.9 defined via the supremum in the Fitzpatrick function.
3. Exemplary Instances: Sparsemax and Logistic Fitzpatrick Losses
Two concrete realizations illustrate the construction:
Fitzpatrick sparsemax loss: Utilizes the squared norm plus simplex indicator; the subdifferential yields the Euclidean projector onto the simplex. Prediction remains with the sparsemax mapping.
Fitzpatrick logistic loss: Leverages the negative Shannon entropy over the simplex; the subdifferential is the softmax. The optimal T=∂Ω1 for the loss is expressed via the Lambert-T=∂Ω2 function and a normalization constant to enforce summation to one.
Both losses maintain the same link function as their Fenchel–Young counterparts, ensuring prediction equivalence while tightening the surrogate.
4. Relationship to Fenchel–Young Losses and Bregman Divergence
Fitzpatrick losses can be interpreted as Fenchel–Young losses generated by a target-dependent generator: T=∂Ω3
where T=∂Ω4 generalizes Bregman divergence.
A principal result states
T=∂Ω5
This suggests that every Fitzpatrick loss is itself a Fenchel–Young loss for a particular, target-sensitive choice of generator. The construction thereby inherits convexity and, under strong convexity of T=∂Ω6, differentiability with Lipschitz gradient.
5. Theoretical Intuition from Maximal Monotone Operators
The Fitzpatrick function represents the tightest possible convex surrogate above the bilinear pairing T=∂Ω7 with the property that its zero level set recovers precisely the graph of the operator T=∂Ω8. In contrast, the usual Fenchel–Young bound T=∂Ω9 is generally looser and fully separable in the two arguments.
Geometrically, while the Fenchel–Young inequality bounds F∂Ω(y,θ)=(y′,θ′)∈∂Ωsup⟨y−y′,θ′⟩+⟨y′,θ⟩.0 using its tangent at F∂Ω(y,θ)=(y′,θ′)∈∂Ωsup⟨y−y′,θ′⟩+⟨y′,θ⟩.1, the Fitzpatrick approach adaptively tilts this tangent via the term F∂Ω(y,θ)=(y′,θ′)∈∂Ωsup⟨y−y′,θ′⟩+⟨y′,θ⟩.2, producing a lower (and thus tighter) vertical intercept and reducing the surrogate’s slack.
A plausible implication is that, whenever the strict coupling between prediction and target is critical, Fitzpatrick losses provide sharper surrogates for learning and optimization.
Empirical studies on multilabel benchmark datasets reveal that linear models trained with Fitzpatrick losses for label-proportion estimation (minimizing F∂Ω(y,θ)=(y′,θ′)∈∂Ωsup⟨y−y′,θ′⟩+⟨y′,θ⟩.3 via L-BFGS) yield:
Fitzpatrick sparsemax often outperforms or matches the traditional sparsemax loss in mean-squared error recovery.
Fitzpatrick logistic is on par with the standard logistic loss in accuracy, with only minor computational overhead due to the use of the Lambert-F∂Ω(y,θ)=(y′,θ′)∈∂Ωsup⟨y−y′,θ′⟩+⟨y′,θ⟩.4 function.
This suggests that Fitzpatrick losses can be substituted for their Fenchel–Young counterparts with minimal modification to code and, potentially, performance improvements (Rakotomandimby et al., 2024).
7. Further Reading and Related Work
The original development of Fitzpatrick functions: S. Fitzpatrick, "Representing monotone operators by convex functions," 1988.
Broad analysis and inequalities for Fitzpatrick functions: H. Bauschke et al., "Fitzpatrick functions: Inequalities, examples …," Journal of Convex Analysis, 2005.
Fenchel–Young losses as the classical family: M. Blondel, A. Martins & V. Niculae, "Learning with Fenchel–Young losses," JMLR 2020.
Primary source for Fitzpatrick losses and experimental evidence: S. Rakotomandimby et al., "Learning with Fitzpatrick losses," ICML 2023 and (Rakotomandimby et al., 2024).
A plausible direction is that Fitzpatrick losses, as convex surrogates with tighter theoretical control, offer a versatile tool in designing prediction-calibrated losses for structured outputs and probabilistic modeling.