---
title: 'Hidden-Convex Losses: Latent Convex Structures'
url: https://www.emergentmind.com/topics/hidden-convex-losses
type: topic
---

# Hidden-Convex Losses: Latent Convex Structures

Searching arXiv for the cited papers and closely related work on hidden-convex losses.
{"query":"all:2606.19891 OR ti:\"Adversarial Bandit Optimization with Globally Bounded Perturbations to Convex Losses\" OR ti:\"Online Learning on Hidden-Convex Losses via Algorithmic Equivalence\" OR ti:\"Stochastic Optimization under Hidden Convexity\" OR ti:\"The Geometry and Calculus of Losses\"","max_results":10}
{"query":"ti:\"Adversarial Bandit Optimization with Globally Bounded Perturbations to Convex Losses\"","max_results":5}
{"query":"ti:\"Online Learning on Hidden-Convex Losses via Algorithmic Equivalence: Optimal Regret, Geometric Barrier, and Bandit Feedback\"","max_results":5}
Hidden-convex losses are loss functions, loss sequences, or loss-generating mechanisms whose observed form is nonconvex, nonsmooth, discrete, or otherwise analytically inconvenient, but whose behavior is controlled by an underlying convex object. The literature uses the term in several related senses rather than a single universal definition. In one line of work, the observed loss is a globally budgeted perturbation of a convex smooth backbone, so the learner sees a nonconvex loss \(f_t\) while analysis proceeds through latent convex losses \(h_t\) [2606.19891]. In another, a nonconvex objective is exactly representable as \(F(x)=H(c(x))\) or \(\ell_t(x)=h_t(q(x))\), where \(c\) or \(q\) is an invertible nonlinear map and \(H\) or \(h_t\) is convex [2401.00108; 2605.26373]. A third viewpoint treats the loss as a subgradient field of a convex-geometric object such as a superprediction set [2209.00238]. A fourth treats convexity as a surrogate or decomposed structure imposed on a fundamentally discrete or non-modular target loss [1402.1792; 1604.03373]. Taken together, these works suggest that hidden-convexity is best understood as a structural relation between an apparent loss and a latent convex representation, rather than as a single formula.

## 1. Formal patterns of hidden convexity

A first formal pattern writes the observed loss as a perturbation of a convex smooth loss:
\[
f_t(x)=h_t(x)+o_t(x),
\]
with \(h_t\) convex and smooth, while the perturbation \(o_t\) may itself be nonconvex and even adaptively chosen after the learner acts. In the bandit model of "Adversarial Bandit Optimization with Globally Bounded Perturbations to Convex Losses", hidden convexity is captured by the global budget condition
\[
\sup_{x_1,\ldots,x_T\in K}\sum_{t=1}^T |f_t(x_t)-h_t(x_t)|\le C,
\]
equivalently
\[
\sup_{x_1,\ldots,x_T\in K}\sum_{t=1}^T |o_t(x_t)|\le C,
\]
so the losses may look nonconvex round by round while remaining uniformly close, in cumulative magnitude, to a convex smooth sequence [2606.19891].

A second pattern is exact nonlinear reparameterization. In "Stochastic Optimization under Hidden Convexity", the original constrained stochastic problem
\[
\min_{x\in X}~F(x)
\]
is hidden convex if there exists an invertible map \(c:X\to U\) such that \(U=c(X)\) is convex and
\[
F(x)=H(c(x)),
\]
with \(H\) convex, or strongly convex, on \(U\) [2401.00108]. In the online analogue, "Online Learning on Hidden-Convex Losses via Algorithmic Equivalence" studies losses of the form
\[
\ell_t(x)=h_t(q(x)),
\]
where \(q:\mathcal X\to\mathcal Z\) is a smooth bijection and each \(h_t\) is convex [2605.26373].

A third pattern is exact value-function convexification. "Conditional Infimum and Hidden Convexity in Optimization" formalizes hidden convexity as equality of optimal values between a nonconvex problem and a convex one:
\[
\inf_{x:\theta(x)\in U} L(x)=\inf_{u\in U}\phi(u), \qquad \phi(u)=\inf\{L(x):\theta(x)=u\},
\]
with \(\phi\) convex on a convex transformed feasible set [2104.05266]. This definition is about the optimization problem, not merely pointwise convexity of a reparameterized formula.

A fourth pattern is convex-geometric representation. In "The Geometry and Calculus of Losses", a proper multiclass loss is induced by a convex superprediction set, and its Bayes risk is the concave support function of that set [2209.00238]. Here convexity is hidden in the geometry that generates the loss, even when the partial losses are not themselves obviously convex in the prediction variable.

## 2. Latent convex backbones under adversarial perturbations

The most direct loss-theoretic formalization of hidden convexity appears in adversarial bandit optimization with globally budgeted perturbations. The learner acts on the Euclidean ball
\[
K=\{x\in\mathbb{R}^d:\|x\|_2\le D\},
\]
observes only the scalar loss at the played point, and faces losses
\[
f_t(x)=h_t(x)+o_t(x),
\]
where each hidden component \(h_t\) is convex, \(\beta\)-smooth, and \(G\)-Lipschitz on \(K\), the observed losses satisfy \(|f_t(x)|\le L\), and the perturbation sequence obeys the pathwise-uniform budget
\[
\sup_{x_1,\ldots,x_T\in K}\sum_{t=1}^T |o_t(x_t)|\le C.
\]
The perturbation may be arbitrary, nonconvex, non-smooth, and chosen after seeing the learner’s action; the control is entirely through the global cumulative budget [2606.19891].

The regret notion is standard static regret, but crucially it is measured on the observed perturbed losses rather than the hidden convex backbone:
\[
\mathrm{Reg}_T:=\sum_{t=1}^T f_t(x_t)-\min_{x\in K}\sum_{t=1}^T f_t(x).
\]
This point is central. Convexity is used only through \(\{h_t\}\), yet the benchmark remains the best fixed action for the actual nonconvex losses \(\{f_t\}\).

Algorithmically, the paper modifies SCRIBLe into Shrunk and Scaled SCRIBLe (SS-SCRIBLe). The internal iterate lies in the shrunk domain
\[
K_\delta=\{x\in\mathbb{R}^d: x/(1-\delta)\in K\},
\]
the barrier is
\[
R(x)=-\log(D^2-\|x\|^2),
\]
and the played point is a locally randomized perturbation
\[
y_t=x_t+A_tu_t,\qquad A_t=b[\nabla^2R(x_t)]^{-1/2}.
\]
The one-point estimator
\[
g_t=d\,f_t(y_t)\,A_t^{-1}u_t
\]
is unbiased for the gradient of a locally smoothed loss, not for \(\nabla h_t(x_t)\), and that mismatch drives the nonstandard analysis.

The main theorem gives an expected regret bound
\[
\mathbb{E}\!\left[\sum_{t=1}^T f_t(y_t)-\sum_{t=1}^T f_t(w)\right]
\le \frac{\beta b^2 D^2 T}{2} +2\eta b^{-2}Td^2L^2 +\frac{2\log(1/\gamma)}{\eta} +db^{-1}C(4+4\sqrt2)\Bigl(\frac{1}{\delta}-1\Bigr) +TGD\,\gamma +2C,
\]
where \(w\in\arg\min_{x\in K}\sum_{t=1}^T f_t(x)\) and \(\gamma=\max\{\delta,\eta d C\}\). With the parameter choice
\[
\delta=\sqrt{CdD}\,T^{-1/3}, \qquad \eta=T^{-2/3}d^{-1}D^{-1}L^{-1}\sqrt{\log T}, \qquad b=T^{-1/3}D^{-1},
\]
this yields
\[
\mathbb{E}\!\left[\sum_{t=1}^T f_t(y_t)-\sum_{t=1}^T f_t(w)\right]
\le \beta T^{2/3} +4T^{2/3}dDL\sqrt{\log T} +(4+4\sqrt2)T^{2/3}\sqrt{C d D} +\max\{1,\sqrt{TCdD}\}\,GD +2C.
\]
When \(C=0\), the perturbation-dependent terms vanish and the result reduces to the standard one-point smooth bandit convex optimization rate of order \(T^{2/3}\). The paper therefore shows that sublinear regret survives even when the observed losses are nonconvex and non-smooth, provided their cumulative deviation from some convex smooth backbone is globally controlled [2606.19891].

This formulation is intentionally restrictive. The perturbation budget is global rather than trajectory-specific; the guarantee is in expectation rather than high probability; the parameter choice depends on \(T\) and \(C\); and no lower bound is given for the approximately convex smooth setting. These limitations delineate one precise meaning of hidden-convex loss: latent convexity corrupted by bounded adversarial distortion, not arbitrary persistent nonconvexity.

## 3. Reparameterization, algorithmic equivalence, and exact convexification

A second major interpretation of hidden-convex loss treats nonconvexity as a coordinate artifact. In stochastic optimization, the basic model is
\[
\min_{x\in X}~F(x)=H(c(x)),
\]
where \(c:X\to U\) is invertible, \(U=c(X)\) is convex, and \(H\) is convex on \(U\). The map \(c\) is required to satisfy
\[
\|c(x)-c(y)\|\ge \mu_c \|x-y\|, \qquad \forall x,y\in X.
\]
If \(H\) is \(\mu_H\)-strongly convex, the problem is \((\mu_c,\mu_H)\)-hidden strongly convex. Under this structure, every stationary point of a hidden-convex function is a global minimum, and simple projected stochastic subgradient or gradient methods achieve global convergence even when the convex reformulation is unavailable explicitly [2401.00108].

In the nonsmooth case, the paper analyzes projected stochastic subgradient updates
\[
x^{t+1} = \Pi_{X}\big(x^t- \eta\, g(x^{t}, \xi^{t})\big),
\]
and in the smooth case projected SGD
\[
x^{t+1} = \Pi_{X}\big(x^t- \eta\, \nabla f(x^{t}, \xi^{t})\big).
\]
For hidden strongly convex problems, the sample complexity improves to an \(O(1/\varepsilon)\) dependence; in the smooth setting, a momentum variant
\[
x^{t+1} = \Pi_{\mathcal{X}}(x^t-\eta g^t), \qquad g^{t+1} = (1-\beta) g^{t} + \beta \nabla f(x^{t+1}, \xi^{t+1})
\]
yields last-iterate guarantees in function value [2401.00108].

The online analogue sharpens this viewpoint into algorithmic equivalence. With
\[
\ell_t(x)=h_t(q(x)),
\]
where \(q:\mathcal X\to\mathcal Z\) is a smooth bijection and each \(h_t\) is convex, the crucial compatibility condition is
\[
\bigl[\nabla^2 R(z)\bigr]^{-1}=J_q(x)J_q(x)^\top,\qquad z=q(x),
\]
for a twice continuously differentiable, strictly convex regularizer \(R\). Under this Hessian compatibility condition, OGD in the original nonconvex variable approximately simulates OMD in hidden convex coordinates, and the one-step discrepancy is quadratic in the stepsize:
\[
\|y_{t+1}-q(x_{t+1})\|_2 \le 6G^5G_F^3\eta^2
\]
in the exact-gradient setting. This refinement is sufficient to recover the optimal full-information regret
\[
R_T\le \sqrt{7D_1G^6G_F^3\,T},
\]
hence \(R_T=O(\sqrt T)\), matching classical adversarial online convex optimization. In the one-point bandit setting, the same framework yields \(\mathbb E[R_T]=O(T^{3/4})\) [2605.26373].

The same paper also clarifies the geometry. The earlier diagonal-Jacobian condition is replaced by the necessary-and-sufficient integrability condition that the induced metric field
\[
M(z):=\left[J_q(q^{-1}(z))J_q(q^{-1}(z))^\top\right]^{-1}
\]
must satisfy
\[
\partial_{z_k} M_{ij}(z)=\partial_{z_j} M_{ik}(z),
\]
so that \(M=\nabla^2R\). Without this compatibility, hidden convexity of the losses is not enough to save Euclidean OGD: the paper constructs a smooth reparameterization and an adversarial hidden-convex loss sequence for which OGD suffers
\[
R_T=\Omega(T).
\]
This establishes that the relevant hidden structure is not merely existence of a convex latent loss, but compatibility between that latent convexity and the algorithmic geometry [2605.26373].

The most abstract formulation of this idea appears in conditional-infimum theory. A nonconvex minimization problem displays hidden convexity if it admits an exact convex representation
\[
\inf_{\uncertain \in \Uncertain} \fonctionuncertain(\uncertain)
=
\inf_{\primal\in\Convex}\fonctionprimal(\primal),
\]
and a sufficient condition is convexity of the conditional infimum
\[
#1{\fonctionuncertain}{\theta}(v)=\inf_{u\in \theta^{-1}(\{v\})} \fonctionuncertain(u)
\]
on a convex transformed feasible set [2104.05266]. The broader survey literature systematizes this as three routes to hidden convex optimization: nonlinear transformation, Lagrangian duality and its variations, and tight primal convex relaxation [1902.10921]. This suggests that exact hidden-convex losses are a special case of a more general phenomenon in which nonconvex optimization inherits convex solvability through transformation, duality, or lifting.

## 4. Convex geometry as the generator of a loss

A more foundational approach does not begin from a reparameterization or perturbation at all. Instead, it treats the loss itself as induced by a convex set. For a finite-outcome proper loss \(\ell:\probm\to\mathbb R_+^n\), define the superprediction set
\[
\super(\ell)\coloneqq \ell(\probm)+\mathbb R_+^n.
\]
Then the conditional Bayes risk satisfies
\[
\minL = \cvsprt_{\cl\conv(\super\ell)}\big|_{\probm},
\]
and
\[
\Big[\forall p\in\probm,\ \ell(p)\in \partial \cvsprt_{\cl\conv(\super\ell)}(p)\Big] \iff \ell \text{ is proper}.
\]
Thus a proper loss is precisely a \(0\)-homogeneous selection of the subgradient of a concave support function associated with a closed convex set with recession cone \(\mathbb R_+^n\). Strict propriety is geometric:
\[
\ell \text{ strictly proper } \iff \cl\conv(\super\ell)\text{ strictly convex}.
\]
This perspective makes convexity the hidden generator of the loss rather than a visible property of its coordinate formula [2209.00238].

The same framework leads to a calculus of losses. Given proper losses with Bayes risks \(\minL_1,\dots,\minL_m\), \(M\)-sums and dual \(M\)-sums of the associated convex sets induce new Bayes risks and therefore new proper losses. Classical operations emerge as special cases: Minkowski sums, convex hulls of unions, and intersections correspond to addition, pointwise maxima, and infimal-convolution-type constructions at the level of support functions. The paper also develops antipolar losses. If \(S=\super(\ell)\), the antipolar loss \(\ell^\apolar\) is induced by \(S^\apolar\) and satisfies a pseudoinverse relation
\[
\ell = \ell\circ \ell^\apolar\circ \ell,\qquad
\ell^\apolar = \ell^\apolar\circ \ell\circ \ell^\apolar.
\]
The canonical composite loss
\[
\tilde\ell=\ell\circ \ell^\apolar
\]
is component-wise convex. This is an explicit instance in which convexity is hidden in the original loss geometry and becomes visible only after canonical reparameterization [2209.00238].

A related functional perspective appears in risk-measure theory. Expected losses
\[
E_\ell(X)=E[\ell(X)]
\]
are modular, and certainty equivalents
\[
\mathrm{CE}_\ell(X)=\ell^{-1}(E[\ell(X)])
\]
are submodular if and only if the underlying scalar loss \(\ell\) is convex. For optimized certainty equivalents,
\[
\mathrm{OCE}_\ell(X)=\inf_{m\in\mathbb R}\{m+E[\ell(X-m)]\},
\]
submodularity always holds when \(\ell\) is increasing and convex. In the coherent law-invariant setting, submodularity characterizes coherent distortion risk measures, including Expected Shortfall [2603.01232]. This does not use the phrase hidden-convex loss in the online-learning sense, but it shows the same structural principle: apparently complicated loss functionals inherit lattice convexity properties from a latent convex scalar generator.

## 5. Surrogates, decomposition, and interpolation between lower and upper objectives

In statistical learning, hidden-convex losses often arise because the objective of real interest is discrete or otherwise nonconvex, and convexity is inserted by surrogate design. The canonical case is binary classification. The true objective is the binary risk
\[
R(f)=\Pr\{y f(\mathbf{x})\le 0\},
\]
with excess risk \(\mathcal E(f)=R(f)-R^*\), but learning is carried out by minimizing a convex surrogate risk
\[
R_\phi(f)=\mathbb E[\phi(yf(\mathbf{x}))].
\]
The classification-calibration framework gives
\[
\psi(\mathcal E(f))\le \mathcal E_\phi(f).
\]
For the smoothed hinge
\[
\phi(z;\gamma)=\frac{1}{\gamma}\log\big(1+\exp(\gamma(1-z))\big),
\]
the paper proves a lower bound
\[
\psi(\eta;\gamma)\ge |\eta|-\frac{1}{\gamma}\log\frac{1}{|\eta|},
\]
and hence
\[
\mathcal{E}(f)\le \mathcal{E}_\phi(f) +\frac{\mathcal{E}_\phi(f)}{1+\gamma \mathcal{E}_\phi(f)} \log\frac{1}{\mathcal{E}_\phi(f)}.
\]
The message is that smooth convex surrogates help optimization and can help generalization, but increased smoothness can worsen the translation from surrogate excess risk to binary excess risk. Hidden convexity here is therefore ambivalent: it is computationally useful but not automatically faithful to the nonconvex target objective [1402.1792].

For structured prediction with non-modular losses, the same idea appears as decomposition. A discrete loss \(\Delta\) is represented by a set function \(l\), canonically decomposed into a submodular part \(f^*\) and an increasing supermodular part \(g^*\) via
\[
\mathbf{D} l = \arg\min_{g \in \mathcal{G}_+} \sum_{A \subseteq V} g(A), \quad \text{s.t. } l-g \in \mathcal{S}.
\]
The resulting convex surrogate is
\[
\mathbf B_{\mathbf D}\Delta = \mathbf L \Delta_{\mathcal S} + \mathbf S \Delta_{\mathcal G},
\]
combining the Lovász hinge for the submodular component and slack-rescaling for the supermodular component. The paper proves that this surrogate is convex, piecewise linear, and, when the canonical submodular component is non-negative, an extension of the original loss on the vertices of the unit cube [1604.03373]. This is another precise meaning of hidden-convex loss: a non-modular discrete loss whose latent decomposition exposes pieces that admit exact convex surrogate machinery.

In verified robustness, the latent convex element is interpolation between lower and upper objectives. For robust loss
\[
\mathcal{L}^* (f(\theta, x), y) := \max_{x' \in \mathcal{C}(x, \epsilon)} \mathcal{L} (f(\theta, x'), y),
\]
adversarial training gives a lower bound \(\mathcal L_{\text{adv}}\), while verification gives an upper bound \(\mathcal L_{\text{ver}}\). The paper defines an expressive family \(\mathcal L_\alpha\) that is continuous, monotone in \(\alpha\in[0,1]\), equals the adversarial loss at \(\alpha=0\), and equals the verified loss at \(\alpha=1\). One concrete family is the literal convex combination
\[
\mathcal{L}_{\alpha, \text{MTL} }(\theta, x, y)
=
( 1 - \alpha) \mathcal{L} (f(\theta, x_{\text{adv}}), y)
+ \alpha\ \mathcal{L}_{\text{ver} }(f(\theta, x), y),
\]
while CC-IBP mixes logit-difference vectors before applying the base loss, and Exp-IBP performs geometric interpolation in log-loss space. The paper argues that expressivity, not simply tighter approximation to the worst-case loss, is key to robustness–accuracy trade-offs [2305.13991].

A final caution comes from boosting and proper composite losses. "What killed the Convex Booster?" argues that the classical Long–Servedio pathology should not be attributed to convexity alone. In class-probability-estimation form, a strictly proper differentiable loss becomes a convex margin surrogate after canonical reparameterization, but the negative result persists across broad proper losses and disappears for richer model classes. The paper’s conclusion is that the decisive variable is often the model class and, more generally, parameterization, not convexity in isolation [2205.09628]. This further narrows the meaning of hidden-convex loss: a convex surrogate or convex representation may reveal the wrong latent structure if the hypothesis class is misaligned.

## 6. Consequences, misconceptions, and open questions

Across these literatures, hidden-convex losses support three distinct kinds of guarantees. The first is optimization or regret equivalence: OGD on nonconvex losses can match mirror descent on hidden convex losses and recover
\[
O(\sqrt T)
\]
regret in adversarial online learning, while one-point bandit OGD recovers the classical
\[
O(T^{3/4})
\]
rate [2605.26373]. The second is robustness to controlled nonconvex corruption: SS-SCRIBLe attains sublinear expected regret on the actual perturbed losses when deviation from a convex smooth backbone is globally budgeted [2606.19891]. The third is exact or approximate convexification in stochastic and statistical learning: projected stochastic methods globally solve nonconvex hidden-convex problems without explicit access to the convex reformulation [2401.00108], and convex-set or decomposition formalisms turn non-modular or proper losses into analyzable convex objects [2209.00238; 1604.03373].

Several misconceptions are explicitly rejected. Hidden convexity does not mean that the observed loss is pointwise convex after a trivial change of variables; in some papers it means exact value equivalence of optimization problems [2104.05266], in others proximity to a latent convex backbone under a pathwise-uniform perturbation budget [2606.19891]. Convexity of a surrogate also does not guarantee fidelity to the target metric: smoothing may worsen the translation from surrogate excess risk to binary excess risk [1402.1792]. Nor is convexity necessarily the main explanation for failure or success: in boosting under label noise, parameterization and model class can dominate the effect of convexity itself [2205.09628].

The open problems are correspondingly varied. In hidden-convex bandit optimization with perturbation budgets, no lower bound is established for the approximately convex smooth setting, and the dependence on \(C\) is not known to be optimal [2606.19891]. In online reparameterized hidden convexity, the Hessian compatibility condition is essential for OGD, but it remains open whether some other online algorithm can achieve sublinear regret without it, and whether stronger bandit guarantees are possible [2605.26373]. In expressive robust training, the paper leaves open a theory explaining which interpolation coefficient \(\alpha\) is optimal for a given verifier and dataset [2305.13991]. More broadly, the survey literature indicates that many polynomially solvable nonconvex problems still lack a known hidden-convex representation, so the boundary between exact hidden convexity and merely useful convex approximation remains incomplete [1902.10921].

The resulting picture is technically narrow but conceptually broad. A hidden-convex loss is not simply a nonconvex loss that “behaves nicely.” It is a loss whose apparent nonconvexity is mediated by a latent convex structure: a globally budgeted convex backbone, an invertible convexifying reparameterization, a convex superprediction set, a submodular–supermodular decomposition, or a tunable interpolation between lower and upper objectives. The unifying claim across these settings is that convexity need not be visible at the surface level of the observed loss for convex-style guarantees to remain possible.

Source: https://www.emergentmind.com/topics/hidden-convex-losses