Local Surrogate Losses for Optimization

• Local surrogate losses are techniques that create differentiable approximations of non-differentiable objectives, enabling gradient-based optimization in complex settings.
• They employ smoothing with Gaussian kernels and locality weighting to ensure accurate local approximations and robust gradient estimates.
• Implementations such as ZeroGrads and SEARNN demonstrate substantial performance gains in graphics, sequence prediction, and segmentation by leveraging adaptive surrogate fitting.

Local surrogate losses are a family of techniques that address the challenge of optimizing objectives that are either non-differentiable, discontinuous, or otherwise incompatible with standard gradient-based methods. These approaches replace or augment the original loss with a locally accurate, differentiable surrogate, enabling the use of gradient descent and related optimization algorithms in otherwise intractable scenarios. The construction and application of local surrogate losses has proven effective across non-differentiable graphics optimization, structured prediction with sequence models, and metric-driven tasks such as semantic segmentation.

1. Formal Framework for Local Surrogate Losses

Formally, consider a target loss $L(p)$ defined over a parameter vector $p \in \Theta \subset \mathbb{R}^n$. In black-box or non-differentiable settings, direct computation of $\nabla_p L(p)$ is unavailable. Local surrogate methodology posits a differentiable surrogate $h(p; \phi)$, parameterized by $\phi$, and locally fitted in the region of current interest—almost always the neighborhood of the current iterate $p_t$. The fit is carried out by minimizing a local regression loss

$$\mathcal{L}_{\mathrm{sur}}(\phi; p_t) = \int \ell(p', p_t) [g(p') - h(p'; \phi)]^2 \, dp'$$

where $g(p')$ is a smoothed version of $L(p')$, obtained via convolution with a smoothing kernel (typically Gaussian), and $\ell(p', p_t)$ is a locality weighting function (locality kernel). This approach explicitly confines the surrogate's approximation capacity to those regions most relevant to the optimization trajectory (Fischer et al., 2023).

2. Smoothing and Locality: Making Non-Differentiable Losses Amenable to Gradient Descent

Central to local surrogate losses is smoothing: the original loss surface, possibly exhibiting plateaus or discontinuities, is convolved with a Gaussian kernel $p \in \Theta \subset \mathbb{R}^n$0, yielding

$p \in \Theta \subset \mathbb{R}^n$1

This operation ensures Lipschitz continuity of the gradients, if they exist, in the smoothed space. The surrogate $p \in \Theta \subset \mathbb{R}^n$2 is then regressed only within a local neighborhood specified by the locality kernel $p \in \Theta \subset \mathbb{R}^n$3. This hierarchy of smoothing and locality fundamentally differentiates the local surrogate regime from global function approximation. The optimization proceeds by differentiating $p \in \Theta \subset \mathbb{R}^n$4 with respect to $p \in \Theta \subset \mathbb{R}^n$5, and updating $p \in \Theta \subset \mathbb{R}^n$6 along $p \in \Theta \subset \mathbb{R}^n$7 (Fischer et al., 2023).

3. Key Algorithmic Instantiations

3.1 Black-Box Graphics Optimization: ZeroGrads

The ZeroGrads framework provides a principled instance of local surrogate losses in the context of non-differentiable graphics and simulation. The framework includes:

• Online, episode-wise fitting of a small multilayer perceptron as local surrogate $p \in \Theta \subset \mathbb{R}^n$8.
• Importance-weighted Monte Carlo estimation for both inner smoothing and outer locality sampling.
• Adaptive hyperparameter schedules: inner bandwith $p \in \Theta \subset \mathbb{R}^n$9; outer bandwidth $\nabla_p L(p)$0–$\nabla_p L(p)$1 of the parameter domain; batch sizes $\nabla_p L(p)$2; learning rates $\nabla_p L(p)$3 (Adam), $\nabla_p L(p)$4.
• Scalability to $\nabla_p L(p)$5 variables, with total runtime overhead of $\nabla_p L(p)$6–$\nabla_p L(p)$7 a pure gradient step.

Empirically, ZeroGrads enabled surrogate gradients for piecewise-constant, discrete, or otherwise non-differentiable objectives, surpassing derivative-free optimizers such as SPSA, CMA-ES, GAs, and simulated annealing, both in convergence speed and final objective value (Fischer et al., 2023).

3.2 Structured Prediction with Sequential Models: SEARNN

The SEARNN algorithm develops a global–local surrogate loss for sequence models by mapping structured loss minimization into a sequence of locally informed, cost-sensitive losses. The global term approximates expected cost under the model policy, while the local surrogate loss at each time step is

$\nabla_p L(p)$8

where $\nabla_p L(p)$9 is a Boltzmann distribution over per-step costs derived from costly roll-outs. SEARNN efficiently realizes global test-metric information at the local cell (token) level, leveraging exploration policies and subsampling to manage the computational cost of expensive roll-outs (Leblond et al., 2017).

4. Automatic Generation and Optimization of Surrogate Losses

Surrogate losses for non-differentiable or misaligned metrics can be generated and tuned automatically. Auto Seg-Loss introduces a search-based framework that parameterizes logical operations (e.g., AND, OR) at the core of evaluation metrics such as mean IoU or boundary F1 using monotonic, truth-table-preserving Bézier curves. The search (using PPO2) jointly identifies parameter settings producing differentiable surrogates that yield maximal empirical performance as measured by the original non-differentiable metric on a validation set. Crucial constraints are imposed:

• Truth-table consistency ($h(p; \phi)$0, $h(p; \phi)$1) ensures exactness at binary values.
• Monotonicity ($h(p; \phi)$2 for $h(p; \phi)$3) prevents pathological and non-convex loss surfaces.
• Piecewise Bézier parameterization with normalized control points allows efficient, few-parameter optimization (Li et al., 2020).

Empirically, automatically searched surrogates outperform manually designed and naïve soft-logic-based surrogates on segmentation benchmarks, with consistent 1–2 mIoU point improvements across models and datasets.

5. Extensions to Local Metric Surrogates in Structured and Spatial Domains

The Auto Seg-Loss framework, while demonstrated for global metrics, naturally allows extension to local surrogates—for instance, via patch-level IoU or local boundary IoU within sliding windows. These involve:

• Defining locality-specific metrics (e.g., aggregation over $h(p; \phi)$4 pixel patches).
• Parameterizing local logical operations with spatial position or neighborhood-aware weights, using constrained Bézier parameterizations.
• Incorporating local-fidelity objectives in the surrogate loss search—directly promoting spatial coherence, local accuracy, and boundary sharpness.
• Multi-scale surrogates aggregate loss contributions across different spatial extents, realized as weighted sums of locally parameterized $h(p; \phi)$5 functions (Li et al., 2020).

This suggests that local surrogate loss methodologies are not limited to parameter-space locality but can be extended to capture locality in data (spatial, temporal, semantic), enhancing their applicability in complex structured domains.

6. Empirical Performance and Scope

Across categories, local surrogate losses show significant empirical efficacy:

• In graphics optimization, surrogate-gradient-based schemes achieve full parameter recovery and visually indistinguishable solutions in problems where finite-difference and evolutionary strategies fail to scale, particularly for $h(p; \phi)$6 variables (Fischer et al., 2023).
• In sequence prediction, local surrogates reduce error metrics (e.g., Hamming loss from 2.8% to 1.0% in OCR, edit distance from 19.3% to 17.7% in spelling correction, BLEU score gain of +2 on NMT tasks) and enjoy scalability via roll-out and action subsampling (Leblond et al., 2017).
• In segmentation, searched surrogate losses yield substantial performance gains over cross-entropy and Lovász-Softmax, including mIoU improvements up to +2.28 points on PASCAL VOC and substantial increases in boundary-based metrics. Transference of searched surrogates is observed across datasets (Cityscapes, VOC) and architectures (ResNet, PSPNet, HRNet) (Li et al., 2020).

The meta-parameters and locality mechanisms are robust over a wide range of dimensionalities and problem classes, providing a general-purpose toolset for handling non-differentiable, locally relevant optimization targets.

7. Theoretical and Practical Considerations

The local surrogate paradigm provides tractable, differentiable approximations for otherwise inaccessible loss functions, with unbiased gradient estimates in the sense that stochastic gradient steps align with the minimization of the (local) surrogate. Complexity is dominated by surrogate fitting and forward-model evaluations, with efficiency achieved by careful sampling and batching strategies. In the case of SEARNN, local surrogates inherit no-regret analysis from Lols/Searn, and the overall procedure is empirically stable without warm-start requirements.

Practical hyperparameter choices such as locality and smoothing bandwidths, learning rates, and network sizes for surrogates are empirically robust. Adherence to truth-table and monotonicity constraints is critical: ablations removing these yield divergent or highly suboptimal loss landscapes. Efficient search algorithms (e.g., PPO2) are necessary to find high-performing parameterizations of surrogate logical operators in a reasonable time frame. These properties make local surrogate losses a powerful, scalable solution for optimization in domains where the derivative of the true objective is zero, undefined, or too costly to estimate by brute-force finite differencing.