Frank-Wolfe with Moreau Envelope Smoothing for Nonsmooth Nonconvex Problems

Abstract: We present and analyze Frank-Wolfe with Moreau Envelope Smoothing (FRAMES) for solving nonsmooth nonconvex constrained optimization problems, taking advantage of iterative smoothing via the Moreau envelope followed by one Frank-Wolfe step per iteration. The problem template we consider encompasses splitting problems with multiple convex constraint sets as well as problems with nonsmooth weakly convex regularizers like the MCP or SCAD penalties. We prove convergence, with rates, for both of these cases under a variety of mild assumptions, including inconsistent problems. Additionally, we highlight a new relationship between the Frank-Wolfe gap for a problem with nonsmooth objective and the Frank-Wolfe gap for a smoothed surrogate problem, demonstrating suboptimality of prior analyses. Numerical experiments are performed for matrix factorization problems and nonconvex quadratic splitting over multiple convex constraint sets, where the improvements in analysis are empirically observed.

• The paper introduces FRAMES, which integrates Moreau envelope smoothing with the Frank-Wolfe method for effective projection-free optimization in nonsmooth, nonconvex settings.
• It establishes quantifiable finite-time convergence rates and explicit gap-based stationarity guarantees under various convex and weakly convex constraints.
• Empirical validations on matrix factorization and feasibility problems demonstrate optimal polynomial decay and highlight trade-offs in parameter tuning.

Frank-Wolfe with Moreau Envelope Smoothing for Nonsmooth Nonconvex Problems

Introduction and Motivation

Classical Frank-Wolfe (FW) algorithms and extensions are fundamental first-order, projection-free methods for constrained smooth optimization. However, many relevant problems in high-dimensional statistical learning, signal recovery, and matrix factorization require handling composite objectives with nonsmooth nonconvex or weakly convex regularizers and potentially multiple constraints. Existing theory for applying projected or conditional gradient schemes in this regime is quite limited—most analyses either focus on smooth or convex settings, or provide only asymptotic results for specific composite forms.

This work presents a rigorous framework for projection-free minimization of general constrained nonsmooth nonconvex objectives by combining FW with Moreau envelope smoothing, resulting in the FRAMES algorithm. The method is justified by explicit computable gap-based stationarity guarantees for both the smooth surrogate and the original nonsmooth problem, covering multiple cases: convex constraints, weakly convex and nonconvex regularizers, and inconsistent feasibility systems. The paper emphasizes the subtleties in how progress on smoothed FW gaps translates to nonsmooth problem stationarity, and establishes finite-time convergence rates under open-loop parameter schedules.

Algorithmic Framework and Problem Structure

The fundamental objective template is

$$\min_{x \in \mathcal{C} \subset \mathbb{R}^n} f(x) + g(Tx)$$

where $\mathcal{C}$ is convex, compact, and admits an efficient LMO; $f$ is $L$-smooth, possibly nonconvex; $T$ is linear; and $g$ is either a (weakly) convex/nonsmooth penalty or an indicator for additional constraints.

The key methodological advance is applying the FW update not directly to the nonsmooth term, but to an auxiliary sequence of surrogates where $g$ is replaced by its Moreau envelope $g^{\beta_k}$, with $\beta_k\to0$ as $k$ increases. At each iteration:

• The surrogate $\mathcal{C}$0 is optimized by a single FW step over $\mathcal{C}$1.
• Parameters $\mathcal{C}$2 are decreased (typically polynomially) to ensure the surrogate sequence converges to the original, and step sizes $\mathcal{C}$3 are chosen accordingly for optimal convergence rate balance.

This surrogate preserves LMO-accessible updates (requiring only proximal access to $\mathcal{C}$4), extends to weakly convex/nonconvex $\mathcal{C}$5, and decouples the computational requirements from the structure of $\mathcal{C}$6 and $\mathcal{C}$7.

Theoretical Results and Analysis

Convergence Guarantees for the Smoothed Gap

The main analytic innovation is an explicit nonasymptotic convergence rate for the best and average smoothed Frank-Wolfe gap over $\mathcal{C}$8 iterations: $\mathcal{C}$9 for open-loop step and smoothing schedules $f$0, $f$1 with appropriate $f$2. It is highlighted that $f$3 and $f$4 is optimal for ensuring stationarity transfer to the original problem. Selecting $f$5 too small (decaying $f$6 too slowly) leads to faster surrogate convergence but suboptimal guarantees for the original nonsmooth problem.

Figure 1: Smoothed gap decay for various smoothing schedules; power-law decay outperforms logarithmic schedules, consistent with theoretical rates.

The established $f$7 rate matches and generalizes the best-known FW rates for nonconvex smooth objectives, and crucially, these rates are carried through even when $f$8 is a nonsmooth indicator or weakly convex penalty.

Stationarity Guarantees for the Nonsmooth Problem

A central technical contribution is a rigorous transfer principle from smoothed FW gap convergence to (generalized) stationarity of the original nonsmooth problem. Two primary forms are covered:

1. Convex Indicator $f$9 (splitting/feasibility): The smoothed gap controls feasibility violation and stationarity over $L$0 via explicit error bounds. The iterates satisfy

$L$1

for appropriate step and smoothing choices, independently of global convexity.

2. Lipschitz Weakly Convex $L$2: When $L$3 is weakly convex and admits a tractable prox, the transfer relies on a lifting property: for every $L$4, the point $L$5 with $L$6 is controlled. The subgradient gap at $L$7 again decays as $L$8.

In both cases, any cluster point of a vanishing smoothed-gap subsequence is a stationary point for the nonsmooth, constrained composite objective. The extension to inconsistent systems (when $L$9) recovers relaxed best-approximation results: limit points minimize the objective over the closest feasible set.

Numerical Validation and Practical Trade-offs

Empirical results across nonnegative matrix factorization (NMF), trend-filtered matrix factorization, and quadratic splitting illustrate the impact of the smoothing schedule, the sensitivity to the initial $T$0 parameter, and the precise matching of theoretical and observed convergence behaviors.

Figure 2: Convergence of smoothed and signed gaps, and feasibility error for FRAMES applied to nonnegative matrix factorization. Lower $T$1 leads to better feasibility at the cost of slower objective improvement.

Figure 3: FRAMES applied to trend-filtered matrix factorization with weakly convex (SCAD/MCP) penalties. Both reconstruction error and smoothed gap exhibit optimal polynomial decay.

In particular:

• The polynomial smoothing schedule outperforms logarithmic decay, both in theory and practical nonsmooth gap convergence.
• The parameter $T$2 must balance between rapidly achieving feasibility (with smaller values) and maintaining favorable smoothness constants, as smaller $T$3 worsens the conditioning of the surrogate problem.
• For inconsistent quadratic splitting problems, iterates converge to the set of best approximations and secondary optimality is achieved.

Figure 4: Trajectories for FRAMES applied to an inconsistent feasibility problem, converging toward the best-approximate face and achieving secondary minimization.

Implications and Future Directions

The FRAMES methodology establishes a principled, projection-free framework with verifiable gap-based convergence for an essential class of nonsmooth, nonconvex, and potentially inconsistent constrained problems. The approach enables leveraging existing highly efficient Frank-Wolfe infrastructure for a new universe of problems otherwise inaccessible to standard projection-based or subgradient variants.

On the theoretical front, the analysis reveals that step size and smoothing schedules must be tuned in concert for optimal nonsmooth stationarity certification—a direct contradiction to earlier smoothing analyses that prioritize only surrogate progress. This insight suggests the applicability of similar methods in other variational inequalities, composite optimization, and structured nonconvex regularization regimes in machine learning.

Practical extension avenues include:

• Removing or weakening the proximal-lifting assumption for weakly convex $T$4,
• Developing two-sided convergence guarantees without requiring subsequence extraction,
• Adaptation to large-scale and stochastic frameworks, especially for deep learning or adversarially constrained architectures requiring only LMO-access,
• Application to unconstrained analogs and implicit Frank-Wolfe-type methods.

The approach lays technical groundwork for integrating truly nontrivial nonsmooth constraints and regularizers into large-scale, interpretable AI systems with theoretical convergence certifications.

Conclusion

Through Moreau envelope smoothing and careful step-size/schedule tuning, FRAMES extends the Frank-Wolfe paradigm to broad classes of nonsmooth, nonconvex, and composite-constrained optimization problems. The paper provides quantifiable, explicit convergence rates for meaningful stationarity certificates of both the surrogate and original objectives, validated by comprehensive empirical results. These advances facilitate rigorous and practical optimization for structured learning and signal processing applications where nonsmoothness and complex constraints are pervasive (2606.00853).