Projected Gradient Descent in Function Space

• PGD in function space is an iterative optimization method that extends finite-dimensional PGD to infinite or high-dimensional constraint sets, enabling advanced signal recovery and learning.
• The approach applies to inverse problems, super-resolution, and variational optimization by leveraging neural network and measure-based formulations for complex projection tasks.
• Recent advances, such as learned projectors and variable metrics, boost convergence rates and computational efficiency, achieving speed-ups up to 175x in tasks like compressed sensing.

Projected gradient descent (PGD) in function space refers to the class of iterative optimization methods where each iterate is projected onto an infinite- or high-dimensional constraint set defined by function space or nonlinear manifold structure. This approach generalizes classical finite-dimensional PGD to a variety of applications involving signal recovery, inverse problems, structural optimization, and learning with generative priors. PGD in function space encompasses both concrete measure- or function-space formulations as well as neural network and generator-driven manifolds, and supports varying degrees of differentiability and constraint complexity.

1. Problem Formulations in Function Space

Projected gradient descent in function space arises in inverse problems, super-resolution, and variational optimization. The general framework involves solving

$\min_{x \in \mathcal{C}} L(y, Ax)$

where $x$ is a function or measure constrained to a feasible set $\mathcal{C}$. In inverse problems with data-driven priors, $\mathcal{C}$ can be the range of a conditional generator $G$ (parametrized, e.g., by a conditional GAN) or an atomic measure space for super-resolution (Damara et al., 2021, Bénard et al., 2022).

For example, super-resolution from Fourier measurements is modeled as minimizing

$F(\mu) = \frac{1}{2} \|y - A(\mu)\|_2^2$

where $\mu$ lies in the cone of nonnegative measures with a mass budget constraint, and $A$ is the linear measurement operator mapping measures to data (Bénard et al., 2022).

In variational optimization, the objective $j(\phi)$ is defined over an admissible set $\mathrm{ad} \subset X \cap \mathbb{B}^*$ in a Banach space, with convex or pointwise constraints (Blank et al., 2015).

2. Classical and Modern PGD Methodologies

Traditional PGD alternates between a descent step in the ambient space and a projection onto the feasible set:

• In generator prior inverse problems, the update is

$x$0

where projection requires finding $x$1 closest to $x$2, often by solving a nonconvex inner minimization over latent codes (Damara et al., 2021).

• In measure space,

$x$3

The projection $x$4 in the set of nonnegative measures with mass constraint is effected by nonnegative thresholding and mass normalization (Bénard et al., 2022).

In Banach spaces, the variable-metric projected gradient type (VMPT) method adapts the projected subproblem to general norms: $x$5 with relaxed update $x$6 (Blank et al., 2015).

3. Acceleration via Parametric and Learned Projectors

A core computational bottleneck for manifold-constrained PGD is the projection step. When $x$7 is defined by the range of a generative network $x$8, the projection

$x$9

requires solving a costly nonconvex minimization in $\mathcal{C}$0 at every iteration.

To address this, network-projected gradient descent (NPGD) employs a learned neural network $\mathcal{C}$1 to approximate the inverse mapping from $\mathcal{C}$2 to the latent code. The NPGD update replaces the inner optimization with fast inference: $\mathcal{C}$3 $\mathcal{C}$4 is trained to minimize reconstruction and inversion error, satisfying $\mathcal{C}$5 (Damara et al., 2021).

The replacement of iterative projection by a single forward pass through $\mathcal{C}$6 leads to empirical speed-ups of 140–175$\mathcal{C}$7 in reconstruction time on MNIST and CelebA datasets for compressed sensing problems with measurement-conditional GAN priors (Damara et al., 2021).

4. Convergence Theory and Approximation in Function Spaces

Convergence analyses extend from Hilbert to Banach and measure spaces. In generator-constrained inverse problems, if the operator $\mathcal{C}$8 satisfies a Restricted Eigenvalue Condition (REC) on the generator manifold and $\mathcal{C}$9 acts as a $\mathcal{C}$0-approximate projector, NPGD achieves linear convergence up to a $\mathcal{C}$1-sized error floor: $\mathcal{C}$2 where $\mathcal{C}$3 and $\mathcal{C}$4 satisfies REC$\mathcal{C}$5 with $\mathcal{C}$6 (Damara et al., 2021).

For measure-space PGD, under Lipschitz gradient conditions and an initial point within the basin of attraction, local linear convergence is observed. Step size selection through line search further improves stability (Bénard et al., 2022).

In Banach space settings, the VMPT method, equipped with Armijo backtracking and variable metrics, provides global convergence guarantees to stationary points even when the objective's differentiability is limited to only $\mathcal{C}$7 (rather than Hilbert spaces), and supports convergence under mesh refinement (Blank et al., 2015).

5. Computational Aspects and Practical Acceleration

Performance of PGD in function spaces is influenced by the structure and implementation of projection and initialization:

Method	Projection	Complexity (per iter)	Empirical speed-up
Standard PGD (GAN prior)	Iterative, over latent	$\mathcal{C}$8	Baseline
NPGD (GAN prior)	Single pass via $\mathcal{C}$9	$G$0	$G$1–$G$2 (MNIST/CelebA)
PGD (measure space)	Weight thresholding	$G$3	$G$4–$G$5 (OMP-init+PGD vs OMP)

Over-parametrized initializations, e.g., via continuous OMP with $G$6, facilitate rapid convergence and grid-free operation in super-resolution (Bénard et al., 2022). In Banach space variational problems, mesh-independence and superlinear convergence are obtained using $G$7 or Hessian-type metrics and BFGS updates (Blank et al., 2015).

6. Implications and Extensions in Infinite-Dimensional Settings

PGD in function and measure spaces addresses projections onto highly nonlinear manifolds, which in infinite dimensions entails computational and analytical challenges. The use of parametric or learned projectors regularizes and accelerates convergence by exploiting the global geometric structure induced by data-driven or problem-specific priors (Damara et al., 2021).

The VMPT framework generalizes Hilbert space methods to Banach spaces, supporting variable metrics, second-order information, limited differentiability, and practical accelerations such as mesh-independence and BFGS updates (Blank et al., 2015).

A plausible implication is that the evolution of PGD in function space, especially with neural or over-parametrized projectors, broadens applicability to complex inverse problems, high-dimensional constraint sets, and nonconvex feasible regions, while retaining convergence guarantees and computational efficiency.