FreeMCG: Derivative-Free Inverse Solver

• FreeMCG is a derivative-free framework that fuses model-constrained Gauss–Newton concepts with ensemble Kalman filtering to solve inverse problems without explicit gradient computation.
• It approximates Newton-type updates using empirical covariances from an ensemble, ensuring stability through adaptive regularization and controlled update sizes.
• The method is highly parallelizable and extends to complex applications such as PDE inversion, scientific imaging, and explainable AI by leveraging ensemble assimilation.

Derivative-Free MCG via Ensemble Kalman Filters (FreeMCG) is a framework for solving inverse problems and black-box optimization that leverages the interplay between model-constrained Gauss–Newton (MCG) principles and ensemble Kalman filtering methodology. FreeMCG refers to any approach that constructs a derivative-free, data-driven approximation to a Newton or conjugate-gradient update by using empirical covariances over an ensemble of candidates and Kalman-style assimilation, rather than explicit adjoint or gradient information. The approach is particularly suited to scenarios where only black-box evaluation of the forward model is available, gradients are inaccessible or expensive, and the objective is either regularized least-squares, maximum-likelihood/posterior estimation, or similar composite optimization. It generalizes to modern tasks such as PDE-constrained inversion, scientific imaging, diffusion-model-guided inference, feature attribution, and counterfactual generation in explainable AI. FreeMCG incorporates regularization, empirical adaptivity, and is naturally parallelizable across ensemble members and model evaluations.

1. Formal Definition and Algorithmic Structure

FreeMCG algorithms estimate solution updates for inverse or data-assimilation problems of the form

$$y = G(u) + \xi, \quad \xi \sim \mathcal{N}(0, \Gamma)$$

where $G: X \rightarrow Y$ is a (possibly non-linear) black-box forward operator in Hilbert or Euclidean space, and $u \in X$ is the parameter to be inferred from noisy measurements $y$. The derivative-free objectives express as minimizing a misfit functional (e.g., data discrepancy, Tikhonov-regularized least-squares): $$\Phi(u) = \|\Gamma^{-1/2}[y^\eta - G(u)]\|_Y^2 + \mathrm{Reg}(u)$$ where regularization reflects prior structure.

The core FreeMCG update proceeds as follows (Iglesias, 2015):

• Draw an ensemble $\{u^{(j)}_k\}_{j=1}^{N_e}$.
• Compute ensemble predictions $w^{(j)}_k = G(u^{(j)}_k)$ and empirical means $\bar u_k, \bar w_k$.
• Form sample covariances:

$$C_k^{uw} = \frac{1}{N_e - 1}\sum_j (u_k^{(j)} - \bar u_k) \otimes (w_k^{(j)} - \bar w_k), \quad C_k^{ww} = \frac{1}{N_e - 1}\sum_j (w_k^{(j)} - \bar w_k) \otimes (w_k^{(j)} - \bar w_k)$$

• Update each ensemble member:

$$u_{k+1}^{(j)} = u_k^{(j)} + C_k^{uw} [C_k^{ww} + \alpha_k \Gamma]^{-1} [y^\eta - w_k^{(j)}]$$

where $G: X \rightarrow Y$0 is a regularization parameter adapted at each step (e.g., via a Morozov discrepancy principle enforcing LM-type stability).

This realizes a Gauss–Newton-like step with all derivative information replaced by ensemble covariances, and the inversion restricted to the small data dimension (Iglesias, 2015, Chada et al., 2020, Zheng et al., 2024). Extension to multi-step or conjugate-gradient logic is achieved by recursively constructing Krylov subspaces from ensemble-preconditioned residuals and performing accelerated updates (Calvello et al., 2022, Chada et al., 2020).

2. Regularization and Stability in FreeMCG

Unlike standard ensemble Kalman inversion, naive application can lead to ill-posed updates, ensemble collapse, or instability, especially for small ensembles or ill-posed problems. FreeMCG remedies this via:

• Dynamic selection of a Tikhonov or Levenberg–Marquardt regularization parameter $G: X \rightarrow Y$1, adapted to maintain control over the update size and avoid overfitting (Iglesias, 2015).
• Early stopping in accordance with a Morozov-type discrepancy principle: terminate the iterations when the (ensemble-averaged) data residual matches the known noise level, $G: X \rightarrow Y$2.
• Implicit prior information introduced through the initialization of the ensemble and (optionally) via empirical prior covariance matrices.

Regularizing the gain matrix ($G: X \rightarrow Y$3) ensures well-posed inversion even in high-noise or data-poor regimes, and suppresses the propagation of spurious directions associated with underdetermined components.

3. Generalization: Bayesian Sampling, Optimization, and Model Types

FreeMCG generalizes beyond deterministic optimization to Bayesian posterior sampling and MAP-type estimation:

• In the context of derivative-free Bayesian inversion, multiscale SDEs drive a slow parameter particle via ensemble-calculated drift, optionally combined with Langevin noise for sampling the posterior (Pavliotis et al., 2021, Garbuno-Inigo et al., 2019).
• For the randomized maximum likelihood (RML) approach, FreeMCG evolves each ensemble member to solve a perturbed optimization problem, providing samples from the linear-Gaussian posterior in the observable (row) space (Stavrinides et al., 3 Jul 2025).
• Diffusion-model-based FreeMCG methods combine a score-based prior (from diffusion models) with ensemble Kalman correction to perform derivative-free inverse imaging or counterfactual search, alternating prediction (score-guided movement on the data manifold) with an ensemble-based Kalman correction toward observed data (thus enabling guidance without needing $G: X \rightarrow Y$4) (Zheng et al., 2024, Kim et al., 2024).
• The approach applies regardless of model type (PDE-constrained, neural network, scientific simulator), as long as repeated forward evaluations are feasible.

Typical tasks include:

• Black-box scientific inverse problems (e.g., electrical impedance tomography, Darcy flow) (Iglesias, 2015, Pavliotis et al., 2021).
• Imaging/vision inverse problems with learned generative priors (Zheng et al., 2024, Kim et al., 2024).
• Explainable AI (e.g., saliency attribution, counterfactuals) using diffusion manifolds (Kim et al., 2024).

4. Computational Considerations and Parameter Choices

FreeMCG is highly parallelizable, as all forward model and score evaluations are independent across the ensemble. Per iteration cost:

• $G: X \rightarrow Y$5 forward solves of $G: X \rightarrow Y$6 per step.
• $G: X \rightarrow Y$7 for inversion in the low-dimensional data space ($G: X \rightarrow Y$8), independent of parameter dimension $G: X \rightarrow Y$9.
• For diffusion-model-based guidance: additional cost for ensemble score evaluations and covariance calculations [$u \in X$0 if performed with full matrices, or $u \in X$1 with low-rank structure/exploitation].

Key parameter guidelines:

• Ensemble size $u \in X$2: Must be large enough to resolve covariance directions, typically $u \in X$3 for practical problems, though dimension-reduced or subspace approaches can alleviate this for high-dimensional parameter spaces (Iglesias, 2015, Zheng et al., 2024).
• Regularization parameter $u \in X$4: Chosen adaptively; increasing $u \in X$5 suppresses instability at the expense of slower convergence.
• Prior noise/ensemble spread: The spread of the initial ensemble encodes regularization and effective prior; must match the problem's expected parameter scale.
• Noise covariance $u \in X$6: Must reflect measurement noise, more regularization when larger; inaccurate specification may bias estimates or slow convergence.

Covariance inflation, shrinkage, or localization may be needed for small ensembles or particularly ill-posed/high-dimensional regimes (Zheng et al., 2024).

5. Theoretical Properties and Empirical Behavior

Convergence theory for FreeMCG has been established in several stylized settings:

• For linear-Gaussian inverse problems, exponential convergence of the ensemble to the posterior mean and covariance in the observable subspace has been proved (Stavrinides et al., 3 Jul 2025).
• For nonlinear problems, in the small-ensemble-spread and large-ensemble limit, the update approximates the Gauss–Newton (or second-order) step, and under sufficient regularity converges to a local minimizer (Chada et al., 2020, Pavliotis et al., 2021, Iglesias, 2015).
• For Bayesian sampling, derivative-free Kalman–Langevin or Fokker–Planck limits can be constructed in mean-field settings, showing correct invariant measures, ergodicity, and gradient flow structure, at least for quadratic posteriors or under suitable convexity conditions (Garbuno-Inigo et al., 2019, Pavliotis et al., 2021).
• In practice, performance is robust across problem classes: imaging, PDEs, machine learning, and XAI. The empirical literature confirms that FreeMCG can outperform or match state-of-the-art gradient-based and zero-order alternatives on problems where gradient information is not available (Zheng et al., 2024, Kim et al., 2024, Iglesias, 2015).

Typical observed phenomena include:

• Ensemble collapse and loss of diversity at small $u \in X$7.
• Slower convergence for highly anisotropic or ill-posed inverse mappings unless preconditioning and adaptive regularization are used.
• Near-optimal low-rank recovery of the posterior covariance when using model reduction and balanced truncation tailored to the Bayesian inference problem (Stavrinides et al., 3 Jul 2025).
• For explainable AI, on-manifold gradient estimation, improved robustness to adversarial artifacts, and faithful feature attributions with only black-box access (Kim et al., 2024).

6. Applications, Variants, and Limitations

FreeMCG underpins a wide range of applications:

• Scientific inverse problems: recovery of spatially-varying parameters in PDE models, shape identification via level-set parameterizations (Iglesias, 2015).
• Imaging and deep learning: denoising, super-resolution, phase retrieval guided by learned priors and black-box forward models (Zheng et al., 2024).
• Explainable AI: manifold-constrained, derivative-free feature attribution and counterfactual analysis in vision models, addressing the pitfalls of classical gradients (Kim et al., 2024).
• Machine learning training: robust, parallelizable optimization procedures for non-differentiable or highly regularized empirical risk objectives (Kovachki et al., 2018).

Variants include:

• Multi-step/Conjugate-Gradient FreeMCG: embedding classical momentum or CG logic into the update via ensemble approximations in Krylov subspaces for accelerated convergence (Calvello et al., 2022, Chada et al., 2020).
• Model-reduced FreeMCG: low-rank approximate updates in high-dimensional or dynamic systems, leveraging balanced truncation and prior-informed subspace bases (Stavrinides et al., 3 Jul 2025).
• Diffusion-guided FreeMCG: alternating score-based drift (from diffusion priors) with ensemble-based correction for sample-efficient, derivative-free inverse inference (Zheng et al., 2024).

Limitations arise when the ensemble size is too small for the parameter dimension, leading to subspace collapse and spurious convergence; excessive regularization can stall progress, while too little invites instability. Posterior sampling properties are approximate except in linear or quadratic cases; hybrid or MCMC-corrected versions are required for exact uncertainty quantification (Garbuno-Inigo et al., 2019, Stavrinides et al., 3 Jul 2025).

7. Summary Table: Key FreeMCG Algorithmic Ingredients

Component	Role	Reference(s)
Empirical covariance	Sensitivity/Jacobian surrogate for preconditioning	(Iglesias, 2015)
Ensemble Kalman gain	Data assimilation, update step	(Calvello et al., 2022)
Regularization $u \in X$8	Stability, LM-type damping, adaptive fitting	(Iglesias, 2015)
Discrepancy stopping	Early termination at noise level, prevents overfitting	(Iglesias, 2015)
Ensemble size $u \in X$9	Ensemble diversity, resolution of active subspace	(Stavrinides et al., 3 Jul 2025)
Multi-step CG logic	Acceleration, Krylov subspace updates	(Calvello et al., 2022)
Diffusion prior/score	Manifold constraint, generative prior	(Zheng et al., 2024)

The FreeMCG paradigm synthesizes derivative-free optimization, regularization, empirical preconditioning, and ensemble filtering, providing a flexible framework for inversion and machine learning in non-differentiable or black-box settings, with broad applicability and well-characterized theoretical and empirical behavior.