---
title: 'FreeMCG: Derivative-Free Inverse Solver'
url: https://www.emergentmind.com/topics/derivative-free-mcg-via-ensemble-kalman-filters-freemcg
type: topic
---

# FreeMCG: Derivative-Free Inverse Solver

Derivative-Free MCG via Ensemble Kalman Filters (FreeMCG) is a framework for solving inverse problems and black-box optimization that leverages the interplay between model-constrained Gauss–Newton (MCG) principles and ensemble Kalman filtering methodology. FreeMCG refers to any approach that constructs a derivative-free, data-driven approximation to a Newton or conjugate-gradient update by using empirical covariances over an ensemble of candidates and Kalman-style assimilation, rather than explicit adjoint or gradient information. The approach is particularly suited to scenarios where only black-box evaluation of the forward model is available, gradients are inaccessible or expensive, and the objective is either regularized least-squares, maximum-likelihood/posterior estimation, or similar composite optimization. It generalizes to modern tasks such as PDE-constrained inversion, scientific imaging, diffusion-model-guided inference, feature attribution, and counterfactual generation in explainable AI. FreeMCG incorporates regularization, empirical adaptivity, and is naturally parallelizable across ensemble members and model evaluations.

## 1. Formal Definition and Algorithmic Structure

FreeMCG algorithms estimate solution updates for inverse or data-assimilation problems of the form
\[
y = G(u) + \xi, \quad \xi \sim \mathcal{N}(0, \Gamma)
\]
where $G: X \rightarrow Y$ is a (possibly non-linear) black-box forward operator in Hilbert or Euclidean space, and $u \in X$ is the parameter to be inferred from noisy measurements $y$. The derivative-free objectives express as minimizing a misfit functional (e.g., data discrepancy, Tikhonov-regularized least-squares):
\[
\Phi(u) = \|\Gamma^{-1/2}[y^\eta - G(u)]\|_Y^2 + \mathrm{Reg}(u)
\]
where regularization reflects prior structure.

The core FreeMCG update proceeds as follows [1505.03876]:
- Draw an ensemble $\{u^{(j)}_k\}_{j=1}^{N_e}$.
- Compute ensemble predictions $w^{(j)}_k = G(u^{(j)}_k)$ and empirical means $\bar u_k, \bar w_k$.
- Form sample covariances:
  \[
  C_k^{uw} = \frac{1}{N_e - 1}\sum_j (u_k^{(j)} - \bar u_k) \otimes (w_k^{(j)} - \bar w_k), \quad
  C_k^{ww} = \frac{1}{N_e - 1}\sum_j (w_k^{(j)} - \bar w_k) \otimes (w_k^{(j)} - \bar w_k)
  \]
- Update each ensemble member:
  \[
  u_{k+1}^{(j)} = u_k^{(j)} + C_k^{uw} [C_k^{ww} + \alpha_k \Gamma]^{-1} [y^\eta - w_k^{(j)}]
  \]
where $\alpha_k > 0$ is a regularization parameter adapted at each step (e.g., via a Morozov discrepancy principle enforcing LM-type stability).

This realizes a Gauss–Newton-like step with all derivative information replaced by ensemble covariances, and the inversion restricted to the small data dimension [1505.03876, 2010.13299, 2409.20175]. Extension to multi-step or conjugate-gradient logic is achieved by recursively constructing Krylov subspaces from ensemble-preconditioned residuals and performing accelerated updates [2209.11371, 2010.13299].

## 2. Regularization and Stability in FreeMCG

Unlike standard ensemble Kalman inversion, naive application can lead to ill-posed updates, ensemble collapse, or instability, especially for small ensembles or ill-posed problems. FreeMCG remedies this via:
- Dynamic selection of a Tikhonov or Levenberg–Marquardt regularization parameter $\alpha_k$, adapted to maintain control over the update size and avoid overfitting [1505.03876].
- Early stopping in accordance with a Morozov-type discrepancy principle: terminate the iterations when the (ensemble-averaged) data residual matches the known noise level, $\|\Gamma^{-1/2}[y^\eta - \bar w_k]\|_Y \leq \tau \eta$.
- Implicit prior information introduced through the initialization of the ensemble and (optionally) via empirical prior covariance matrices.

Regularizing the gain matrix ($C_k^{ww} + \alpha_k \Gamma$) ensures well-posed inversion even in high-noise or data-poor regimes, and suppresses the propagation of spurious directions associated with underdetermined components.

## 3. Generalization: Bayesian Sampling, Optimization, and Model Types

FreeMCG generalizes beyond deterministic optimization to Bayesian posterior sampling and MAP-type estimation:
- In the context of derivative-free Bayesian inversion, multiscale SDEs drive a slow parameter particle via ensemble-calculated drift, optionally combined with Langevin noise for sampling the posterior [2102.00540, 1903.08866].
- For the randomized maximum likelihood (RML) approach, FreeMCG evolves each ensemble member to solve a perturbed optimization problem, providing samples from the linear-Gaussian posterior in the observable (row) space [2507.03207].
- Diffusion-model-based FreeMCG methods combine a score-based prior (from diffusion models) with ensemble Kalman correction to perform derivative-free inverse imaging or counterfactual search, alternating prediction (score-guided movement on the data manifold) with an ensemble-based Kalman correction toward observed data (thus enabling guidance without needing $\nabla F$) [2409.20175, 2411.15265].
- The approach applies regardless of model type (PDE-constrained, neural network, scientific simulator), as long as repeated forward evaluations are feasible.

Typical tasks include:
- Black-box scientific inverse problems (e.g., electrical impedance tomography, Darcy flow) [1505.03876, 2102.00540].
- Imaging/vision inverse problems with learned generative priors [2409.20175, 2411.15265].
- Explainable AI (e.g., saliency attribution, counterfactuals) using diffusion manifolds [2411.15265].

## 4. Computational Considerations and Parameter Choices

FreeMCG is highly parallelizable, as all forward model and score evaluations are independent across the ensemble. Per iteration cost:
- $N_e$ forward solves of $G$ per step.
- $O(m^3)$ for inversion in the low-dimensional data space ($m = \mathrm{dim}(Y)$), independent of parameter dimension $d$.
- For diffusion-model-based guidance: additional cost for ensemble score evaluations and covariance calculations [$O(N d^2 + m^3)$ if performed with full matrices, or $O(N m^2)$ with low-rank structure/exploitation].

Key parameter guidelines:
- Ensemble size $N_e$: Must be large enough to resolve covariance directions, typically $N_e \sim 10^2-10^3$ for practical problems, though dimension-reduced or subspace approaches can alleviate this for high-dimensional parameter spaces [1505.03876, 2409.20175].
- Regularization parameter $\alpha_k$: Chosen adaptively; increasing $\alpha_k$ suppresses instability at the expense of slower convergence.
- Prior noise/ensemble spread: The spread of the initial ensemble encodes regularization and effective prior; must match the problem's expected parameter scale.
- Noise covariance $\Gamma$: Must reflect measurement noise, more regularization when larger; inaccurate specification may bias estimates or slow convergence.

Covariance inflation, shrinkage, or localization may be needed for small ensembles or particularly ill-posed/high-dimensional regimes [2409.20175].

## 5. Theoretical Properties and Empirical Behavior

Convergence theory for FreeMCG has been established in several stylized settings:
- For linear-Gaussian inverse problems, exponential convergence of the ensemble to the posterior mean and covariance in the observable subspace has been proved [2507.03207].
- For nonlinear problems, in the small-ensemble-spread and large-ensemble limit, the update approximates the Gauss–Newton (or second-order) step, and under sufficient regularity converges to a local minimizer [2010.13299, 2102.00540, 1505.03876].
- For Bayesian sampling, derivative-free Kalman–Langevin or Fokker–Planck limits can be constructed in mean-field settings, showing correct invariant measures, ergodicity, and gradient flow structure, at least for quadratic posteriors or under suitable convexity conditions [1903.08866, 2102.00540].
- In practice, performance is robust across problem classes: imaging, PDEs, machine learning, and XAI. The empirical literature confirms that FreeMCG can outperform or match state-of-the-art gradient-based and zero-order alternatives on problems where gradient information is not available [2409.20175, 2411.15265, 1505.03876].

Typical observed phenomena include:
- Ensemble collapse and loss of diversity at small $N_e$.
- Slower convergence for highly anisotropic or ill-posed inverse mappings unless preconditioning and adaptive regularization are used.
- Near-optimal low-rank recovery of the posterior covariance when using model reduction and balanced truncation tailored to the Bayesian inference problem [2507.03207].
- For explainable AI, on-manifold gradient estimation, improved robustness to adversarial artifacts, and faithful feature attributions with only black-box access [2411.15265].

## 6. Applications, Variants, and Limitations

FreeMCG underpins a wide range of applications:
- Scientific inverse problems: recovery of spatially-varying parameters in PDE models, shape identification via level-set parameterizations [1505.03876].
- Imaging and deep learning: denoising, super-resolution, phase retrieval guided by learned priors and black-box forward models [2409.20175].
- Explainable AI: manifold-constrained, derivative-free feature attribution and counterfactual analysis in vision models, addressing the pitfalls of classical gradients [2411.15265].
- Machine learning training: robust, parallelizable optimization procedures for non-differentiable or highly regularized empirical risk objectives [1808.03620].

Variants include:
- Multi-step/Conjugate-Gradient FreeMCG: embedding classical momentum or CG logic into the update via ensemble approximations in Krylov subspaces for accelerated convergence [2209.11371, 2010.13299].
- Model-reduced FreeMCG: low-rank approximate updates in high-dimensional or dynamic systems, leveraging balanced truncation and prior-informed subspace bases [2507.03207].
- Diffusion-guided FreeMCG: alternating score-based drift (from diffusion priors) with ensemble-based correction for sample-efficient, derivative-free inverse inference [2409.20175].

Limitations arise when the ensemble size is too small for the parameter dimension, leading to subspace collapse and spurious convergence; excessive regularization can stall progress, while too little invites instability. Posterior sampling properties are approximate except in linear or quadratic cases; hybrid or MCMC-corrected versions are required for exact uncertainty quantification [1903.08866, 2507.03207].

## 7. Summary Table: Key FreeMCG Algorithmic Ingredients

| Component                 | Role                                                       | Reference(s)      |
|---------------------------|------------------------------------------------------------|-------------------|
| Empirical covariance      | Sensitivity/Jacobian surrogate for preconditioning         | [1505.03876]      |
| Ensemble Kalman gain      | Data assimilation, update step                             | [2209.11371]      |
| Regularization $\alpha_k$ | Stability, LM-type damping, adaptive fitting               | [1505.03876]      |
| Discrepancy stopping      | Early termination at noise level, prevents overfitting      | [1505.03876]      |
| Ensemble size $N_e$       | Ensemble diversity, resolution of active subspace           | [2507.03207]      |
| Multi-step CG logic       | Acceleration, Krylov subspace updates                      | [2209.11371]      |
| Diffusion prior/score     | Manifold constraint, generative prior                      | [2409.20175]      |

The FreeMCG paradigm synthesizes derivative-free optimization, regularization, empirical preconditioning, and ensemble filtering, providing a flexible framework for inversion and machine learning in non-differentiable or black-box settings, with broad applicability and well-characterized theoretical and empirical behavior.

Source: https://www.emergentmind.com/topics/derivative-free-mcg-via-ensemble-kalman-filters-freemcg