---
title: 'GVAMP: Generalized VAMP for GLMs'
url: https://www.emergentmind.com/topics/generalized-vector-approximate-message-passing-gvamp
type: topic
---

# GVAMP: Generalized VAMP for GLMs

Generalized Vector Approximate Message Passing (GVAMP), also presented as GLM-VAMP in early work, is an expectation-consistent / expectation-propagation-style iterative inference method for generalized linear models (GLMs) in which an unknown vector \(x\) is observed through a separable output channel acting on \(z=Ax\). It extends Vector Approximate Message Passing (VAMP) from additive white Gaussian noise linear models to arbitrary separable likelihoods \(p(y|z)=\prod_m p_{y|z}(y_m|z_m)\), while retaining the robustness of VAMP to right-rotationally or unitarily invariant sensing matrices rather than restricting \(A\) to the IID Gaussian regime required by classical GAMP analyses [1610.03082][1612.01186]. Later work embedded GVAMP inside broader families such as generalized memory AMP (GMAMP), established Bayes-optimal fixed-point results under unitarily invariant matrices, and developed low-complexity, adaptive, bilinear, and coding-aware extensions [2110.06069][2305.07870].

## 1. Model class and conceptual foundations

GVAMP is formulated for the generalized linear model
\[
\Psi:\quad y = Q(z),\qquad
\Gamma:\quad z = A x,\qquad
\Phi:\quad x_i \sim P_X(x),
\]
where \(A\) is \(M\times N\), the prior is separable \(p(x)=\prod_{i=1}^N P_X(x_i)\), and the observation channel is separable \(p(y|z)=\prod_{i=1}^M p(y_i|z_i)\) [2110.06069]. Equivalent formulations write the posterior as
\[
p(x,z|y)\propto p(x)\,p(y|z)\,\delta(z-Ax),
\]
which makes explicit the factorization into a prior factor on \(x\), a likelihood factor on \(z\), and a linear consistency constraint between them [1612.01186].

The matrix class that underlies GVAMP’s robustness is broader than the IID sub-Gaussian setting of AMP. In the VAMP literature, \(A\) is right-orthogonally invariant, meaning that in the singular value decomposition \(A=USV\), the right singular vectors \(V\) are Haar distributed and the distribution of \(A\) is invariant to right multiplication by any fixed orthogonal matrix [1610.03082]. In the generalized-memory formulation, the analogous complex-valued statement is that
\[
A = U \Sigma V^{\rm H},
\]
with \(U\) and \(V\) unitary, mutually independent with \(\Sigma\), and \(U,V\) Haar distributed; this is the unitarily invariant setting [2110.06069].

This placement of GVAMP between GAMP and VAMP is central. Standard GAMP is efficient for GLMs but is derived for IID sensing matrices; VAMP is robust for right-orthogonally invariant matrices but was introduced for linear AWGN models. GVAMP combines the generalized-output capability of GAMP with the rotational-invariance robustness of VAMP by introducing an auxiliary output variable \(z\) and passing Gaussian-parameterized extrinsic messages between nonlinear scalar modules and a linear LMMSE module [1612.01186][1610.03082].

## 2. Core algorithmic structure

The canonical GVAMP architecture alternates between nonlinear estimators and a linear estimator. In the GLM-VAMP presentation of Schniter, Rangan, and Fletcher, the algorithm maintains pseudo-observations \((r_{1k},\gamma_{1k})\) for \(x\) and \((p_{1k},\tau_{1k})\) for \(z\), applies scalar denoisers, and forms extrinsic messages through Onsager-like orthogonalization [1612.01186].

For the \(x\)-stream,
\[
x_{1k} = g_{x1}(r_{1k},\gamma_{1k}),\qquad
\alpha_{1k} = g'_{x1}(r_{1k},\gamma_{1k}),
\]
followed by
\[
r_{2k} = \frac{x_{1k} - \alpha_{1k}r_{1k}}{1-\alpha_{1k}},\qquad
\gamma_{2k} = \frac{\gamma_{1k}(1-\alpha_{1k})}{\alpha_{1k}}.
\]
For the \(z\)-stream,
\[
z_{1k} = g_{z1}(p_{1k},\tau_{1k}),\qquad
\beta_{1k} = g'_{z1}(p_{1k},\tau_{1k}),
\]
and
\[
p_{2k} = \frac{z_{1k} - \beta_{1k}p_{1k}}{1-\beta_{1k}},\qquad
\tau_{2k} = \frac{\tau_{1k}(1-\beta_{1k})}{\beta_{1k}}.
\]
Here \(g_{x1}\) is the prior denoiser and \(g_{z1}\) is the output-channel denoiser, each acting componentwise under Gaussian pseudo-priors [1612.01186].

The linear module enforces \(z=Ax\) through an LMMSE calculation. With the SVD \(A=USV^T\), diagonal matrix \(D_k\) defined by
\[
[D_k]_{nn} = (\tau_{2k}s_n^2 + \gamma_{2k})^{-1},
\]
the linear update is
\[
x_{2k} = V D_k \big( \tau_{2k} S U^T p_{2k} + \gamma_{2k}V^T r_{2k} \big),\qquad
z_{2k} = A x_{2k},
\]
with average Jacobians
\[
\alpha_{2k} = \frac{1}{N}\sum_{n=1}^N \frac{\gamma_{2k}}{\tau_{2k}s_n^2 + \gamma_{2k}},\qquad
\beta_{2k} = \frac{1}{M}\sum_{n=1}^N \frac{\tau_{2k}s_n^2}{\tau_{2k}s_n^2 + \gamma_{2k}}.
\]
Extrinsic feedback to the nonlinear modules is then formed by the same subtraction-and-rescaling pattern [1612.01186].

A later unitarily invariant formulation describes GVAMP as a two-module non-memory iterative process with symbol-wise MMSE denoisers
\[
\hat{\phi}_t(x_t)=\mathbb{E}\{x\mid x_t,\Phi\},\qquad
\hat{\psi}_t(z_t)=\mathbb{E}\{z\mid z_t,\Psi\},
\]
and an LMMSE linear step
\[
\hat{\gamma}_t(x_t)=A^{\rm H}(\rho_t I + A A^{\rm H})^{-1}(z_t - A x_t),\qquad
\rho_t=\frac{v_t^z}{v_t^x}.
\]
This form makes explicit the matrix inverse \(A^{\rm H}(\rho_t I + A A^{\rm H})^{-1}\), which is the chief source of GVAMP’s computational cost [2110.06069].

That cost is substantial. For the linear estimator, direct inversion requires \(O(M^3)\) per iteration, plus \(O(M^2N)\) multiplications; an SVD-based variant has similar asymptotic burden in memory and still high runtime for large-scale systems [2110.06069]. This computational profile is one of the main reasons later work sought memory-based approximations and low-complexity replacements.

## 3. State evolution, fixed points, and optimality claims

The original theoretical foundation comes from VAMP. For standard linear regression with right-orthogonally invariant \(A\) and separable Lipschitz denoisers, VAMP admits a rigorous scalar state evolution, and its fixed points are consistent with replica predictions of the minimum mean-squared error [1610.03082]. GVAMP inherits this program in the GLM setting by replacing the Gaussian output model with a nonlinear output-channel module.

In the generalized-memory treatment, GVAMP’s state evolution under unitarily invariant \(A\) tracks the pseudo-observation variances through scalar recursions
\[
(v_t^x, v_t^z) = \big(\phi_{\rm SE}(v_t^x), \psi_{\rm SE}(v_t^z)\big),\qquad
(v_{t+1}^x, v_{t+1}^z) = \gamma_{\rm SE}(v_t^x, v_t^z),
\]
with
\[
v_{t+1}^x = \big([\delta\,\epsilon^\gamma_t]^{-1}-1\big)\,v_t^x,\qquad
v_{t+1}^z = \big([\epsilon^\gamma_t]^{-1}-1\big)^{-1}\,v_t^z.
\]
These recursions are rigorous under the rotational-invariance assumptions and the associated orthogonality conditions [2110.06069].

The strongest Bayes-optimality statement in the supplied literature is attached to the memory generalization rather than to the original 2016 GLM-VAMP paper. For unitarily invariant transformation matrices, Bayes-optimal GMAMP is proved to attain the replica minimum, i.e. Bayes-optimal, MSE if it has a unique fixed point, and its fixed points coincide with those of GVAMP’s state evolution [2110.06069]. This places GVAMP’s fixed points inside a later, more explicit replica-optimal framework.

Model mismatch complicates this picture. In the mismatched VAMP/GVAMP analysis, state evolution must track the order parameters \(m\), \(q\), and \(\chi\), rather than a single MSE-like scalar. The fixed-point equations coincide with the replica-symmetric saddle-point equations, while the microscopic instability threshold of GVAMP matches the de Almeida–Thouless instability line [2001.02824]. A common implication is that convergence failures are not purely numerical artifacts: in some regimes they correspond to replica-symmetry breaking and genuine algorithmic instability.

Theoretical coverage is therefore stratified. The original GLM-VAMP paper emphasizes the algorithmic construction and empirical robustness, while explicitly stating that “a rigorous justification of these models is postponed for future work” [1612.01186]. Subsequent work supplied more detailed asymptotic and fixed-point analyses, especially for memory extensions and mismatched inference [2110.06069][2001.02824].

## 4. Generalizations and algorithmic descendants

Several later algorithms should be understood as GVAMP extensions rather than unrelated methods. One line concerns parameter learning. EM-GVAMP combines expectation maximization with GVAMP when the prior \(p_x(\cdot;\theta_x)\) and/or channel \(p(y|z;\theta_z)\) contain unknown deterministic parameters. After GVAMP supplies Gaussian approximate posterior moments, the M-step updates
\[
\theta^{(t+1)} = \arg\max_\theta \left\{ \sum_{j=1}^N \int \mathcal{N}(x_j; \bar{x}_j^{(t)}, 1/\eta^{(t)}) \log p(x_j; \theta_x)\, dx_j
+ \sum_{i=1}^M \int \mathcal{N}(z_i; \bar{z}_i^{(t)}, 1/\zeta^{(t)}) \log p(y_i|z_i; \theta_z)\, dz_i \right\},
\]
thereby yielding an empirical-Bayes variant of GVAMP [1806.10079].

A second line concerns bilinear models. Bilinear Adaptive GVAMP (BAd-GVAMP) addresses measurements
\[
\mathbf{Y} \sim p(\mathbf{Y}\mid \mathbf{Z}) = \prod_{i,j} p(Y_{ij}\mid Z_{ij}),\qquad
\mathbf{Z} = \mathbf{A}(\mathbf{b})\, \mathbf{X},\qquad
\mathbf{A}(\mathbf{b}) = \mathbf{A}_0 + \sum_{k=1}^{G} b_k \mathbf{A}_k,
\]
and decomposes inference into a componentwise MMSE output module on \(\mathbf{Z}\) plus a BAd-VAMP module on a pseudo-linear observation, with EM updates for \(\boldsymbol{\theta}_X\), \(\boldsymbol{\theta}_A\), and \(\boldsymbol{\theta}_Y\) [1810.08129]. When \(\mathbf{A}(\mathbf{b})\) is fixed, the model collapses to GVAMP on the GLM; when the output channel is linear Gaussian, BAd-GVAMP reduces to BAd-VAMP [1810.08129].

A third line concerns long-memory processing. GMAMP generalizes AMP/VAMP/GVAMP by allowing arbitrary-length memory in all local processors and imposing stricter orthogonality: the current output error must be orthogonal to the entire memory of input errors. In this framework, choosing memory lengths equal to \(1\) and selecting LMMSE polynomial coefficients recovers GVAMP exactly [2110.06069]. The same paper constructs a low-complexity Bayes-optimal GMAMP with memory linear recursions
\[
B = \lambda^\dagger I - AA^{\rm H},\qquad
\hat{z}_t = \theta_t\,B\,\hat{z}_{t-1}+\xi_t\,(z_t-Ax_t),\qquad
\hat{x}_t = A^{\rm H}\hat{z}_t,
\]
thereby replacing the inverse-based GVAMP linear step with matrix-vector recursions of \(O(MN)\) complexity per iteration [2110.06069].

A fourth line concerns coding and achievable rates. For generalized linear systems with unitarily invariant matrices, GOAMP/GVAMP has been analyzed through a variational state evolution obtained by transforming the fully unfolded state evolution into an equivalent single-input single-output recursion. Using that reduction and the I-MMSE lemma, the literature derives a maximum achievable rate and an optimal coding principle, then designs LDPC codes with reported gains of \(0.8\sim 2.8\) dB over existing methods [2305.07870]. This suggests that GVAMP is not only a detector but also a basis for end-to-end coded-system design.

## 5. Applications and empirical behavior

GVAMP has been used across a broad range of GLM tasks. The original GLM-VAMP paper emphasizes robust regression, binary classification, quantized compressed sensing, phase retrieval, photon-limited imaging, and inference from neural spike trains as motivating examples [1612.01186]. In one-bit compressed sensing experiments with \(N=512\), \(M=2048\), 16 nonzeros, and SNR \(=40\) dB, GLM-VAMP maintained accurate recovery up to \(\kappa(A)=10^6\), whereas adaptively damped GAMP was accurate for \(\kappa(A)<10^3\) and degraded for larger condition numbers; GLM-VAMP converged in approximately \(10\)–\(20\) iterations with weak dependence on \(\kappa\), while AD-GAMP required approximately \(30\)–\(40\) iterations when it converged [1612.01186].

EM-GVAMP was demonstrated on phase retrieval with unknown measurement-noise variance. For the complex model \(y_i = |z_i + w_i|\), \(w_i \sim \mathcal{CN}(0,\nu_w)\), the paper derives an EM update for \(\nu_w\) using the GVAMP Gaussian posterior approximation and reports accurate estimation of unknown \(\nu_w\) on large problems with \(A\in\mathbb{C}^{8192\times 1024}\) having IID circular Gaussian entries [1806.10079].

BAd-GVAMP extends these ideas to generalized bilinear recovery. The reported applications include quantized compressed sensing with matrix uncertainty, blind self-calibration from quantized measurements, and structured dictionary learning from quantized measurements. In the experiments summarized in the supplied material, BAd-GVAMP converged in approximately \(20\)–\(30\) outer iterations across tested scenarios, achieved near-oracle behavior in quantized compressed sensing, and remained effective under both 1-bit and multi-bit quantization [1810.08129].

More recent work places GVAMP in high-rate communication systems. For bandlimited direct-detection optical fiber channels, a GVAMP detector combined with multi-level coding and successive interference cancellation was reported to operate within about \(0.3\) bits per channel use of the real-alphabet coherent capacity for optically amplified links, to improve the best existing theory-based gap of \(1\) bpcu, and to deliver approximately \(6\) dB power-efficiency gain for bipolar over unipolar modulation in amplified links and \(3\) dB in unamplified links. In that setting, the receiver required approximately \(38\) iterations to achieve \(5\) bpcu with \(80\) multiplications per information bit [2508.01438].

These results also clarify the empirical role of GVAMP’s assumptions. When the matrix model matches the right-rotational or unitary-invariance setting, GVAMP combines competitive accuracy with unusually strong robustness to ill-conditioning; when the model moves into bilinear, adaptive, or coded regimes, the literature tends to preserve GVAMP’s extrinsic-message structure while modifying the modules around it [1612.01186][1810.08129].

## 6. Limitations, misconceptions, and related methods

GVAMP is often described informally as a “general-matrix GAMP,” but that formulation is too broad. Its strongest guarantees rely on right-rotationally invariant or unitarily invariant \(A\), separable priors and output channels, and module regularity such as Lipschitz continuity and well-behaved derivatives [1610.03082][2110.06069]. It is therefore more accurate to view GVAMP as a rotational-invariance counterpart to GAMP than as a universally robust GLM solver.

A related misconception is that the original GLM-VAMP paper already supplied a full rigorous state-evolution theory for the generalized model. It did not: that paper explicitly deferred rigorous justification of the GLM extension, even while showing strong numerical robustness [1612.01186]. Rigorous scalar state evolution is classical for VAMP in the AWGN linear model, and later papers extend asymptotic analysis in several directions, but the theoretical development proceeded incrementally rather than all at once [1610.03082][2001.02824].

Matrix structure remains a practical fault line. A unitary-transform alternative, GUAMP, was proposed for general measurement matrices, in particular highly correlated matrices. In quantized compressed sensing with doubly correlated \(A\), the reported numerical evidence shows that GAMP diverges for moderate correlation \(\rho>0.05\), GVAMP begins to diverge in the same regime, and both GAMP and GVAMP fail completely at larger \(\rho\), such as \(\rho=0.35\), while GUAMP remains stable and accurate [2210.08861]. This does not negate GVAMP’s value; it delineates the boundary of the variance-tracking mechanism on which GVAMP depends.

Complexity is the second major limitation. The inverse-based LMMSE step that gives GVAMP its robustness is also its bottleneck, with \(O(M^3)\) direct-inversion cost per iteration in the unitarily invariant formulation [2110.06069]. Low-complexity descendants such as BO-GMAMP aim to preserve GVAMP’s fixed points while replacing inversions by memory recursions and matrix-vector products. A plausible implication is that much of the later AMP literature is best read as an effort to retain GVAMP’s extrinsic-information geometry while weakening its matrix assumptions or reducing its linear-module cost.

In the broader AMP taxonomy, AMP and GAMP remain attractive for IID Gaussian \(A\) because of simplicity and low cost; VAMP and GVAMP trade additional linear-algebra structure for greater robustness under rotational invariance; OAMP is closely related through orthogonality-based design; GMAMP adds explicit memory; EM-GVAMP addresses unknown parameters; and bilinear extensions such as BAd-GVAMP and BiG-VAMP transport the same message-passing logic to structured matrix-factorization problems [1612.01186][2110.06069]. Within that lineage, GVAMP occupies the pivotal position at which VAMP’s rotational-invariance machinery first meets the full generalized-linear observation model.

Source: https://www.emergentmind.com/topics/generalized-vector-approximate-message-passing-gvamp