Eliminate the inverse in the inner KL term of the alternative inverse-free SVGP bound

Develop an inverse-free optimisation scheme that removes the dependence on K_uu^{-1} from the inner Kullback–Leibler term KL[N(0, R R^T) || N(0, K_uu^{-1})] in the alternative inverse-free bound derived from the marginal SVGP parameterisation with m = K_uu R \tilde{m} and S = K_uu R \tilde{S} R^T K_uu. Demonstrate that the matmul-only natural-gradient techniques used to optimise the auxiliary matrix T in the R-SVGP construction can be adapted to this setting so that updates of the model and variational parameters are effectively inverse-free.

Background

Section 2 outlines a construction that generalises the R-SVGP idea to parameterisations beyond L-SVGP by introducing a lower-triangular auxiliary matrix R in the marginal SVGP parameterisation. This yields an ELBO whose predictive variances and most terms avoid matrix inversions.

However, the resulting Kullback–Leibler term still includes an inner KL divergence between zero-mean Gaussians with covariances R R^T and K_uu^{-1}, which introduces an explicit dependence on the inverse of K_uu. The authors conjecture that their matmul-only natural-gradient approach for the auxiliary T in R-SVGP could be extended to eliminate this remaining inverse, making all required updates inverse-free.