Stop Marginalizing My Dreams: Model Inversion via Laplace Kernel for Continual Learning

Abstract: Data-free continual learning (DFCIL) relies on model inversion to synthesize pseudo-samples and mitigate catastrophic forgetting. However, existing inversion methods are fundamentally limited by a simplifying assumption: they model feature distributions using diagonal covariance, effectively ignoring correlations that define the geometry of learned representations. As a result, synthesized samples often lack fidelity, limiting knowledge retention. In this work, we show that modeling feature dependencies is a key ingredient for effective DFCIL. We introduce REMIX, a structured covariance modeling framework that enables scalable full-covariance modeling without the prohibitive cost of dense matrix inversion and log-determinant computation. By leveraging a Laplace kernel parameterization, REMIX captures structured feature dependencies using memory that scales linearly with the feature dimensionality, while requiring only an additional logarithmic factor in computation. Modeling these correlations produces more coherent synthetic samples and consistently improves performance across standard DFCIL benchmarks. Our results demonstrate that moving beyond diagonal assumptions is essential for effective and scalable data-free continual learning. Our code is available at https://github. com/pkrukowski1/REMIX-Model-Inversion-via-Laplace-Kernel.

• The paper demonstrates that full-covariance modeling via a Laplace kernel significantly improves synthetic sample fidelity and knowledge retention in continual learning.
• It introduces REMIX, which efficiently parameterizes covariance matrices with learnable channel positions, reducing computational costs while capturing structured dependencies.
• Experimental results show statistically significant gains in incremental accuracy on benchmarks like CIFAR-100 and Tiny-ImageNet across CNN and Transformer models.

Structured Covariance Modeling for Data-Free Continual Learning: An Expert Overview of “Stop Marginalizing My Dreams: Model Inversion via Laplace Kernel for Continual Learning”

Motivation and Limitations of Prior DFCIL Inversion Approaches

Data-free class-incremental learning (DFCIL) targets the challenge of continual learning under stringent data privacy constraints, where original training data cannot be stored, compelling frameworks to rely on synthetic sample generation via model inversion. Existing model inversion methods, such as DeepInversion and PMI, critically employ diagonal covariance assumptions, presuming complete independence among feature dimensions within stored distribution statistics. This marks a significant simplification; deep representations learned by neural networks display structured and high-order dependencies across feature channels and spatial locations. Diagonalization marginalizes these spatial and inter-channel correlations, which severely constrains synthetic sample fidelity, leading to information loss and knowledge decay across sequential tasks.

Figure 1: The shared feature extraction pipeline contrasting diagonal and full-covariance modeling in DFCIL; PMI's diagonal assumptions ignore all inter-feature correlations, while the proposed Laplace Covariance Model (LCM) captures full covariance structure and spatial dependencies.

Empirical observations demonstrate suboptimal preservation of previously acquired knowledge and reduced representation faithfulness under diagonal modeling. There exists a clear need for scalable methods that explicitly capture and utilize structured correlation within high-dimensional learned representations for robust pseudo-sample synthesis.

REMIX: Laplace Kernel-Based Structured Covariance Estimator

The REMIX framework addresses the core limitation of previous inversion-based DFCIL methods through a novel full-covariance modeling strategy. By parameterizing covariance matrices via a Laplace kernel, REMIX enables the retention of local feature dependencies while maintaining tractable complexity. This method is formalized as the Laplace Covariance Model (LCM), which assigns a one-dimensional latent coordinate per feature channel and defines pairwise correlations as $K_{ij} = \exp(-|a_i - a_j|)$, where $a_i$ are learnable channel positions. A learnable per-channel scaling and a residual diagonal noise term complete the covariance parameterization:

Figure 2: Construction of the REMIX covariance matrix $\Sigma \in \mathbb{R}^{C \times C}$ using $\mathcal{O}(C)$ parameters, encoding local dependencies through learned channel distances in latent space.

This induces a dense yet structured covariance with a corresponding tridiagonal precision matrix, enabling efficient quadratic forms, log-determinants, and inverses in $\mathcal{O}(C \log C)$ time with strictly linear storage. REMIX thereby resolves the prohibitive computational and memory costs of standard full-covariance Gaussian models, making structured modeling feasible even for $D \sim 10^5 - 10^6$.

Likelihood-based statistical matching, enabled by this covariance model, allows pseudo-samples to be explicitly optimized under the true global feature distribution, rather than collapsing to marginally independent BatchNorm statistics.

Model Inversion Pipeline and Integration in Continual Learning

REMIX integrates into data-free continual learning as follows:

• Statistics Aggregation: For each new task, LCM parameters (diagonal variance $\mathbf{d}$, scaling $\mathbf{w}$, latent positions $\mathbf{a}$) are fitted layer-wise to the empirical feature covariance.
• Covariance Aggregation Across Tasks: The accumulated covariance integrates new classes through weighted averaging, followed by re-fitting LCM parameters.
• Inversion Objective: During sample synthesis, generated features are regularized by the exact Gaussian log-likelihood under the LCM, with the precision matrix’s tridiagonal structure providing computational efficiency.

The framework adapts seamlessly to both convolutional (using BatchNorm statistics) and transformer-based (e.g., ViT) architectures (using explicit feature recording), and does not require architectural changes or protocol modifications from existing strong baselines such as PMI. Empirical results evidence that this approach enables superior pseudo-sample fidelity and improved knowledge retention.

Figure 3: Qualitative comparison of synthetic samples using diagonal vs. full-covariance modeling on ResNet-34 and ViT-B/16 architectures; structured dependency modeling yields semantically coherent samples with robust object structure.

Experimental Evidence and Ablation

REMIX achieves state-of-the-art retention and incremental accuracy across CIFAR-100, Tiny-ImageNet, and CUB-200 on both ResNet and CLIP/ViT backbones. On CIFAR-100 under a 10-task split, REMIX surpasses PMI by +1.48 percentage points in average incremental accuracy, and by up to +1.44 points on Tiny-ImageNet (Table results described in the paper). Gains are even more pronounced in high-task-count regimes, where retention of long-tail knowledge is essential.

Figure 4: Task-wise performance on Tiny-ImageNet (ResNet-32); REMIX improves old-task retention while preserving learning capacity on new tasks.

On CLIP-based architectures with MoE-Adapter, REMIX marginally improves both average and last-task accuracy (+0.12 and +0.31 points on CIFAR-100, respectively), and displays more robust retention on CUB-200 (+0.43).

Qualitative analysis highlights visually significant improvements in pseudo-sample consistency, reduction of artifacts, and enhanced semantic alignment, especially under high-dimensional feature regimes. Log-likelihood analyses demonstrate that LCM modeling consistently yields higher feature likelihoods across all depths of both ResNet and ViT models, quantifying the improved fit to real data statistics.

Figure 5: ViT backbone: Prior diagonal-covariance methods yield localized artifacts; REMIX’s feature-structure modeling confers coherence and global semantic consistency.

Figure 6: ResNet-34 backbone: REMIX enables generation of structurally faithful samples across spatial dimensions, which is unattainable with purely diagonal modeling.

Ablation on the Frobenius regularization $\lambda_{\mathrm{F}}$ confirms stable optimization dynamics and an interpretable plasticity–stability trade-off, with optimal values supporting retention without sacrificing adaptation.

Practical, Theoretical, and Future Implications

This work demonstrates that full-covariance modeling, even under stringent computational budgets, is feasible and essential for representation faithfulness in DFCIL. The core claim—that moving beyond diagonal covariance is necessary for effective, scalable data-free continual learning—is strongly supported both by robust, cross-architecture numerical improvements and qualitative gains in generated pseudo-samples.

From a practical perspective, REMIX provides an immediately applicable module for inversion-based learning pipelines without introducing prohibitive memory or computational burdens. Its approach generalizes across both CNN and Transformer architectures, requiring only minimal adaptation.

Theoretically, this work establishes the critical importance of preserving feature correlations for pseudo-sample generation—a key insight for any generative or inversion-based continual learning algorithm. It also emphasizes the versatility of structured kernel parameterizations (Laplace kernel/1D GMRFs) for high-dimensional Gaussian modeling in deep networks.

Potential future directions include:

• Integrating more flexible kernel parameterizations or learning non-stationary correlation structures.
• Combining REMIX with other generative replay mechanisms or advanced diffusion-based inversion models to further enhance expressivity.
• Investigation of long-range or higher-order feature correlation structures for spatially richer representations.
• Applying REMIX-like covariance modeling to other applications requiring data-free distribution alignment or privacy-preserving sample generation.

Conclusion

REMIX provides a theoretically principled and computationally efficient approach to full-covariance modeling in data-free continual learning, eliminating the independence assumptions of prior methods and empirically enhancing both the retention of historical knowledge and the quality of synthetic sample generation. Its success motivates further investigation into advanced structured modeling of deep representations under continual and privacy-constrained machine learning paradigms.