Susceptibilities and Patterning: A Primer on Linear Response in Bayesian Learning

Abstract: These notes introduce the theory of susceptibilities as developed in [arXiv:2504.18274, arXiv:2601.12703] for interpreting neural networks. The susceptibility of an observable $φ$ to a data perturbation is defined as a derivative of a posterior expectation, which by the fluctuation--dissipation theorem equals a posterior covariance. Different choices of $φ$ yield different objects: per-sample losses give the influence matrix (the Bayesian influence function of [arXiv:2509.26544]), while component-localized observables give the structural susceptibility matrix that pairs model components with data patterns. The susceptibility matrix is (up to a factor of $nβ$) the Jacobian of the map from data distributions to structural coordinates; its pseudo-inverse provides a linearized solution to the patterning problem of [arXiv:2601.13548]: finding data perturbations that produce a desired structural change. We motivate the theory from its statistical-mechanical foundations, then give a detailed exposition of susceptibilities, their empirical estimators, and their connection to the geometry of the loss landscape.

• The paper develops a novel susceptibility framework that uses covariance-based linear response to probe Bayesian learning dynamics.
• It leverages a statistical mechanics analogy to derive influence matrices and structural susceptibilities for targeted data interventions.
• The approach remains effective in singular settings, enabling principled interpretability and model steering through empirical estimators.

Susceptibilities and Linear Response in Bayesian Learning: A Technical Exposition

Statistical Mechanics Framework and Bayesian Analogy

The paper develops a formal framework for susceptibilities in Bayesian learning through a deep analogy to statistical mechanics. In statistical mechanics, a system is described by a configuration space and a Hamiltonian, with statistical properties derived from the Boltzmann distribution. Perturbations of the Hamiltonian yield susceptibilities, which are captured as derivatives of expectation values—interpreted through the fluctuation-dissipation theorem (FDT)—with responses characterized by covariances.

This formalism is ported to Bayesian learning via the identification of the parameter space as configuration space, the (population) loss as an energy function, and the posterior as an annealed Boltzmann distribution. Perturbations of the data distribution induce a family of posteriors, and susceptibilities characterize how posterior expectations of observables change under such perturbations. This importation of thermodynamic structure enables principled and computable probes of model internals in Bayesian neural networks.

Definition and Computation of Susceptibilities

A susceptibility is defined for any observable $\varphi$ as the first-order response of its posterior expectation to a data-distribution perturbation. Utilizing the FDT, this derivative is shown to be a covariance:

$$\chi(\varphi; q') = -\mathrm{Cov}[\varphi, \Delta L]$$

where $\Delta L$ is the data-directional perturbation of the loss.

Special choices of observable and perturbation yield two key empirical objects:

• Influence Matrix: With per-sample loss observables, the susceptibility matrix recovers the Bayesian influence function, quantifying the functional coupling between data points ($-\mathrm{Cov}[l_{z'}, l_z - L]$), thus providing a Hessian-free, posterior-covariance-based generalization of classical influence functions.
• Structural Susceptibility Matrix: With component-localized observables, the susceptibility matrix pairs model components (e.g., attention heads, layers) with data patterns, indicating which data perturbations affect which structural components most.

This covariance-based definition is both general—the same construction applies regardless of singularities in the loss landscape—and practically computable, as the covariance may be estimated via posterior sampling approximation (e.g., SGLD). The paper argues that this formulation supersedes traditional influence-function-based approaches, especially in the generic singular setting of deep neural networks.

Geometric and Asymptotic Structure

The analysis unveils the geometric content of susceptibilities by Taylor-expanding the loss landscape around minima. In the regular case (nondegenerate Hessian), Laplace expansion shows that leading-order asymptotics of susceptibilities correspond to influence functions involving the inverse Hessian. Flat directions in the loss surface dominate the susceptibility structure, and higher-order Taylor coefficients (cubic and beyond) are probed using observables with higher vanishing order at the minimum. For singular models—ubiquitous in overparameterized networks—singular learning theory is invoked. Here, the effective dimension $d/2$ is replaced by the real log canonical threshold (RLCT) $\lambda$, governing the leading-order Bayesian asymptotics and the local geometry of the vanishing set of the loss.

Notably, the susceptibility formalism, defined through covariances, remains operational even when standard notions of curvature break down (i.e., when the Hessian is degenerate). The framework generalizes to arbitrary distributional observables and directly connects the local learning coefficient to the RLCT, which has direct observable effect on asymptotic generalization error.

Patterning: The Dual Problem of Interpretability

A central contribution is the duality between interpretability and patterning:

• Interpretability (Forward Problem): The susceptibility matrix describes how model structural coordinates (expectation values of selected observables) respond to infinitesimal data distribution perturbations. Its singular value decomposition exposes principal directions in both data and component space, enabling structured and interpretable probing of model internals.
• Patterning (Inverse Problem): For any desired first-order change in structural coordinates, the Moore-Penrose pseudo-inverse of the susceptibility matrix prescribes the minimal-norm change in the data distribution required to realize the desired internal structural change. This yields a practical recipe for targeted data reweighting (batch reweighting), invertible via the SVD of the susceptibility matrix.

This duality promotes a rigorous, linear-response-based approach to model steering, where interpretability and targeted modification are two sides of the same functional relationship.

Empirical Estimation and Implementation

The paper articulates the transition from population-level theory to empirical estimation. Susceptibilities defined under the population posterior and true data distribution are estimated empirically by replacing both with their empirical counterparts. The empirical susceptibility estimator is rigorously justified both as an exact per-sample weight derivative of the empirical posterior and as a consistent estimator for the population susceptibility in the asymptotic regime (large $n$, nontrivial $\beta$). The connection between theoretical covariance-based susceptibilities and practical empirical estimators (with centering and standardization) is clarified.

For component observables, empirical estimation requires weight-restricted posterior sampling—sampling component parameters with other parameters clamped at their reference values. The standardization procedures (column z-scoring and row centering) are shown to absorb both scale and normalization indeterminacies, ensuring consistent interpretation across components.

Batch reweighting operationalizes patterning in practice by realizing theoretical data distribution shifts as per-sample loss reweightings. The pseudo-inverse amplification of small singular values is addressed by ridge regularization, and empirical studies confirm the practical viability and effectiveness of patterning for structured interventions in neural networks.

Implications and Future Directions

By providing a unified, statistically principled framework for probing and manipulating the internal structure of Bayesian neural networks, the susceptibility formalism advances both theoretical understanding and practical interpretability. The approach fosters:

• Systematic construction of interpretable axes in model space,
• Data-driven identification of functionally specialized model components,
• Targeted interventions to modify or steer internal computation via data manipulation,
• Direct connections to the geometric and asymptotic theory of learning via the RLCT.

Future developments may explore advanced applications in model debiasing, adversarial robustness, fine-tuning through patterning, and deeper connections to the algebraic geometry of deep learning landscapes. The continuous generalization to singular settings and unresolved compositional structures, as well as scalable implementations in large-scale deep networks, remain compelling avenues for theoretical and empirical research.

Conclusion

The paper establishes a rigorous theory of susceptibilities in Bayesian learning, importing and adapting linear response theory from statistical mechanics and grounding it in singular learning theory. By systematically using posterior covariances to characterize internal structure and enabling both interpretability and targeted interventions, this framework provides a powerful, geometrically-motivated toolkit for both theorists and practitioners seeking principled understanding and control over complex neural systems (2605.07980).