SPN-Guided Latent Space Manipulation

• The paper introduces a method where an SPN models both class posteriors and latent densities, enabling controlled counterfactual search within a VAE’s latent space.
• It integrates a semi-supervised VAE with an SPN to simultaneously guide latent edits towards target labels and maintain plausibility under the learned distribution.
• Empirical results on chest radiograph data show that SPN guidance improves counterfactual validity and image quality compared to traditional MLP-based approaches under weak regularization.

Searching arXiv for the cited SPN-guided paper and closely related latent-manipulation work. SPN-guided latent space manipulation denotes a class of methods in which edits are performed in a learned latent representation while a sum-product network (SPN) supplies structured guidance over that latent space. In the clearest explicit formulation, the SPN is fit over the joint latent-label distribution of a semi-supervised variational autoencoder (VAE), so that latent optimization is driven simultaneously by target-class attainment and by plausibility under the learned latent distribution (Siekiera et al., 25 Jul 2025). In adjacent latent-editing literature, closely related goals are pursued with other guidance sources rather than an SPN—attention-derived priors, sparse user constraints, weakly induced semantic directions, or anchored concept subspaces—but these systems share the objective of manipulating selected semantic factors while preserving non-target content and realism (Xu et al., 26 May 2025, Li et al., 2021).

1. Probabilistic definition and latent-space setting

The explicit SPN-guided formulation is presented for counterfactual explanations in medical imaging. The task is to generate a nearby alternative image that would be assigned a different class, while satisfying validity, similarity or minimality, plausibility, and interpretability. Rather than optimizing in image space, the method performs search in the latent space of a VAE, where small moves are expected to decode to more plausible image changes (Siekiera et al., 25 Jul 2025).

The representation-learning stage uses a semi-supervised VAE with latent variable $z$, reconstruction model $p_\theta(x\mid z)$, encoder $q_\phi(z\mid x)$, and classifier $q_{\phi_1}(y\mid z)$. The loss is

$$\begin{aligned} \mathcal{L} = \;& \beta_0 \cdot \Big(-\mathbb{E}_{q_\phi(z\mid x)}[\log p_\theta(x\mid z)]\Big) \ &+ \beta_1 \cdot D_{KL}\!\big(q_\phi(z\mid x)\,\|\,p(z)\big) \ &+ \beta_2 \cdot \mathcal{H}\big(q_{\phi_1}(y\mid z)\big). \end{aligned} \tag{1}$$

After VAE training, the latent classifier MLP is replaced by an SPN learned over the joint representation $(z,y)$, using LearnSPN and Gaussian leaves for the latent variables. The SPN is therefore both a classifier and a latent-space density model. Its class posterior is written as

$p_{\text{spn}(Y=y_0 \mid Z_0,\ldots,Z_{N-1}) = \frac{ p_{\text{spn}(Z_0,\ldots,Z_{N-1}\mid Y=y_0)\,p_{\text{spn}(Y=y_0) }{ \sum_k p_{\text{spn}(Z_0,\ldots,Z_{N-1}\mid Y=y_k)\,p_{\text{spn}(Y=y_k) }. \tag{2}$

The SPN structure obeys the standard validity constraints of completeness and decomposability, which ensure tractable exact inference. In this setting, “SPN-guided” does not merely mean that an SPN predicts labels from latent codes; it means that the same model supplies both $p_{\text{spn}}(y\mid z)$ and a likelihood term over $z$, allowing latent editing to be constrained by class consistency and by latent plausibility within one probabilistic object (Siekiera et al., 25 Jul 2025).

2. Optimization as SPN-guided counterfactual search

Counterfactual generation is formulated as continuous latent optimization. For an original latent code $z$, a target label $p_\theta(x\mid z)$0, and a candidate latent point $p_\theta(x\mid z)$1, the general objective is

$p_\theta(x\mid z)$2

In the SPN-guided version this becomes

$p_\theta(x\mid z)$3

The three terms have distinct roles. The posterior term drives the latent point toward the target class. The quadratic penalty $p_\theta(x\mid z)$4 enforces closeness or minimality. The likelihood-difference term $p_\theta(x\mid z)$5 discourages moves into latent regions whose plausibility differs strongly from that of the original sample (Siekiera et al., 25 Jul 2025).

Operationally, the encoder produces latent samples for an input image, the target label is specified, and the initial counterfactual code is set to the original sample. Optimization then proceeds directly on the latent variables by SGD, with all latent variables manipulated simultaneously. The SPN is converted into a TensorFlow model so that the objective can be differentiated by backpropagation. After optimization, each optimized latent sample is decoded, and the final counterfactual image is averaged across replicate samples:

$p_\theta(x\mid z)$6

The same averaging is used for the reconstruction of the original image, and the difference $p_\theta(x\mid z)$7 is visualized as a heatmap-like explanation. This procedure is neither discrete SPN inference nor latent direction arithmetic; it is continuous latent search in which the SPN acts as a differentiable guidance model (Siekiera et al., 25 Jul 2025).

3. Empirical instantiation in medical imaging

The explicit SPN-guided study uses the CheXpert chest-radiograph dataset in a balanced binary setting with only cardiomegaly and no finding. Images are preprocessed with CLAHE and resized to $p_\theta(x\mid z)$8. The reported split sizes are approximately 21,572 training, 5,532 validation, and 6,896 test, with patient-level separation and 3-fold cross-validation within the training portion (Siekiera et al., 25 Jul 2025).

The VAE has a CNN/ResNet-style encoder and decoder with 7 layers, filters $p_\theta(x\mid z)$9, ReLU activations, Gaussian noise in each layer with $q_\phi(z\mid x)$0 and $q_\phi(z\mid x)$1, latent dimension 62, Adam with learning rate $q_\phi(z\mid x)$2, batch size 50, and 100 epochs. The SPN is learned afterward over $q_\phi(z\mid x)$3 using SPFlow and LearnSPN, then converted into TensorFlow 2 for differentiable optimization (Siekiera et al., 25 Jul 2025).

The evaluation emphasizes validity, proximity, and plausibility. Validity is the prediction flip rate,

$q_\phi(z\mid x)$4

while plausibility is measured with FID and proximity with image-space L2 distance. The study varies the KL weight $q_\phi(z\mid x)$5, exposing a strong regularization trade-off.

$q_\phi(z\mid x)$6	MLP validity	SPN validity
0.1	1.00	1.00
0.01	0.09	0.51–0.56
0.001	0.08	0.86–0.89
0.0001	0.03	0.43–0.55

These results show that the SPN is most useful when latent regularization is weak. Under weak regularization, the MLP baseline often remains a stronger classifier in terms of accuracy or entropy, but its counterfactual manipulability collapses. By contrast, the SPN retains much higher validity because it contributes an explicit latent likelihood term in addition to class posterior guidance (Siekiera et al., 25 Jul 2025).

The same trade-off appears in image quality metrics. Strong regularization at $q_\phi(z\mid x)$7 gives perfect validity for both methods, but with larger changes, such as L2 around 25.5 and FID around 394–398. Weak regularization improves reconstruction and lowers L2 and FID, but severely harms MLP validity. The SPN partly stabilizes this regime. Qualitatively, the SPN explanations consistently focus on the heart region across all regularization settings, whereas MLP explanations are convincing mainly when KL regularization is strong (Siekiera et al., 25 Jul 2025).

4. Functional role of the SPN

Within this framework, the SPN has a dual role. As a classifier, it supplies $q_\phi(z\mid x)$8, which drives the counterfactual toward the target decision boundary. As a density model or descriptor of the latent space, it supplies $q_\phi(z\mid x)$9 or class-conditional latent structure, which provides a plausibility signal that a pure neural classifier does not naturally furnish (Siekiera et al., 25 Jul 2025).

This dual role resolves a common tension in latent editing. A neural latent classifier can produce strong discrimination while offering no explicit notion of whether a candidate latent point remains near the empirical latent distribution. In the SPN-guided formulation, the target-class force and plausibility force are derived from the same joint model over $q_{\phi_1}(y\mid z)$0. That makes the SPN a latent controller in a stronger sense than a post hoc scoring head. It does not merely veto or score edits after they are proposed; it directly shapes the optimization landscape.

A common misconception is that SPN-guided manipulation is simply “classifier-guided latent optimization with an SPN classifier.” The probabilistic construction shows otherwise. The distinctive property is the joint latent-label model and the use of its likelihood structure during optimization. The reported results are consistent with that distinction: the MLP often shows better raw classification performance, but the SPN is the better guide for counterfactual search under weakly regularized latents (Siekiera et al., 25 Jul 2025).

5. Neighboring paradigms without explicit SPNs

Explicit SPN-guided latent manipulation remains relatively specialized. Much of the neighboring literature pursues the same general objective—guided editing of latent or hidden representations—without an SPN. This suggests that the topic sits inside a broader family of guided latent manipulation methods distinguished by the source of guidance rather than by the fact of latent editing itself.

Paper	Guidance source	Explicit SPN
"Counterfactual Explanations in Medical Imaging: Exploring SPN-Guided Latent Space Manipulation" (Siekiera et al., 25 Jul 2025)	Joint latent-label SPN over VAE latents	Yes
"In-Context Brush: Zero-shot Customized Subject Insertion with Context-Aware Latent Space Manipulation" (Xu et al., 26 May 2025)	Attention-derived prompt, subject, and mask priors in MM-DiT	No
"User-Controllable Latent Transformer for StyleGAN Image Layout Editing" (Endo, 2022)	Sparse point-and-vector user constraints with anchor points	No

In diffusion-based image insertion, "In-Context Brush" reformulates customized subject insertion as in-context learning inside a pretrained MM-DiT inpainting network and applies test-time latent manipulation inside transformer attention blocks. Its intra-head latent feature shifting and inter-head attention reweighting act as implicit semantic priors derived from prompt, subject, and mask attention, but the paper explicitly states that it does not introduce or mention an explicit SPN module (Xu et al., 26 May 2025).

In GAN latent editing, "User-Controllable Latent Transformer for StyleGAN Image Layout Editing" uses a transformer encoder-decoder to convert sparse user drags, anchor points, and pseudo-$q_{\phi_1}(y\mid z)$1-direction inputs into residual edits in $q_{\phi_1}(y\mid z)$2. Here the guidance is an external sparse structural prior supplied by the user rather than a probabilistic latent model (Endo, 2022).

Other neighboring formulations substitute different guidance mechanisms for the SPN. "Surrogate Gradient Field for Latent Space Manipulation" learns an auxiliary mapping network $q_{\phi_1}(y\mid z)$3 whose Jacobians induce a target-conditioned latent vector field, enabling multidimensional manipulation by attributes, keypoints, and captions without relying on gradients of $q_{\phi_1}(y\mid z)$4 (Li et al., 2021). "3DLatNav" discovers part-level semantic clusters in a part latent space, transfers them to an object latent space, and uses linear SVM normals as semantic traversal directions for 3D point-cloud generators (Dharmasiri et al., 2022). "Sparse Concept Anchoring" shapes latent geometry during training so that selected concepts occupy fixed directions or axis-aligned subspaces, enabling reversible projection-based suppression and permanent weight ablation with minimal supervision (Fraser et al., 13 Dec 2025). "Semantic and Geometric Unfolding of StyleGAN Latent Space" learns an invertible proxy space $q_{\phi_1}(y\mid z)$5 via normalizing flows so that Euclidean distances better align with perceptual distances and attributes become more linearly separable (Shukor et al., 2021). "Spatial Latent Representations in Generative Adversarial Networks for Image Generation" replaces vector latents with spatial latent spaces such as $q_{\phi_1}(y\mid z)$6 and $q_{\phi_1}(y\mid z)$7, supporting masked spatial mixing and part-aware manipulation on pretrained StyleGAN2 models (Sypetkowski, 2023).

These works do not instantiate SPN-guidance, but they clarify the design space into which SPN-guided methods fit. Guidance may come from an explicit probabilistic latent model, from interactive structural constraints, from attention-derived priors, from weakly supervised semantic subspaces, or from training-time geometric anchoring.

6. Limitations, misconceptions, and plausible extensions

The explicit SPN-guided formulation has several stated limitations. The current empirical evidence is confined to one dataset and one binary CheXpert task. The evaluation metrics are also limited: both L2 and FID are strongly affected by the VAE’s reconstruction quality, so low L2 may partly reflect better reconstruction rather than better counterfactual explanation, and FID compares counterfactuals to the global original distribution rather than class-conditional realism (Siekiera et al., 25 Jul 2025).

The method also relies on a two-stage pipeline rather than end-to-end VAE+SPN training. The latent representation is first shaped by a VAE with an MLP classifier, and only afterward is the SPN fit over $q_{\phi_1}(y\mid z)$8. This keeps the method modular, but it means the latent geometry was not originally optimized under SPN structure. A plausible implication is that tighter integration between probabilistic latent modeling and representation learning could further stabilize counterfactual search, although such a claim is not established in the reported experiments.

Another misconception is that SPN-guided manipulation is the default form of guided latent editing. The surrounding literature shows the opposite. Explicit SPNs are rare; most systems implement guidance through architectural bias, auxiliary maps, user constraints, or learned semantic directions. What distinguishes the SPN-based case is the simultaneous use of tractable latent density and class posterior within the same latent optimization problem.

A second plausible implication, suggested by comparison with neighboring work, is that SPN-guided latent manipulation could benefit from stronger latent geometry. Proxy latent unfolding improves linear separability and perceptual distance alignment in StyleGAN editing (Shukor et al., 2021), while anchored directions and subspaces make inference-time projection and deletion reliable by design (Fraser et al., 13 Dec 2025). Spatial latent fields make region-level structure explicit (Sypetkowski, 2023). None of these papers combines its mechanism with an SPN, but together they indicate that future SPN-guided systems need not be restricted to flat latent vectors or to purely post hoc optimization.

In its current explicit form, SPN-guided latent space manipulation is therefore best understood as a probabilistically constrained latent editing paradigm: a VAE supplies a continuous latent representation, an SPN models its class-conditional geometry, and latent optimization searches for a target-class counterfactual that remains small and plausible under that geometry (Siekiera et al., 25 Jul 2025). The broader literature suggests that this paradigm is one member of a larger research program concerned with how external structure—probabilistic, spatial, semantic, or user-specified—can be converted into controlled movement through latent space.