Joint Learning of Directional Basis & Reconstructor

• The paper presents a joint learning framework where both a directional basis and its reconstructor are optimized simultaneously to enhance semantic manipulation.
• The methodology employs end-to-end training with adversarial constraints and innovative gradient checkpointing to manage VRAM usage efficiently.
• Empirical evaluations demonstrate improved reconstructor accuracy and sample fidelity on diffusion model benchmarks, contrasting with fixed operator techniques.

Joint learning of directional basis and reconstructor is a paradigm in which a representation encoding directional or semantic shifts (the "basis") is learned in tandem with a network or algorithm that can recognize, quantify, or reconstruct the effect of those shifts (the "reconstructor"), with end-to-end training and mutual supervision. This approach has emerged recently in the context of semantic manipulation in pre-trained diffusion models, and stands in contrast to earlier frameworks where directionality is imposed via fixed operators computed a priori, as in tomographic reconstruction and morphological decomposition. Modern joint learning frameworks are characterized by simultaneous optimization of both basis and reconstructor, often with additional adversarial constraints to maintain sample fidelity.

1. Definitions and Historical Context

The joint learning framework is exemplified in the unsupervised discovery of interpretable directions in the h-space of pre-trained diffusion models (Zhang et al., 2023). Here, a shift control module parameterizes a basis of directions in latent feature space $\{h_t\}$ at each diffusion step. Unlike static approaches, both the basis and the reconstructor are optimized together, enabling the system to adaptively discover semantically meaningful directions that support scalable manipulation and robust inversion.

In classical tomographic decomposition (Kongskov et al., 2017), directionality is encoded via hand-engineered operators, specifically rotated finite-difference matrices $R_\theta$, with the main angle $\theta$ estimated via a global scan before optimization. No "learning" of the directional basis occurs during reconstruction.

2. Directional Basis: Construction and Learning

In the diffusion model framework, at each timestep $t$, the latent feature $h_t$ is subjected to a learnable shift: $\hat{h}_t = h_t + s\cdot\Delta h_t^k$ where $\Delta h_t^k$ is the $k$-th column of a basis matrix $B_t \in \mathbb{R}^{d\times K}$ produced by a parameterized shift control module $f_t^\varphi$, and $R_\theta$0 samples the shift magnitude. The basis is realized via 1x1 convolutions with $R_\theta$1 heads, with end-to-end gradients flowing from the reconstructor and adversarial losses back to $R_\theta$2 (Zhang et al., 2023).

In contrast, in tomographic decomposition, the basis is specified by rotated finite-difference operators: $R_\theta$3 and is fixed after initial estimation of $R_\theta$4 using an FFT-based algorithm (Kongskov et al., 2017).

3. Reconstrutor Design and Objective

The reconstructor $R_\theta$5 is a neural network module that, given the original and manipulated images $R_\theta$6, predicts the applied direction index $R_\theta$7 (classification) and shift magnitude $R_\theta$8 (regression): $R_\theta$9 where $\theta$0, $\theta$1 (Zhang et al., 2023). The corresponding loss is

$\theta$2

In tomographic decomposition, there is no explicit reconstructor; the estimation of components $\theta$3 (fibres) and $\theta$4 (cracks) proceeds via convex optimization of the variational objective, with all directionality held fixed (Kongskov et al., 2017).

4. Joint Optimization and Training Dynamics

The key innovation in the diffusion-based approach is joint optimization of the basis and reconstructor. The total loss combines the reconstructor loss $\theta$5, an adversarial loss $\theta$6 to preserve data fidelity using discriminator $\theta$7, and regularization: $\theta$8 Optimization proceeds by freezing the pre-trained diffusion model, and updating only $\theta$9 (basis) and $t$0 (reconstructor) in a two-player min-max game with the discriminator $t$1. This results in learned directions that are both semantically disentangled and interpretable, while shifts remain close to the data manifold (Zhang et al., 2023).

In the classical tomographic setting, after estimating directionality from data, optimization relies on convex composite methods (primal–dual splitting, Chambolle–Pock/PDHG) where the basis remains fixed throughout (Kongskov et al., 2017).

5. Architectural and Computational Implementation

The shift control module $t$2 is implemented as a 1x1 convolutional block with $t$3 heads per timestep, projecting $t$4 into the $t$5 basis vectors. Training is stabilized by adversarial initialization (heads at zero) so that initial manipulations do not perturb outputs (Zhang et al., 2023). The reconstructor takes image pairs as input, using architectures such as Siamese/concatenated LeNet for small datasets or ResNet-18 for higher-resolution datasets.

Due to the iterative nature of diffusion models, memory consumption is a challenge. A novel two-pass, node-by-node gradient checkpointing scheme is proposed to reduce VRAM requirements from $t$6 to $t$7 per sample, at the cost of doubling computational overhead. This scheme stores only final activations of each reverse step, recomputing necessary steps during backpropagation, enabling feasible multi-step training (Zhang et al., 2023).

In tomographic decomposition, the ASTRA toolbox is used for efficient GPU-based Radon transform operations, enabling primal–dual methods to scale to high-resolution data (Kongskov et al., 2017).

6. Empirical Evaluation and Metrics

Empirical validation on standard benchmarks (MNIST32, AnimeFaces64, CelebAHQ128, AFHQ-dog128) demonstrates that the jointly optimized system achieves high reconstructor classification accuracy (RCA = [0.77, 0.85, 0.93, 0.87]) and moderately positive mean opinion scores (MOS = [0.52, 0.36, 0.28, 0.31]), while the fidelity (measured by FID) remains close to the unshifted baseline (Zhang et al., 2023). Memory benchmarking shows constant VRAM usage up to trajectory lengths of $t$8, with nearly the same throughput as vanilla backpropagation.

In tomographic fibre/crack decomposition, the proposed joint variational models yield robust separation of orthogonal structural components, as observed numerically and in real-data reconstruction figures. The FFT-based direction estimation reliably identifies main object directions for robust initialization (Kongskov et al., 2017).

Metric	Dataset	Joint Basis+Reconstructor (Zhang et al., 2023)	Tomographic Decomposition (Kongskov et al., 2017)
Classification Acc	MNIST32–AFHQ128	0.77–0.93	Not applicable
FID (shifted)	Anime64	9.21	Not applicable
Decomposition	Fibre/crack RMSE	Not applicable	Robust, visually interpretable

7. Limitations and Differentiators

In (Zhang et al., 2023), end-to-end joint learning is central: if either the basis learning or reconstructor loss is ablated, degenerate or trivial solutions arise (e.g., all directions collapse to identity or to fixed noise patterns). Maintaining high mutual information between direction indices and output samples is critical, reminiscent of InfoGAN's regularization role. The discriminator ensures learned directions remain non-destructive.

In (Kongskov et al., 2017), no joint learning of the directional basis is performed. Directional operators are estimated once using FFT and then frozen, with optimization focused solely on the reconstruction variables. All directionality is imposed externally, and no classifier or regressor for directionality is present.

A plausible implication is that joint end-to-end learning of direction and recognizer (as in diffusion h-space) facilitates the unsupervised emergence of global and semantically meaningful directions that are robustly quantifiable, which is not generally achievable with fixed or a priori direction encoding schemes.

---

For further details and implementation specifics, see "Tomographic Reconstruction Methods for Decomposing Directional Components" (Kongskov et al., 2017) and "Unsupervised Discovery of Interpretable Directions in h-space of Pre-trained Diffusion Models" (Zhang et al., 2023).