Double-Factorized Representation

• Double-Factorized Representation is a design pattern that decomposes models along two complementary axes to separate distinct sources of variation while preserving relevant structure.
• It is applied across domains—from latent variable models and 3D graphics to quantum chemistry and operator learning—to improve disentanglement, efficiency, and performance.
• Its effectiveness relies on domain-specific trade-offs, such as balancing support versus density or shared versus private structures, though it may miss higher-order interactions.

“Double-Factorized Representation” does not denote a single universally fixed formalism. In the literature, it names a recurring design pattern in which a model, operator, or algebraic object is decomposed along two complementary axes rather than represented in a single entangled form. Depending on the domain, those axes may be pairwise support versus density factorization, shared versus private topic structure, coordinates versus rendering attributes, space versus time together with modality, view versus illumination, dynamic versus persistent physical responses, or a two-stage factorization of electronic Hamiltonians. Across these settings, the common objective is to separate heterogeneous mechanisms while preserving the structure most relevant to the target task (Roth et al., 2022, Zhang et al., 2013, Sun et al., 2024, Ma et al., 2023, Lee et al., 2022, Tang et al., 15 Jun 2026, Cohn et al., 2021, Stair et al., 2022).

1. Pairwise support factorization in disentanglement

In representation learning, “double-factorized” is used to describe a pairwise factorized support criterion rather than full statistical independence. In “Disentanglement of Correlated Factors via Hausdorff Factorized Support,” the learned representation is $z=(z_1,\dots,z_k)$, and the relevant condition is that for any pair $(i,j)$, the support of the joint distribution over $(z_i,z_j)$ equals the Cartesian product of their marginal supports. This criterion permits arbitrary distributions over the support, including correlations, and is therefore weaker than a factorial distribution $p(z)=\prod_i p(z_i)$, even though independence implies factorized support (Roth et al., 2022).

The paper formalizes this relaxation through the Hausdorff Factorized Support criterion. For a mini-batch $X$ with encoded latent matrix $Z=f_\phi(X)\in\mathbb{R}^{b\times k}$, the observed pair set is $B=Z_{:,(i,j)}$, while the factorized pair set is $A=Z_{:,i}\times Z_{:,j}$. The loss uses the Hausdorff distance

$$d_H(A,B)=\max\left\{\sup_{a\in A}\inf_{b\in B}d(a,b),\sup_{b\in B}\inf_{a\in A}d(a,b)\right\},$$

with Euclidean metric in latent space, and in the relevant setting reduces to the directed distance from $A$ to $(i,j)$0. The pairwise batch approximation is

$(i,j)$1

This penalty is combined with a stochastic autoencoder reconstruction term, and can also be added to $(i,j)$2-VAE, FactorVAE, $(i,j)$3-TCVAE, and DIP-VAE as a support regularizer (Roth et al., 2022).

Empirically, the method is reported to improve disentanglement on Shapes3D, MPI3D, and dSprites, measured primarily by DCI-D. The reported gains include “up to +61% relative DCI-D improvement over standard methods under correlated training,” “+30% relative gains without correlations and over +60% under shared confounders” on Shapes3D when HFS is added to $(i,j)$4-VAE or $(i,j)$5-TCVAE, and “up to +80% relative improvement” for single-entry factor prediction. The paper also reports that lower $(i,j)$6 correlates with higher DCI-D and that downstream classification under correlation shift improves in both accuracy and sample efficiency (Roth et al., 2022).

This usage makes “double-factorized representation” a support-level notion rather than a density-level one. A plausible implication is that the term here emphasizes completeness of pairwise combinations in latent support, not probabilistic independence.

2. Shared–private decompositions in latent variable models

A second usage appears in supervised topic modeling. In “Factorized Topic Models,” the representation is “softly” split into two parts: class-private topics and class-shared topics. The model introduces a prior over topic space so that highly class-correlated topics encode within-class variations, while weakly class-correlated topics encode class-independent variation described as “structured noise.” The topic mixture of a document is written as

$(i,j)$7

where $(i,j)$8 is the private part and $(i,j)$9 the shared part (Zhang et al., 2013).

The partition is driven by an entropy-like statistic over classes for each topic,

$(z_i,z_j)$0

where low $(z_i,z_j)$1 indicates concentration in one class and high $(z_i,z_j)$2 indicates uniformity across classes. The factorizing prior uses

$(z_i,z_j)$3

and an auto-annealing variant replaces this with

$(z_i,z_j)$4

The effect is to encourage topics to become either class-specific or class-unspecific rather than intermediate (Zhang et al., 2013).

The reported empirical behavior is domain-general. On toy object classification, Factorized LDA achieved 81.25% accuracy versus 34.38% for regular LDA with class label and 0% for SLDA. On Reuters 21578 R8, Factorized LDA achieved 83.91% versus 74.63% for regular LDA and 63.75% for SLDA. On natural scene classification, it achieved 84.50% versus 80.50% for regular LDA and 84.00% for SLDA. On KTH action classification, it achieved 65.22% versus 38% for regular LDA and 51.33% for SLDA. The paper attributes part of the gain to the fact that “only $(z_i,z_j)$5 contributes effectively to classification,” while $(z_i,z_j)$6 explains away nuisance variation (Zhang et al., 2013).

A conceptually related but architecturally distinct form occurs in RGB-D action and gesture recognition. “Multi-stage Factorized Spatio-Temporal Representation for RGB-D Action and Gesture Recognition” does not use the exact phrase “double-factorized,” but its MFST architecture factorizes representation along two axes: space versus time within each stage, and RGB versus depth at the input through modality-aware CDC stems. Each stage applies an MSC-Trans spatial block followed by a WMS-Trans temporal block, while RGB uses CDC-ST and depth uses CDC-T. The paper describes this as constructing “dimension-independent higher-order semantic primitives” (Ma et al., 2023).

The architecture reports 92.1 / 95.8 accuracy for MFST-Large RGB on NTU RGB-D under CS/CV, 93.8 / 97.3 for depth, and 94.6 / 98.2 for RGB+DEP, improving over De+Recouple by +0.4 / +0.9 absolute in the RGB+DEP setting. On IsoGD, MFST-Large RGB+DEP reaches 68.47 versus 66.79 for De+Recouple. Ablations report that WMS-Trans outperforms a vanilla Transformer by +1.1% for RGB and +1.6% for depth, and that residual connections and modality-specific CDC stems are beneficial (Ma et al., 2023).

Across these models, “double-factorized” refers to a partition into two complementary latent roles rather than merely a low-rank matrix or tensor decomposition.

3. Dual-axis factorizations in 3D graphics and controllable generation

In 3D Gaussian Splatting, “double-factorized” is literal: both coordinates and per-Gaussian attributes are factorized. “F-3DGS: Factorized Coordinates and Representations for 3D Gaussian Splatting” factorizes Gaussian centers using either Canonical Polyadic or Vector-Matrix constructions and factorizes attributes such as scale, rotation, opacity, and color through axis- or plane-wise factors composed by outer products and decoded by a small MLP. The Gaussian covariance is parameterized as

$(z_i,z_j)$7

and rendering uses front-to-back alpha compositing

$(z_i,z_j)$8

The coordinate factorization includes constructions such as

$(z_i,z_j)$9

for CP, and

$p(z)=\prod_i p(z_i)$0

for VM (Sun et al., 2024).

The storage-quality tradeoff is quantitatively emphasized. On Synthetic-NeRF, Ours-CP-16 is 6.06 MB with PSNR 32.42 and 237.4 FPS, while 3DGS is 68.88 MB with PSNR 33.31 and 345.8 FPS; Ours-VM-16 is 28.75 MB with PSNR 33.24 and 275.5 FPS. On Tanks & Temples, Ours-CP-16 is 10.94 MB with PSNR 30.29 and 138.8 FPS, while 3DGS is 105.15 MB with PSNR 30.88 and 170.8 FPS. On Mip-NeRF360 indoor, Ours is 70.50 MB versus 334.75 MB for 3DGS. The ablations attribute roughly 70% of compression to color/opacity factorization and 25% to scale/rotation factorization, while learnable masking raises rendering speed from 125.6 FPS to 237.4 FPS with nearly identical PSNR (Sun et al., 2024).

A different but related use appears in 3D-aware image generation. “3D-GIF: 3D-Controllable Object Generation via Implicit Factorized Representations” factorizes along view dependence and lighting. Instead of a NeRF-style view-dependent color field, it learns view-independent fields for density $p(z)=\prod_i p(z_i)$1, albedo $p(z)=\prod_i p(z_i)$2, normal proxy features $p(z)=\prod_i p(z_i)$3, and specular parameters $p(z)=\prod_i p(z_i)$4, and introduces lighting only through an explicit shading model. The final pixel color is

$p(z)=\prod_i p(z_i)$5

with ambient-plus-directional shading and a learned specular term. Lighting is randomly sampled during training but is not fed into the implicit fields, and view direction is removed from the fields in stage 2, so the representation is both light-disentangled and view-independent (Lee et al., 2022).

The reported results include geometry improvements on BFM, where SIDE is 1.41 for 3D-GIF versus 1.78 for pi-GAN, and projection FID on CelebA with best value about 50.17 versus about 53.84 for pi-GAN. The method also extracts albedo-textured meshes by marching cubes on a $p(z)=\prod_i p(z_i)$6 grid and supports relighting with new $p(z)=\prod_i p(z_i)$7 without re-running the generator (Lee et al., 2022).

In these graphics settings, the term denotes a simultaneous factorization of geometric structure and appearance-related structure, but the two axes differ by application: coordinates versus attributes in Gaussian splatting, and view versus illumination in 3D-aware generation.

4. Mechanism-separating operator factorizations in scientific computing

In operator learning for physical systems, “double-factorized representation” is used to separate distinct physical mechanisms. “Factorized Neural Operators Decompose Dynamic and Persistent Responses” introduces FaNO under a Unified Green’s Function Framework. The model splits the learned operator into an equivariant dynamic branch and an invariant persistent branch. In spectral space, the dynamic branch is diagonal, for example

$p(z)=\prod_i p(z_i)$8

on Euclidean domains, while the persistent branch uses a geometry-aware global integral such as

$p(z)=\prod_i p(z_i)$9

The two responses are concatenated and mixed through

$X$0

The dynamic branch is designed to be equivariant, while the persistent branch is invariant by construction (Tang et al., 15 Jun 2026).

The paper reports consistent specialization: the equivariant branch captures rapidly varying transient dynamics, while the invariant branch captures coherent persistent structures. Quantitatively, on shallow-water equations at $X$1 train resolution and 20-step rollout, FaNO achieves MRE $X$2 of 1.198 versus 5.130 for SFNO; under cross-resolution evaluation from $X$3 to $X$4, FaNO achieves 2.183 versus 27.70. On WeatherBench at $X$5, 5-day average ACC is 56.1 for FaNO versus 50.9 for SFNO with 1.6M versus 5.4M parameters, and at $X$6 test resolution the 5-day average ACC is 55.8 versus 50.2. On Navier–Stokes at $X$7, FaNO reports step-average error $X$8 of 161.4 versus 178.1 for FNO and full-rollout error $X$9 of 189.0 versus 205.5 (Tang et al., 15 Jun 2026).

This suggests a more mechanistic reading of double factorization: not merely decomposing tensors, but decomposing inductive biases. One branch is constrained to transformation-sensitive responses and the other to transformation-insensitive responses.

5. Double factorization in quantum chemistry Hamiltonians

In quantum algorithms for electronic structure, “double factorization” has a specific tensor-algebraic meaning. “Quantum Filter Diagonalization with Double-Factorized Hamiltonians” represents the two-electron integral tensor as

$Z=f_\phi(X)\in\mathbb{R}^{b\times k}$0

with orthonormal leaf matrices $Z=f_\phi(X)\in\mathbb{R}^{b\times k}$1 and symmetric core matrices $Z=f_\phi(X)\in\mathbb{R}^{b\times k}$2. The one-body term is also diagonalized,

$Z=f_\phi(X)\in\mathbb{R}^{b\times k}$3

yielding a Hamiltonian in which transformed number-preserving operators are diagonal in the qubit $Z=f_\phi(X)\in\mathbb{R}^{b\times k}$4 basis (Cohn et al., 2021).

The paper further introduces compressed double factorization, in which the factors are optimized globally by least squares against the ERI tensor. The objective is

$Z=f_\phi(X)\in\mathbb{R}^{b\times k}$5

with a two-step procedure: initialize by explicit DF, analytically refit $Z=f_\phi(X)\in\mathbb{R}^{b\times k}$6, then optimize the antisymmetric generators of the $Z=f_\phi(X)\in\mathbb{R}^{b\times k}$7 matrices with L-BFGS while solving for optimal $Z=f_\phi(X)\in\mathbb{R}^{b\times k}$8 at each step. The paper states that Stage-2 C-DF improves the objective by at least one order of magnitude and the ERI maximum absolute deviation by about one order of magnitude across $Z=f_\phi(X)\in\mathbb{R}^{b\times k}$9, and that $B=Z_{:,(i,j)}$0 C-DF outperforms $B=Z_{:,(i,j)}$1 X-DF in the naphthalene example (Cohn et al., 2021).

The factorized form is then used inside Quantum Filter Diagonalization. Time evolution is approximated by Trotterization over diagonal one-body $B=Z_{:,(i,j)}$2 rotations and diagonal two-body $B=Z_{:,(i,j)}$3 rotations bracketed by Givens-rotation basis changes, and number-preserving post-selection and echo-sequencing are used for mitigation. In noisy simulations of ethylene torsion, combining post-selection and echo-sequencing yields 1–2 mHa statistical uncertainties, and the deviations are reported as statistically compatible with zero. Hardware experiments use a HOMO–LUMO active space, $B=Z_{:,(i,j)}$4, and a single Trotter step (Cohn et al., 2021).

A closely related use appears in “A stochastic quantum Krylov protocol with double factorized Hamiltonians.” There, the two-electron operator is explicitly transformed into a short sum of squares of one-body operators, so that each block is fast-forwardable. After a first factorization

$B=Z_{:,(i,j)}$5

each $B=Z_{:,(i,j)}$6 is spectrally decomposed,

$B=Z_{:,(i,j)}$7

leading to the canonical form

$B=Z_{:,(i,j)}$8

The explicit XDF Hamiltonian is then written in rotated bases in terms of $B=Z_{:,(i,j)}$9 and $A=Z_{:,i}\times Z_{:,j}$0 (Stair et al., 2022).

This structure is exploited inside randomized quantum Krylov diagonalization. The paper reports that on naphthalene with a $A=Z_{:,i}\times Z_{:,j}$1 active space, the XDF(3) interleaved ansatz with optimal weights achieved ground-state energy within 1 kcal/mol of CASCI at total evolution time $A=Z_{:,i}\times Z_{:,j}$2, with maximal circuit depth orders of magnitude smaller than first- or second-order Trotterized QKD at similar accuracy. For hydrogen chains $A=Z_{:,i}\times Z_{:,j}$3 with $A=Z_{:,i}\times Z_{:,j}$4 to $A=Z_{:,i}\times Z_{:,j}$5, the reported maximum rQK(3) circuit depths are 1296, 2016, 2520, 3024, and 3528, compared with 13248, 28448, 46760, 69552, and 94864 for TS1, with corresponding rQK(3) energy errors ranging from about 0.5 to 8.8 m$A=Z_{:,i}\times Z_{:,j}$6 (Stair et al., 2022).

In quantum chemistry, then, “double factorization” is a literal two-stage decomposition of the Hamiltonian tensor that is chosen because it aligns with qubit-native diagonal structure and shallow implementations.

6. Algebraic, geometric, and topological meanings

Outside machine learning and numerical physics, the phrase also appears in genuinely algebraic senses. In “Polynomial-Value Sieving and Recursively-Factorable Polynomials,” a “double-factorized” representation is simply a presentation of a polynomial value as

$A=Z_{:,i}\times Z_{:,j}$7

The paper studies when all non-trivial such factorizations are generated by a recursive structure, and for quadratic polynomials establishes the identity

$A=Z_{:,i}\times Z_{:,j}$8

for matrices $A=Z_{:,i}\times Z_{:,j}$9 under the determinant-type condition $$d_H(A,B)=\max\left\{\sup_{a\in A}\inf_{b\in B}d(a,b),\sup_{b\in B}\inf_{a\in A}d(a,b)\right\},$$0 or $$d_H(A,B)=\max\left\{\sup_{a\in A}\inf_{b\in B}d(a,b),\sup_{b\in B}\inf_{a\in A}d(a,b)\right\},$$1. It also gives a geometric correspondence between factorizations of $$d_H(A,B)=\max\left\{\sup_{a\in A}\inf_{b\in B}d(a,b),\sup_{b\in B}\inf_{a\in A}d(a,b)\right\},$$2 and lattice points on the conic

$$d_H(A,B)=\max\left\{\sup_{a\in A}\inf_{b\in B}d(a,b),\sup_{b\in B}\inf_{a\in A}d(a,b)\right\},$$3

The paper proves that $$d_H(A,B)=\max\left\{\sup_{a\in A}\inf_{b\in B}d(a,b),\sup_{b\in B}\inf_{a\in A}d(a,b)\right\},$$4, $$d_H(A,B)=\max\left\{\sup_{a\in A}\inf_{b\in B}d(a,b),\sup_{b\in B}\inf_{a\in A}d(a,b)\right\},$$5, and $$d_H(A,B)=\max\left\{\sup_{a\in A}\inf_{b\in B}d(a,b),\sup_{b\in B}\inf_{a\in A}d(a,b)\right\},$$6 are recursively-factorable and works out explicit examples such as $$d_H(A,B)=\max\left\{\sup_{a\in A}\inf_{b\in B}d(a,b),\sup_{b\in B}\inf_{a\in A}d(a,b)\right\},$$7 at $$d_H(A,B)=\max\left\{\sup_{a\in A}\inf_{b\in B}d(a,b),\sup_{b\in B}\inf_{a\in A}d(a,b)\right\},$$8, where $$d_H(A,B)=\max\left\{\sup_{a\in A}\inf_{b\in B}d(a,b),\sup_{b\in B}\inf_{a\in A}d(a,b)\right\},$$9 (Burns, 2014).

In knot theory, “Factorization of differential expansion for non-rectangular representations” uses a different double-factorized structure. For double-braid knots, the differential expansion coefficients factorize into products of single-braid contributions,

$A$0

and the paper extends this from rectangular representations to $A$1 and $A$2. For $A$3, the double-braid DE is written explicitly as a sum of $A$4-blocks multiplied by products such as $A$5 and $A$6. The paper states that this factorization “fixes the shape of the DE” and that it is checked, among other ways, by the Alexander reduction (Morozov, 2016).

In group theory, “A factorization result for classical and similitude groups” studies elements factored into two “(almost) isometries” whose squares are scalar maps. For $A$7 with similitude multiplier $A$8, the paper proves that there exist $A$9 such that $(i,j)$00, $(i,j)$01 is an anti-unitary involution with $(i,j)$02, and $(i,j)$03 is an anti-unitary similitude with multiplier $(i,j)$04 and $(i,j)$05. When $(i,j)$06, both factors can be chosen involutions. This factorization is then used to re-prove and extend a result on dualizing involutions for $(i,j)$07-adic classical groups (Roche et al., 2016).

These examples show that “double-factorized representation” is sometimes not a latent representation at all. It can instead denote a two-factor decomposition of a value, polynomial, braid invariant, or group element.

7. Common structure, distinctions, and limitations

Across fields, the phrase consistently signals decomposition into two complementary components, but the semantics of “factorized” differ sharply. In HFS it refers to pairwise factorized support rather than a factorial density (Roth et al., 2022). In factorized topic models it refers to class-private versus class-shared topic structure (Zhang et al., 2013). In F-3DGS it means factorized coordinates and factorized rendering attributes (Sun et al., 2024). In 3D-GIF it means separating viewpoint dependence from illumination effects (Lee et al., 2022). In FaNO it means separating equivariant dynamic responses from invariant persistent responses (Tang et al., 15 Jun 2026). In quantum chemistry it denotes a two-stage decomposition of the electronic Hamiltonian into qubit-friendly blocks (Cohn et al., 2021, Stair et al., 2022). In algebraic settings it can simply mean factorization into two multiplicative factors or two factorized braid contributions (Burns, 2014, Morozov, 2016, Roche et al., 2016).

The main advantages are likewise domain-specific but structurally similar: better disentanglement under correlation shift, cleaner separation of signal and nuisance variation, lower storage cost with comparable rendering quality, more controllable relighting and geometry extraction, better long-horizon and cross-resolution generalization in physical modeling, shallower quantum circuits, or explicit parametrizations of algebraic factorizations. A plausible implication is that the term is best understood as a methodological template rather than a single theory.

The main limitations are equally consistent with that interpretation. Pairwise factorization may miss higher-order inconsistencies in HFS (Roth et al., 2022). Shared–private topic splits can mis-handle class imbalance or poorly chosen topic counts (Zhang et al., 2013). CP-style coordinate factorization can struggle with irregular geometry, and densification is not directly compatible with the structured parameterization in F-3DGS (Sun et al., 2024). 3D-GIF uses a simplified lighting model and a learned normal proxy rather than full global illumination or strict surface normals (Lee et al., 2022). FaNO provides empirical rather than formal long-horizon guarantees and may underuse the invariant branch when persistent structure is negligible (Tang et al., 15 Jun 2026). Compressed double factorization in quantum chemistry can converge slowly, and the information content remains rank-3 rather than exact rank-2 in the reported formulations (Cohn et al., 2021).

Accordingly, “Double-Factorized Representation” is most precise when read in its local disciplinary context. What unifies the term is not a universal formula, but a recurrent decision to model two qualitatively different sources of variation, structure, or algebraic contribution separately and then recombine them through a controlled rule.