Factorized Multi-View Probing

• The paper introduces methods that decompose multi-view data into distinct factors (e.g., row/column and first-/second-order views), leading to enhanced diagnostic performance.
• Factorized multi-view probing separates shared and unique task-relevant information, offering clear interpretability across various domains and modalities.
• Empirical results demonstrate improved accuracy and efficiency in tasks like object recognition and multimodal retrieval through structured factorization.

Factorized multi-view probing is best understood, in this literature, as an umbrella designation for methods that decompose multi-view structure into explicitly modeled factors and then interrogate those factors through prediction, reconstruction, alignment, clustering, retrieval, or layerwise visualization. The factors vary by domain: they may be viewpoint and object code, shared and unique task-relevant information, row and column geometry, first- and second-order interactions, shared and private self-representation matrices, or view-specific partition variables. Correspondingly, the “probe” may be a learned decoder, a classifier, a contrastive head, a transport-weighted fusion rule, a clustering factorization, or a local nonlinear visualization map (Heo et al., 22 May 2026, Zhang et al., 2015, Liang et al., 2023).

1. Scope and terminological status

The phrase does not denote a single standardized framework. Rather, the supplied works use closely related but non-identical constructions. In weight-space learning, MVProbe is not presented as a named “factorized” operator, yet it is explicitly interpretable as a decomposition across axis and interaction order: row-side versus column-side, and first-order versus second-order views (Heo et al., 22 May 2026). In multimodal retrieval, “factorized” refers instead to a separable transport coupling, while the same paper explicitly states that it is not about probing in the diagnostic-classifier sense (Chen et al., 19 Dec 2025). In manifold-based recognition, an early formulation factorizes viewpoint from object identity by representing object-specific appearance manifolds as deformations of a unified manifold, which the paper itself describes as a strong early example of what one might now call factorized multi-view probing (Zhang et al., 2015).

This variation matters because the three constituent words shift across subfields. “Factorized” may refer to explicit tensor-style decompositions, additive shared/private splits, separable transport plans, or branchwise probe banks. “Multi-view” may denote multiple camera views, modalities, feature sources, or complementary algebraic views of a weight matrix. “Probing” may mean lightweight diagnostic prediction, explicit generative inversion, cluster extraction, or supervised visualization of internal latent states. A recurrent misconception is therefore to treat the phrase as synonymous with one specific architectural recipe. The cited literature instead supports a broader family resemblance: multi-view structure is decomposed into interpretable factors, and those factors are separately tested or fused (Yao et al., 2019).

2. Canonical decompositions of multi-view structure

A first canonical form is geometric latent factorization. In the view-object manifold formulation, an observation is modeled as a composition of a canonical viewpoint coordinate and a view-invariant object code. The paper instantiates this as

$$y \approx \gamma(x,s) = \mathcal{A}\times_2 s \times_3 \psi(x),$$

where $x$ lies on a unified manifold encoding viewpoint and $s$ is an object/style latent. The key claim is that object-specific view manifolds are geometrically different but topologically equivalent, so each can be represented as a deformed version of a common manifold (Zhang et al., 2015).

A second canonical form is information-theoretic factorization. FactorCL decomposes multimodal task-relevant information into shared and modality-unique components:

$$I(X_1,X_2;Y)=\underbrace{I(X_1;X_2;Y)}_{S}+\underbrace{I(X_1;Y\mid X_2)}_{U_1}+\underbrace{I(X_2;Y\mid X_1)}_{U_2}.$$

This formulation is explicitly designed to go beyond the multi-view redundancy assumption by separating task-relevant shared information from task-relevant unique information, and by distinguishing both from task-irrelevant shared dependence $I(X_1;X_2\mid Y)$ (Liang et al., 2023).

A third form is additive shared/private decomposition. In multi-view multiple clustering, the latent self-representation for clustering $k$ is

$\mathbf Z^k = \mathbf U + \mathbf D^k,$

with $\mathbf U$ a shared/commonality matrix and $\mathbf D^k$ an individuality matrix. Each clustering is then generated from the composite factor $\mathbf U+\mathbf D^k$, so quality is tied to the shared part and diversity to the individual part (Yao et al., 2019). A closely related partition-level factorization appears in deep matrix-factorization clustering, where each view keeps its own hierarchical latent representation and only the final partition-level representation is aligned to a consensus partition matrix $x$0 through orthogonal alignment matrices $x$1 (Zhang et al., 2021).

A fourth form is operator factorization over a single matrix. MVProbe separates a representative weight matrix into row-side versus column-side summaries and first-order versus second-order summaries. This suggests that, in some domains, “multi-view” need not mean multiple raw observations; it can mean multiple structurally distinct interrogations of the same object (Heo et al., 22 May 2026).

3. Weight-space factorized multi-view probing

In weight-space learning, the concrete task is model identification from parameters: given a checkpoint, predict properties such as its training label set directly from one empirically informative weight matrix $x$2. The standard first-order probe uses a learned probe matrix $x$3 and computes

$x$4

The paper argues that this single-view design is limited because it is row-centric and first-order: $x$5 only interrogates the row space through a few directions in input space, and any perturbation $x$6 satisfying $x$7 is invisible to the probe (Heo et al., 22 May 2026).

MVProbe addresses this by using four complementary branches:

$x$8

$x$9

The first two are first-order “external-direction” views, and the second two are Gram-based “intrinsic-geometry” views. The paper’s theoretical results justify both dimensions of the decomposition. One theorem shows that second-order probes can distinguish matrices that collapse under first-order probing:

$s$0

with distinct $s$1 satisfying $s$2 but $s$3. A transpose-complement theorem analogously shows that the column view $s$4 is non-redundant with the row view $s$5. Structurally, the method therefore factorizes probing across axis and order.

The same paper also gives an explicit conditioning argument. Under an i.i.d. Gaussian model for $s$6, the expected Frobenius norm of second-order outputs can dominate first-order outputs by a factor

$s$7

Branch-wise per-sample standardization,

$s$8

is therefore presented as necessary rather than cosmetic. After standardization, each branch is flattened, projected, passed through ReLU, concatenated, and fused by a shared encoder and classifier. The second-order branches preserve ProbeX-like scalability because

$s$9

so each branch remains $$I(X_1,X_2;Y)=\underbrace{I(X_1;X_2;Y)}_{S}+\underbrace{I(X_1;Y\mid X_2)}_{U_1}+\underbrace{I(X_2;Y\mid X_1)}_{U_2}.$$0 rather than requiring explicit Gram formation.

Empirically, the design is motivated not merely by more parameters but by complementary views. On the Model Jungle benchmark, MVProbe exceeds ProbeX$$I(X_1,X_2;Y)=\underbrace{I(X_1;X_2;Y)}_{S}+\underbrace{I(X_1;Y\mid X_2)}_{U_1}+\underbrace{I(X_2;Y\mid X_1)}_{U_2}.$$1 at the best layer for each family, including ResNet $$I(X_1,X_2;Y)=\underbrace{I(X_1;X_2;Y)}_{S}+\underbrace{I(X_1;Y\mid X_2)}_{U_1}+\underbrace{I(X_2;Y\mid X_1)}_{U_2}.$$2 versus $$I(X_1,X_2;Y)=\underbrace{I(X_1;X_2;Y)}_{S}+\underbrace{I(X_1;Y\mid X_2)}_{U_1}+\underbrace{I(X_2;Y\mid X_1)}_{U_2}.$$3, SupViT $$I(X_1,X_2;Y)=\underbrace{I(X_1;X_2;Y)}_{S}+\underbrace{I(X_1;Y\mid X_2)}_{U_1}+\underbrace{I(X_2;Y\mid X_1)}_{U_2}.$$4 versus $$I(X_1,X_2;Y)=\underbrace{I(X_1;X_2;Y)}_{S}+\underbrace{I(X_1;Y\mid X_2)}_{U_1}+\underbrace{I(X_2;Y\mid X_1)}_{U_2}.$$5, MAE $$I(X_1,X_2;Y)=\underbrace{I(X_1;X_2;Y)}_{S}+\underbrace{I(X_1;Y\mid X_2)}_{U_1}+\underbrace{I(X_2;Y\mid X_1)}_{U_2}.$$6 versus $$I(X_1,X_2;Y)=\underbrace{I(X_1;X_2;Y)}_{S}+\underbrace{I(X_1;Y\mid X_2)}_{U_1}+\underbrace{I(X_2;Y\mid X_1)}_{U_2}.$$7, and DINO $$I(X_1,X_2;Y)=\underbrace{I(X_1;X_2;Y)}_{S}+\underbrace{I(X_1;Y\mid X_2)}_{U_1}+\underbrace{I(X_2;Y\mid X_1)}_{U_2}.$$8 versus $$I(X_1,X_2;Y)=\underbrace{I(X_1;X_2;Y)}_{S}+\underbrace{I(X_1;Y\mid X_2)}_{U_1}+\underbrace{I(X_2;Y\mid X_1)}_{U_2}.$$9. On $I(X_1;X_2\mid Y)$0, first-order probing collapses to $I(X_1;X_2\mid Y)$1 in-distribution and $I(X_1;X_2\mid Y)$2 zero-shot, while MVProbe achieves $I(X_1;X_2\mid Y)$3 and $I(X_1;X_2\mid Y)$4. The ablations further show that all four branches together are always best, that standardization gives an average gain of $I(X_1;X_2\mid Y)$5, and that higher-order branches beyond second order help some architectures but hurt or barely help others; the paper therefore treats second order as the robust default (Heo et al., 22 May 2026).

4. Shared/private multimodal formulations

FactorCL provides the clearest information-theoretic account of why factorization may be necessary. Standard contrastive learning is criticized for assuming multi-view redundancy, formalized as small conditional task information $I(X_1;X_2\mid Y)$6 and $I(X_1;X_2\mid Y)$7. In the non-redundant case, the paper derives a uniqueness gap:

$I(X_1;X_2\mid Y)$8

so asymptotically shared-only contrastive representations recover only the shared task-relevant part. FactorCL therefore assigns separate branches to shared and unique factors, denoted $I(X_1;X_2\mid Y)$9, and combines MI lower bounds on desired information with MI upper bounds on nuisance shared information. In the supplied summary, the main reusable point is not only the algorithm but the ontology: a factorized multi-view probe should distinguish task-relevant shared information, task-relevant unique information, and task-irrelevant shared dependence (Liang et al., 2023).

A more explicitly architectural version appears in incomplete multi-view multi-label classification. MVFD learns a view-consistent representation $k$0 and view-specific representations $k$1 in two stages. The shared branch is trained first from masked inputs using reconstruction, semantic contrastive alignment across views, and partial-label supervision. The specific branch is then trained with the shared encoders frozen, using per-view reconstruction and a graph disentangling loss that penalizes cosine similarity both between shared and specific factors and among specific factors from different views. The final fused representation is

$k$2

This design makes the intended semantics explicit: shared factors are cross-view, task-relevant, and label-bearing; private factors are residual, view-idiosyncratic, and nonredundant with the shared branch. The same framework is built to tolerate missing views and missing labels through masks $k$3 and $k$4 (Xie et al., 11 Jan 2025).

Factorized transport alignment uses a different object of factorization: not the representation itself, but the cross-view coupling matrix. With image views $k$5 and text views $k$6, the fused embeddings are

$k$7

and the interaction matrix is restricted to the rank-1 form

$k$8

This is presented as a lightweight optimal-transport-inspired approximation. Training emphasizes primary views and samples auxiliary ones, so complexity drops from $k$9 to $\mathbf Z^k = \mathbf U + \mathbf D^k,$0 per listing per step, while inference fuses all views offline into a single cached embedding. Although the paper explicitly notes that it is not about probing in the diagnostic sense, its cross-view retrieval formulation can be read as probing distributed semantics across views, because auxiliary-view gains quantify what information primary views miss (Chen et al., 19 Dec 2025).

5. Clustering and partition-level factorization

In multi-view multiple clustering, factorization is used to generate alternative high-quality and diverse clusterings rather than label predictions. Each view matrix $\mathbf Z^k = \mathbf U + \mathbf D^k,$1 is reconstructed by a self-representation matrix of the form $\mathbf Z^k = \mathbf U + \mathbf D^k,$2, where $\mathbf Z^k = \mathbf U + \mathbf D^k,$3 is shared across clusterings and $\mathbf Z^k = \mathbf U + \mathbf D^k,$4 is individuality encoding for clustering $\mathbf Z^k = \mathbf U + \mathbf D^k,$5. Redundancy among the individuality matrices is reduced with the Hilbert-Schmidt Independence Criterion:

$\mathbf Z^k = \mathbf U + \mathbf D^k,$6

using $\mathbf Z^k = \mathbf U + \mathbf D^k,$7. Shared information is modeled by forcing $\mathbf Z^k = \mathbf U + \mathbf D^k,$8 to be smooth over similarity graphs from all views through a summed graph Laplacian. Each clustering is then produced by semi-NMF factorization of $\mathbf Z^k = \mathbf U + \mathbf D^k,$9 as $\mathbf U$0. The empirical conclusion reported in the supplied summary is that diversity is mainly driven by the individual factors, while the shared factor improves quality (Yao et al., 2019).

Deep matrix factorization and partition alignment takes a different route. Each view keeps its own deep semi-NMF hierarchy,

$\mathbf U$1

and only the top-layer partition matrices are aligned to a consensus partition $\mathbf U$2. The core objective combines per-view reconstruction with a trace-alignment term

$\mathbf U$3

where $\mathbf U$4 are orthogonal alignment matrices, $\mathbf U$5 weight reconstruction, and $\mathbf U$6 weight partition fusion. The conceptual point is that multi-view data should not be forced to share identical latent coordinates at every layer; instead, view-specific deep structure is preserved and agreement is enforced only at the partition level. For factorized multi-view probing, this provides a template in which private structure is maintained until an explicitly shared decision-level factor is formed (Zhang et al., 2021).

These clustering papers are not supervised probing papers, and they do not evaluate linear decodability or diagnostic selectivity. Their relevance is architectural and conceptual. They show that factorized multi-view analysis can be organized around shared/common factors, nonredundant private factors, and a downstream extraction mechanism that reads out one or more meaningful organizations of the data.

6. Geometric view manifolds and transformer latent-state probing

The view-object manifold model is an early geometric formulation of factorized multi-view inference. For each object instance $\mathbf U$7, the mapping from a canonical manifold coordinate $\mathbf U$8 to descriptor space is

$\mathbf U$9

and across objects the coefficient matrices are factorized as

$\mathbf D^k$0

Pose is encoded by the coordinate $\mathbf D^k$1 on a known unified manifold such as a unit circle or sphere, while identity/category is encoded by the deformation parameters or style vector $\mathbf D^k$2. Inference is generative:

$\mathbf D^k$3

The supplied summary emphasizes a central inversion of viewpoint and identity: viewpoint is treated as the content manifold because its topology is known, and object identity/category appears as the deformation of that manifold (Zhang et al., 2015).

A contemporary probing-oriented analysis appears in work on multi-view transformers such as DUSt3R. Here the object being probed is not an input manifold or a learned shared/private embedding, but the residual state inside a multi-view decoder. Separate local nonlinear probes decode pointmaps from the skip-connected state after self-attention, cross-attention, and MLP sublayers, with no cross-patch communication in the probe. The paper reports that the shared encoder output already provides a strong geometric estimate of the first view, while the decoder progressively aligns and reconstructs the second view in the first-view coordinate system. Cross-attention behaves like a correspondence-driven transport mechanism, and self-attention primarily restores coherent intra-view geometry around those transported anchors. Quantitatively, after Procrustes alignment of the second-view pointmap to isolate intra-view geometry, self-attention layers decrease aligned second-view error by $\mathbf D^k$4 on average, while cross-attention layers increase it by $\mathbf D^k$5 and MLP layers by $\mathbf D^k$6. Correspondence quality improves from roughly $\mathbf D^k$7 at decoder input to over $\mathbf D^k$8 after the first six decoder blocks under a 16-pixel threshold. The paper also argues, via register-token interventions, that the investigated DUSt3R variant is unlikely to utilize global pose information explicitly and instead relies heavily on correspondences refined with reconstructed geometry (Stary et al., 28 Oct 2025).

Taken together, these two lines illustrate two very different versions of geometric factorization. The earlier manifold work imposes a canonical equivariant coordinate system and learns object-conditioned deformations of it. The transformer analysis, by contrast, extracts factor-like roles retrospectively from a distributed latent state. A plausible implication is that factorized multi-view probing can be either constructive, by building the factorization into the model, or analytic, by probing factor-like structure after training.

7. Empirical regularities, misconceptions, and unresolved issues

One strong regularity is that increasing capacity within a single view is not equivalent to adding structurally distinct views. MVProbe isolates this most cleanly: ProbeX$\mathbf D^k$9 increases first-order probe count, yet the main gains come from adding row/column and Gram-based branches rather than from more probes alone. The same paper also reports that all four branches together are always best, whereas adding 3rd- and 4th-order branches yields architecture-dependent effects; second order is therefore presented as the sweet spot rather than a monotone ladder of improvement (Heo et al., 22 May 2026).

A second regularity is that explicit conditioning or redundancy reduction is central whenever factors differ in scale or semantics. In MVProbe, per-sample standardization is theoretically motivated by incompatible branch norms. In MVFD, disentangling is enforced by similarity penalties between shared and specific factors and among specific factors across views. In multi-view multiple clustering, HSIC reduces dependence among individuality matrices. These methods differ mechanistically, but they converge on the same design principle: factorization without conditioning tends to collapse into a dominant branch or redundant branches (Xie et al., 11 Jan 2025).

A third regularity is that lightweight factorization often trades expressivity for scalability. Factorized transport alignment is explicit about this tradeoff. The rank-1 coupling

$\mathbf U+\mathbf D^k$0

cannot express arbitrary pairwise dependencies, fixed primary/auxiliary weights may be suboptimal, and categories with a single representative image show little improvement. Yet the same restriction yields a practical $\mathbf U+\mathbf D^k$1 rolling-sampling training scheme and preserves two-tower retrieval with no additional online overhead (Chen et al., 19 Dec 2025).

Several common misconceptions also recur. Factorized multi-view probing is not always an explicit tensor factorization; MVProbe, for example, does not introduce a named factorized fusion module. It is not always diagnostic probing; transport alignment and clustering methods use retrieval or unsupervised factor extraction rather than label probes. Nor does successful probing automatically imply causal use. The transformer analysis explicitly notes that most probe results are correlational rather than causal, and FactorCL and MVFD both stop short of strong identifiability guarantees for their shared/private decompositions (Liang et al., 2023, Stary et al., 28 Oct 2025).

The supplied literature suggests three persistent open questions. The first is the factorization target: operators, latents, transport plans, self-representation matrices, and residual streams lead to different notions of “view.” The second is the supervision regime: some methods rely on pose labels, some on category labels, some on weak pairing only, and some probe pretrained internals after the fact. The third is leakage: shared branches may retain private information, private branches may retain label information, and a decoded factor may be recoverable without being the mechanism actually used. These questions explain why factorized multi-view probing remains less a settled method than a technical perspective: represent multi-view structure through nonredundant factors, test what each factor contains, and determine which combinations are necessary for robust inference.