Generalized Multi-view Auto-Encoder (GMAE)

• GMAE is a deep multi-view clustering framework that disentangles view-specific and view-common representations to mitigate view-distribution entanglement.
• It employs a dual-path autoencoder architecture with separate encoders and projection MLPs to fuse representations before performing k-means clustering.
• The framework handles incomplete views by zero-filling missing modalities and leveraging reconstruction, adversarial, and neighbor mutual-information losses.

Generalized Multi-view Auto-Encoder (GMAE) is a deep multi-view clustering framework introduced for complete and incomplete multi-view clustering, with the explicit aim of mitigating view-distribution entanglement by disentangling each view into view-specific and view-common representations before fusion (Zou et al., 15 May 2026). In this usage, GMAE denotes Generalized Multi-view Auto-Encoder. The acronym is not unique in the recent literature: “Gaussian Masked Autoencoders” also uses GMAE, but for a single-image masked autoencoding method with a 3D Gaussian bottleneck rather than a multi-view autoencoder (Rajasegaran et al., 6 Jan 2025).

1. Scope, terminology, and problem formulation

GMAE is formulated in the standard multi-view clustering setting. Given

$$\Big\{\boldsymbol{x}_i^1\in\mathbb{R}^{d_1},\boldsymbol{x}_i^2\in\mathbb{R}^{d_2},\dots,\boldsymbol{x}_i^V\in\mathbb{R}^{d_V}\Big\}_{i=1}^N,$$

with $N$ samples and $V$ views, the objective is to partition the samples into $K$ clusters (Zou et al., 15 May 2026). The motivating claim is that existing deep MVC methods often fuse heterogeneous views into a shared space before properly separating what is common across views from what is view-dependent. The resulting latent space can mix cluster-relevant common semantics, useful complementary cues, and nuisance variation, which the paper identifies as view-distribution entanglement.

The framework is designed for both complete MVC, where every sample has all views, and incomplete MVC, where some views are missing. The paper does not introduce a specialized masked objective for missing views. Instead, incomplete data are generated by selecting a missing ratio $m_r\in[0,1]$, randomly selecting $m_r$ proportion of samples, randomly choosing between $1$ and $n-1$ views for each selected sample, and setting those views to zero while ensuring each sample retains at least one view. Training then proceeds with the same architecture and objective as in the complete setting. This makes GMAE “generalized” primarily in the sense that it is defined for an arbitrary number of views and empirically evaluated under both complete and incomplete-view regimes.

A second terminological issue is external to the clustering method itself. In “Gaussian Masked Autoencoders,” the acronym GMAE refers to a masked autoencoder that predicts 3D Gaussian primitives from a single masked image and reconstructs pixels through splatting; that model is explicitly not a multi-view autoencoder (Rajasegaran et al., 6 Jan 2025). In the multi-view clustering literature, however, GMAE denotes the disentanglement-driven clustering framework introduced in 2026 (Zou et al., 15 May 2026).

2. Dual-path architecture and representation structure

The architectural core of GMAE is a dual-path autoencoder design that decomposes each view into two latent pathways (Zou et al., 15 May 2026). For sample $i$ and view $v$, a view-specific encoder $N$0 produces a view-dependent hidden feature,

$N$1

while a cross-view shared encoder $N$2 produces a hidden feature intended to capture common structure,

$N$3

Projection MLPs then map these hidden features into a view-specific code

$N$4

and a view-common code

$N$5

The paper denotes the aggregated matrices as

$N$6

Reconstruction is view-wise. For each view, the latent used by the decoder is the concatenation

$N$7

and the reconstruction is

$N$8

This choice is important: neither latent branch is allowed to become vacuous, because both contribute to reconstructing the source view.

After disentanglement, GMAE forms a comprehensive representation

$N$9

by concatenating the aligned common representation with all view-specific codes,

$V$0

Clustering is then performed by applying $V$1-means to $V$2. The method therefore separates representation learning from the final clustering step: the network optimizes reconstruction, disentanglement, adversarial, and neighborhood-mutual-information objectives, and clustering itself is an external $V$3-means stage rather than an embedded pseudo-label refinement module.

The paper describes this as preserving cross-view complementarity through disentangled representation learning. Architecturally, that means GMAE does not compress all modalities directly into a single undifferentiated shared embedding. It first factorizes each view into a private path and a shared path, then fuses them only after this decomposition.

3. Objective functions and learning mechanism

The loss design combines four components: reconstruction, correlation minimization, cross-view adversarial discrimination, and neighborhood mutual-information modulation (Zou et al., 15 May 2026).

The reconstruction term is motivated as a variational lower bound on maximizing mutual information between the input $V$4 and the latent embedding $V$5. The implemented objective is the per-view squared reconstruction error

$V$6

This keeps the concatenated latent $V$7 informative about the source view.

Disentanglement between common and specific information is enforced by a covariance-style correlation penalty,

$V$8

where $V$9 is the mean vector of $K$0. The paper interprets this as a relaxed surrogate for minimizing dependence between $K$1 and $K$2. The intended effect is that view-common and view-specific channels should not encode the same factors redundantly.

A distinctive component is the set of cross-view adversarial discriminators $K$3. For view $K$4, a code from the same view is treated as real and codes from other views are treated as fake:

$K$5

The discriminator loss is written as

$K$6

The stated purpose is twofold: to regularize distribution mismatch across views and to guide view-specific encoders toward more discriminative features.

The final fused embedding $K$7 is further regularized by an InfoNCE-style neighborhood mutual-information term. The paper defines

$K$8

and uses

$K$9

Here $m_r\in[0,1]$0 is the number of nearest neighbors and $m_r\in[0,1]$1 uses the remaining samples as negatives. This term is described as maximizing mutual information between each point and its $m_r\in[0,1]$2-nearest neighbors, thereby making within-cluster samples more cohesive and between-cluster samples more separated.

The full objective is

$m_r\in[0,1]$3

with $m_r\in[0,1]$4. Equivalently, the text clarifies the operational form as

$m_r\in[0,1]$5

This bidirectional use of mutual information is central to the method’s interpretation. Useful mutual information is increased through input reconstruction and neighbor consistency, while unwanted mutual information between common and specific branches is reduced through the correlation penalty.

4. Optimization protocol, incomplete-view handling, and complexity

Training is joint rather than staged. The algorithm initializes the model, optimizes reconstruction, correlation, adversarial, and MI-neighbor losses for a fixed number of epochs, forms the comprehensive representation $m_r\in[0,1]$6, and then runs $m_r\in[0,1]$7-means for the final partition (Zou et al., 15 May 2026). The implementation details reported are: PyTorch 1.13.1 + CUDA 11.6, Python 3.8.15, Adam optimizer, learning rate $m_r\in[0,1]$8, 500 epochs, random seed 42, fully connected autoencoders on vectorized inputs, and batch size = full dataset size. The paper recommends $m_r\in[0,1]$9 and reports that 64 is the most generally effective latent dimension.

The method’s incomplete-view protocol is empirical rather than explicitly probabilistic. Missing views are created by zero-filling selected modalities while keeping at least one available view per sample, and the same loss is applied without introducing masks, imputation variables, or view-conditional decoders. The paper’s interpretation is that internal feature alignment and disentanglement make the model robust when some views are absent. A plausible implication is that GMAE’s robustness in incomplete MVC arises from representation design rather than from an explicit missing-data likelihood.

The paper studies sensitivity of $m_r$0 and $m_r$1 over $m_r$2 and reports that overly large $m_r$3 causes $m_r$4 and $m_r$5 to dominate, overly large $m_r$6 causes $m_r$7 to dominate, and overly small values make reconstruction dominate too much. Complexity is stated as linear in $m_r$8, with contributions from common-feature prior generation $m_r$9, view-specific features $1$0, and AE processing $1$1, so the dominant term is effectively

$1$2

where $1$3 is the largest hidden width and $1$4 the maximum specific latent dimension.

Several limitations are built into this optimization protocol. Incomplete-view handling is not modeled with explicit masks. The adversarial training description does not provide stabilization devices such as gradient penalty or alternate update ratios. Clustering is still external $1$5-means, not an integrated deep clustering objective. The implementation is full-batch and based on fully connected autoencoders, which is reproducible but narrow relative to convolutional or graph-structured alternatives.

5. Empirical results and ablation evidence

The empirical study covers 13 datasets spanning image-text, image, omics, and synthetic domains, including Wikipedia, RGB-D, STL-10, LandUse-21, Out-Scene, ALOI-100, NUSWIDEOBJ, Digits, MSRCV1, Dermatology, LGG, BRCA, and Synthetic3D (Zou et al., 15 May 2026). Performance is reported with ACC, NMI, and PUR, averaged over 10 runs. The paper states that on the main 20-test summary GMAE achieved 15 first places, 3 second places, and 2 third places.

Several headline results are explicit. On LGG, GMAE reaches ACC 92.16 / NMI 75.48 / PUR 98.91. On Wikipedia, it achieves ACC 62.18 / NMI 56.67 / PUR 70.35. On STL-10, it reports 96.25 / 90.26 / 96.37, accompanied by t-SNE plots showing clearer clusters than competitors. The Digits experiments are especially central to the “generalized” claim because performance improves as the number of views increases:

• 2 views: 95.55 / 90.88 / 95.90
• 3 views: 95.90 / 91.51 / 96.00
• 4 views: 96.50 / 92.07 / 96.65
• 6 views: 97.45 / 94.16 / 97.45

For incomplete MVC at missing rate 0.5, representative results include:

• BRCA: 51.01 / 32.42 / 64.82
• LGG: 74.16 / 34.91 / 74.16
• MSRCV1: 54.62 / 49.00 / 57.62

The ablation study isolates four settings: reconstruction only; reconstruction plus $1$6; reconstruction plus $1$7; and the full model. On Digits-6V, the scores are:

• reconstruction only: ACC 64.10 / NMI 61.88 / PUR 65.40
• • disentanglement/adversarial: 82.35 / 85.46 / 85.50
• • neighbor MI: 83.05 / 84.57 / 83.05
• full model: 97.45 / 94.16 / 97.45

On RGB-D, the same pattern appears:

• reconstruction only: 17.46 / 5.20 / 29.74
• • disentanglement/adversarial: 16.77 / 5.78 / 28.30
• • neighbor MI: 37.06 / 34.84 / 52.17
• full model: 45.07 / 40.06 / 58.94

These numbers support two conclusions stated in the source material. First, $1$8 is often the strongest single add-on. Second, the correlation-plus-adversarial block and the neighborhood-MI block are complementary; their combination is far stronger than either alone.

6. Position within multi-view autoencoder research

GMAE belongs to a broader family of multi-view autoencoding models, but its specific combination of common/private disentanglement, cross-view adversarial discrimination, and neighbor-based mutual-information modulation gives it a distinct place in that landscape.

The 2024 tutorial on multi-view autoencoders does not define a model class called GMAE, but it provides a unified framework that distinguishes shared-latent models from shared-plus-private latent models (Aguila et al., 2024). This suggests that GMAE is structurally closest to the shared-plus-private side of that taxonomy, because it explicitly learns both view-common codes $1$9 and view-specific codes $n-1$0. By contrast, the tutorial’s baseline deterministic multi-view AE is organized around same-view and cross-view reconstruction, while its variational family emphasizes latent fusion mechanisms such as PoE, MoE, generalized PoE, MoPoE, and shared/private latent-variable models.

Relative to graph-based multi-view autoencoders, GMAE is less relationally explicit. CMGEC’s multiple graph auto-encoder (M-GAE) reconstructs per-view adjacency matrices from a shared embedding, uses a learned consensus graph $n-1$1, and adds a multi-view mutual information maximization module for neighbor preservation (Wang et al., 2021). MVGCCA similarly couples per-view graph encoders with a shared latent representation and reconstructs both views and graph structure under an ELBO (Kaloga et al., 2020). GMAE does not reconstruct graphs, learn a consensus graph, or use graph convolutions; its inductive bias is disentanglement across views rather than graph structure.

Relative to supervised or hybrid frameworks, GMAE is also distinct. ACMVL alternates between per-view autoencoder reconstruction and a supervised fusion network with shared projection weights, refining encoders through classification loss (Lu et al., 2022). GMAE, by contrast, remains an unsupervised clustering framework whose final partition is produced by $n-1$2-means on $n-1$3. Theoretical work on multi-view representation learning further indicates that joint latent priors can improve generalization and that a Gaussian product mixture prior can encourage redundant cross-view features rather than penalizing them away (Sefidgaran et al., 25 Apr 2025). A plausible implication is that GMAE’s common/specific design could, in principle, be paired with more structured joint latent regularization than the present objective uses.

Two recurring misconceptions deserve emphasis. First, GMAE in the multi-view clustering literature is not the Gaussian bottlenecked masked autoencoder of (Rajasegaran et al., 6 Jan 2025). Second, GMAE is not a canonical “all-in-one” formulation of multi-view autoencoding. Its incomplete-view robustness comes from disentanglement and regularization under zero-filled views, not from a mask-aware probabilistic model; its clustering is performed by external $n-1$4-means; and its theory of common-code recoverability is presented in stylized form. These constraints define its current scope as a disentanglement-driven deep MVC method rather than a universal multi-view autoencoder formalism.