Intrinsic Dimension Estimating Autoencoder (IDEA)

• The paper introduces IDEA, a neural autoencoder that estimates intrinsic dimension by pruning redundant latent coordinates using a re-weighted double CancelOut bottleneck.
• It integrates three tasks—dimension selection, representation learning, and data reconstruction—into a single end-to-end training procedure with a projected reconstruction loss.
• IDEA demonstrates robust performance on synthetic and manifold benchmarks, accurately recovering intrinsic dimensions and enabling interpretable, efficient latent representations.

Intrinsic Dimension Estimating Autoencoder (IDEA) is a neural-network-based method for intrinsic dimension (ID) estimation that targets datasets whose samples lie on either linear or nonlinear manifolds. Introduced in "Intrinsic Dimension Estimating Autoencoder (IDEA) Using CancelOut Layer and a Projected Loss" (Orioua et al., 12 Sep 2025), it combines an autoencoder architecture with a re-weighted double CancelOut bottleneck and a projected reconstruction loss. The method is designed not only to estimate the integer-valued intrinsic dimension of a dataset, but also to reconstruct the original data after projection into the latent space identified by the network.

1. Conceptual basis

IDEA addresses a standard difficulty in nonlinear dimensionality analysis: dimensionality reduction and dimension estimation are related, but not identical. A conventional autoencoder can compress and reconstruct data, yet its latent width does not by itself provide an intrinsic dimension estimate. Earlier work on autoencoder-based dimension estimation emphasized precisely this point, noting that autoencoders do not natively provide proxies for intrinsic dimension and that all latent variables may remain nonzero unless explicit regularization and post-processing are imposed (Bahadur et al., 2019).

IDEA turns the latent bottleneck itself into an estimator. Its predicted ID, denoted $\tilde{d}$, is the final number of active latent coordinates, namely the coordinates in the first CancelOut layer that are not zeroed out by regularization. In that sense, the method ties three tasks into a single training objective: selecting the effective latent dimension, learning a low-dimensional representation, and reconstructing the ambient-space data from that representation (Orioua et al., 12 Sep 2025).

This design places IDEA within a broader family of geometry-aware autoencoders, but with a distinct operational definition of intrinsic dimension. Rather than infer ID from latent variances, pullback-metric rank, singular-value surrogates, or neighborhood statistics, IDEA estimates ID by continuous pruning of latent coordinates during end-to-end training. This suggests a close connection between intrinsic dimension estimation and model selection in the bottleneck itself.

2. Network architecture and the re-weighted double CancelOut bottleneck

IDEA is built on a classical autoencoder with a customizable bottleneck layer. The encoder and decoder each comprise five fully connected layers, each except the last followed by normalization and SiLU nonlinearities. The architectural novelty lies in the bottleneck, which is implemented as a re-weighted double CancelOut layer (Orioua et al., 12 Sep 2025).

For a latent vector $X \in \mathbb{R}^l$ and learned weights $W_{\mathrm{CO}} \in \mathbb{R}^l$, the CancelOut layer operates as

$$\mathrm{CancelOut}(X) = X \odot g(W_{\mathrm{CO}}),$$

where $g$ is an elementwise activation, taken as ReLU. Each latent coordinate is therefore modulated independently. If a learned weight becomes zero, the corresponding latent dimension is effectively cancelled.

The bottleneck contains two such layers in sequence. The first, Co$_1$, is regularized to push redundant latent dimensions' weights to zero; the summary specifies L1 regularization as the example mechanism for sparsity induction. The second, Co$_2$, is unregularized and rescales the active latent variables back to a comparable scale, compensating for magnitude suppression introduced by Co$_1$. The effective latent dimension $l_{\mathrm{eff}}$ is the number of nonzero weights after this pruning process.

This double structure is central to IDEA’s interpretation. Co$_1$ performs dimension selection, whereas Co$X \in \mathbb{R}^l$0 preserves the usability of the surviving coordinates for decoding. A plausible implication is that the model separates “whether a dimension is needed” from “how large its numerical scale should be,” which is important because latent shrinkage alone need not correspond to genuine redundancy.

3. Projected reconstruction loss and training dynamics

The defining methodological contribution of IDEA is the projected reconstruction loss. In addition to the standard reconstruction loss used in autoencoders, IDEA continuously evaluates what would happen if one more latent dimension were removed (Orioua et al., 12 Sep 2025).

The total loss is given in the paper as

$X \in \mathbb{R}^l$1

Here, $X \in \mathbb{R}^l$2 is the mean-squared reconstruction loss, $X \in \mathbb{R}^l$3 is the reconstruction obtained after simulating the removal of the last active latent variable, $X \in \mathbb{R}^l$4 is the L1 regularization term with a small $X \in \mathbb{R}^l$5 offset for numerical stability, and $X \in \mathbb{R}^l$6 is an orthogonality penalty on latent codes that encourages disentanglement.

The projected term is constructed by taking the current effective dimension $X \in \mathbb{R}^l$7, setting the weight of the last nonzero latent variable to zero, and computing the MSE of the resulting adjusted reconstruction. The purpose is to push the model toward using as few active latent variables as possible, while halting further pruning when the removal of an additional coordinate causes a marked degradation in reconstruction quality.

The training process is described as cycling through four coupled pressures:

1. minimizing reconstruction error with as few active latent variables as possible;
2. penalizing extra redundant latent variables;
3. testing the effect of eliminating the least contributive variable through the projected loss;
4. regularizing toward low dimension, but stopping when further shrinkage irreparably hurts reconstruction.

The orthogonality term adds a second criterion beyond sparsity. Latent coordinates are encouraged to be as independent as possible, or orthogonal in the linear-correlation sense adopted by the loss. This is not identical to a full disentanglement guarantee, but it establishes a geometric bias against redundant latent directions.

The reported hyperparameters are: overall learning rate $X \in \mathbb{R}^l$8, learning rate for Co$X \in \mathbb{R}^l$9 equal to $W_{\mathrm{CO}} \in \mathbb{R}^l$0, batch size 256, Adam optimizer, $W_{\mathrm{CO}} \in \mathbb{R}^l$1, $W_{\mathrm{CO}} \in \mathbb{R}^l$2, and $W_{\mathrm{CO}} \in \mathbb{R}^l$3. The initial latent dimension $W_{\mathrm{CO}} \in \mathbb{R}^l$4 is flexible and can be overparameterized before pruning.

4. Benchmarks on synthetic and manifold data

IDEA was first evaluated on theoretical benchmarks intended to test both reconstruction ability and intrinsic-dimension estimation. On synthetic data formed by linear combinations of scaled Legendre polynomials with known intrinsic dimension $W_{\mathrm{CO}} \in \mathbb{R}^l$5, the model exactly recovered $W_{\mathrm{CO}} \in \mathbb{R}^l$6 for the tested cases $W_{\mathrm{CO}} \in \mathbb{R}^l$7. In those experiments, the mean-squared reconstruction error rose sharply when fewer than $W_{\mathrm{CO}} \in \mathbb{R}^l$8 dimensions were active, while additional dimensions beyond $W_{\mathrm{CO}} \in \mathbb{R}^l$9 did not improve the loss (Orioua et al., 12 Sep 2025).

The paper then evaluates IDEA on standard manifold benchmarks drawn from the scikit-dimension toolbox, including linearly and nonlinearly embedded spheres, helices, Swiss roll, affine and linear synthetic distributions, and multidimensional paraboloids. The reported outcome is that IDEA exactly matches or outperforms leading ID estimators such as MLE, TwoNN, DANCo, and lPCA, whose estimates often err significantly on complex nonlinear manifolds. The same set of experiments is presented as evidence of robustness across numerous manifold types with little tuning.

This benchmark profile is significant because it situates IDEA among methods that attempt to estimate intrinsic dimension directly from geometry rather than solely from reconstruction heuristics. A related line of work uses variational autoencoders and information-geometric diagnostics to show that bottleneck size interacts strongly with data ID: once the bottleneck exceeds the data’s intrinsic dimension, the representation profile changes qualitatively, producing a “double hunchback” ID pattern across encoder and decoder layers (Camboulin et al., 2024). IDEA addresses a different problem—directly estimating the dataset’s ID—but the two results are compatible in suggesting that intrinsic dimension is a practically useful control parameter for bottleneck design.

5. Scientific application to vertically resolved free-surface flow

Beyond synthetic benchmarks, IDEA was applied to data generated from the numerical solution of a vertically resolved one-dimensional free-surface flow, following a pointwise discretization of the vertical velocity profile in the horizontal direction, vertical direction, and time (Orioua et al., 12 Sep 2025).

In this setting, the input profiles are high-dimensional—for example, the summary mentions $$\mathrm{CancelOut}(X) = X \odot g(W_{\mathrm{CO}}),$$0—but are often well approximated by a low number of moments, such as Legendre polynomials. IDEA is reported to recover the intrinsic dimension corresponding to the minimally sufficient number of dynamical parameters and then reconstruct the original solution by working directly within the projection space identified by the network.

The scientific result emphasized in the summary is compression efficiency. IDEA required only two latent dimensions to match the accuracy obtained by 4–7 Legendre moments. The same application also yielded interpretable latent coordinates: the model could directly disentangle the representation, for example by reserving a latent for water height while two others corresponded to dominant velocity modes.

These findings matter because many intrinsic-dimension estimators stop at returning a scalar estimate. IDEA instead provides an ID estimate together with an operational latent representation and a decoder that reconstructs the original data. That reconstruction capability is comparatively unusual among ID estimators and is especially relevant in scientific computing, where the reduced coordinates are often useful only if they support inverse mapping to physically meaningful fields.

6. Relation to adjacent autoencoder-based ID methods

IDEA belongs to a growing literature that uses autoencoders or variational autoencoders for intrinsic-dimension analysis, but it differs in where the dimensionality signal is encoded and how it is extracted.

One earlier strategy regularizes latent activations and then constructs singular value proxies from sorted latent magnitudes. In "Dimension Estimation Using Autoencoders" (Bahadur et al., 2019), strong L1/L2 regularization on the latent layer, together with normalization and post-hoc aggregation, is used to turn autoencoder representations into proxies analogous to PCA singular values. That approach is post-hoc and threshold-based, whereas IDEA makes the number of active bottleneck coordinates the estimate itself.

A second family uses reconstruction-error knees. "An Additive Autoencoder for Dimension Estimation" (Kärkkäinen et al., 2022) estimates intrinsic dimension by varying bottleneck size and identifying the knee or plateau in the autoencoding error after serial bias, linear-trend, and nonlinear-residual estimation. IDEA avoids repeated retraining across candidate bottleneck widths by starting from an overparameterized latent space and pruning during a single training procedure.

Other methods impose geometric constraints directly on learned representations. "Autoencoders with Intrinsic Dimension Constraints for Learning Low Dimensional Image Representations" (Zheng et al., 2023) regularizes both global and local intrinsic dimension through reconstruction of data representations. "Thinner Latent Spaces: Detecting dimension and imposing invariance through autoencoder gradient constraints" (Kevrekidis et al., 2024) uses orthogonality of latent gradients to infer the number of functionally independent coordinates, but the paper highlights limitations for manifolds that cannot be covered by a single smooth chart. IDEA shares the use of an orthogonality bias, but its central estimator is the active-coordinate count in the CancelOut bottleneck, not gradient activity.

Variational approaches locate ID in yet other quantities. "Estimating Dataset Dimension via Singular Metrics under the Manifold Hypothesis: Application to Inverse Problems" (Causin et al., 9 Jul 2025) estimates intrinsic dimension from the numerical rank of the VAE decoder pullback metric and then uses a mixture of invertible VAEs to construct an atlas of local charts. "Learning Ordered Representations in Latent Space for Intrinsic Dimension Estimation via Principal Component Autoencoder" (Zhan et al., 27 Jan 2026) learns ordered latent coordinates through non-uniform variance regularization and an isometric constraint, allowing a PCA-like explained-variance criterion to estimate ID post hoc. Compared with these methods, IDEA is distinguished by combining direct latent pruning, projected reconstruction assessment, and reconstruction in the identified projection space (Orioua et al., 12 Sep 2025).

A recurring misconception in this area is that an autoencoder bottleneck width automatically equals intrinsic dimension. The broader literature does not support that view: autoencoders can overuse latent variables, latent coordinates may be unordered, and topology can obstruct globally faithful low-dimensional charts (Bahadur et al., 2019). IDEA should therefore be understood not as a generic property of autoencoders, but as a specific estimator built around a structured bottleneck and a specialized loss.

7. Limitations, failure modes, and open questions

The most explicit limitation reported for IDEA is topological. On the 1D helix embedded in 3D, denoted $$\mathrm{CancelOut}(X) = X \odot g(W_{\mathrm{CO}}),$$1, the method predicted ID $$\mathrm{CancelOut}(X) = X \odot g(W_{\mathrm{CO}}),$$2 although the correct value is $$\mathrm{CancelOut}(X) = X \odot g(W_{\mathrm{CO}}),$$3. The stated reason is that the autoencoder’s latent space contained the data but did not map it homeomorphically; the induced manifold became a 2D strip around the underlying 1D helix (Orioua et al., 12 Sep 2025).

This failure mode is consistent with a broader issue in autoencoder-based manifold learning: nonlinear manifolds with nontrivial topology may resist faithful parameterization by a single Euclidean latent chart. Related work on conformal autoencoders reports an analogous limitation on the circle, where the method correctly identifies local intrinsic dimension but fails to globally reconstruct the manifold because the circle cannot be described by a single smooth chart (Kevrekidis et al., 2024). In IDEA, the helix example shows that reconstruction success alone does not guarantee correct topological dimension.

The paper also notes practical limitations. Deep models may require larger sample sizes and longer compute times for difficult manifolds. Latent interpretability is encouraged but not guaranteed: the orthogonality penalty targets correlations in latent space, yet more general nonlinear disentanglement measures, such as mutual information, may be needed for stronger independence. Suggested improvements include adaptive learning rates and more general disentanglement measures, with the explicit caveat that such changes would impose an additional computational cost (Orioua et al., 12 Sep 2025).

These limitations clarify the scope of the method. IDEA is presented as a flexible neural method for intrinsic-dimension estimation on both linear and nonlinear manifolds, with the added benefit of reconstructing data from the learned projection space. At the same time, the available evidence indicates that its estimate is still shaped by representational geometry and topology: when the latent chart wraps the manifold in a non-homeomorphic way, the active-coordinate count can overestimate the true intrinsic dimension.