Entropy-Ordered Flows

• Entropy-Ordered Flows (EOFlows) are a normalizing flow framework that orders latent dimensions by explained entropy, creating a clear separation of semantic core and noise detail.
• They employ entropy-based regularization and Jacobian orthogonality to achieve nonlinear PCA-like behavior, ensuring effective unsupervised disentanglement and robust density modeling.
• EOFlows enable adaptive inference by allowing flexible retention of top core latents for compression, denoising, and controlled generation without needing retraining.

Entropy-Ordered Flows (EOFlows) are a normalizing-flow framework in which latent dimensions are ordered by their explained entropy, analogously to PCA’s explained variance. In this formulation, a learned representation separates into a low-dimensional core subspace that captures semantic structure and a complementary detail subspace that captures fine details and noise, while retaining a fully bijective flow during training. After training, the model can be used as an adaptive injective flow: one may retain only the top (C) latent variables at inference time, with (C) chosen flexibly rather than fixed during training. EOFlows were introduced for unsupervised disentanglement, compression, and denoising, and were developed from ideas in Independent Mechanism Analysis (IMA), Principal Component Flows (PCF), and Manifold Entropic Metrics (MEM) [2602.06940].

1. Conceptual definition and scope

EOFlows use standard invertible flows,
[
\mathbf{z}=\mathbf{f}(\mathbf{x}),\qquad \mathbf{x}=\mathbf{g}(\mathbf{z})=\mathbf{f}^{-1}(\mathbf{z}),\qquad \mathbf{z},\mathbf{x}\in\mathbb{R}^D,
]
with a standard Gaussian prior (p(\mathbf{Z})=\mathcal{N}(0,I)). Their distinction from vanilla flows lies in three features: they add entropy-based regularizers to maximum likelihood, they shape the decoder Jacobian so that its columns are nearly orthogonal, and they compute an entropy per latent dimension and sort dimensions by that quantity [2602.06940].

This construction yields a nonlinear PCA-like model. In the linear case, EOFlows reduce exactly to PCA: total disentanglement together with maximum likelihood produces a decoder whose columns align with eigenvectors and are ordered by eigenvalues, while recursive core–detail compression produces the same PCA solution. In the nonlinear case, the corresponding ordered coordinates become curvilinear coordinates on the learned manifold rather than linear principal axes.

A common misconception is to identify EOFlows with injective flows trained on a fixed bottleneck. EOFlows instead learn a full bijection (\mathbb{R}^D\leftrightarrow\mathbb{R}^D) and only acquire injective behavior post hoc, after entropy ordering. The practical consequence is that a single trained model can be queried at multiple intrinsic dimensions (C), rather than requiring retraining for each bottleneck size.

2. Manifold entropy, explained entropy, and latent ordering

The central object in EOFlows is the manifold entropy associated with a subset of latent coordinates. For an index set (S\subseteq{1,\dots,D}), the decoder and latent prior induce a manifold PDF on the reconstruction manifold obtained by varying only (\mathbf{z}S) while fixing (\mathbf{z}{S^c}). With (\mathbf{J}S(\mathbf{z})) denoting the Jacobian submatrix containing only columns indexed by (S), the injective change-of-variables formula is
[
q_S(\mathbf{U}_S=\mathbf{u}_S)=p_S(\mathbf{Z}_S=\mathbf{z}_S)\,\big|\mathbf{J}_S([\mathbf{z}_S,\mathbf{z}{S^c}])\big|^{-1}.
]
The associated pointwise manifold entropy is
[
\mathcal{L}S(\mathbf{x})\coloneq -\log q_S(\mathbf{U}_S=\mathbf{x})
= \frac{1}{2}\big|\mathbf{f}_S(\mathbf{x})\big|_2^2
+ \log\big|\mathbf{J}_S(\mathbf{f}(\mathbf{x}))\big|
+ \frac{|S|}{2}\log(2\pi),
]
and the manifold entropy is
[
H_S\coloneq H(\mathbf{U}_S)=\mathbb{E}{\mathbf{x}\sim q(\mathbf{X})}\big[\mathcal{L}_S(\mathbf{x})\big].
]
For single dimensions (S={j}), the resulting (H_j) are the explained entropies used for ordering [2602.06940].

After training, EOFlows sort latent coordinates so that
[
H_1\ge H_2\ge \dots \ge H_D.
]
High-(H_j) dimensions encode large, structured variation, while low-(H_j) dimensions encode small-scale or noise-like variation. This ordering is directly analogous to PCA’s eigenvalue ordering, except that EOFlows use entropy rather than variance. Once sorted, the top coordinates define the core and the remainder define the detail.

Disentanglement is quantified through manifold mutual information and manifold total correlation. For subsets (S) and (T), EOFlows define a pointwise manifold mutual information (\mathcal{L}{S\perp T}) and its expectation (\mathcal{I}{S,T}). For all individual dimensions,
[
\mathcal{L}{\mathrm{TC}}(\mathbf{x})=\sum{j=1}^D \mathcal{L}j(\mathbf{x})-\mathcal{L}*(\mathbf{x}),
\qquad
\mathcal{I}{\mathrm{TC}}=\mathbb{E}\big[\mathcal{L}{\mathrm{TC}}(\mathbf{x})\big].
]
A key structural fact is
[
\mathcal{L}{\mathrm{TC}}(\mathbf{x})=0
\iff
\mathbf{J}_j(\mathbf{f}(\mathbf{x}))\perp \mathbf{J}{j'}(\mathbf{f}(\mathbf{x}))
\quad \forall j\neq j'.
]
When (\mathcal{I}{S,T}\approx 0), manifold entropy becomes approximately additive,
[
H{ST}\approx H_S+H_T,
]
which makes per-dimension entropies meaningful and comparable.

This additivity underlies the core/detail decomposition. After sorting, one defines the core index set ({1,\dots,C}) and the detail index set ({C+1,\dots,D}). Empirically, the entropy spectrum often drops rapidly and then saturates at the noise entropy level; for Gaussian noise of variance (\sigma^2), the data state that, up to constants, (H_{\text{noise}}=\tfrac12+\log\sigma). Dimensions beyond that plateau mostly encode noise rather than structured variation.

3. Objective function, Jacobian regularization, and training procedure

The base likelihood model is the standard flow objective. With (\mathbf{J}(\mathbf{z})=\partial \mathbf{g}(\mathbf{z})/\partial \mathbf{z}), the data density is
[
q(\mathbf{X}=\mathbf{g}(\mathbf{z}))=p(\mathbf{Z}=\mathbf{z})\,|\mathbf{J}(\mathbf{z})|^{-1},
]
and the negative log-likelihood per data point is
[
\mathcal{L}(\mathbf{x})=
\frac12|\mathbf{f}(\mathbf{x})|_2^2+\log|\mathbf{J}(\mathbf{f}(\mathbf{x}))|+\text{const.}
]
EOFlows generalize maximum likelihood through **Maximum Manifold Likelihood (MML)*, which augments (\mathcal{L}) with manifold entropy and manifold mutual-information terms. The practical objective used for entropy ordering is **Total Disentanglement,
[
\mathcal{L}(\mathbf{x})=\mathcal{L}_(\mathbf{x})+\lambda_{\mathrm{TC}}\mathcal{L}{\mathrm{TC}}(\mathbf{x}),\qquad \lambda{\mathrm{TC}}>0,
]
where
[

\mathcal{L}_{\mathrm{TC}}(\mathbf{x})

\sum_{j=1}^D \log|\mathbf{J}_j(\mathbf{f}(\mathbf{x}))|

\log|\mathbf{J}(\mathbf{f}(\mathbf{x}))|.
]
This is the local Jacobian regularization term that penalizes deviations from orthogonality between decoder Jacobian columns [2602.06940].

Noise augmentation is integral rather than auxiliary. EOFlows train on noise-inflated data
[
\tilde{\mathbf{x}}=\mathbf{x}+\sigma\boldsymbol{\epsilon},
\qquad
\boldsymbol{\epsilon}\sim\mathcal{N}(0,I_D),
]
and the full loss is
[

\mathcal{L}_{\text{EOFlow}}(\tilde{\mathbf{x}})

\mathcal{L}*(\tilde{\mathbf{x}})
+
\lambda{\mathrm{TC}}\mathcal{L}_{\mathrm{TC}}(\tilde{\mathbf{x}}).
]
The stated rationale is geometric: if the true data lie near a low-dimensional manifold, a bijective flow matching them exactly tends to fold and overfit the manifold structure; adding isotropic noise inflates the distribution into a full-dimensional density, and later reconstructions through a low-entropy bottleneck effectively deflate and denoise it.

In the reported experiments, EOFlows use fully invertible coupling-based flows, specifically RealNVP-style affine coupling, with 8–12 blocks. Each block consists of a fixed (D\times D) rotation/permutation and an affine coupling layer with fully connected MLPs; the hidden size is 1024 for EMNIST and 3072 for CelebA. A final learned linear rotation layer is added because maximum likelihood is invariant to this rotation whereas the entropy-based MML is not, so the rotation becomes the locus where the model chooses a canonical coordinate system. No convolutions are used.

The principal computational challenge is evaluating (\mathcal{L}_{\mathrm{TC}}), which naïvely requires all columns of the decoder Jacobian and scales as (O(D^2)). EOFlows address this with a stochastic estimator based on Jacobian–vector products (JVPs). Since
[
\log|\mathbf{J}_i|=\frac12\log|\mathbf{J}_i|_2^2,
]
each column norm can be obtained from one JVP with a one-hot vector. Rather than computing all (D) columns per sample, the method samples one dimension index per sample in a batch of size (B\ge D), computes one batched JVP per sample, and reweights by the multiplicities of sampled indices. The data state that this yields an unbiased estimate of (\sum_i \mathcal{L}_i), scales to (28\times 28\times 3) CelebA images with (D=2352), and increases training time by a factor of about (2.5) over vanilla maximum-likelihood flows, essentially independent of dimensionality under the stated batching regime.

4. Adaptive injective behavior, core/detail separation, and inference-time use

EOFlows are described as adaptive injective flows because a bijection learned in (D) dimensions can be used, after training, as an injective model of varying intrinsic dimension (C). After entropy ordering, the latent vector is partitioned as
[
\mathbf{z}=[\mathbf{z}_C,\mathbf{z}_D],
\qquad
C={1,\dots,C},
\qquad
D={C+1,\dots,D}.
]
Given a data point (\mathbf{x}), one computes (\mathbf{z}=\mathbf{f}(\mathbf{x})), retains (\mathbf{z}_C), and reconstructs from the core only by setting the detail coordinates to zero:
[
\hat{\mathbf{x}}=\mathbf{g}([\mathbf{z}_C,\mathbf{0}]).
]
This defines the core reconstruction manifold
[
\mathcal{M}_C(0)={\mathbf{g}([\mathbf{z}_C,0]) : \mathbf{z}_C\in\mathbb{R}^C}.
]
All points sharing the same core latents but differing in detail latents project to the same reconstruction [2602.06940].

The operational interpretation is that the core stores semantic structure while the detail stores fine texture, local perturbations, and noise. The entropy spectrum supplies a model-internal criterion for choosing (C): one may select the point where (H_j) reaches the noise level (H_{\sigma}=\tfrac12+\log\sigma), or choose (C) by rate–distortion measurements such as PSNR and SSIM on a validation set. The reported CelebA experiments with (\sigma=0.1) place this plateau around a few dozen dimensions.

This inference-time flexibility supports several uses. For compression, one stores only (\mathbf{z}C). For denoising, one encodes a noisy input, keeps the core latents, sets the detail latents to zero, and decodes; this is described as projecting the input onto the learned core manifold. For controllable generation, one may sample (\mathbf{z}_C\sim\mathcal{N}(0,I_C)) and either set (\mathbf{z}_D=0) for minimal distortion or sample (\mathbf{z}_D\sim\mathcal{N}(0,I{D-C})) to restore correct global data statistics. The data further state that, using the theory of Blau and Michaeli (2019), sampling (\mathbf{z}_D) increases distortion by at most a factor of (2) in MSE relative to zeroing (\mathbf{z}_D), and that this was verified empirically with rate–distortion plots.

EOFlows also support latent editing. On CelebA, the reported procedure fixes all latent variables at zero except one coordinate (z_i), sets that coordinate to (\pm 4), decodes both images, and studies their difference. The resulting dimension-wise traversals are described as archetypes and are used to identify latent dimensions associated with head pose, hair color and style, glasses, beard, illumination, color temperature, eyebrows, and lips.

5. Empirical behavior on EMNIST, Entangled Digits, and CelebA

The reported experiments evaluate EOFlows on EMNIST digits, a synthetic Entangled Digits dataset, and CelebA faces. Across these settings, the main empirical pattern is that increasing (\lambda_{\mathrm{TC}}) decreases manifold total correlation, sharpens the entropy spectrum, and yields a more pronounced core/detail separation than unregularized flows or PCA [2602.06940].

On EMNIST ((D=784), (\sigma\in{0.01,0.03,0.1})), increasing (\lambda_{\mathrm{TC}}) reduces (\mathbb{E}[\mathcal{L}{\mathrm{TC}}]), with only modest degradation of average NLL until (\lambda{\mathrm{TC}}\approx 0.1)–(1). Larger noise makes the likelihood–disentanglement tradeoff more favorable. The qualitative analysis reports that average Jacobian columns for high-entropy dimensions capture global factors such as stroke thickness, slant, and scaling, whereas low-entropy dimensions recover structured EMNIST preprocessing artifacts, including grid-like patterns that are not visible to the eye and are not recovered by simple injective models that discard low-entropy directions.

On Entangled Digits, the dataset is constructed by superposing EMNIST 0s and 1s with a mixing coefficient (\alpha\sim\mathcal{U}[0,1]). After entropy ordering, the top latent dimension is reported to correlate with (\alpha) at approximately (0.92) on test data. Editing that latent from (-2) to (+2) moves reconstructions smoothly between the underlying digits, indicating that EOFlows can isolate a single mixing factor without supervision.

On CelebA, faces are center-cropped and resized to (28\times 28\times 3) with (D=2352), and EOFlows are trained with noise inflation (\sigma=0.1) and (\lambda_{\mathrm{TC}}\in{0,0.01,0.1,1.0}). The reported entropy spectra separate the methods clearly: an unregularized normalizing flow has a flat entropy spectrum, EOFlows exhibit a sharp decrease in the first tens of dimensions followed by saturation at the noise entropy level, and PCA shows a more gradual decay. This is presented as evidence that EOFlows concentrate entropy into a few core dimensions whereas PCA spreads it more evenly.

(\lambda_{\mathrm{TC}})	(\mathbb{E}[\mathcal{L}_*])	(\mathbb{E}[\mathcal{L}_{\mathrm{TC}}])
0	(-1.729)	(1.104)
0.01	(-1.730)	(0.048)
0.1	(-1.717)	(0.033)
1.0	(-1.691)	(0.012)

These numbers support two conclusions stated in the data. First, a small regularization weight, (\lambda_{\mathrm{TC}}=0.01), slightly improves NLL while drastically lowering total correlation, which is interpreted as regularization aiding generalization. Second, larger (\lambda_{\mathrm{TC}}) values continue to reduce total correlation but produce mild NLL degradation.

The qualitative CelebA results emphasize three properties. The first is interpretability: EOFlows produce sharp archetypes that are described as less blurry and more cartoonish than PCA eigenvectors. The second is stability across runs: different EOFlow runs with the same (\lambda_{\mathrm{TC}}) discover very similar archetypes. The third is useful bottleneck behavior: for regularized EOFlows, reconstructions with (C\in{5,10,20}) are heavily smoothed but recognizable, around (C\approx 50) they recover high-quality clean faces and denoise strongly, and at (C\approx 500) they begin to reintroduce noise because those dimensions primarily encode fine noise. By contrast, for (\lambda_{\mathrm{TC}}=0), varying (C) has little effect because information is spread across dimensions. The rate–distortion comparisons further state that EOFlows with (\lambda_{\mathrm{TC}}=0.01) or (0.1) outperform PCA and a denoising baseline based on Tweedie’s formula at low rates.

6. Relation to PCA, disentanglement models, and other entropy-based flow notions

EOFlows occupy a specific position within the broader landscape of latent-variable models. Relative to standard normalizing flows, they retain exact likelihood and invertibility but modify the objective so that the latent basis becomes ordered and approximately disentangled. Relative to PCA, they are a nonlinear generalization in which the variance spectrum is replaced by an entropy spectrum. Relative to (\beta)-VAEs and disentanglement VAEs, they do not impose a bottleneck during training and do not rely on approximate posteriors; instead, they regularize geometry through Jacobian orthogonality and derive a bottleneck post hoc from entropy ordering. Relative to manifold flows and rectangular flows, they do not fix the intrinsic dimension (C) during training. Relative to PCF, EOFlows inherit related information-theoretic quantities but add a stochastic Jacobian estimator that scales to high-dimensional images [2602.06940].

The term “entropy-ordered” also appears in adjacent but distinct senses. In Risk-Entropic Flow Matching, the log-exponential transform of the flow-matching residual induces Gibbs weights over microscopic velocity targets, and the accompanying synthesis states that this “defines an implicit ordering / prioritization of paths” in which rare or high-loss trajectories are emphasized. There, “entropy-ordered” refers to an ordering of trajectories by entropic risk rather than an ordering of latent dimensions by explained entropy [2512.03078].

A different usage appears in a turbulence context. A synthesis of "Topological Entropy of Stationary Turbulent Flows" proposes ordering turbulent flows by topological entropy, using the Eulerian formula
[

\mathcal{S}

\frac{1}{\tau}
\left[
\frac{1}{2}\left\langle G(\lambda,\zeta;\tau)\right\rangle
+
\frac{1}{4}\ln\left\langle H(\lambda,\zeta;\tau)\right\rangle
\right],
]
where the ordering compares entire flow configurations by mixing complexity. In that setting, an EOFlows perspective denotes an entropy-based classification of stationary incompressible turbulent flows rather than a generative latent-variable model [2506.20973].

A third neighboring construction is the "Entropy flow" introduced for one-parameter families of vector fields on (M\times I). There, the extended vector field
[
E(s,\tau)=(F_\tau(s)+V(s,\tau),P(s,\tau))
]
adds a drift in parameter space designed to move trajectories toward more complex dynamical regimes along routes to chaos. Complexity is encoded topologically through bifurcation structure, isolated invariant sets, and Conley-index transport rather than through latent entropies. This use is conceptually related but formally distinct from the normalizing-flow framework introduced in EOFlows [2606.24289].

Taken together, these papers show that “entropy-ordered flows” is not a single universal formalism. In the specific sense of EOFlows, the defining object is an entropy-ordered latent coordinate system in an invertible generative model. The broader literature uses related language for trajectory reweighting, turbulence ranking, and parameter-drift constructions, but those are separate frameworks.

7. Limitations, caveats, and open problems

The principal limitation reported for EOFlows is a tradeoff between exact density modelling and disentanglement. If (\lambda_{\mathrm{TC}}) is too small, ordering and disentanglement are weak; if it is too large, NLL degrades and expressivity may be reduced. The CelebA table makes this tradeoff explicit: (\mathbb{E}[\mathcal{L}{\mathrm{TC}}]) improves monotonically as (\lambda{\mathrm{TC}}) increases, while (\mathbb{E}[\mathcal{L}_*]) eventually worsens [2602.06940].

A second limitation is residual entanglement. The data state that high-entropy dimensions retain some residual interactions even as off-diagonal pairwise manifold mutual information shrinks with increasing (\lambda_{\mathrm{TC}}). This is interpreted as an expressivity-versus-disentanglement tradeoff: the most informative dimensions cannot always be made perfectly orthogonal without loss.

A third limitation is computational. The JVP-based regularizer requires a batch size (B\ge D) in the reported implementation, which is memory-intensive for high-dimensional data; on CelebA this means (B\ge 2352). Training is also approximately (2.5\times) slower than maximum-likelihood flows.

A fourth caveat is noise dependence. The entropy cutoff, the apparent intrinsic dimension, and the interpretation of low-entropy coordinates are tied to the inflation noise level (\sigma). The data explicitly state that intrinsic dimension is not absolute but limited by measurement accuracy, and the entropy spectrum’s saturation point is correspondingly noise-level dependent.

Theoretical identifiability also remains incomplete. EOFlows inherit IMA-style identifiability arguments, but the conditions under which the method recovers true generative factors are stated not to be completely analyzed. This suggests that the ordered coordinates should be interpreted as a stable, geometry-aware coordinate system on the learned data manifold rather than, in general, as a formally unique decomposition into ground-truth causal factors.

Despite these caveats, EOFlows establish a precise framework for combining exact-likelihood normalizing flows with manifold entropy, Jacobian orthogonality, and inference-time adaptive dimensionality reduction. Their distinctive contribution is to replace an undifferentiated latent space with an entropy spectrum, making the distinction between semantic core and detail an explicit property of the trained flow rather than a manually fixed architectural prior.