i-RevNet: Deep Invertible Networks (1802.07088v1)

Abstract: It is widely believed that the success of deep convolutional networks is based on progressively discarding uninformative variability about the input with respect to the problem at hand. This is supported empirically by the difficulty of recovering images from their hidden representations, in most commonly used network architectures. In this paper we show via a one-to-one mapping that this loss of information is not a necessary condition to learn representations that generalize well on complicated problems, such as ImageNet. Via a cascade of homeomorphic layers, we build the i-RevNet, a network that can be fully inverted up to the final projection onto the classes, i.e. no information is discarded. Building an invertible architecture is difficult, for one, because the local inversion is ill-conditioned, we overcome this by providing an explicit inverse. An analysis of i-RevNets learned representations suggests an alternative explanation for the success of deep networks by a progressive contraction and linear separation with depth. To shed light on the nature of the model learned by the i-RevNet we reconstruct linear interpolations between natural image representations.

• The paper demonstrates that maintaining invertibility via homeomorphic transformations preserves complete input information throughout the network.
• It innovatively replaces non-invertible layers with fully reversible modules, achieving competitive performance on benchmarks like ImageNet.
• The reversible architecture enhances model interpretability and suggests new avenues for generative modeling and robust analysis.

An Analysis of $i$-RevNet: Deep Invertible Networks

The paper "i-RevNet: Deep Invertible Networks" explores the development and implications of invertible neural networks, specifically the $i$-RevNet model. Contrary to conventional CNN architectures, which traditionally lose information across layers to extract meaningful features, $i$-RevNet proposes maintaining the input data's information throughout the network. This essay offers a synthesized account of the paper, examining its methodological strengths and the broader implications of its findings in deep learning.

Motivation and Innovation

A core motivation behind this work is to demonstrate that discarding information via non-invertible transformations is not necessary for achieving high-performance models. The authors introduce $i$-RevNet, a network that ensures invertibility through homeomorphic transformations. By maintaining data fidelity until the final classification layer, $i$-RevNet challenges the perceived trade-off between the complexity of representations and information retention. The network accomplishes this through the use of an innovative cascade of invertible layers, which ensures that no information is discarded throughout the entire network processing.

Architecture of $i$-RevNet

The $i$-RevNet is grounded in the reversible network architecture introduced by RevNet, but it extends this concept by replacing non-invertible components with fully invertible layers. Each processing block in the network consists of a combination of specialized operations, including split and merge procedures, convolutional transformations, and invertible down-sampling. Such a structure ensures that the transformation at every stage can be inverted, thereby formally preserving information.

Two variants of $$i-RevNet were examined: one injective, involving increased channel size, and one bijective, maintaining constant dimensionality. Both models were benchmarked on ImageNet, demonstrating competitive performance to existing architectures like RevNet and ResNet. The injective variant, in particular, illustrates that even when channels are expanded to support inversion, comparable accuracy remains achievable.</p> <h3 class='paper-heading'>Empirical Findings</h3> <p>The authors performed a thorough empirical evaluation of$$i$$-RevNet, showing that it successfully maintains complete input information up to the final classification projection. The paper highlights several crucial findings:</p> <ul> <li><strong>Ill-conditioned Inversion:</strong> The differential of the$$i$$-RevNet transformations suggests a locally confined invertible space, reinforcing that the network parses input into a manageable, albeit high-dimensional, interpretable form.</li> <li><strong>Linear Separation and Contraction:</strong> The$$i$-RevNet exhibits progressive linear separability and compact feature distributions across layers, comparable to those seen in non-invertible architectures.</li> </ul> <p>Additionally, a cross-examination of feature linearity suggests that the network&#39;s feature space truly resides in a notably lower-dimensional subspace, facilitating efficient separation of classes without loss of information.</p> <h3 class='paper-heading'>Implications and Future Directions</h3> <p>The $i$$-RevNet architecture opens several new research pathways in the field of deep learning:</p> <ul> <li><strong>Invertibility as a Primary Design Principle:</strong> It highlights the viability of constructing invertible architectures that retain informative decision boundaries, challenging the necessity of information discard.</li> <li><strong>Interpretability and Robustness:</strong> By allowing backmapping from feature to input space,$$i$-RevNet holds promise for enhancing model interpretability and robustness, particularly in tasks requiring high fidelity recoveries, such as medical imaging or scientific computing.</li> <li><strong>Generative Modeling:</strong> i-RevNet&#39;s invertible structure can inspire development in generative model design, potentially merging powerful discriminative capabilities with generative expressiveness.</li> </ul> <p>It bears noting that, while the increased computational overhead associated with channel expansion and depth could be a limitation, continued optimization and hardware advances could alleviate such concerns. Subsequent research would benefit from examining the balance between model complexity, computational resources, and interpretability.</p> <p>In conclusion, $i$-RevNet represents a valuable contribution to the discourse around invertible networks, setting a foundational framework for models that maintain high levels of data integrity from input to inference.