Orthogonium : A Unified, Efficient Library of Orthogonal and 1-Lipschitz Building Blocks

Abstract: Orthogonal and 1-Lipschitz neural network layers are essential building blocks in robust deep learning architectures, crucial for certified adversarial robustness, stable generative models, and reliable recurrent networks. Despite significant advancements, existing implementations remain fragmented, limited, and computationally demanding. To address these issues, we introduce Orthogonium , a unified, efficient, and comprehensive PyTorch library providing orthogonal and 1-Lipschitz layers. Orthogonium provides access to standard convolution features-including support for strides, dilation, grouping, and transposed-while maintaining strict mathematical guarantees. Its optimized implementations reduce overhead on large scale benchmarks such as ImageNet. Moreover, rigorous testing within the library has uncovered critical errors in existing implementations, emphasizing the importance of standardized and reliable tools. Orthogonium thus significantly lowers adoption barriers, enabling scalable experimentation and integration across diverse applications requiring orthogonality and robust Lipschitz constraints. Orthogonium is available at https://github.com/deel-ai/orthogonium.

• The paper introduces Orthogonium, a comprehensive library implementing orthogonal and 1-Lipschitz layers for robust neural network design.
• It presents multiple orthogonalization techniques such as Björck–Bowie, Cayley Transform, and Modified Gram–Schmidt with rigorous unit testing and spectral normalization.
• The library enables robust adversarial defenses and stable training in RNNs, GANs, and other architectures while supporting modern convolutional features.

Orthogonium: A Unified, Efficient Library for Orthogonal and 1-Lipschitz Building Blocks

Overview

This paper introduces "Orthogonium" (2601.13776), a comprehensive PyTorch library that provides standardized, high-performance implementations of orthogonal and 1-Lipschitz neural network layers. These layers are crucial for adversarial robustness, stable training of generative models, isometry-preserving recurrent architectures, and theoretical analysis requiring strict norm preservation. The Orthogonium library consolidates diverse methodologies, supports essential features (stride, dilation, grouping, transposition), and prioritizes both implementation efficiency and extensibility. Extensive unit testing also uncovered errors in earlier works, underlining the importance of reliable, open implementations.

Figure 1: Orthogonium provides a standardized API for constructing, evaluating, and extending 1-Lipschitz layers, enabling efficient network design and experimentation.

Motivation and Library Design

Orthogonal and 1-Lipschitz layers have become indispensable for certified adversarial robustness, normalizing flows, Wasserstein GANs, stability in deep RNNs, and applications where bounded operator norms are structurally required. Despite numerous advances, practical deployment has been hindered by fragmented codebases, non-uniform APIs, a lack of support for current deep learning features (strides, groupings), and inefficient kernels. Existing approaches often restrict to canonical $3 \times 3$ convolutions, cannot scale to modern architectures, or fail to validate mathematical constraints rigorously in implementation.

Orthogonium addresses these issues by centralizing recent methods, exposing a coherent and flexible API, supporting state-of-the-art convolutional primitives, and imposing strict validation. It makes rigorous method comparison and adoption feasible even on large-scale datasets like ImageNet, and enables cross-domain integration where robust or stable architectures are a prerequisite.

Dense Layer Support and Orthogonalization Algorithms

The dense layer API, exposed as OrthoLinear, builds on PyTorch's module interface, supporting multiple orthogonalization parameterizations:

• Björck–Bowie Iterative Projection: Iteratively projects onto orthogonality with rapid convergence and spectral preconditioning.
• Exponential Map: Exploits the Lie algebraic structure to generate orthogonal matrices from skew-symmetric bases.
• Modified Gram–Schmidt: Numerically stable, suitable for full-rank cases and rectangular matrices.
• Cayley Transform: Maps skew-symmetric matrices to orthogonal matrices, extended for the non-square case.
• Cholesky Decomposition: Enforces orthogonality via Gram matrix normalization, exploiting high-performance linear algebra.

Spectral normalization is optionally applied as a stabilization step to all these parametrizations, providing both theoretical and empirical control over layer norms. The API is modular, extensible, and can be rapidly integrated into research codebases.

Orthogonal and 1-Lipschitz Convolutions

Orthogonium implements a spectrum of orthogonal and Lipschitz-controlled convolutional layers. These include:

• Adaptive Orthogonal Convolution (AOC): Extends BCOP kernels, supporting arbitrary kernel sizes, strides, dilations, and groupings natively. Yields $\leq 1.13\times$ wall-time overhead on ImageNet-scale data with batch size 256, making strict orthogonality practical at scale.
• Adaptive-SOC: Combines Skew Orthogonal Convolution with AOC's stride-awareness, decoupling cost from batch size and supporting grouped/dilated/transposed convolutions. Uses improved normalization for faster training convergence.
• Almost-Orthogonal Layers (AOL): Implements efficient nearly orthogonal convolutions with Lipschitz constants $\leq 1$ via the Gram iteration, providing a favorable accuracy-speed tradeoff for applications not requiring strict orthogonality.
• SDP-based Lipschitz Layers (SLL): Implements provably 1-Lipschitz residual blocks, extended for grouped/dilated/strided convolutions. Efficient blocks can seamlessly replace ResNet downsampling branches.
• Sandwich-AOC: Removes the Fourier-domain Cayley bottleneck from "sandwich layers" by leveraging explicit AOC kernels, supporting costly configurations in pixel space for large input images.

  Figure 3: Block convolution composition enables SLL variants with stride and changing channel dimensions, maintaining the ResNet-style residual block structure while preserving 1-Lipschitz constraints.

All modules are PyTorch-native, maintain feature parity with standard layers, and are validated for orthogonality or Lipschitz properties under all configuration permutations.

Additional Components: Nonlinearities, Normalization, Residuals

The library centralizes components required for truly 1-Lipschitz architectures:

• Activations: Custom gradient-norm-preserving units (Abs, SoftHuber, MaxMin, HouseHolder, and variants), all tested for $1$-Lipschitzness and corrected where necessary (e.g., missing scaling factors in HouseHolder from prior works).
• Normalization: BatchCentering and LayerCentering replace BatchNorm, preserving norm control without variance scaling.
• Residual Wrappers: Implement exact 1-Lipschitz skip connections (ConcatResidual, L2NormResidual, AdditiveResidual, PrescaledAdditiveResidual), ensuring that skip-connection benefits are not traded for loss of global norm guarantees.

Validation and Unit Testing

Empirical validation is essential due to implementation-level nuances and possible numerical instability. Orthogonium's unit tests combine:

• Explicit SVD on Toeplitz matrices for exhaustive validation on small inputs,
• Parametrization-aware scalable spectral methods for practical, large-scale testing.

All layer configurations (stride, group, dilation, transposition) are systematically tested. Nonlinear blocks are checked via randomized empirical Jacobian SVDs. Notably, flaws were found and corrected in previously published HouseHolder activations. The library reaches 94% test coverage, certifies tight operator norms, and programmatically rejects configurations known to violate theoretical constraints.

Implications and Future Perspectives

Orthogonium provides a standardized foundation for research and deployment of provable robust, stable, or invertible networks, immediately facilitating:

• Certified Adversarial Robustness: Tighter robustness certificates due to strict norm control.
• Reliable Generative Modeling: Improved training stability in normalizing flows and Wasserstein GANs.
• Stable RNNs and PINNs: Ensures long-sequence stability or backward uniqueness, addressing vanishing/exploding gradients and physical law enforcement.
• Safety-Critical Systems: Reduction of empirical risk of misconfiguration in high-assurance deployments (e.g., perception, privacy, finance).

Open-sourcing the library also acts as a community "reference implementation," supporting continuous verification, correction, and method extension. Given the modular design and full feature parity with unconstrained networks, rapid experimentation with new architectural ideas under provable constraints is enabled.

On the theoretical side, Orthogonium's flexible architecture supports ongoing research into the interplay of Lipschitz-constrained operators, nonlinearities, and global norm bounds—particularly important as scalable network certification and SDP-based analysis (2601.13776) become more tractable for deep and wide models.

Conclusion

Orthogonium (2601.13776) unifies a vast landscape of orthogonal and Lipschitz-constrained methods, bridging the gap between mathematical guarantee and practical deployment. By providing rigorously tested, efficient, and fully-featured building blocks, the library facilitates robust, scalable network construction. Its adoption is expected to advance empirical research on certified robustness, generative modeling, and stable learning paradigms, while simultaneously reducing implementation errors in high-stakes applications. Future advances in norm-constrained architectures can be quickly evaluated, standardized, and disseminated using this framework, accelerating both practical and foundational progress in robust machine learning.