Orthogonium: Efficient Orthogonal & Lipschitz Layers

• Orthogonium is a unified PyTorch library providing orthogonal and 1-Lipschitz layers with exact spectral-norm control for robust and stable deep learning architectures.
• It integrates multiple orthogonalization methods like QR, Cayley, and Björck–Bowie while natively supporting modern convolution features such as strides, dilations, and groups without shape hacks or FFTs.
• The library offers efficient CUDA kernels and in-place parametrizations that minimize computational overhead, enabling reproducible research in certified adversarial robustness, generative modeling, and recurrent networks.

Searching arXiv for the cited Orthogonium papers and closely related records. Orthogonium is a unified, efficient, and comprehensive PyTorch library providing orthogonal and 1-Lipschitz layers for deep learning architectures in which exact or tight spectral-norm control is a central design requirement. It is positioned as infrastructure for certified adversarial robustness, stable generative models, and reliable recurrent networks, while also aiming to remove practical barriers created by fragmented implementations, incomplete operator support, and computational overhead. The library is open-source at https://github.com/deel-ai/orthogonium and is presented both as a general library of orthogonal and 1-Lipschitz building blocks and as the software vehicle for the Adaptive Orthogonal Convolution scheme, or AOC (Boissin et al., 20 Jan 2026, Boissin et al., 14 Jan 2025).

1. Role within orthogonal and 1-Lipschitz deep learning

Orthogonal and 1-Lipschitz layers are described as critical components for provable robustness and stability in deep learning. The stated rationale is that if weight matrices or convolutional operators satisfy exact or tight spectral-norm constraints, then one can obtain certified adversarial robustness, stable generative models, and reliable recurrent architectures. In the formulation used by Orthogonium, certified adversarial robustness is linked to the fact that a 1-Lipschitz network cannot amplify small perturbations; stable generative modeling is linked to normalizing flows and Wasserstein GANs, which rely on invertibility or tight Lipschitz constants; and recurrent reliability is linked to orthogonality preventing exploding or vanishing gradients (Boissin et al., 20 Jan 2026).

The motivating problem is not the absence of methods, but the fragmentation of their implementations. The source material enumerates “over a dozen papers” proposing orthogonalization schemes such as BCOP, RKO, skew-exponential, Cayley, and Björck–Bowie, while also emphasizing that their implementations are “scattered, incomplete, or incompatible with modern convolutional features such as stride, dilation, grouping, and transposed variants.” Reimplementing and validating each method is described as prohibitively time-consuming, and naïve projections or power-iterations are described as capable of incurring large computational overheads that can halt large-scale experiments on ImageNet or semantic segmentation. Orthogonium is therefore presented as a standardizing layer over a previously heterogeneous methodological landscape.

A common misconception in this area is that strict orthogonality and modern convolutional functionality are intrinsically at odds. Orthogonium is explicitly designed against that assumption: it states native support for strides, dilations, groups, and transposed convolutions, with no shape hacks or FFTs, while maintaining strict mathematical guarantees. This suggests that the library’s contribution is as much about systems integration and operator coverage as about individual parametrization schemes.

2. Mathematical foundations

Orthogonium adopts the standard Lipschitz definition: an operator $f$ is $L$-Lipschitz if

$$\|f(x) - f(y)\|_2 \;\le\; L\,\|x-y\|_2\quad\forall x,y.$$

For a linear layer with weight matrix $W\in\mathbb R^{m\times n}$, the spectral norm $\|W\|_2=\sigma_{\max}(W)$ must satisfy $\sigma_{\max}(W)\le L$. An orthogonal layer is the special case $L=1$ and $m=n$ with

$W^\top W = I_n.$

These definitions establish the library’s dual emphasis: orthogonality as a special case of 1-Lipschitzness, and spectral control as the mechanism by which both are enforced (Boissin et al., 20 Jan 2026).

For convolutions, the AOC presentation makes the induced linear operator explicit. A standard convolution kernel $W\in\mathbb R^{c_o\times c_i\times k_h\times k_w}$ corresponds to a Toeplitz matrix $L$0. Orthogonality is then expressed either as row-orthogonality,

$L$1

or as column-orthogonality,

$L$2

When $L$3, the two coincide; otherwise one chooses the regime. The AOC text gives the example $L$4 column-orthogonal (Boissin et al., 14 Jan 2025).

In practice, Orthogonium enforces these constraints by several mechanisms. The first is spectral normalization by power iteration:

$L$5

followed by $L$6. The second is orthogonal retractions or parametrizations, specifically QR (Modified Gram–Schmidt), Cayley transform, Lie-exponential of a skew matrix, Cholesky-based solve, and Björck–Bowie fixed-point iterations. The Cayley construction is also stated in the AOC source as

$L$7

Together, these formulations situate Orthogonium within the established literature on exact and approximate spectral control rather than as a single-method package.

3. Library organization and exposed building blocks

Orthogonium covers both dense and convolutional layers under a single, explicit API. Its major layer classes include dense orthogonal layers, multiple convolutional orthogonalizations, and 1-Lipschitz residual constructions (Boissin et al., 20 Jan 2026).

Component	Description
OrthoLinear	Extends torch.nn.Linear with an OrthoParams object registering one of five orthogonalizations and optional spectral norm preconditioning
AdaptiveOrthoConv2d / ConvTranspose2d (AOC)	Explicit convolutional kernel combining BCOP and RKO techniques, supporting stride, dilation, groups, and transposed natively
AdaptiveSOCConv2d	Fuses skew-exponential SOC with AOC’s stride-aware kernel; caches the exponential once per update and uses the “AOL” normalization to converge in 3–4 iterations
AOLConv2d	“almost-orthogonal” multi-step projection that guarantees Lipschitz≤1, implemented via PyTorch’s parametrize API and refined using Gram iterations
SDPBasedLipschitzConv	Tight SDP normalization for 1-Lipschitz convolutional constructions
SLLxAOCResBlock	1-Lipschitz residual block embedding a nonlinearity inside convolution plus tight SDP normalization; extended to support stride and channel changes by fusing AOC blocks
SandwichAOC	Under development; replaces costly FFT/Cayley steps in sandwich layers with pairwise AOC kernels for direct pixel-space operations

The library also defines nonlinearity and residual-wrapper primitives intended to preserve unit-Lipschitz structure. The unit-Lipschitz activations listed are Abs, SoftHuber, MaxMin, and Householder(2nd order). The centering normalizations are BatchCentering and LayerCentering, with the explicit note that variance is untouched to preserve norms. The Lipschitz-preserving residual wrappers are ConcatResidual, L2NormResidual, AdditiveResidual, and PrescaledAdditiveResidual, with the corresponding formulas

$L$8

for L2NormResidual,

$L$9

for AdditiveResidual, and

$$\|f(x) - f(y)\|_2 \;\le\; L\,\|x-y\|_2\quad\forall x,y.$$0

for PrescaledAdditiveResidual.

This organization is significant because it places linear operators, nonlinearities, normalizations, and residual pathways under the same Lipschitz-aware design discipline. A plausible implication is that Orthogonium is intended not merely as a set of isolated kernels, but as a compositional library for end-to-end 1-Lipschitz or orthogonal network design.

4. Adaptive Orthogonal Convolution and operator generality

AOC is the core convolutional scheme associated with Orthogonium. It is described as a scalable method extending BCOP and overcoming limitations in the construction of orthogonal convolutions for large-scale applications. The central design choice is that AOC never projects at run time; instead it constructs an explicitly orthogonal convolutional kernel in the spatial domain by composing BCOP-style factorization with RKO reshaped-kernel orthogonalization (Boissin et al., 14 Jan 2025).

The BCOP-style component factorizes convolutions into 1×1, 1×2, and 2×1 elementary orthogonal convolutions. The construction starts with an orthogonal 1×1 kernel $$\|f(x) - f(y)\|_2 \;\le\; L\,\|x-y\|_2\quad\forall x,y.$$1, forms a 1×2 kernel $$\|f(x) - f(y)\|_2 \;\le\; L\,\|x-y\|_2\quad\forall x,y.$$2 with $$\|f(x) - f(y)\|_2 \;\le\; L\,\|x-y\|_2\quad\forall x,y.$$3 as a projector, forms a 2×1 kernel $$\|f(x) - f(y)\|_2 \;\le\; L\,\|x-y\|_2\quad\forall x,y.$$4 similarly, and fuses these pieces using the block-convolution operator $$\|f(x) - f(y)\|_2 \;\le\; L\,\|x-y\|_2\quad\forall x,y.$$5. The source states the proposition that if $$\|f(x) - f(y)\|_2 \;\le\; L\,\|x-y\|_2\quad\forall x,y.$$6 and $$\|f(x) - f(y)\|_2 \;\le\; L\,\|x-y\|_2\quad\forall x,y.$$7 are both row-orthogonal then $$\|f(x) - f(y)\|_2 \;\le\; L\,\|x-y\|_2\quad\forall x,y.$$8 is row-orthogonal. The RKO component supports native stride $$\|f(x) - f(y)\|_2 \;\le\; L\,\|x-y\|_2\quad\forall x,y.$$9 by orthogonalizing a flattened kernel $W\in\mathbb R^{m\times n}$0 via a matrix orthogonalization such as QR or Björck, then reshaping it back to $W\in\mathbb R^{m\times n}$1. The stated proposition is that if $W\in\mathbb R^{m\times n}$2 then the resulting strided convolution is exactly orthogonal.

The fused AOC kernel is written as

$W\in\mathbb R^{m\times n}$3

where $W\in\mathbb R^{m\times n}$4, $W\in\mathbb R^{m\times n}$5, and $W\in\mathbb R^{m\times n}$6. The associated block-convolution operator is defined by

$W\in\mathbb R^{m\times n}$7

Its naive cost is stated as $W\in\mathbb R^{m\times n}$8, while Orthogonium implements it as “a single chained 2D convolution (CUDA kernel), fusing loops and exploiting batch/group parallelism.”

The operator coverage claims are unusually broad. AOC is stated to be the first orthogonal convolution scheme to support natively strides $W\in\mathbb R^{m\times n}$9, dilations, group convolutions, and transposed convolutions. In Orthogonium’s library-level wording, this appears as support for strides, dilation, grouping, and transposed convolution features “without shape hacks or FFTs,” and with forward execution as a single torch.nn.Conv2d call on the materialized weight. This is central to Orthogonium’s identity: it treats strict orthogonality as compatible with the standard operational vocabulary of modern CNNs rather than as a special-case constraint limited to atypical architectures.

5. Efficiency, verification, and corrections

Orthogonium emphasizes efficiency and correctness jointly. Its implementation uses fused CUDA kernels and in-place parametrizations to limit extra memory or compute. It also uses explicit block-convolution operators to allow multi-kernel fusions, so that exponentials or residual fusions occur once per update, not per sample, and native stride/dilation/group support avoids tensor reshaping or FFT overhead. The library’s validation strategy combines SVD on Toeplitz matrices, described as exact on small inputs, with scalable spectral-norm checks via conv.singular_values, enforcing $\|W\|_2=\sigma_{\max}(W)$0 with tolerance $\|W\|_2=\sigma_{\max}(W)$1 (Boissin et al., 20 Jan 2026).

At ImageNet scale, the headline efficiency claim in the Orthogonium paper is that on ImageNet-1K with batch size 256 and mixed precision, AOC layers run within $\|W\|_2=\sigma_{\max}(W)$2 the wall-time of nn.Conv2d, yielding an average overhead of approximately $\|W\|_2=\sigma_{\max}(W)$3. The AOC paper provides the following representative runtime and memory measurements for ResNet-34 / ImageNet (224×224):

Setting	Conv2D reference	AOCConv2d
Batch 128	137 ms train, 4.7 GB	239 ms (1.75×), 5.3 GB (1.15×)
Batch 512	550 ms, 17.9 GB	622 ms (1.13×), 18.6 GB (1.04×)

The same source reports construction overhead of each layer’s kernel as approximately $\|W\|_2=\sigma_{\max}(W)$4–$\|W\|_2=\sigma_{\max}(W)$5 on small images (32×32), falling to approximately $\|W\|_2=\sigma_{\max}(W)$6–$\|W\|_2=\sigma_{\max}(W)$7 on ImageNet-scale (224×224) as $\|W\|_2=\sigma_{\max}(W)$8 grows, with memory overhead $\|W\|_2=\sigma_{\max}(W)$9 due to no large FFT buffers, and with overhead percentage shrinking as batch size $\sigma_{\max}(W)\le L$0 increases (Boissin et al., 14 Jan 2025).

The library also foregrounds corrective work on prior implementations. The examples given are that the Householder activation was missing a $\sigma_{\max}(W)\le L$1 scaling and is now fixed, SOC normalization switched from “fantastic four” to AOL, halving iteration count, and SLL was extended to arbitrary stride/channel via block-fusion. In the layer description, AdaptiveSOCConv2d is said to use the “AOL” normalization to converge in 3–4 iterations. These points matter because Orthogonium presents rigorous unit testing not only as quality assurance for its own code, but as a mechanism that uncovered subtle bugs in existing implementations. This suggests that standardization is treated as a correctness issue, not merely a usability issue.

6. Integration, applications, limitations, and reported impact

Orthogonium is designed to integrate seamlessly with PyTorch. The provided example uses $\sigma_{\max}(W)\le L$6 and constructs a LipschitzResNetBlock in which AdaptiveOrthoConv2d supplies an orthogonal 3×3 convolution with stride, MaxMin provides a unit-Lipschitz activation, and ConcatResidual ensures 1-Lipschitzness. A dense example uses OrthoLinear(128, 128, bias=False, ortho_method='qr'). Configuration options stated for AdaptiveOrthoConv2d are stride, dilation, groups, padding_mode, and transposed, with the note that they “all plug into AdaptiveOrthoConv2d with no loss of orthogonality.” The ortho_method choices listed are 'aoc', 'soc', 'aol', 'qr', 'cayley', 'cholesky', and 'bjorck', and spectral-norm-only layers use spectral=True in OrthoParams (Boissin et al., 20 Jan 2026).

The applications named across the two sources are adversarially robust 1-Lipschitz classifiers with provable bounds, normalizing flows and invertible ResNets, WGAN critics requiring Lipschitz continuity, and deep or recurrent networks where gradient norm preservation is critical. The AOC paper also reports representative downstream benchmarks: on CIFAR-10 with 84 M parameters, BCOP is reported at 72.2% clean accuracy and 58.3% provable accuracy at $\sigma_{\max}(W)\le L$2, AOC in “pure accuracy mode” at 80.0% clean accuracy and 60.1% provable accuracy, and AOC in “robustness mode” at 74.0% clean accuracy and 64.3% provable accuracy. On ImageNet-1K with 53 M parameters, “AOC + Cosine similarity” is reported at 68.2% Top-1 accuracy, while “AOC + margin $\sigma_{\max}(W)\le L$3” is reported at 42.1% Top-1 accuracy with certified $\sigma_{\max}(W)\le L$4 of 26.3% (Boissin et al., 14 Jan 2025).

The stated limitations are specific. BCOP parametrization is described as incomplete for certain small 2×2 kernels, resolved in practice by using 3×3 kernels. Construction overhead remains nonzero, with small-resolution applications such as 8×8 toy problems seeing a larger relative slowdown. At very large kernel sizes $\sigma_{\max}(W)\le L$5, pre-compute cost grows; future work may exploit low-rank factorizations. For exact theory, Orthogonium is stated to assume circular padding, while zero padding is also supported empirically but may introduce tiny boundary effects. These qualifications are important because they delimit the scope of the guarantees and clarify that “strict, explicit spatial-domain orthogonality” is paired with nontrivial implementation assumptions.

In its own impact framing, Orthogonium is described as providing “a field-wide foundation for deploying orthogonal and 1-Lipschitz architectures at scale,” from certified-robust classifiers to stable GANs and invertible normalizing flows, and as lowering adoption barriers by standardizing APIs, rigorously testing each block, and delivering optimized kernels. This suggests a role not only as a software package, but as a reference implementation layer for reproducible research on orthogonality and robust Lipschitz constraints in PyTorch (Boissin et al., 20 Jan 2026).