Orthogonal Band Convolutions Overview

• Orthogonal band convolutions are a family of convolution operators that impose orthogonality on finite-support, banded matrices, including architectural variants like InceptionMamba’s band branches.
• They achieve exact operator orthogonality through methods such as paraunitary factorization, operator regularization, and Lie-algebraic constructions, ensuring norm preservation and near-isometry.
• They span diverse applications from convolutional network design to noncommutative probability, highlighting the need for terminological clarity across fields.

Orthogonal band convolutions occupy a heterogeneous terminological space spanning at least two distinct CNN lineages and one unrelated noncommutative-probability lineage. In convolutional-network theory, the phrase can denote finite-support convolutional operators whose induced global matrix is orthogonal or semi-orthogonal and whose locality makes that matrix banded, Toeplitz, or block-circulant under appropriate boundary conditions (Achour et al., 2021). In InceptionMamba, by contrast, “orthogonal band convolutions” denotes a depthwise local modeling branch formed by the summed pair $DWConv_{3\times 11}+DWConv_{11\times 3}$, intended as a replacement for one-dimensional strip convolutions (Wang et al., 10 Jun 2025). In operator-valued free probability, “orthogonal convolution” is instead a noncommutative additive convolution characterized by reciprocal Cauchy transforms rather than spatial filtering (Liu, 2018).

1. Terminological scope and disambiguation

The available literature suggests that “orthogonal band convolutions” is not a single standardized construction. One CNN usage is structural: finite-support convolutions induce localized, hence banded, global linear operators, and orthogonality is imposed on that operator. Another usage is architectural: elongated kernels with mutually orthogonal orientations are combined to improve local spatial modeling. These usages overlap only partially.

In the architectural usage introduced by InceptionMamba, “band” refers to asymmetric kernels such as $3\times 11$ and $11\times 3$, each having long extent in one spatial direction and thickness $3$ in the other. The term “orthogonal” refers to the pairing of horizontal-like and vertical-like bands inside the same branch. In the operator-theoretic usage, orthogonality is instead a property of the full convolution map, typically written as row orthogonality or column orthogonality of the induced matrix. A common conflation is therefore to identify orthogonal-orientation band kernels with exact orthogonal convolutions; the available definitions do not support that equivalence (Wang et al., 10 Jun 2025, Achour et al., 2021).

This distinction matters because the two usages target different deficiencies. Exact orthogonal-convolution work is motivated by norm preservation, stable gradient propagation, $1$-Lipschitz design, invertibility, and certified robustness. The InceptionMamba formulation is motivated by the limited directional coverage of one-dimensional strip convolutions and by the need for more cohesive local spatial modeling.

2. Orthogonality as a property of banded convolution operators

In the operator-theoretic formulation, a convolutional layer with architecture $(M,C,k,S)$ is represented by a global matrix

$\mathcal K \in \mathbb R^{MN^2 \times CS^2N^2},$

acting on vectorized inputs by

$\mathrm{Vec}(Y)=\mathcal K\,\mathrm{Vec}(X).$

Because $\mathcal K$ is generally rectangular, orthogonality splits into two cases. In the row-orthogonal case, when $M\le CS^2$,

$3\times 11$0

In the column-orthogonal case, when $3\times 11$1,

$3\times 11$2

Finite support implies bandedness: a kernel of size $3\times 11$3 mixes only nearby spatial locations, so each single-channel convolution matrix is banded. Under circular padding, the single-channel operators become circulant in $3\times 11$4D and doubly block-circulant in $3\times 11$5D, and the full layer matrix is assembled from banded block-circulant blocks. This is the strongest precise sense in which orthogonal convolutional layers may be called orthogonal band convolutions.

A tractable kernel-space characterization is obtained through channel-summed correlations. Let

$3\times 11$6

Then orthogonality in the row-orthogonal case is equivalent to

$3\times 11$7

This encodes two simultaneous conditions: at zero relative shift, output-channel filters are orthonormal across channels; at all other sampled shifts, the overlaps vanish. The associated regularizer $3\times 11$8 is defined directly from this tensor condition and satisfies

$3\times 11$9

The existence theory is unusually sharp. For $11\times 3$0 and $11\times 3$1, orthogonal convolutional layers exist in the row-orthogonal case if and only if

$11\times 3$2

and in the column-orthogonal case if and only if

$11\times 3$3

For circular padding, this covers almost all practical architectures. The favorable theory does not extend unchanged to other boundary conditions: with valid padding there exists no orthogonal convolutional layer in the column-orthogonal case when $11\times 3$4, and with same zero-padding in $11\times 3$5D and $11\times 3$6, exact orthogonality forces the kernel to be trivial center-delta channel mixing. The same work also proves stability and scalability results, including

$11\times 3$7

under circular padding and $11\times 3$8, together with size-independent spectral bounds of the form

$11\times 3$9

showing that small $3$0 implies near-isometry even at large input resolutions (Achour et al., 2021).

3. Principal construction frameworks for exact orthogonal CNN layers

One line of work formulates orthogonality directly at the level of the convolution operator $3$1, not the flattened kernel tensor. In “Orthogonal Convolutional Neural Networks,” the layer is written as

$3$2

with $3$3 interpreted as a doubly block-Toeplitz operator. The central claim is that kernel orthogonality is only necessary but not sufficient for orthogonal convolution, because the im2col representation $3$4 introduces an additional structured linear transform from $3$5 to $3$6 whose spectrum is not necessarily uniform. This operator-level view is used to motivate a convolutional orthogonal regularizer that imposes orthogonality on the full structured map rather than on flattened filters alone (Wang et al., 2019).

A second line of work gives an exact spectral characterization through paraunitary systems. In this framework, convolution with transfer matrix $3$7 is orthogonal if and only if

$3$8

For finite-length $3$9D systems, a complete factorization is available: $1$0 where $1$1 is orthogonal, each $1$2 is column-orthogonal, and

$1$3

This yields SC-Fac, an exact and complete parameterization for $1$4D finite-length orthogonal convolutions. The same framework extends the paraunitary characterization to strided, dilated, and group convolutions, and supports deep orthogonal architectures such as ResNet, WideResNet, and ShuffleNet. Its multi-dimensional coverage is complete for separable $1$5D paraunitary systems rather than for all non-separable multi-dimensional cases (Su et al., 2021).

A third exact strategy is Lie-algebraic rather than factorized. Skew Orthogonal Convolution constructs a convolution filter whose Jacobian $1$6 is skew-symmetric by setting

$1$7

then defines the layer through the exponential operator $1$8, exploiting the fact that the exponential of a skew-symmetric matrix is orthogonal. The resulting convolution exponential is

$1$9

In practice, SOC uses the truncated series

$(M,C,k,S)$0

with explicit approximation guarantee

$(M,C,k,S)$1

The reported implementation uses $(M,C,k,S)$2 terms during training and $(M,C,k,S)$3 during evaluation, handles unequal channel counts by padding or projection, and handles stride through invertible downsampling (Singla et al., 2021).

These frameworks share the objective of exact or controlled-approximate operator orthogonality, but they instantiate it through different algebraic mechanisms: operator regularization, paraunitary factorization, or skew-symmetric exponentiation.

4. BCOP, AOC, and explicit finite-support orthogonal kernels

Adaptive Orthogonal Convolution is best understood as a direct extension of the explicit orthogonal convolution line typified by BCOP rather than as a new theory of orthogonal band convolutions. BCOP constructs finite-support orthogonal kernels through a block-composition scheme based on the associative block-convolution operator $(M,C,k,S)$4, defined by

$(M,C,k,S)$5

or equivalently

$(M,C,k,S)$6

The coefficients are

$(M,C,k,S)$7

BCOP builds larger kernels by composing elementary orthogonal pieces. A $(M,C,k,S)$8 orthogonal convolution is an orthogonal matrix reshaped as a kernel. A $(M,C,k,S)$9 or $\mathcal K \in \mathbb R^{MN^2 \times CS^2N^2},$0 factor is built from a half-rank projector: if $\mathcal K \in \mathbb R^{MN^2 \times CS^2N^2},$1 is column-orthogonal, then

$\mathcal K \in \mathbb R^{MN^2 \times CS^2N^2},$2

and

$\mathcal K \in \mathbb R^{MN^2 \times CS^2N^2},$3

is an orthogonal $\mathcal K \in \mathbb R^{MN^2 \times CS^2N^2},$4 convolution satisfying

$\mathcal K \in \mathbb R^{MN^2 \times CS^2N^2},$5

BCOP then alternates $\mathcal K \in \mathbb R^{MN^2 \times CS^2N^2},$6 and $\mathcal K \in \mathbb R^{MN^2 \times CS^2N^2},$7 factors plus a final $\mathcal K \in \mathbb R^{MN^2 \times CS^2N^2},$8 factor.

AOC extends this construction by introducing an RKO-style strided factor. The key observation is that RKO becomes exactly orthogonal when kernel size equals stride, $\mathcal K \in \mathbb R^{MN^2 \times CS^2N^2},$9. A strided orthogonal kernel is then factored as

$\mathrm{Vec}(Y)=\mathcal K\,\mathrm{Vec}(X).$0

For target kernel size $\mathrm{Vec}(Y)=\mathcal K\,\mathrm{Vec}(X).$1 and stride $\mathrm{Vec}(Y)=\mathcal K\,\mathrm{Vec}(X).$2,

$\mathrm{Vec}(Y)=\mathcal K\,\mathrm{Vec}(X).$3

with intermediate width

$\mathrm{Vec}(Y)=\mathcal K\,\mathrm{Vec}(X).$4

This preserves strict orthogonality while adding native stride, transposed convolution, groups, and dilation. The paper also formalizes transposed convolution as

$\mathrm{Vec}(Y)=\mathcal K\,\mathrm{Vec}(X).$5

and notes that grouped orthogonality reduces to orthogonality of each group kernel, while dilation is inherited from the standard-convolution case.

The practical motivation is computational. The naive block-convolution weight computation has theoretical cost

$\mathrm{Vec}(Y)=\mathcal K\,\mathrm{Vec}(X).$6

and the available BCOP implementation used nested loops. AOC rewrites block convolution as a padded conv2d-style operation, uses batching and grouped convolution for parallelization, and exploits associativity to reduce the sequential chain of $\mathrm{Vec}(Y)=\mathcal K\,\mathrm{Vec}(X).$7 BCOP compositions to $\mathrm{Vec}(Y)=\mathcal K\,\mathrm{Vec}(X).$8 stages via a parallel associative scan. In a ResNet-34/ImageNet-scale setting, the reported training overhead relative to standard Conv2D is $\mathrm{Vec}(Y)=\mathcal K\,\mathrm{Vec}(X).$9 for AOC versus $\mathcal K$0 for BCOP at batch size $\mathcal K$1, and $\mathcal K$2 versus $\mathcal K$3 at batch size $\mathcal K$4. The paper summarizes this as roughly a $\mathcal K$5–$\mathcal K$6 slowdown versus unconstrained models, reports an “8x reduction of the original overhead,” and emphasizes that the parameterization cost is independent of input image size. Reported results include CIFAR-10 performance up to $\mathcal K$7 clean accuracy and $\mathcal K$8 provable accuracy at $\mathcal K$9, and ImageNet-1K performance of $M\le CS^2$0 top-1 and $M\le CS^2$1 provable accuracy (Boissin et al., 14 Jan 2025).

5. Orthogonal band convolutions in InceptionMamba

In InceptionMamba, orthogonal band convolutions are not introduced as exact orthogonal operators in the row-orthogonal or column-orthogonal sense. They are introduced as a targeted replacement for the one-dimensional strip convolutions used in InceptionNeXt, with the explicit purpose of improving local spatial modeling while retaining the efficiency of Inception-style multi-branch depthwise convolution.

The orthogonal band branch lives inside the ConvMixer component of the InceptionMamba block. The overall block has two parts: a ConvMixer for local spatial modeling and a GlobalMixer based on bottleneck Mamba for long-range contextual interaction. The network follows a four-stage hierarchical architecture, and the ConvMixer splits

$M\le CS^2$2

into three channel groups,

$M\le CS^2$3

which are processed as

$M\le CS^2$4

and fused by

$M\le CS^2$5

The band branch is therefore defined operationally as the sum of two depthwise convolutions with orthogonal orientations. This differs from standard strip convolution, which uses $M\le CS^2$6 or $M\le CS^2$7 kernels and therefore samples essentially a single row or column line. A $M\le CS^2$8 kernel covers an $M\le CS^2$9-long horizontal band with thickness $3\times 11$00, and an $3\times 11$01 kernel covers a vertical band with thickness $3\times 11$02. Because they are summed on the same channel group,

$3\times 11$03

the branch is intended to capture horizontal and vertical neighborhood patterns jointly, with broader local support in the perpendicular direction than a strict one-dimensional strip.

The branch is deliberately lightweight. Across all four stages, the ConvMixer uses the kernel set

$3\times 11$04

with convolution group ratio $3\times 11$05 and GlobalMixer bottleneck ratio $3\times 11$06. The best-performing branch allocation is

$3\times 11$07

meaning $3\times 11$08 of channels are sent to the square branch, $3\times 11$09 to the band branch, and $3\times 11$10 to the identity branch. The paper reports that this setting gives the best accuracy-efficiency tradeoff relative to $3\times 11$11 and $3\times 11$12.

The empirical evidence for the band branch is deliberately narrow and controlled. In the ConvMixer ablation at identical complexity, four alternatives yield $3\times 11$13, $3\times 11$14, $3\times 11$15, and $3\times 11$16 Top-1, respectively, all at

$3\times 11$17

The progression is $3\times 11$18, InceptionDWConv2d, strip convolution, and the proposed design. The absolute improvement over strip convolution is therefore $3\times 11$19 Top-1 at unchanged parameter count and FLOPs. Qualitative CAM visualizations are presented as additional evidence: InceptionNeXt tends to activate scattered, eye-centric regions, whereas InceptionMamba produces more cohesive activations over semantically relevant object areas.

A central misconception is to read this branch as an exact orthogonal convolution in the BCOP, AOC, paraunitary, or SOC sense. The paper’s exact definition is the depthwise sum $3\times 11$20 applied to a subset of channels and fused with $3\times 11$21 and identity branches. It is motivated by orthogonal orientation coverage and thicker local bands, not by a formal condition such as $3\times 11$22 or $3\times 11$23 (Wang et al., 10 Jun 2025).

6. Orthogonal convolution in operator-valued free probability

A mathematically distinct usage appears in operator-valued free probability, where orthogonal convolution is an additive convolution on distributions rather than a spatial convolutional layer. In the $3\times 11$24-independence framework, free, Boolean, monotone, orthogonal, and s-free or subordination convolutions arise by different choices of projection data in reduced free products of Hilbert $3\times 11$25-modules. Orthogonal additive convolution is the specialization with

$3\times 11$26

and is denoted

$3\times 11$27

This convolution is neither commutative nor associative. Its importance lies in its analytic characterization by fully matricial reciprocal Cauchy transforms. For a $3\times 11$28-valued distribution $3\times 11$29,

$3\times 11$30

and the fully matricial family $3\times 11$31 determines $3\times 11$32. Orthogonal convolution is then characterized by

$3\times 11$33

This places it between Boolean and monotone convolution, since

$3\times 11$34

with the corresponding transform laws

$3\times 11$35

Its relation to free convolution is mediated by subordination. The paper proves

$3\times 11$36

so orthogonal convolution can recover free convolution when the second argument is the appropriate subordination distribution. It also gives a concrete operator-valued example: $3\times 11$37 This literature is analytically precise and deeply developed, but it is unrelated to the CNN notion of band kernels except at the level of the shared word “convolution.” Terminological caution is therefore essential when crossing between the two fields (Liu, 2018).

Orthogonal band convolutions are thus best understood as a family resemblance rather than a single object. In exact orthogonal CNN theory, the central object is a banded structured operator whose full map is orthogonal or semi-orthogonal. In InceptionMamba, the term names a lightweight depthwise branch that sums orthogonal-orientation band kernels for more cohesive local spatial modeling. In operator-valued free probability, orthogonal convolution is a transform-governed additive law with no spatial semantics. The technical content attached to the phrase depends entirely on which of these lineages is meant.