Channel Independent Directional Convolution

• Channel Independent Directional Convolution (CIDC) is a neural network operation that models directional dependencies while preserving channel independence.
• It employs uni-directional temporal filtering and grouped convolutions to drastically reduce parameters and improve computational efficiency.
• Empirical results in speech separation, time series forecasting, and action recognition demonstrate significant gains in key performance metrics.

Channel Independent Directional Convolution (CIDC) refers to a class of neural network operations or architectural modules designed to model directional, often temporal, dependencies within multidimensional data while retaining a channel-wise decomposition at the convolutional or filtering stage. Unlike standard convolutions that combine information across feature channels, CIDC operations maintain independence between channels during directional processing, facilitating efficient modeling of evolutionary or spatial cues. These modules have proven effective in speech separation, time series forecasting, and action recognition, leading to substantial empirical performance gains.

1. Mathematical Formulation and Core Principles

Several CIDC variants appear in the literature, but they share a common structure: per-channel or channel-pair convolutions impose directional (often temporal or spatial) processing without full inter-channel mixing. The key mathematical forms are found in action recognition and speech separation domains.

Temporal CIDC for Video and Sequential Data

Given a feature tensor $\mathbf F\in\mathbb R^{C\times T\times H\times W}$, a CIDC operation is defined channel-wise as

$$\mathbf F'_c(t',i,j) = \sum_{t=1}^{T} w^c_{t',t}\; \mathbf F_c(t,i,j)$$

subject to $w^c_{t',t} = 0$ for $t > t'$, and $\sum_{t=1}^{t'} w^c_{t',t} = 1$. The weights are softmax-normalized. This uni-directional formulation enforces information flow only from past to present, enabling strict modeling of causality or directed evolution within each channel (Li et al., 2020). Bi-directional CIDC concatenates forward and reverse outputs.

Channel-Independent Convolution for Time Series

For multivariate sequences $X_{in}\in\mathbb R^{C\times T}$, channel-independent convolution applies $M$ unique 1D kernels per channel: $$H^{(c,m)}_n = \mathrm{ReLU} \left( \sum_{p=1}^P K_{cp}^{(c,m)}[p] \cdot \mathrm{Norm}(\tilde Y^{(c)}_{nS+p-1}) \right),$$ where $K_{cp}^{(c,m)} \in \mathbb R^P$, $n = 0\ldots N-1$, and $$\mathbf F'_c(t',i,j) = \sum_{t=1}^{T} w^c_{t',t}\; \mathbf F_c(t,i,j)$$0 stacks all outputs (Lee et al., 25 Sep 2025).

Inter-Channel Convolution Differences (ICDs)

For spatial filtering in MCSS, let $$\mathbf F'_c(t',i,j) = \sum_{t=1}^{T} w^c_{t',t}\; \mathbf F_c(t,i,j)$$1 be two microphone waveforms. The $$\mathbf F'_c(t',i,j) = \sum_{t=1}^{T} w^c_{t',t}\; \mathbf F_c(t,i,j)$$2-th ICD map is

$$\mathbf F'_c(t',i,j) = \sum_{t=1}^{T} w^c_{t',t}\; \mathbf F_c(t,i,j)$$3

where $$\mathbf F'_c(t',i,j) = \sum_{t=1}^{T} w^c_{t',t}\; \mathbf F_c(t,i,j)$$4 denotes 1D convolution, $$\mathbf F'_c(t',i,j) = \sum_{t=1}^{T} w^c_{t',t}\; \mathbf F_c(t,i,j)$$5 is a learnable temporal filter shared across both channels, and $$\mathbf F'_c(t',i,j) = \sum_{t=1}^{T} w^c_{t',t}\; \mathbf F_c(t,i,j)$$6, $$\mathbf F'_c(t',i,j) = \sum_{t=1}^{T} w^c_{t',t}\; \mathbf F_c(t,i,j)$$7 implement (possibly soft) subtraction (Gu et al., 2020). This operation acts as a data-driven, channel-independent, directional filter.

2. Architectural Integration and Parameterization

Speech Separation

In the end-to-end MCSS pipeline, CIDC (ICD modules) appears as a 2D convolution block after the encoder. The kernel shape is $$\mathbf F'_c(t',i,j) = \sum_{t=1}^{T} w^c_{t',t}\; \mathbf F_c(t,i,j)$$8, height dimension spanning two microphones, with each filter separately convolving and (soft-)subtracting paired channels. These spatial-feature maps are concatenated with “multi-channel sums” and encoder features, which are subsequently processed by a TCN separation network and decoded to waveform (Gu et al., 2020).

Multivariate Time Series

The IConv module consists of a “Channel Independent Patcher Compressor” (CIPC), in which each channel independently processes temporal dependencies using several large 1D kernels. A lightweight “Inter-Channel Mixer” (ICM) applies two linear layers (interpreted as $$\mathbf F'_c(t',i,j) = \sum_{t=1}^{T} w^c_{t',t}\; \mathbf F_c(t,i,j)$$9 convolutions) post-convolution, enabling minimal but sufficient cross-variable information sharing. This is followed by an upsampling “expand” step that reconstructs local fluctuations, with IConv blocks interleaved with MLP-based trend estimators (Lee et al., 25 Sep 2025).

Action Recognition

CIDC is implemented as a grouped $w^c_{t',t} = 0$0 convolution with temporal kernels masked to enforce directionality and grouped by channel. Multiple CIDC branches can be attached at different stages of a ResNet/I3D backbone, propagating context across scales, sometimes with spatial attention from late stages fused back into earlier spatial resolutions. Final features are concatenated along the temporal axis and pooled for classification (Li et al., 2020).

3. Computational Complexity and Channel Independence

Channel independence restricts kernel learning to per-channel (or channel-pair) filtering, reducing parameter count and computational cost substantially relative to standard, fully-mixing convolutions.

Operation Type	Parametric Complexity	FLOPs
Full 3D Conv	$w^c_{t',t} = 0$1	$w^c_{t',t} = 0$2
CIDC/Grouped	$w^c_{t',t} = 0$3	$w^c_{t',t} = 0$4

For $w^c_{t',t} = 0$5, $w^c_{t',t} = 0$6, parameter savings exceed two orders of magnitude (Li et al., 2020).

In time series, CIPC parameter count scales as $w^c_{t',t} = 0$7 (channels × kernels × kernel-size), which enables much larger temporal kernels and high-dimensional channel processing with a fraction of the computation (Lee et al., 25 Sep 2025).

4. Applications and Empirical Impact

Multi-Channel Speech Separation

CIDC blocks, specifically ICDs, offer a learnable generalization of hand-crafted inter-channel phase difference (IPD) features. Whereas IPD computes analytic phase shifts on fixed STFT bins, ICD blocks rely on learned time-domain bandpass filters and soft subtraction, trained end-to-end. Empirically, the addition of ICD to the MCSS model raised SI-SDR improvement from 10.8 dB (with only multi-channel sum) to 11.9 dB, outperforming both fixed and STFT-learned IPD variants. Overall, ICD yielded a 10.4% relative SI-SDRi boost over fixed IPD models (Gu et al., 2020).

Multivariate Time Series Forecasting

In the IConv framework, channel-independent convolutions capture fine-grained, non-stationary local variations and periodicity per channel, where MLP-based approaches typically underperform. On datasets including ECL, ETTh1/2, ETTm1/2, Solar, Traffic, and Weather, IConv obtained 45 first-place and 9 second-place results across 64 settings, with 5–10% lower MAE/MSE compared to MLP- and Transformer-based baselines. Efficiency improves due to the highly favorable scaling of channel-wise convolutions (Lee et al., 25 Sep 2025).

Action Recognition

CIDC modules applied to video backbones (ResNet/I3D) consistently yielded top-1 accuracy gains: for instance, on UCF-101, top-1 increased from 92.9% to 97.2%, and on Something-Something V2, from 49.6% to 56.3%, with similar improvements on other major datasets. Qualitative activation maps indicated a shift in focus towards semantically relevant foreground regions and stronger foreground-to-background activation ratios (+15–20% over I3D baselines) (Li et al., 2020).

5. Interpretation and Relationships to Classical Methods

CIDC operations generalize classical analytic operations. In speech separation, ICDs operate as a data-driven analog of IPD, but rather than relying on analytically determined Fourier kernels, they learn frequency bands and spatial filtering directly, adaptive to the rest of the network (Gu et al., 2020). In time series modeling, channel-wise convolutions with large receptive fields draw on the inductive biases of local, shift-invariant feature learning, while a light inter-channel mixer reintroduces dependencies among variables that pure channel-independence would miss (Lee et al., 25 Sep 2025).

The core property unifying CIDC-style operations is their strict architectural decoupling of channels during directional (temporal, spatial) filtering, yielding parameter and efficiency advantages while preserving or enhancing modeling capacity through learnable, data-driven filters.

6. Ablations, Visualizations, and Empirical Validation

Ablation experiments consistently show significant drops in task performance upon removal of CIDC modules. In IConv, ablating the channel-independent convolution layer results in highest error, with subsequent layers (ICM, upsampling) incrementally improving results (Lee et al., 25 Sep 2025). Visualizations of attention maps in CIDC-enabled video networks demonstrate sharper localization on action-relevant regions and higher foreground-to-background activation ratios, which are quantitatively confirmed on human-annotated benchmarks (Li et al., 2020). In speech separation, the addition of ICD is necessary to match or exceed the best prior models on SI-SDRi and SDRi metrics (Gu et al., 2020).

7. Prospects and Significance

The emergence of CIDC modules across modalities—speech, time series, and video—suggests a general utility of channel-independent, directionally constrained convolutional operations for efficiently capturing structured evolution within complex signals. A plausible implication is that such modules may serve as universal building blocks for future neural architectures requiring scalable, interpretable, and locally adaptive modeling of multidimensional, temporally or spatially ordered data. Further, their empirical advantages are demonstrated not only in accuracy metrics but also in parameter efficiency and computational throughput, which is a critical consideration for high-channel datasets and real-time applications.