---
title: Interactive Convolution Block (ICB) Overview
url: https://www.emergentmind.com/topics/interactive-convolution-block-icb
type: topic
---

# Interactive Convolution Block (ICB) Overview

An Interactive Convolution Block (ICB) is a convolutional neural network (CNN) architectural module designed to improve feature representation and adaptive capacity by introducing data-dependent interactions between multi-scale or multi-branch convolutional features. In contrast to standard convolutional blocks that apply single-filter, single-path convolutions followed by simple nonlinearities (typically ReLU), ICBs systematically leverage parallel convolutions, nonlinear activations, element-wise gating, or global mixing to enable context-sensitive, nonlinear, and often cross-scale feature fusion. Used across image, audio, and sequence modeling, ICBs have appeared in several forms: as dual-branch nonlinear fusion modules [2410.17172], as mixer-augmented conv blocks for efficient global pattern extraction [2201.05863], as time-series local interaction layers [2404.08472], and as physics-inspired inter-layer collision structures [1911.08252]. Their adoption has led to consistent, measurable performance improvements and enhanced robustness over classical implementations.

## 1. Core Architectures and Mathematical Formulations

ICBs exhibit several distinct architectural patterns, unified by the design principle of explicit feature interaction between parallel convolutional or nonlinear paths.

### 1.1. Multi-Branch Convolution with Multiplicative Gating

KANICE introduces an ICB where two convolutions with different kernel sizes (3×3 and 5×5) operate in parallel on the same input tensor $X \in \mathbb{R}^{C_{\text{in}} \times H \times W}$, producing feature maps $A$ and $B$ after GELU activation. The block output is the element-wise product:
\[
Y = \mathrm{GELU}(W_1 * X + b_1) \odot \mathrm{GELU}(W_2 * X + b_2)
\]
This module promotes co-activation across scales; output is strongly gated unless both branch activations are high [2410.17172].

### 1.2. Mixer-Augmented Depthwise Blocks

ConvMixer employs an ICB composed of:
- 2D depthwise-separable convolution (local spatial extraction),
- Followed by 1D depthwise-separable convolution (temporal/channel refinement),
- Followed by two MLP mixer layers that implement (a) temporal mixing and (b) frequency/channel mixing, each with GELU activation and LayerNorm,
- Residual connections encompassing both the input and intermediate branches.

The complete forward operator
\[
\hat{y} = x + y_1 + f_3(y_2)
\]
where $y_1$ is post-2D conv, $y_2$ post-1D conv, $f_3$ is the mixer, enables both local and global interaction at efficient computational cost [2201.05863].

### 1.3. Interactive Gated Fusion for Time Series

TSLANet's ICB for time-series reconstructs local temporal detail lost in spectral-domain processing. It applies two parallel 1D convolutions of distinct kernel sizes to input $\mathbf{S}'$, followed by dual interactive gates:
\[
\begin{aligned}
\mathbf{X}_1 &= W_1 * \mathbf{S}' + b_1, \quad \mathbf{X}_2 = W_2 * \mathbf{S}' + b_2 \\
\mathbf{A}_1 &= \mathrm{GELU}(\mathbf{X}_1) \odot \mathbf{X}_2, \quad
\mathbf{A}_2 = \mathrm{GELU}(\mathbf{X}_2) \odot \mathbf{X}_1 \\
\mathbf{O}_{\mathrm{ICB}} &= W_3 * (\mathbf{A}_1 + \mathbf{A}_2) + b_3
\end{aligned}
\]
This structure enables rich cross-scale and local feature mixing downstream of adaptive spectral blocks [2404.08472].

### 1.4. Physics-Inspired Collision-Based Fusion

IC-Networks define an ICB using a collision term inspired by elastic collisions between layers. For an input $\mathbf{X}$, the IC layer merges direct convolution with a non-linear residual computed between the standard response and a "rough-feature" (per-channel sum or average) term:
\[
\mathbf{u}_i = \mathbf{w}_i * \mathbf{X} + \sigma(\mathbf{w}_i * \mathbf{X} - w'_i (\mathbf{I} ** \mathbf{X}))
\]
where $**$ is a depthwise separable convolution, $w'_i$ a scalar gating parameter, and $\sigma$ is ReLU [1911.08252].

## 2. Adaptive Feature Fusion and Nonlinear Capacity

ICBs commonly employ either multiplicative gating, interaction terms, or global MLP-based mixing to produce context- or input-dependent activations.

- **Multiplicative Gating**: Element-wise multiplication (Hadamard product) between branch activations enforces strict co-activation, increasing selectivity and enabling complex, higher-order interactions [2410.17172, 2404.08472].
- **Additive-Residual and Nonlinear Mixer**: MLP-based mixer layers globally recombine tokens/channels, effectively substituting for attention at an order of magnitude lower computational cost [2201.05863].
- **Collision Mechanism**: The explicit "collision" nonlinearity can flexibly carve out up to three linear regions (standard ReLU neurons provide two), thus increasing local representational power [1911.08252].

These mechanisms contrast with the standard Conv–ReLU pipeline, which applies a single convolution and a fixed pointwise nonlinearity, lacking feature-dependent gating or cross-scale mixing.

## 3. Implementation Protocols and Computational Analysis

### Parameter and FLOP Costs

| ICB Variant          | Added Parameters (per block)     | FLOPs Overhead                | Activation Scheme |
|----------------------|----------------------------------|-------------------------------|-------------------|
| KANICE ICB           | $34\,C_{\text{in}}C_{\text{out}}$ | $2\,HW\,C_{\text{in}}C_{\text{out}}\times 34$ | GELU, Hadamard   |
| ConvMixer ICB        | $\sim$100–143K for $C\leq 64$     | $\sim$22.2M      | Swish/GELU, Mixer |
| TSLANet ICB          | $3 \times (d \times d \times k)$  | Dominated by 1D convs; low    | GELU, Gating      |
| IC-Network ICB       | $+O(C)$                          | $\tfrac{1}{C}+\tfrac{1}{k^2}$ | ReLU, collision   |

- KANICE's ICB is $\approx3.8\times$ heavier than a single 3×3 conv for the same input/output channels [2410.17172].
- ConvMixer blocks scale to meet tight parameter/MAC budgets (100K param, 22.2M MACs for full KWS pipeline) and use depthwise/pointwise separable convolutions for efficiency [2201.05863].
- TSLANet and IC-Network ICBs can be configured with arbitrary kernel sizes to balance locality and cost [2404.08472, 1911.08252].

### Initialization and Integration

- He (Kaiming) normal initialization is used for convolution weights in KANICE [2410.17172].
- BatchNorm, GELU/Swish activations, and residual connections are employed variously across implementations to stabilize training and improve convergence [2201.05863, 1911.08252].
- Dropout and explicit regularization are not used within the ICB itself in current KANICE experiments; standard weight decay is global [2410.17172].

## 4. Empirical Performance and Ablation Analysis

ICB integration yields consistent, sometimes substantial, improvements in accuracy and robustness across application domains.

### Image Classification (KANICE)

| Model        | MNIST | Fashion | EMNIST | SVHN |
|--------------|-------|---------|--------|------|
| CNN (std)    | 98.55 | 92.36   | 85.38  | 84.04|
| **ICB only** | 98.98 | 92.05   | 86.43  | 86.70|
| ICB_CNN      | 98.92 | 92.94   | 87.00  | 89.60|

Replacing the first conv with an ICB increases accuracy by +0.43% (MNIST), +1.32% (EMNIST), and +2.66% (SVHN). "ICB_CNN", a hybrid pattern, performs even better, especially for challenging data [2410.17172].

### Speech/Audio Robustness (ConvMixer)

- Google Speech Commands V2-12 accuracy: 98.2% (<120K params).
- Far-field robustness (down to SNR –10 dB): 71.88% vs 64.5% for MatchboxNet of similar size.
- Removing the mixer from ICB produces a ~7% accuracy decline at low SNRs, confirming the global interaction's criticality [2201.05863].

### Time Series Tasks (TSLANet)

| Variant      | FordA (ACC%) | UWaveGL (ACC%) | ETTh1 (MSE) | Exchange (MSE) |
|--------------|--------------|----------------|-------------|----------------|
| w/o ICB      | 91.3         | 86.2           | 0.419       | 0.376          |
| Full TSLANet | 93.1         | 91.3           | 0.413       | 0.369          |

ICB removal consistently reduces classification accuracy (–1.8% / –5.1%) and modestly worsens forecasting error [2404.08472].

### Large-Scale Image Recognition (IC-Network)

- Integrating IC blocks into ResNet-50 reduces top-1 error from 22.85% to 21.49% (ImageNet 10-crop), outperforming ResNet-101 with fewer FLOPs.
- On CIFAR-10, gains of 0.3–0.9% are observed; however, very deep ICNets may require additional regularization [1911.08252].

## 5. Contextualization Across Domains and Use Cases

ICBs have been adapted and validated across image classification [2410.17172, 1911.08252], keyword spotting in noisy environments [2201.05863], and time-series learning [2404.08472]. Patterns emerge:

- In CNNs: Multi-branch and collision-based ICBs enhance discrimination and convergence at modest computational cost.
- In audio/sequential processing: Mixer-based ICBs compete with transformer architectures while maintaining small footprints.
- In time series: ICBs restore local detail lost to global spectral blocks, improving classification and forecasting by dynamic, scale-aware nonlinearity.

The architectural modularity of ICBs makes them suitable replacements or supplements to both standard convolution and transformer FFN/attention layers in settings where cross-scale, nonlinear, or global-local feature fusion is critical.

## 6. Implementation Considerations and Limitations

- Architecture: ICBs may be implemented via standard 2D/1D convolutions, mixer MLPs, or collision-based fusion, depending on domain and desired behavior.
- Kernel size and parameter scaling must be chosen relative to task statistics (e.g. sequence length, local vs. global feature prominence) [2404.08472].
- ICBs typically increase parameters and FLOPs (up to 4× per block for KANICE), but selective deployment (only in first layers or as hybrid blocks) offers a favorable performance/efficiency trade-off.
- Very deep integration (e.g., overly many IC bottleneck blocks) may cause overfitting and mandate stronger regularization schemes [1911.08252].
- Certain ICBs (e.g., TSLANet's) do not contain residual skips by default; further empirical work could illuminate optimal placement and structure for arbitrary domains [2404.08472].

## 7. References

- "KANICE: Kolmogorov-Arnold Networks with Interactive Convolutional Elements" [2410.17172]  
- "ConvMixer: Feature Interactive Convolution with Curriculum Learning for Small Footprint and Noisy Far-field Keyword Spotting" [2201.05863]  
- "TSLANet: Rethinking Transformers for Time Series Representation Learning" [2404.08472]  
- "IC-Network: Efficient Structure for Convolutional Neural Networks" [1911.08252]

Source: https://www.emergentmind.com/topics/interactive-convolution-block-icb