CIFFN: Channel-Interactive Feed-Forward Network

• Channel-Interactive Feed-Forward Network (CIFFN) is a feed-forward layer that enhances inter-channel communication by using feature decomposition, adaptive scaling, and convolution-based operations.
• It replaces the standard transformer FFN with a combination of 1x1 convolutions, depth-wise convolutions, and residual connections to mitigate channel redundancy and compensate for sparse attention.
• Experimental results show that incorporating CIFFN within SAEViT improves Top-1 accuracy by up to 1.3% while adding minimal computational overhead, making it a key component in lightweight ViT architectures.

Channel-Interactive Feed-Forward Network (CIFFN) is a feed-forward layer introduced within SAEViT, a lightweight convolution–Vision Transformer integrated architecture designed to improve inter-channel information exchange while mitigating redundancy in traditional feed-forward networks. In the formulation presented for SAEViT, CIFFN replaces the vanilla transformer FFN with a module built from $1 \times 1$ convolutions, depth-wise convolution, activation, feature decomposition, channel-wise scaling, and residual connection. Its stated role is to enhance expressive channel communication, compensate for information loss associated with sparse attention, and do so with limited additional computational cost (Zhang et al., 23 Aug 2025).

1. Definition and architectural position

CIFFN is described as a Channel-Interactive Feed-Forward Network layer developed “to enhance inter-channel information exchange through feature decomposition and redistribution, mitigating redundancy in traditional feed-forward networks (FNN)” (Zhang et al., 23 Aug 2025). Within SAEViT, it appears as the feed-forward component of the transformer block, replacing the vanilla FFN immediately after the sparse attention mechanism.

The motivating contrast is with the standard Vision Transformer FFN, characterized in the source as a stack of fully connected layers applied equally and independently to each channel and token. In that account, such a design lacks explicit modeling of inter-channel relationships, which is associated with redundant channel information, inadequate discrimination between important and unimportant features, and reduced expressivity and generalization capability. The proposed CIFFN is therefore positioned as a channel-aware alternative that supplements sparse spatial modeling with explicit channel communication (Zhang et al., 23 Aug 2025).

A plausible implication is that CIFFN is not intended as an isolated module, but as part of a broader balancing strategy in SAEViT: sparse attention reduces attention complexity, while CIFFN recovers representational richness in the channel domain.

2. Internal structure and mathematical formulation

The paper describes CIFFN as introducing explicit feature decomposition and channel reweighting (redistribution) and using a combination of $1 \times 1$ convolutions (Conv), depth-wise convolutions (DWConv), and learnable, channel-wise scaling factors (Zhang et al., 23 Aug 2025).

Its high-level structure is given as follows:

1. Initial $1 \times 1$ Conv to project feature channels to higher dimension.
2. Depth-wise Conv to capture spatial information within each channel.
3. Activation such as GELU.
4. Feature Decomposition (FD) module to apply per-channel scaling and residual connections.
5. Final $1 \times 1$ Conv to restore the original channel dimension.
6. Residual connection.

Let $X_{in} \in \mathbb{R}^{H \times W \times C}$ denote the input feature map. The initial transformation is written as

$$X_1 = \text{GELU}(\text{DWConv}(\text{Conv}_1(\text{Norm}(X_{in}))))$$

where $\text{Conv}_1$ is a $1 \times 1$ convolution for expansion, $\text{DWConv}$ is a depth-wise $3 \times 3$ convolution, and $1 \times 1$0 denotes normalization (Zhang et al., 23 Aug 2025).

The feature decomposition stage is expressed as

$1 \times 1$1

with

$1 \times 1$2

where $1 \times 1$3 is a $1 \times 1$4 convolution for additional channel mixing and $1 \times 1$5 is a learnable channel-wise scaling factor initialized to zero (Zhang et al., 23 Aug 2025).

The final output is

$1 \times 1$6

where $1 \times 1$7 restores the original channel dimension and the residual addition uses the input feature map. The source also presents the compact expression

$1 \times 1$8

as the full CIFFN function (Zhang et al., 23 Aug 2025).

This formulation suggests a deliberate division of labor: the initial projection and DWConv provide expanded and spatially aware features, the FD module performs adaptive channel redistribution, and the output projection returns the representation to the model’s working dimensionality.

3. Mechanism of inter-channel interaction

The central claim associated with CIFFN is that it enhances inter-channel information exchange. The mechanism is described through three coupled operations: feature decomposition, adaptive recalibration, and convolution-based channel mixing (Zhang et al., 23 Aug 2025).

First, the FD module performs explicit channel decomposition and learnable rescaling via $1 \times 1$9, rather than treating all channels identically. This is presented as an explicit alternative to channel-independent processing. Second, the channel-wise scaling factors are learned during training and are said to dynamically boost or suppress channels based on relevance. Third, the insertion of $1 \times 1$0 convolution before and inside the FD block provides channel mixing in a form that is coupled to the spatially processed representation delivered by DWConv (Zhang et al., 23 Aug 2025).

The paper contrasts this with the classic FFN in transformers, for which “all channels [are] transformed identically,” with “no interaction across channels within the FFN.” By contrast, CIFFN is characterized by “inter-channel mixing, adaptive weighting, and explicit mechanism to address channel redundancy” (Zhang et al., 23 Aug 2025).

The source further states that CIFFN “prunes redundant inter-channel correlations” while encouraging channel diversity and feature discriminativeness. This suggests that the module is intended not merely to amplify channels, but to regulate correlation structure among them. In that sense, its function is both redistributive and regularizing: it seeks a middle ground between overly redundant channel representations and overly decoupled ones.

4. Relation to sparse attention and the SAEViT pipeline

In SAEViT, sparse attention is implemented by the Sparsely Aggregated Attention (SAA) module, which adaptively performs sparse sampling based on image redundancy and reconstructs the feature map by deconvolution. The stated purpose is to reduce the computational complexity of attention operations (Zhang et al., 23 Aug 2025).

CIFFN is explicitly motivated in relation to this sparse attention design. The paper states that because SAA reduces spatial computation, it can potentially omit important feature interactions. CIFFN is introduced to “recover and enhance representational power by focusing on multi-channel interactions, reducing chances of losing critical information and boosting discriminative ability” (Zhang et al., 23 Aug 2025).

This division of functionality is central to the SAEViT design. SAA addresses the efficiency problem of global attention, while CIFFN addresses representational degradation that may follow from sparsification. The hierarchical pyramid structure and embedded depth-wise separable convolutional blocks strengthen convolutional features at the architectural level, but CIFFN specifically operates as the channel-domain compensatory mechanism inside each transformer block (Zhang et al., 23 Aug 2025).

A plausible implication is that CIFFN should be understood less as a generic drop-in FFN variant and more as a component whose significance is amplified in architectures where attention has been made sparse or otherwise compressed.

5. Experimental evidence and ablation findings

The paper reports an ablation of core components involving LFE, SAA, and CIFFN. The tabulated Top-1 accuracy values are reproduced below from the source (Zhang et al., 23 Aug 2025).

LFE	SAA	CIFFN	Top-1 Accuracy (%)
			77.2
✓			77.8
✓	✓		78.3
✓	✓	✓	79.6

The accompanying observation is that adding CIFFN to a backbone already using SAA and LFE raises accuracy from 78.3% to 79.6%, a gain of +1.3%, identified in the source as the largest single-component gain among the three modules (Zhang et al., 23 Aug 2025).

The paper also reports a channel correlation study on Tiny ImageNet comparing vanilla FFN, DWFFN, and CIFFN. In that analysis, vanilla FFN exhibits high inter-channel correlations, interpreted as redundancy; DWFFN exhibits very low correlations, interpreted as insufficient channel interaction or diversity; and CIFFN exhibits moderate, well-distributed correlations, interpreted as pruning redundancy while retaining diversity and useful interaction (Zhang et al., 23 Aug 2025).

On computational overhead, the source states that CIFFN adds only 0.4M parameters and 0.03 GFLOPs over the baseline, while yielding a +2.4% topline improvement when all core modules are included, with less than 1M total parameter increase (Zhang et al., 23 Aug 2025). In the broader SAEViT model, extensive experiments on mainstream datasets report Top-1 accuracies of 76.3\% and 79.6\% on the ImageNet-1K classification task with only 0.8 GFLOPs and 1.3 GFLOPs, respectively (Zhang et al., 23 Aug 2025).

The source further states that improvements in classification accuracy transfer to detection on COCO and segmentation on ADE20K, with consistently leading performance among tested lightweight hybrid ViT models (Zhang et al., 23 Aug 2025). Since the detailed downstream metrics are not included in the provided material, the significance of this claim is best understood qualitatively: CIFFN is presented as beneficial not only for classification but also for transfer to standard dense prediction benchmarks.

6. Distinctions, interpretation, and scope

CIFFN is distinguished from both classical FFN and DWFFN by the specific combination of adaptive feature redistribution, explicit channel mixing, learnable per-channel weighting, and built-in spatial context via DWConv (Zhang et al., 23 Aug 2025). In the source summary, these properties underpin improved inter-channel interaction, efficient feature encoding, and reduced redundancy at minimal extra computational cost.

One possible misconception is to regard CIFFN as simply a convolutional FFN. The description provided does not support that reduction. Although convolutions are essential to its implementation, the defining element is the feature decomposition and redistribution mechanism with learnable channel-wise scaling. Conversely, another possible misconception is to interpret CIFFN as exclusively a channel-attention block akin to a standalone recalibration mechanism. The source instead presents it as a full feed-forward replacement integrating expansion, spatial mixing, decomposition, redistribution, projection, and residual learning in a single module (Zhang et al., 23 Aug 2025).

The paper’s own interpretation of the correlation analysis is that CIFFN occupies an intermediate regime: it avoids the high redundancy of vanilla FFN without collapsing into the insufficient interaction associated with DWFFN. This suggests a broader design principle for lightweight hybrid ViTs: when attention is made sparse for efficiency, the FFN can be re-engineered to shoulder part of the burden of representation learning through structured channel interaction.

Within the scope of SAEViT, CIFFN is therefore best understood as the channel-domain complement to sparse attention and convolutional feature enhancement. Its reported function is not to replace attention’s long-range modeling role, but to preserve discriminative capacity under efficiency constraints by improving how channels communicate, redistribute information, and suppress redundancy (Zhang et al., 23 Aug 2025).