Channel-wise Score Unit (CSU)

• Channel-wise Score Unit (CSU) is a neural attention module that selectively reweights CNN feature channels to emphasize key visual attributes in image captioning tasks.
• It operates by applying global average pooling, context fusion with LSTM hidden states, and channel scoring via softmax normalization to modulate each channel’s contribution.
• Empirical evaluations show that integrating CSU, especially in joint attention frameworks like SCA-CNN, significantly improves BLEU scores in datasets such as Flickr8k and MSCOCO.

The Channel-wise Score Unit (CSU) is a neural attention module designed for convolutional networks, introduced in the context of image captioning. It enables a model to modulate the relevance of individual feature channels in CNN activations, thereby learning “what” semantic attributes to emphasize when generating textual descriptions. The CSU is a core component of the SCA-CNN model, which combines spatial and channel-wise attentions for improved visual feature selection during sequence generation (Chen et al., 2016).

1. Mathematical Formulation

The CSU operates on the convolutional feature map $V \in \mathbb{R}^{C \times H \times W}$ at a given CNN layer, treating $V$ as $C$ distinct $H \times W$ channel maps. Channel-wise attention is computed as follows:

1. Channel Pooling: Global average pooling reduces each channel to a scalar:

$$v_i = \frac{1}{H\cdot W}\sum_{x=1}^H \sum_{y=1}^W V_{i, x, y}$$

Collect $v = [v_1, \ldots, v_C]^T \in \mathbb{R}^C$.

2. Context Fusion: Fuse $v$ with the prior LSTM hidden state $h_{t-1} \in \mathbb{R}^d$ via a gating network:

$$b = \tanh\big( W_c v + b_c \oplus W_{hc} h_{t-1} \big) \in \mathbb{R}^k$$

where $W_c \in \mathbb{R}^{k \times C}$, $V$0, $V$1, and $V$2 denotes broadcast addition.

3. Channel Scoring: Produce pre-activation channel scores:

$V$3

Where $V$4, $V$5.

4. Softmax Normalization: Convert scores to activations:

$V$6

These scalar weights $V$7 satisfy $V$8, $V$9.

5. Feature Map Reweighting: Output feature map channels are reweighted:

$C$0

2. Stepwise CSU Module Architecture

The operational procedure can be itemized as follows:

Step	Operation	Output
1	Global average pooling (GAP)	$C$1
2	Linear projection of $C$2	$C$3
3	Linear projection of $C$4	$C$5
4	Non-linear fusion	$C$6
5	Linear to channel space	$C$7
6	Channel-wise softmax	$C$8

The CSU thus produces a normalized channel attention vector $C$9, which is then broadcast and multiplied channel-wise across the feature map $H \times W$0 before subsequent processing.

3. Integration in SCA-CNN Architectures

CSUs are inserted at selected convolutional layers to modulate channels before further attention or recurrent stages. In SCA-CNN, typical placements include:

• VGG-19: conv5_4 (single-layer), and conv5_3, conv5_2 for multilayer variants.
• ResNet-152: res5c (single-layer), and optionally at res5c_branch2b/res5c_branch2a.

Following channel-wise weighting, spatial attention may be composed in two canonical orders:

• C-S (Channel first): Apply CSU, yielding $H \times W$1, then spatial attention over $H \times W$2.
• S-C (Spatial first): Compute spatial attention, then apply CSU to the spatially attended map.

Empirical evaluations favor C-S ordering, yielding slightly superior or comparable performance. After attention, the final attended feature $H \times W$3 is flattened and provided as visual input to the LSTM-based decoder at each time step.

4. Quantitative Effects and Ablation

The contribution of channel-wise attention was evaluated against spatial-only attention, two-stream variants (C-S, S-C), and the “hard” spatial attention (SAT) baseline. On Flickr8k, Flickr30k, and MSCOCO datasets using ResNet-152 (2048 channels), the CSU delivered quantifiable improvements:

• Spatial-only (S): BLEU-4 ≈ 20.5–28.3.
• Channel-only (C): BLEU-4 ≈ 24.4–29.5, with a +4 BLEU improvement on Flickr8k.
• C-S (joint attention): BLEU-4 ≈ 25.7–30.4, the best overall performance.

This reveals that channel-only attention outperforms spatial-only attention, especially as the channel count increases. Combining both results in maximal gains (Chen et al., 2016).

5. Genericity and Applicability

The CSU is not confined to the SCA-CNN framework; it can be integrated into any CNN–RNN caption-generation architecture where channel salience (“what” features to emphasize) is beneficial. It operates independently of spatial attention and can be used recurrently across multiple layers. Placement is flexible, depending on the depth and design of the CNN backbone.

6. Interpretation and Significance

The CSU enables a network to learn context-dependent selection among high-level visual attributes encoded in channel maps, supplementing standard spatial “where” attention. This supports a richer dynamic visual encoding, especially critical in tasks like image captioning, where identifying key semantic elements per decoding step enhances descriptive output. A plausible implication is that CSU makes the attention mechanism more expressive for architectures with wide or deep feature representations. The generic recipe may influence the design of attention modules for broader vision-language tasks requiring channel “emphasis” (Chen et al., 2016).