Deep Features from InceptionResNet-v2

• Deep features from InceptionResNet-v2 are the internal representations capturing multi-scale semantic and spatial patterns through combined inception and residual modules.
• Feature extraction involves processing preprocessed 299x299 images through sequential blocks, culminating in L2-normalized global descriptors or MLSP vectors tailored for high-level tasks.
• These features effectively enhance tasks such as transfer learning, image retrieval, aesthetic assessment, and conditional generation in computer vision pipelines.

Deep features from InceptionResNet-v2 refer to the internal activation representations generated by intermediate and terminal layers of the InceptionResNet-v2 convolutional neural network, typically pretrained on large-scale datasets such as ImageNet. These feature vectors, extracted prior to the final classification layers or aggregated through various spatial pooling schemes, have become central to a wide range of downstream computer vision tasks including transfer learning, image retrieval, aesthetic assessment, and conditional generation. Their utility arises from their aggregation of hierarchical semantic and spatial cues, the architectural innovations of residual and inception modules, and their well-characterized preprocessing and normalization protocols.

1. InceptionResNet-v2 Architectural Overview

InceptionResNet-v2 combines the parallel multi-scale convolutional pathways of “Inception” modules with the stabilizing effects of residual connections. It is structured as a stack of sequential blocks, each capturing increasingly high-level features while maintaining computational efficiency. The network consists of:

• Input Stem: Sequential conv/pooling layers reducing a $299\times299\times3$ input to an activation tensor.
• Inception-ResNet-A/B/C Blocks: Each block consists of parallel convolutional branches merged and then connected via a scaled residual addition. Residual connections are modulated by a scalar $\alpha$ (typically 0.1 or 0.2), which is essential for stable training at high channel widths.
• Reduction Blocks (A and B): Downsample the spatial dimensions, increasing receptive field and abstraction; e.g., $35\times 35 \to 17\times 17$, $17\times 17\to8\times8$.
• Final Head: Global average pooling produces a fixed-size feature vector (typically 1760- or 2080-dim), followed by dropout and a fully connected classification layer (Szegedy et al., 2015, Szegedy et al., 2016).

The table below summarizes representative block shapes for key feature-extraction locations:

Extraction Layer	Feature Shape
After last A-block	$35\times35\times320$
After Reduction-A/pre-B	$17\times17\times1088$ (varies by variant)
After Reduction-B/last C-block	$8\times8\times1760{-}2080$
After global average pooling	$1\times1\times1760{-}2080$

2. Preprocessing and Feature Extraction Protocols

Input images are typically resized to $299\times299$, mapped from [0, 255] to approximately $[-1, +1]$ via channel-wise centering and scaling. No ZCA or PCA whitening is used. The network is traversed up to a chosen layer, and activations are read out as deep features (Szegedy et al., 2015, Szegedy et al., 2016).

For fixed-length global descriptors, the output of global average pooling is preferred. This produces a vector of dimensionality matching the number of output channels in the last convolutional block (e.g., 1760 or 2080). These vectors are commonly L2-normalized before downstream use: $f_{\rm norm} = \frac{f}{\|f\|_2}$ Optionally, features can be dimensionality-reduced by PCA. For tasks requiring spatially-aware features, one may extract the full feature map (e.g., $8\times8\times1760$) or intermediate-stage activations (Szegedy et al., 2015, Szegedy et al., 2016).

3. Multi-level Spatially Pooled Feature Strategies

The MLSP (Multi-level Spatially Pooled) strategy concatenates pooled activations from all sequential blocks in InceptionResNet-v2 to construct extremely high-dimensional semantic signatures (Hosu et al., 2019). There are two principal pooling approaches:

• Narrow MLSP: Applies global average pooling to each block’s activation, producing 1×1×Cℓ features per block. Concatenation across L=43 blocks produces a 16,928-dim vector.
• Wide MLSP: Each activation is resized to $5\times5\times C_\ell$ via area-interpolation (OpenCV INTER_AREA), concatenated across all blocks to create a $5\times5\times16,928$ (423,200-dimensional) tensor.

This protocol enables a unified representation aggregating low, mid, and high-level cues. Key advantages include:

• Preservation of high-frequency detail by avoiding pre-extraction warping or cropping.
• The ability to support images of variable resolution and aspect ratio due to block-level pooling and interpolation.
• Compatibility with lightweight, custom head networks for transfer learning, as shown in shallow CNN configurations such as Single-3FC and Pool-3FC (Hosu et al., 2019).

4. Fusion of Deep Features in Conditional Generation Tasks

A representative methodology for incorporating deep features from InceptionResNet-v2 in conditional, non-classification pipelines appears in colorization networks such as Deep Koalarization (Baldassarre et al., 2017). The process involves:

1. Colorization Input Preparation: Greyscale luminance image in CIELab, $X_L$, is stacked and resized for InceptionResNet-v2 input.
2. Feature Extraction: The pre-trained network is run up to the last pre-softmax layer. The resulting embedding vector $F$ ($F\in\mathbb R^{1001}$) is obtained by feeding the triplicate-stacked, preprocessed luminance image.
3. Encoder-Decoder Fusion: The colorization encoder processes $X_L$ into mid-level feature maps ($E\in\mathbb R^{\tfrac H8\times\tfrac W8\times 512}$). The deep feature vector $F$ is spatially broadcast, concatenated with encoder maps, and reduced by a $1\times1$ convolution to fusable mid-level vectors ($Z\in\mathbb R^{\tfrac H8\times\tfrac W8\times256}$).
4. Decoding: The fused representation is upsampled and decoded into ab color channels, reconstructing the full-color image by combining with the original luminance channel.

Training is performed via standard mean squared error on the output color channels. The protocol does not utilize adversarial or perceptual losses; optimization is performed with Adam ($\beta_1=0.9, \beta_2=0.999$), batch size 100, and a fixed learning rate ($\eta=0.001$) (Baldassarre et al., 2017).

5. Representative Applications and Empirical Performance

Deep features from InceptionResNet-v2 have demonstrated efficacy in a variety of tasks:

• Aesthetic Quality Assessment: Extraction of MLSP features enables state-of-the-art performance on AVA, the leading aesthetics dataset. The Pool-3FC head trained on wide InceptionResNet-v2 MLSP features achieved SRCC 0.756 and accuracy 81.72%, surpassing prior models (NIMA, SRCC 0.612). The improvements are attributed to the full-resolution nature of feature extraction and the aggregation across all 43 network blocks (Hosu et al., 2019).
• Image Colorization: Conditional inference in Deep Koalarization leverages high-level semantic features from InceptionResNet-v2 to successfully infer plausible colorization, utilizing an encoder-decoder with explicit mid-/high-level fusion (Baldassarre et al., 2017).
• Transfer Learning: A canonical transfer learning workflow involves extracting the global-pooled feature vector from a pretrained network, L2-normalizing it, and using it as input to a linear/classification/regression model for a downstream task. Features are typically drawn just prior to the final classifier layer (Feng et al., 2015, Szegedy et al., 2016).
• Metric Learning and Retrieval: L2-normalized deep features from global pooling layers are widely used as compact image descriptors for retrieval, clustering, and similarity comparison (Szegedy et al., 2015, Szegedy et al., 2016).

6. Practical and Computational Considerations

InceptionResNet-v2 contains approximately 55M parameters and requires on the order of 13–20B multiply-add operations for a single 299×299 inference, depending on the variant (Szegedy et al., 2015, Szegedy et al., 2016). Feature extraction for a single image typically requires 1–1.5GB of GPU memory (batch 1). Tensor shapes for feature extraction depend on the extraction point within the network, with global pooled features offering the most compact descriptors. Batch-wise processing is standard for amortizing memory bandwidth and computational latency.

When integrating features into pipelines, normalization (L2 or PCA), and occasionally whitening, is advised, particularly in metric learning and SVM scenarios. For end-to-end fine-tuning, it is recommended to reduce the optimizer learning rate for all pretrained layers and to re-initialize or drop running mean/variance for batch normalization layers.

7. Extraction Recipes and Implementation Details

Feature extraction from InceptionResNet-v2 follows a standard protocol:

1. Load pretrained InceptionResNet-v2 weights.
2. Preprocess images: resize/crop to 299×299, subtract 128, divide by 128.
3. Run forward pass through the network.
4. Extract activations at the chosen tap (e.g., pre-logits for [2080], final convolutional map for [8,8,2080], or MLSP protocol for multi-level descriptors).
5. Optionally normalize features for transferability.

No additional batch normalization or activation scaling is typically performed during extraction; all per-layer batch normalization is fixed during inference. Systematic MLSP extraction—saving features from all blocks for all image augmentations prior to training lightweight head models—permits large-scale or variable-resolution datasets to be processed efficiently (Hosu et al., 2019).

• Deep Koalarization: Image Colorization using CNNs and Inception-ResNet-v2 (Baldassarre et al., 2017)
• Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning (Szegedy et al., 2016)
• Effective Aesthetics Prediction with Multi-level Spatially Pooled Features (Hosu et al., 2019)
• Rethinking the Inception Architecture for Computer Vision (Szegedy et al., 2015)