Hierarchical Convolutional Patch Embedding

• HCPE is a multi-layer convolutional embedding module that replaces traditional single-layer patch embeddings in ViTs, injecting strong locality and inductive bias.
• It employs a compact stack of convolutional blocks (MBConv or Fused-MBConv) to progressively down-sample features, enhancing the effective receptive field and semantic representation.
• The design integrates seamlessly into hierarchical backbones, improving performance across benchmarks like ImageNet, COCO, and ADE20K without significant extra computational cost.

Hierarchical Convolutional Patch Embedding (HCPE) is a multi-layer convolutional embedding module designed to replace traditional patch embedding layers in hierarchical Vision Transformers (ViTs). HCPE is constructed to inject strong locality and inductive bias at each scale of the network, enhancing both the effective receptive field and semantic feature representation. It is primarily employed at each stage transition in hierarchical ViT backbones, resulting in measurable accuracy improvements across image classification, detection, and segmentation benchmarks without increasing FLOPs or parameter count in a significant manner (Wang et al., 2022).

1. Motivation and Conceptual Framework

Traditional ViT architectures utilize a single linear projection or a shallow convolution (often 1–2 layers) to perform patch embedding, typically lacking spatial locality and constrained effective receptive field (ERF). This design often limits data efficiency and the network’s local semantic representation. In contrast, hierarchical ViT designs (such as PVT and Swin) benefit from progressive spatial down-sampling and feature pyramid construction. HCPE generalizes the embedding module by introducing a compact stack of convolutional blocks—either MBConv (depthwise separable, pointwise expansion) or Fused-MBConv—at the beginning of each stage. This approach delivers:

• Stronger locality and increased ERF per scale,
• Inductive bias propagated to deeper stages, not just the initial input,
• Significant accuracy uplift with preserved computational budget.

2. Macro-Architecture and Stage Design

HCPEs are inserted at the start of each stage within a 5-stage hierarchical backbone. The progression of spatial resolutions and channel expansions for a $224 \times 224$ input image follows a regular pattern, as shown below:

Stage	Spatial Transformation	Output Channels ($C^i$)
S0	$224 \to 56$	$C^0$ (conv stem)
S1	$56 \to 28$	$C^1$ (HCPE$^1$)
S2	$28 \to 14$	$C^2$ (HCPE$^2$)
S3	$C^i$0	$C^i$1 (HCPE$C^i$2)
S4	$C^i$3	$C^i$4 (HCPE$C^i$5)

The initial stem (S0) comprises four Fused-MBConv layers followed by a $C^i$6 convolution and LayerNorm, transforming RGB input into feature maps. Each subsequent HCPE takes feature maps, halves spatial resolution (except at S4) and increases channel dimension prior to transformer (ViT) blocks.

3. HCPE Layer Configuration and Forward Path

Each HCPE comprises $C^i$7 convolutional blocks:

• The first block: MBConv (stages S2–S4) or Fused-MBConv (S0/S1), stride $C^i$8 for down-sampling.
• Remaining blocks: MBConv (or Fused-MBConv in S1), stride $C^i$9, maintaining spatial size.
• All blocks: 3×3 depthwise convolution with BatchNorm (BN) and GELU activation (ReLU for Fused-MBConv), sandwiched between pointwise ($224 \to 56$0) expansion and projection.
• After 5 blocks, a terminal $224 \to 56$1 convolution with LayerNorm maps to the target channel dimension.
• The final output is flattened spatially to produce a sequence of tokens for the transformer stage.

The typical configuration values for channels are $224 \to 56$2=64, $224 \to 56$3=128, $224 \to 56$4=256, $224 \to 56$5=512, $224 \to 56$6=1024 (for the “base” model).

4. Mathematical Formulation

Given input $224 \to 56$7, the HCPE is a composition of L convolutional layers:

$224 \to 56$8

for $224 \to 56$9, with kernel sizes $C^0$0, strides $C^0$1 (down-sampling on $C^0$2), and paddings $C^0$3. $C^0$4 is BatchNorm for internal layers; the final $C^0$5 convolution’s output passes through LayerNorm. Activations $C^0$6 are GELU in MBConv, ReLU in Fused-MBConv. After $C^0$7 layers, flattening across spatial dimensions yields token embeddings:

$C^0$8

where $C^0$9 and $56 \to 28$0 is the channel size after $56 \to 28$1 layers.

5. Integration into Vision Transformers

HCPE modules are integrated into the transformer backbone as follows:

• The input image passes through the convolutional stem (4 × Fused-MBConv + $56 \to 28$2 conv + LayerNorm).
• At each subsequent stage, an HCPE module down-samples and re-embeds features, which are then tokenized.
• Tokens go through $56 \to 28$3 blocks of windowed (often shifted) self-attention + MLP transformer layers.
• The process is repeated for four hierarchical stages, forming a spatial feature pyramid for downstream tasks.
• Final tokens are processed by classification, detection, or segmentation heads.

The pseudocode for this process is:

$C^1$0

6. Comparison with Standard ViT Patch Embedding

The original ViT employs a single $56 \to 28$4 convolution (or linear projection) with stride 16 to transform a $56 \to 28$5 image directly into $56 \to 28$6 tokens, leading to a lack of progressive spatial hierarchy. In contrast, HCPE introduces:

• Multi-layer design: Five small convolutional blocks per stage, instead of a single patchification layer.
• Multi-scale pyramid: Inserted at every stage, with progressive spatial reduction (56→28→14→7), forming hierarchical feature maps.
• Enhanced inductive bias: Depthwise convolutions enlarge the effective receptive field and embed locality, which a single projection cannot achieve.

7. Empirical Performance and Best Practices

Substituting original patch embedding/merging modules in hierarchical ViTs (CvT-13, PVT-S, Swin-T, CSWin-T) with HCPE leads to consistent accuracy increases on ImageNet-1K:

Model	Orig Top-1	+HCPE Top-1	Δ
CvT-13	81.6%	82.1%	+0.5
PVT-S	79.8%	81.1%	+1.3
Swin-T	81.3%	82.5%	+1.2
CSWin-T	82.7%	83.6%	+0.9

On COCO (Mask-R-CNN, $56 \to 28$7 schedule):

Backbone	box AP	mask AP
Swin-T	43.7	39.1
HCPE-Swin-T	45.5	40.7

On ADE20K (UperNet, single-scale):

Backbone	mIoU
Swin-T	44.5
HCPE-Swin-T	46.5

For implementation, the following are recommended: use five conv blocks per HCPE (first with stride 2); prefer Fused-MBConv for S0/S1 and MBConv elsewhere; expansion ratios $56 \to 28$8; always append LayerNorm after the final $56 \to 28$9 conv; GELU activation for MBConv, ReLU for Fused-MBConv; AdamW optimizer with recommended hyperparameters (initial lr 1e–3, weight-decay 0.05, 300 epochs, batch 1024, input 224, 20-epoch warmup, cosine schedule), and fine-tuning at 384×384 resolution for 30 epochs with lr 2e–5. Maintaining windowed self-attention but projecting queries/keys via local depthwise convolution (LEWin) further strengthens inductive bias.

HCPE thus provides a lightweight, scalable, and empirically validated alternative to single-layer patch embeddings, enhancing both performance and feature hierarchy in modern hierarchical ViTs (Wang et al., 2022).