Patchify Capsule Layer (PatchifyCaps)

• PatchifyCaps is a primary-capsule mechanism that partitions multi-scale convolutional feature maps into non-overlapping patches to form localized capsule tokens.
• It employs average pooling, 1x1 convolution, positional embedding, and layer normalization to efficiently transform each patch into a capsule vector while preserving spatial identity.
• The approach improves computational efficiency and multi-scale feature integration compared to global capsule extraction, addressing limitations in prior capsule network designs.

Searching arXiv for the cited capsule-network papers and related PatchifyCaps work. {"queries":[{"query":"id:(Amer et al., 2019)"},{"query":"id:(Hu et al., 23 Aug 2025)"},{"query":"capsule network primary capsules patchify multi-scale routing"}]} Patchify Capsule Layer, usually abbreviated PatchifyCaps, is a primary-capsule construction mechanism introduced in "MSPCaps: A Multi-Scale Patchify Capsule Network with Cross-Agreement Routing for Visual Recognition" (Hu et al., 23 Aug 2025). In that architecture, PatchifyCaps converts multi-scale convolutional feature maps into localized capsule tokens by partitioning each feature map into non-overlapping patches, projecting each patch to a capsule vector, and preserving patch identity through positional embedding and layer normalization. The mechanism is designed to address two limitations identified in prior capsule-network practice: deriving primary capsules from a single high-level feature map, and neglecting complementary information across resolutions. In the literature provided here, the closest earlier precursor is "Path Capsule Networks" (Amer et al., 2019), which changes how primary capsules are formed but does not introduce anything explicitly named Patchify Capsule Layer or PatchifyCaps.

1. Definition and conceptual scope

PatchifyCaps is one of the three key components of MSPCaps, alongside a Multi-Scale ResNet Backbone (MSRB) and Cross-Agreement Routing (CAR) blocks (Hu et al., 23 Aug 2025). Its role is to transform the backbone’s multi-scale feature representations into primary capsules in a way that is spatially localized and computationally efficient. The paper characterizes this as a "one-patch, one-capsule" pipeline: each capsule corresponds to a patch at a specific scale rather than to an entire feature map.

The mechanism is motivated by a capsule-theoretic objection to global primary-capsule construction. Prior work such as Efficient-CapsNet is described as instantiating primary capsules directly from an entire feature map, which, in the authors’ account, makes each primary capsule too entangled with global context. PatchifyCaps instead treats a primary capsule as a localized visual primitive. This suggests a shift from feature-map-level capsule instantiation toward explicitly region-level capsule instantiation.

PatchifyCaps is also defined by its multi-scale character. MSPCaps uses three feature maps, and PatchifyCaps converts each of them into a scale-specific capsule set. The resulting capsule system therefore preserves both fine-grained details and coarse semantic context rather than forcing routing to operate on a single-resolution representation (Hu et al., 23 Aug 2025).

2. Motivation in relation to earlier capsule designs

The immediate rationale for PatchifyCaps is twofold. First, the paper argues that global feature map to primary capsule conversion is conceptually too coarse: low-level capsules should represent localized parts rather than entire scenes. Second, it argues that single-scale features miss complementary information, because high-resolution maps preserve edges, textures, and fine parts, whereas low-resolution maps preserve object-level semantics and context (Hu et al., 23 Aug 2025).

This positioning is clearer when contrasted with the earlier PathCapsNet architecture. PathCapsNet is described as "fundamentally different in how the PrimaryCapsules are constructed" because "each PrimaryCapsule is formed by a deep CNN, named a path" (Amer et al., 2019). Its design uses multiple parallel paths, each path outputs one 8D PrimaryCapsule, and all PrimaryCapsules are arranged with spatial dimensions $7 \times 7$. That mechanism is the closest material in the supplied literature to a patchify-style interpretation, but the paper explicitly states that it is not patchification in the Vision Transformer sense.

The distinction is consequential. PathCapsNet decomposes primary-capsule extraction into several deep, narrow CNN branches, whereas PatchifyCaps partitions multi-scale feature maps into non-overlapping patches of uniform size and maps each patch directly to a capsule vector. A plausible implication is that both approaches seek parameter-efficient localized capsule formation, but they operationalize locality differently: PathCapsNet through parallel deep feature extractors, PatchifyCaps through explicit patch partitioning (Amer et al., 2019, Hu et al., 23 Aug 2025).

3. PatchifyCaps pipeline and mathematical formulation

MSPCaps first produces three feature maps

$$f_i \in \mathbb{R}^{C_i \times H_i \times W_i}, \quad i \in \{1,2,3\},$$

with spatial sizes halving across scales,

$H_{i+1} = \frac{H_i}{2}.$

PatchifyCaps then converts each $f_i$ into a sequence of primary capsules through four steps (Hu et al., 23 Aug 2025).

In the first step, each feature map is divided into non-overlapping patches of uniform size $p \times p$. The implementation uses average pooling with kernel and stride equal to the patch size,

$\text{AvgPool2d}(f_i, k=p, s=p).$

This produces a local summary for each patch receptive field. If the feature map is divisible by $p$ in both spatial dimensions, the number of capsules generated from scale $i$ is

$n_i = \frac{H_i \cdot W_i}{p^2}.$

In the second step, a $1 \times 1$ convolution projects each pooled patch feature from $$f_i \in \mathbb{R}^{C_i \times H_i \times W_i}, \quad i \in \{1,2,3\},$$0 channels to the target capsule dimension $$f_i \in \mathbb{R}^{C_i \times H_i \times W_i}, \quad i \in \{1,2,3\},$$1,

$$f_i \in \mathbb{R}^{C_i \times H_i \times W_i}, \quad i \in \{1,2,3\},$$2

This is the operation by which a patch becomes a vector-valued capsule.

In the third step, the projected tensor is reshaped into a capsule sequence

$$f_i \in \mathbb{R}^{C_i \times H_i \times W_i}, \quad i \in \{1,2,3\},$$3

so that each of the $$f_i \in \mathbb{R}^{C_i \times H_i \times W_i}, \quad i \in \{1,2,3\},$$4 capsules has dimensionality $$f_i \in \mathbb{R}^{C_i \times H_i \times W_i}, \quad i \in \{1,2,3\},$$5.

In the fourth step, explicit spatial identity is reintroduced through a learnable positional embedding,

$$f_i \in \mathbb{R}^{C_i \times H_i \times W_i}, \quad i \in \{1,2,3\},$$6

followed by layer normalization,

$$f_i \in \mathbb{R}^{C_i \times H_i \times W_i}, \quad i \in \{1,2,3\},$$7

The paper also summarizes the procedure algorithmically: $$f_i \in \mathbb{R}^{C_i \times H_i \times W_i}, \quad i \in \{1,2,3\},$$8

$$f_i \in \mathbb{R}^{C_i \times H_i \times W_i}, \quad i \in \{1,2,3\},$$9

$H_{i+1} = \frac{H_i}{2}.$0

$H_{i+1} = \frac{H_i}{2}.$1

$H_{i+1} = \frac{H_i}{2}.$2

The output is therefore not a single capsule tensor but three scale-specific capsule sets. This preserves the backbone’s scale hierarchy while making it directly accessible to capsule routing (Hu et al., 23 Aug 2025).

4. Capsule dimensionality, patch size, and computational profile

PatchifyCaps uses scale-specific capsule dimensions that differ between the tiny and large variants of MSPCaps. For MSPCaps-T,

$H_{i+1} = \frac{H_i}{2}.$3

whereas for MSPCaps-L,

$H_{i+1} = \frac{H_i}{2}.$4

(Hu et al., 23 Aug 2025).

The default patch size is $H_{i+1} = \frac{H_i}{2}.$5. With this setting, the number of primary capsules is reduced to 84, compared with 1152 in the original CapsNet setup, which the paper describes as a reduction of over 90%. Because routing cost scales with the number of capsule votes, this reduction lowers the number of transformation matrices, vote computations, and associated memory and computation overhead. The paper further emphasizes that PatchifyCaps replaces heavier primary capsule construction with average pooling, a single $H_{i+1} = \frac{H_i}{2}.$6 convolution, reshape, positional embedding, and layer norm (Hu et al., 23 Aug 2025).

Patch size directly controls capsule count. For $H_{i+1} = \frac{H_i}{2}.$7, the model uses 84 capsules; for $H_{i+1} = \frac{H_i}{2}.$8, it uses 336. The paper tests $H_{i+1} = \frac{H_i}{2}.$9 on CIFAR-10. For MSPCaps-T, the reported accuracies are 88.01% for $f_i$0, 86.34% for $f_i$1, and 88.71% for $f_i$2, with $f_i$3 selected as best. For MSPCaps-L, the reported accuracies are 93.16% for $f_i$4, 93.30% for $f_i$5, and 92.88% for $f_i$6; despite the slight preference for $f_i$7, the authors choose $f_i$8 as the global setting for consistency and efficiency (Hu et al., 23 Aug 2025).

The paper describes the associated trade-off explicitly. Too small a patch size increases parameterization and may overfit. Too large or uneven patching can lose border information or miss local detail. This suggests that patch size functions as the principal granularity control for the primary-capsule layer.

5. Coupling with MSRB and Cross-Agreement Routing

PatchifyCaps is structurally positioned between the Multi-Scale ResNet Backbone and the Cross-Agreement Routing block (Hu et al., 23 Aug 2025). MSRB produces the three feature maps; PatchifyCaps converts them into capsule sequences $f_i$9; CAR then performs progressive fine-to-coarse fusion. The paper describes this routing schedule as: first route fine capsules $p \times p$0 with mid-level capsules $p \times p$1, then route the fused result with coarse capsules $p \times p$2.

The coupling between PatchifyCaps and CAR is not merely sequential but geometric. The routing stage depends on spatial correspondence created by patchification. The paper defines a group size

$p \times p$3

so that a group of finer capsules corresponds to one coarser capsule. PatchifyCaps provides the structured patch grid that makes this region-wise correspondence well defined.

CAR operates on prediction capsules generated by capsule-to-capsule transformations: $p \times p$4 For the shared-weight version used in the tiny model, the same matrix is applied to both scales: $p \times p$5 Agreement is then computed through the maximum similarity within each corresponding fine-scale group,

$p \times p$6

The output capsule is

$p \times p$7

followed by the standard capsule squash nonlinearity,

$p \times p$8

Within this design, PatchifyCaps is the mechanism that makes cross-scale grouping and agreement comparison operationally meaningful. Without the patch grid, CAR would not have the same structured basis for aligning finer and coarser capsules (Hu et al., 23 Aug 2025).

6. Empirical role, interpretation, and relation to PathCapsNet

The empirical contribution attributed to PatchifyCaps is indirect but substantial. The paper notes that there is no ablation removing only the patchify operation while keeping everything else fixed, but several results speak to its effect. In the scale ablation, using only the $p \times p$9 scale yields 74.81%, only $\text{AvgPool2d}(f_i, k=p, s=p).$0 yields 81.90%, only $\text{AvgPool2d}(f_i, k=p, s=p).$1 yields 87.48%, and using all three scales yields 88.71% (Hu et al., 23 Aug 2025). The paper interprets this as confirmation that multi-scale capsule construction adds value beyond any single scale.

The broader model-level results are also presented as evidence for the design. MSPCaps is reported to achieve configurations ranging from a Tiny model with 344.3K parameters to a Large model with 10.9M parameters, while consistently surpassing multiple baseline methods in terms of classification accuracy and exhibiting superior robustness (Hu et al., 23 Aug 2025). Since PatchifyCaps is one of the three defining components of MSPCaps, a plausible implication is that its efficiency and spatial organization are integral to those outcomes.

The comparison with PathCapsNet clarifies a potential misconception. PatchifyCaps should not be conflated with the path-based primary-capsule extractor in PathCapsNet. PathCapsNet does not define a module named PatchifyCaps, and its primary capsules are formed by deep CNN paths rather than by explicit patch partitioning (Amer et al., 2019). In PathCapsNet, each path is a deep narrow CNN with two initial $\text{AvgPool2d}(f_i, k=p, s=p).$2 convolution layers, a $\text{AvgPool2d}(f_i, k=p, s=p).$3 max-pooling layer, two more $\text{AvgPool2d}(f_i, k=p, s=p).$4 convolution layers, and another $\text{AvgPool2d}(f_i, k=p, s=p).$5 max-pooling layer; the resulting primary capsules are then routed to DigitCaps using either fan-out or the newly introduced fan-in routing. Its fan-in coefficients are defined as

$\text{AvgPool2d}(f_i, k=p, s=p).$6

in contrast to CapsNet-style fan-out routing,

$\text{AvgPool2d}(f_i, k=p, s=p).$7

(Amer et al., 2019). The two papers therefore address related questions—how to construct and route primary capsules efficiently—but by different means.

Taken together, the supplied literature positions PatchifyCaps as a specialized primary-capsule layer for multi-scale visual recognition. Its defining characteristics are non-overlapping patch partitioning, lightweight per-patch projection, positional encoding, and a scale-aware capsule layout tailored for cross-scale routing. Its nearest precursor in the provided record is not a patchify layer in the strict sense, but a path-based rethinking of primary-capsule construction that similarly seeks more efficient and more structured capsule formation (Amer et al., 2019, Hu et al., 23 Aug 2025).