MSPCaps: Multi-Scale Patchify Capsule Net

• The paper introduces a novel capsule architecture that integrates multi-scale feature extraction and uniform patchification to preserve spatial locality.
• It employs cross-scale agreement routing (CAR) to selectively fuse fine and coarse capsule predictions, improving coherence and parameter efficiency.
• Empirical results on MNIST, CIFAR-10, SVHN, and FashionMNIST demonstrate competitive accuracy gains over previous capsule designs with a favorable parameter tradeoff.

Multi-Scale Patchify Capsule Network (MSPCaps) is a capsule-based visual recognition architecture that combines multi-scale feature extraction, localized capsule formation, and cross-scale agreement-based routing to address two recurrent limitations of prior capsule networks: dependence on a single high-level feature map and shallow multi-scale fusion by addition or concatenation (Hu et al., 23 Aug 2025). The model is organized around three components—Multi-Scale ResNet Backbone (MSRB), Patchify Capsule Layer (PatchifyCaps), and Cross-Agreement Routing (CAR)—that together preserve fine-grained detail, retain global contextual information, and route only the most coherent part-to-whole predictions across scales. For $32\times 32$ inputs, the canonical pipeline extracts feature maps at $32\times 32$, $16\times 16$, and $8\times 8$, converts them into localized primary capsules through uniform patchification, and fuses them in two non-iterative CAR stages before classification by class capsule norm (Hu et al., 23 Aug 2025).

1. Problem setting and design rationale

MSPCaps is motivated by a tension internal to the capsule-network program. Standard CapsNets form primary capsules directly from a single high-level feature map, often a global feature representation that dilutes spatial locality. In the formulation adopted by MSPCaps, this weakens the capsule objective of encoding parts through pose and activation so that higher-level capsules can be inferred from consistent votes. The paper further argues that downsampling in deep backbones reduces equivariance to local transformations, thereby compounding the loss of localized structure (Hu et al., 23 Aug 2025).

A second motivation concerns multi-scale fusion. Addition and concatenation combine features with different receptive fields and semantic levels without explicitly testing whether those features are mutually consistent. Under this view, fine-scale capsules may be dominated by local textures while coarse-scale capsules encode semantic context, and shallow fusion can therefore produce incoherent votes. MSPCaps is designed to resolve this by extracting multi-scale feature maps, patchifying each scale with a uniform patch size so that capsules remain localized and spatially aligned, and routing via cross-scale maximum agreement so that only the most coherent prediction pair per spatial region contributes to the final vote.

This design places MSPCaps within the broader line of capsule architectures that emphasize equivariant representation and part-whole hierarchy, but it departs from earlier single-scale or concatenation-based formulations by making cross-scale consistency a routing primitive rather than a post hoc fusion step. A plausible implication is that MSPCaps treats multi-scale structure not merely as added capacity, but as a constraint on which votes are permitted to survive.

2. Architectural organization

The end-to-end architecture has three stages: multi-scale feature extraction, per-scale capsule formation, and two-stage cross-scale routing (Hu et al., 23 Aug 2025). For a $32\times 32$ image, MSRB emits three feature maps $f_1, f_2, f_3$ at resolutions $32\times 32$, $16\times 16$, and $8\times 8$. PatchifyCaps converts each feature map into a capsule set $u_1, u_2, u_3$. CAR then first fuses $32\times 32$0 to produce an intermediate capsule set $32\times 32$1, and subsequently fuses $32\times 32$2 to produce class capsules. Prediction uses the class capsule norms.

Within MSRB, each stage begins with a downsampling $32\times 32$3 convolution with stride $32\times 32$4, followed by BatchNorm and ReLU, then $32\times 32$5 identical $32\times 32$6 convolutional modules with stride $32\times 32$7 and residual connections. The backbone is explicitly asymmetric across resolution: shallower parametrization at high resolution is intended to preserve localized primitives and reduce overfitting to textures, while deeper parametrization at lower resolution is intended to enrich global semantics. The paper also states that sharing early residual blocks across scales regularizes learning by exposing shared kernels to gradients from both fine and coarse outputs.

Two named instantiations are provided.

Variant	Core configuration	Total parameters
MSPCaps-T	$32\times 32$8, $32\times 32$9, $16\times 16$0, $16\times 16$1, $16\times 16$2, patch size $16\times 16$3, CAR with weight sharing	$16\times 16$4
MSPCaps-L	$16\times 16$5, $16\times 16$6, $16\times 16$7, $16\times 16$8, $16\times 16$9, patch size $8\times 8$0 as global choice, CAR without weight sharing	$8\times 8$1

Fusion is not performed inside MSRB. BatchNorm and ReLU operate within residual blocks, while capsule formation applies LayerNorm after patchification and positional embedding. The architecture therefore separates feature extraction from cross-scale reconciliation: MSRB produces scale-specific representations, and CAR performs the actual multi-scale fusion.

3. PatchifyCaps and localized capsule formation

PatchifyCaps enforces a strict “one-patch, one-capsule” mapping (Hu et al., 23 Aug 2025). Each feature map $8\times 8$2 is partitioned into non-overlapping $8\times 8$3 patches by AvgPool2d with kernel $8\times 8$4 and stride $8\times 8$5. A $8\times 8$6 convolution then projects channels to capsule dimension $8\times 8$7. The resulting grid is flattened into $8\times 8$8 capsules,

$8\times 8$9

Positional embeddings $32\times 32$0 are added to preserve spatial identity, and LayerNorm normalizes the capsule vectors.

The paper attributes three roles to this operation. First, it preserves locality by tying each primary capsule to a bounded spatial patch rather than to a globally mixed feature tensor. Second, because the patch size is uniform in feature space across scales, it induces spatial correspondence between fine and coarse capsule grids. Third, it is computationally efficient relative to conventional primary-capsule constructions: AvgPool2d plus $32\times 32$1 projection replaces parallel convolutions, and with $32\times 32$2 the model yields $32\times 32$3 capsules instead of $32\times 32$4 in the cited earlier capsule formulation.

Capsule pose is represented as a vector of dimension $32\times 32$5, not a matrix, and activation is implicit in the vector norm. The output nonlinearity is the standard squash function,

$32\times 32$6

where $32\times 32$7 is the pre-activation vote sum and $32\times 32$8 is the output capsule. This preserves the usual capsule interpretation in which vector orientation carries pose information while vector magnitude encodes activation strength.

A common misconception is to treat patchification here as generic tokenization analogous to transformer patch embeddings. In MSPCaps, patchification is not merely a preprocessing convenience; it is the mechanism that makes cross-scale spatial grouping deterministic and thereby enables the subsequent agreement calculation in CAR.

4. Cross-Agreement Routing

CAR is the central routing mechanism of MSPCaps and is explicitly non-iterative (Hu et al., 23 Aug 2025). It takes two capsule sets from layer $32\times 32$9: a fine set $f_1, f_2, f_3$0 and a coarse set $f_1, f_2, f_3$1. For each output capsule $f_1, f_2, f_3$2, linear predictions are formed from both inputs,

$f_1, f_2, f_3$3

In the weight-sharing option used by MSPCaps-T, $f_1, f_2, f_3$4 is eliminated and $f_1, f_2, f_3$5 is reused for grouped fine capsules:

$f_1, f_2, f_3$6

Uniform patchification induces a local grouping relation between scales. Each coarse capsule $f_1, f_2, f_3$7 corresponds to a group of $f_1, f_2, f_3$8 fine capsules. CAR computes raw agreement for the $f_1, f_2, f_3$9-th output capsule and the $32\times 32$0-th coarse capsule by taking the maximum similarity over the fine predictions within that group,

$32\times 32$1

This is the model’s “maximum agreement” operation: only the best fine-to-coarse prediction pair within a spatial group determines the coupling for that region. The weighted aggregation then uses coarse predictions,

$32\times 32$2

followed by squash to obtain the output capsule.

The paper contrasts CAR with both self-routing and concatenation-based fusion. Self-routing over a single scale does not explicitly check cross-scale coherence. Concatenation mixes features with incompatible receptive fields but does not resolve disagreement. CAR instead acts as a hard attention mechanism over local fine-scale explanations, selecting the most coherent part-to-whole pair and suppressing incoherent votes. Its stated complexity scales with

$32\times 32$3

and memory is reduced because the weighted sum is formed over coarse predictions. Dropout with rate $32\times 32$4 may be applied inside CAR after computing coupling coefficients, although it is omitted for SVHN in the reported experiments.

5. Training objective and implementation protocol

MSPCaps uses margin loss and omits reconstruction loss (Hu et al., 23 Aug 2025). For $32\times 32$5 classes, class capsule $32\times 32$6, and target $32\times 32$7, the loss is

$32\times 32$8

with

$32\times 32$9

The paper does not numerically specify $16\times 16$0, $16\times 16$1, or $16\times 16$2, and notes that defaults from the earlier capsule literature can be used if needed. Cross-entropy is not used; optimization targets capsule norms, and evaluation uses top-1 prediction by largest class capsule $16\times 16$3 norm.

The reported implementation uses PyTorch 2.5.1, Python 3.12, and CUDA 12.4 on a single NVIDIA RTX 4090D GPU. Training typically runs for $16\times 16$4 epochs, with an appendix note that MNIST was trained for $16\times 16$5 epochs in one setting. The optimizer is AdamW with base learning rate $16\times 16$6, weight decay $16\times 16$7, batch size $16\times 16$8, and a learning-rate schedule consisting of a $16\times 16$9-epoch linear warmup from $8\times 8$0 of the base value followed by cosine annealing to $8\times 8$1.

Preprocessing and augmentation are dataset-specific. MNIST and FashionMNIST are resized to $8\times 8$2. Augmentations include random crop with padding, random horizontal flip except for SVHN, and random rotation of $8\times 8$3 for MNIST and SVHN. Normalization is $8\times 8$4, $8\times 8$5 for MNIST, FashionMNIST, and SVHN; CIFAR-10 uses channel-wise mean $8\times 8$6 and std $8\times 8$7. Initialization is Xavier normal for the Patchify $8\times 8$8 convolution and Kaiming for other convolutions and projection layers.

The paper does not report inference speed, latency, or FLOPs. This omission matters because CAR is presented as more efficient than iterative routing, but deployment under strict real-time constraints would still require explicit profiling.

6. Empirical performance, interpretation, and limitations

The reported benchmarks are MNIST, FashionMNIST, SVHN, and CIFAR-10, with classification accuracy computed from the largest class capsule $8\times 8$9 norm (Hu et al., 23 Aug 2025). MSPCaps-T, with $u_1, u_2, u_3$0 parameters, achieves $u_1, u_2, u_3$1 on MNIST, $u_1, u_2, u_3$2 on SVHN, and $u_1, u_2, u_3$3 on CIFAR-10. MSPCaps-L, with $u_1, u_2, u_3$4 parameters, achieves $u_1, u_2, u_3$5 on MNIST, $u_1, u_2, u_3$6 on FashionMNIST, and $u_1, u_2, u_3$7 on CIFAR-10.

Model	Reported results
MSPCaps-T	$u_1, u_2, u_3$8 MNIST, $u_1, u_2, u_3$9 SVHN, $32\times 32$00 CIFAR-10, $32\times 32$01 params
MSPCaps-L	$32\times 32$02 MNIST, $32\times 32$03 FashionMNIST, $32\times 32$04 CIFAR-10, $32\times 32$05 params

The paper reports that MSPCaps-T outperforms CapsNet, DA-CapsNet, AA-CapsNet, Efficient-Caps, and OrthCaps-S on CIFAR-10, including a $32\times 32$06 gain over OrthCaps-S. MSPCaps-L is reported to outperform DeepCaps by $32\times 32$07 on CIFAR-10 and RS-CapsNet by $32\times 32$08, while also exceeding the listed results for AR-CapsNet and OrthCaps-D.

Ablation studies isolate the contribution of each design decision. For MSPCaps-T on CIFAR-10, using only the $32\times 32$09 scale yields $32\times 32$10, only $32\times 32$11 yields $32\times 32$12, only $32\times 32$13 yields $32\times 32$14, and all three scales together yield $32\times 32$15. The paper interprets this as evidence that $32\times 32$16 contributes global context while $32\times 32$17 refines fine detail. CAR is also compared against a Dynamic Routing baseline with four blocks and multi-scale independent routing plus concatenation: MSPCaps-T with CAR attains $32\times 32$18 using $32\times 32$19 parameters versus $32\times 32$20 with $32\times 32$21 for the DR baseline, and MSPCaps-L with CAR attains $32\times 32$22 with $32\times 32$23 parameters versus $32\times 32$24 with $32\times 32$25.

Weight sharing in CAR exhibits a scale-dependent tradeoff. In MSPCaps-T, sharing improves performance to $32\times 32$26 with $32\times 32$27 parameters versus $32\times 32$28 with $32\times 32$29 without sharing, suggesting that sharing is beneficial under tight parameter budgets. In MSPCaps-L, the non-sharing variant improves to $32\times 32$30 with $32\times 32$31 parameters versus $32\times 32$32 with $32\times 32$33, indicating that larger models benefit from additional projection expressivity.

Patch size is similarly nontrivial. For MSPCaps-T, $32\times 32$34 gives the best reported accuracy, $32\times 32$35, and the lowest parameter count among the tested settings, while $32\times 32$36 increases parameters to $32\times 32$37 and mildly reduces accuracy to $32\times 32$38, and $32\times 32$39 reduces accuracy to $32\times 32$40 due to uneven borders. For MSPCaps-L, $32\times 32$41 achieves $32\times 32$42 with $32\times 32$43 parameters, $32\times 32$44 yields $32\times 32$45 with $32\times 32$46 and overfitting risk, and $32\times 32$47 yields $32\times 32$48 with $32\times 32$49; the global choice remains $32\times 32$50 for consistency and efficiency.

The robustness study applies FGSM and BIM with $32\times 32$51 steps on CIFAR-10. MSPCaps-L is reported to exhibit strong stability across increasing $32\times 32$52, whereas MSPCaps-T degrades for $32\times 32$53 and eventually falls below original CapsNet. The paper attributes the stronger behavior of the large model to multi-scale agreement and progressive fusion, although this remains an empirical interpretation rather than a formal robustness guarantee.

Several limitations are explicitly acknowledged. Patchification with a patch size that does not evenly tile a feature map can lose border information and reduce accuracy. The Tiny variant is less robust under stronger adversarial perturbation. FLOPs and latency are not reported. Future work proposed in the paper includes adaptive or learned patch sizes per scale, deformable patch boundaries, alternative agreement metrics such as cosine similarity or learned bilinear forms, top-$32\times 32$54 rather than top-1 within-group selection, integration of explicit equivariance modules, and calibration analyses.

Taken together, MSPCaps is best understood as a capsule architecture that recasts multi-scale fusion as selective agreement-based routing. Its empirical contribution lies not only in the reported accuracies and parameter tradeoffs, but also in the architectural claim that spatially aligned patchified capsules permit fine-to-coarse part-whole reasoning without iterative routing.