---
title: 'MSPCaps: Multi-Scale Patchify Capsule Net'
url: https://www.emergentmind.com/topics/multi-scale-patchify-capsule-network-mspcaps
type: topic
---

# MSPCaps: Multi-Scale Patchify Capsule Net

Multi-Scale Patchify Capsule Network (MSPCaps) is a capsule-based visual recognition architecture that combines multi-scale feature extraction, localized capsule formation, and cross-scale agreement-based routing to address two recurrent limitations of prior capsule networks: dependence on a single high-level feature map and shallow multi-scale fusion by addition or concatenation [2508.16922]. The model is organized around three components—Multi-Scale ResNet Backbone (MSRB), Patchify Capsule Layer (PatchifyCaps), and Cross-Agreement Routing (CAR)—that together preserve fine-grained detail, retain global contextual information, and route only the most coherent part-to-whole predictions across scales. For \(32\times 32\) inputs, the canonical pipeline extracts feature maps at \(32\times 32\), \(16\times 16\), and \(8\times 8\), converts them into localized primary capsules through uniform patchification, and fuses them in two non-iterative CAR stages before classification by class capsule norm [2508.16922].

## 1. Problem setting and design rationale

MSPCaps is motivated by a tension internal to the capsule-network program. Standard CapsNets form primary capsules directly from a single high-level feature map, often a global feature representation that dilutes spatial locality. In the formulation adopted by MSPCaps, this weakens the capsule objective of encoding parts through pose and activation so that higher-level capsules can be inferred from consistent votes. The paper further argues that downsampling in deep backbones reduces equivariance to local transformations, thereby compounding the loss of localized structure [2508.16922].

A second motivation concerns multi-scale fusion. Addition and concatenation combine features with different receptive fields and semantic levels without explicitly testing whether those features are mutually consistent. Under this view, fine-scale capsules may be dominated by local textures while coarse-scale capsules encode semantic context, and shallow fusion can therefore produce incoherent votes. MSPCaps is designed to resolve this by extracting multi-scale feature maps, patchifying each scale with a uniform patch size so that capsules remain localized and spatially aligned, and routing via cross-scale maximum agreement so that only the most coherent prediction pair per spatial region contributes to the final vote.

This design places MSPCaps within the broader line of capsule architectures that emphasize equivariant representation and part-whole hierarchy, but it departs from earlier single-scale or concatenation-based formulations by making cross-scale consistency a routing primitive rather than a post hoc fusion step. A plausible implication is that MSPCaps treats multi-scale structure not merely as added capacity, but as a constraint on which votes are permitted to survive.

## 2. Architectural organization

The end-to-end architecture has three stages: multi-scale feature extraction, per-scale capsule formation, and two-stage cross-scale routing [2508.16922]. For a \(32\times 32\) image, MSRB emits three feature maps \(f_1, f_2, f_3\) at resolutions \(32\times 32\), \(16\times 16\), and \(8\times 8\). PatchifyCaps converts each feature map into a capsule set \(u_1, u_2, u_3\). CAR then first fuses \((u_1, u_2)\) to produce an intermediate capsule set \(u_{1-2}\), and subsequently fuses \((u_{1-2}, u_3)\) to produce class capsules. Prediction uses the class capsule norms.

Within MSRB, each stage begins with a downsampling \(3\times 3\) convolution with stride \(2\), followed by BatchNorm and ReLU, then \(N\) identical \(3\times 3\) convolutional modules with stride \(1\) and residual connections. The backbone is explicitly asymmetric across resolution: shallower parametrization at high resolution is intended to preserve localized primitives and reduce overfitting to textures, while deeper parametrization at lower resolution is intended to enrich global semantics. The paper also states that sharing early residual blocks across scales regularizes learning by exposing shared kernels to gradients from both fine and coarse outputs.

Two named instantiations are provided.

| Variant | Core configuration | Total parameters |
|---|---|---|
| MSPCaps-T | \((C_1,C_2,C_3)=(32,64,128)\), \(N=2\), \((d_1,d_2,d_3)=(8,8,16)\), \(d_{\text{mid}}=16\), \(d_{\text{out}}=32\), patch size \(p=4\), CAR with weight sharing | \(\approx 344.3\text{K}\) |
| MSPCaps-L | \((C_1,C_2,C_3)=(128,256,512)\), \(N=3\), \((d_1,d_2,d_3)=(16,32,64)\), \(d_{\text{mid}}=64\), \(d_{\text{out}}=128\), patch size \(p=4\) as global choice, CAR without weight sharing | \(\approx 10.9\text{M}\) |

Fusion is not performed inside MSRB. BatchNorm and ReLU operate within residual blocks, while capsule formation applies LayerNorm after patchification and positional embedding. The architecture therefore separates feature extraction from cross-scale reconciliation: MSRB produces scale-specific representations, and CAR performs the actual multi-scale fusion.

## 3. PatchifyCaps and localized capsule formation

PatchifyCaps enforces a strict “one-patch, one-capsule” mapping [2508.16922]. Each feature map \(f_i \in \mathbb{R}^{C_i \times H_i \times W_i}\) is partitioned into non-overlapping \(p\times p\) patches by `AvgPool2d` with kernel \(p\) and stride \(p\). A \(1\times 1\) convolution then projects channels to capsule dimension \(d_i\). The resulting grid is flattened into \(n_i\) capsules,
$$
n_i = \frac{H_i \cdot W_i}{p^2}, \qquad u_i \in \mathbb{R}^{n_i \times d_i}.
$$
Positional embeddings \(E_{\text{pos}} \in \mathbb{R}^{n_i \times d_i}\) are added to preserve spatial identity, and LayerNorm normalizes the capsule vectors.

The paper attributes three roles to this operation. First, it preserves locality by tying each primary capsule to a bounded spatial patch rather than to a globally mixed feature tensor. Second, because the patch size is uniform in feature space across scales, it induces spatial correspondence between fine and coarse capsule grids. Third, it is computationally efficient relative to conventional primary-capsule constructions: `AvgPool2d` plus \(1\times 1\) projection replaces parallel convolutions, and with \(p=4\) the model yields \(84\) capsules instead of \(1152\) in the cited earlier capsule formulation.

Capsule pose is represented as a vector of dimension \(d_i\), not a matrix, and activation is implicit in the vector norm. The output nonlinearity is the standard squash function,
$$
v = \frac{\|s\|^2}{1+\|s\|^2}\frac{s}{\|s\|},
$$
where \(s\) is the pre-activation vote sum and \(v\) is the output capsule. This preserves the usual capsule interpretation in which vector orientation carries pose information while vector magnitude encodes activation strength.

A common misconception is to treat patchification here as generic tokenization analogous to transformer patch embeddings. In MSPCaps, patchification is not merely a preprocessing convenience; it is the mechanism that makes cross-scale spatial grouping deterministic and thereby enables the subsequent agreement calculation in CAR.

## 4. Cross-Agreement Routing

CAR is the central routing mechanism of MSPCaps and is explicitly non-iterative [2508.16922]. It takes two capsule sets from layer \(l\): a fine set \(u_1^l \in \mathbb{R}^{n_{\text{in}}^{(1)} \times d_{\text{in}}^{(1)}}\) and a coarse set \(u_2^l \in \mathbb{R}^{n_{\text{in}}^{(2)} \times d_{\text{in}}^{(2)}}\). For each output capsule \(j\), linear predictions are formed from both inputs,
$$
\hat{u}_{j|i}^{(1)} = W_{j,i}^{(1)}u_{1,i}^{l}, \qquad
\hat{u}_{j|k}^{(2)} = W_{j,k}^{(2)}u_{2,k}^{l}.
$$
In the weight-sharing option used by MSPCaps-T, \(W^{(1)}\) is eliminated and \(W^{(2)}\) is reused for grouped fine capsules:
$$
\hat{u}_{j|m,k}^{(1)} = W_{j,k}^{(2)}u_{1,m,k}^{l}.
$$

Uniform patchification induces a local grouping relation between scales. Each coarse capsule \(k\) corresponds to a group of \(s = n_{\text{in}}^{(1)} / n_{\text{in}}^{(2)}\) fine capsules. CAR computes raw agreement for the \(j\)-th output capsule and the \(k\)-th coarse capsule by taking the maximum similarity over the fine predictions within that group,
$$
c_{jk} = \mathrm{Softmax}_k\!\left(
\max_{m=1,\dots,s}
\left(
\frac{\hat{u}_{j|m,k}^{(1)} \cdot (\hat{u}_{j|k}^{(2)})^\top}{\sqrt{d_{\mathrm{out}}}}
\right)
\right).
$$
This is the model’s “maximum agreement” operation: only the best fine-to-coarse prediction pair within a spatial group determines the coupling for that region. The weighted aggregation then uses coarse predictions,
$$
v_j = \sum_k c_{jk}\hat{u}_{j|k}^{(2)},
$$
followed by squash to obtain the output capsule.

The paper contrasts CAR with both self-routing and concatenation-based fusion. Self-routing over a single scale does not explicitly check cross-scale coherence. Concatenation mixes features with incompatible receptive fields but does not resolve disagreement. CAR instead acts as a hard attention mechanism over local fine-scale explanations, selecting the most coherent part-to-whole pair and suppressing incoherent votes. Its stated complexity scales with
$$
O(n_{\text{out}} \cdot n_{\text{in}}^{(2)} \cdot d_{\text{out}} + n_{\text{out}} \cdot n_{\text{in}}^{(2)} \cdot s),
$$
and memory is reduced because the weighted sum is formed over coarse predictions. Dropout with rate \(0.1\) may be applied inside CAR after computing coupling coefficients, although it is omitted for SVHN in the reported experiments.

## 5. Training objective and implementation protocol

MSPCaps uses margin loss and omits reconstruction loss [2508.16922]. For \(K\) classes, class capsule \(v_k\), and target \(T_k \in \{0,1\}\), the loss is
$$
L_k = T_k \max(0, m^+ - \|v_k\|)^2 + \lambda (1 - T_k)\max(0, \|v_k\| - m^-)^2,
$$
with
$$
L = \sum_{k=1}^{K} L_k.
$$
The paper does not numerically specify \(m^+\), \(m^-\), or \(\lambda\), and notes that defaults from the earlier capsule literature can be used if needed. Cross-entropy is not used; optimization targets capsule norms, and evaluation uses top-1 prediction by largest class capsule \(L_2\) norm.

The reported implementation uses PyTorch 2.5.1, Python 3.12, and CUDA 12.4 on a single NVIDIA RTX 4090D GPU. Training typically runs for \(300\) epochs, with an appendix note that MNIST was trained for \(100\) epochs in one setting. The optimizer is AdamW with base learning rate \(5\times 10^{-4}\), weight decay \(1\times 10^{-4}\), batch size \(128\), and a learning-rate schedule consisting of a \(5\)-epoch linear warmup from \(10\%\) of the base value followed by cosine annealing to \(1\times 10^{-6}\).

Preprocessing and augmentation are dataset-specific. MNIST and FashionMNIST are resized to \(32\times 32\). Augmentations include random crop with padding, random horizontal flip except for SVHN, and random rotation of \(\pm 15^\circ\) for MNIST and SVHN. Normalization is \(\text{mean}=0.5\), \(\text{std}=0.5\) for MNIST, FashionMNIST, and SVHN; CIFAR-10 uses channel-wise mean \((0.4914, 0.4822, 0.4465)\) and std \((0.2023, 0.1994, 0.2010)\). Initialization is Xavier normal for the Patchify \(1\times 1\) convolution and Kaiming for other convolutions and projection layers.

The paper does not report inference speed, latency, or FLOPs. This omission matters because CAR is presented as more efficient than iterative routing, but deployment under strict real-time constraints would still require explicit profiling.

## 6. Empirical performance, interpretation, and limitations

The reported benchmarks are MNIST, FashionMNIST, SVHN, and CIFAR-10, with classification accuracy computed from the largest class capsule \(L_2\) norm [2508.16922]. MSPCaps-T, with \(\approx 344.3\text{K}\) parameters, achieves \(99.69\%\) on MNIST, \(95.79\%\) on SVHN, and \(88.71\%\) on CIFAR-10. MSPCaps-L, with \(\approx 10.9\text{M}\) parameters, achieves \(99.73\%\) on MNIST, \(95.05\%\) on FashionMNIST, and \(92.88\%\) on CIFAR-10.

| Model | Reported results |
|---|---|
| MSPCaps-T | \(99.69\%\) MNIST, \(95.79\%\) SVHN, \(88.71\%\) CIFAR-10, \(\approx 344.3\text{K}\) params |
| MSPCaps-L | \(99.73\%\) MNIST, \(95.05\%\) FashionMNIST, \(92.88\%\) CIFAR-10, \(\approx 10.9\text{M}\) params |

The paper reports that MSPCaps-T outperforms CapsNet, DA-CapsNet, AA-CapsNet, Efficient-Caps, and OrthCaps-S on CIFAR-10, including a \(+1.87\%\) gain over OrthCaps-S. MSPCaps-L is reported to outperform DeepCaps by \(+1.87\%\) on CIFAR-10 and RS-CapsNet by \(+3.07\%\), while also exceeding the listed results for AR-CapsNet and OrthCaps-D.

Ablation studies isolate the contribution of each design decision. For MSPCaps-T on CIFAR-10, using only the \(8\times 8\) scale yields \(87.48\%\), only \(32\times 32\) yields \(74.81\%\), only \(16\times 16\) yields \(81.90\%\), and all three scales together yield \(88.71\%\). The paper interprets this as evidence that \(8\times 8\) contributes global context while \(32\times 32\) refines fine detail. CAR is also compared against a Dynamic Routing baseline with four blocks and multi-scale independent routing plus concatenation: MSPCaps-T with CAR attains \(88.71\%\) using \(344.3\text{K}\) parameters versus \(87.46\%\) with \(483.5\text{K}\) for the DR baseline, and MSPCaps-L with CAR attains \(92.88\%\) with \(10.9\text{M}\) parameters versus \(92.56\%\) with \(14.6\text{M}\).

Weight sharing in CAR exhibits a scale-dependent tradeoff. In MSPCaps-T, sharing improves performance to \(88.71\%\) with \(344.3\text{K}\) parameters versus \(87.59\%\) with \(557.3\text{K}\) without sharing, suggesting that sharing is beneficial under tight parameter budgets. In MSPCaps-L, the non-sharing variant improves to \(92.88\%\) with \(10.9\text{M}\) parameters versus \(92.50\%\) with \(7.9\text{M}\), indicating that larger models benefit from additional projection expressivity.

Patch size is similarly nontrivial. For MSPCaps-T, \(p=4\) gives the best reported accuracy, \(88.71\%\), and the lowest parameter count among the tested settings, while \(p=2\) increases parameters to \(899.3\text{K}\) and mildly reduces accuracy to \(88.01\%\), and \(p=3\) reduces accuracy to \(86.34\%\) due to uneven borders. For MSPCaps-L, \(p=3\) achieves \(93.30\%\) with \(13.7\text{M}\) parameters, \(p=2\) yields \(93.16\%\) with \(39.4\text{M}\) and overfitting risk, and \(p=4\) yields \(92.88\%\) with \(10.9\text{M}\); the global choice remains \(p=4\) for consistency and efficiency.

The robustness study applies FGSM and BIM with \(10\) steps on CIFAR-10. MSPCaps-L is reported to exhibit strong stability across increasing \(\epsilon\), whereas MSPCaps-T degrades for \(\epsilon > 0.1\) and eventually falls below original CapsNet. The paper attributes the stronger behavior of the large model to multi-scale agreement and progressive fusion, although this remains an empirical interpretation rather than a formal robustness guarantee.

Several limitations are explicitly acknowledged. Patchification with a patch size that does not evenly tile a feature map can lose border information and reduce accuracy. The Tiny variant is less robust under stronger adversarial perturbation. FLOPs and latency are not reported. Future work proposed in the paper includes adaptive or learned patch sizes per scale, deformable patch boundaries, alternative agreement metrics such as cosine similarity or learned bilinear forms, top-\(k\) rather than top-1 within-group selection, integration of explicit equivariance modules, and calibration analyses.

Taken together, MSPCaps is best understood as a capsule architecture that recasts multi-scale fusion as selective agreement-based routing. Its empirical contribution lies not only in the reported accuracies and parameter tradeoffs, but also in the architectural claim that spatially aligned patchified capsules permit fine-to-coarse part-whole reasoning without iterative routing.

Source: https://www.emergentmind.com/topics/multi-scale-patchify-capsule-network-mspcaps