Cross-Agreement Routing in Capsule Networks

• Cross-Agreement Routing (CAR) is a non-iterative mechanism that fuses multi-scale capsules by selecting the most coherent fine-to-coarse prediction pairs.
• It employs a max-selection step and softmax weighting over localized capsule predictions to outperform conventional addition, concatenation, and iterative routing methods.
• CAR improves parameter efficiency and robustness against adversarial attacks while achieving superior accuracy on benchmarks like MNIST, CIFAR-10, and SVHN.

Searching arXiv for the cited CAR paper and related routing-by-agreement literature. Cross-Agreement Routing (CAR) is a non-iterative routing mechanism introduced in the Multi-Scale Patchify Capsule Network (MSPCaps) for visual recognition. It is designed to fuse multi-scale capsules adaptively by identifying cross-scale prediction pairs with maximum agreement, so that only coherent predictions contribute to class capsule formation. In MSPCaps, CAR operates after a Multi-Scale ResNet Backbone (MSRB) and a Patchify Capsule Layer (PatchifyCaps), and replaces simple multi-scale fusion strategies such as addition and concatenation with an agreement-based selection rule over fine-scale and coarse-scale capsule votes (Hu et al., 23 Aug 2025).

1. Conceptual lineage and problem setting

CAR belongs to the broader family of routing-by-agreement methods in capsule-style architectures. In routing-by-agreement, lower-level entities act as “parts” and higher-level entities act as “wholes”; the central operation is to determine how strongly each part should contribute to each whole on the basis of agreement between lower-level predictions and emerging higher-level representations. In multi-head attention, an analogous formulation was used to treat the partial representations learned from attention heads as parts and output capsules as wholes, with routing coefficients updated iteratively according to agreement scores rather than fixed by a single linear transformation (Li et al., 2019).

The motivation for CAR is specifically multi-scale visual representation learning. Existing CapsNet and variants often rely on a single high-level feature map, while conventional multi-scale fusion strategies such as addition and concatenation struggle to reconcile multi-scale feature discrepancies. Independent routing blocks also provide no explicit mechanism to identify which features across scales best align as parts and wholes. CAR addresses this by using cross-scale agreement to decide which prediction pairs should influence the final voting stage (Hu et al., 23 Aug 2025).

A related theoretical line of work argues that routing is necessary for learning compositional, parse-tree-like structure in capsule networks. That work formalizes routing coefficients as the mechanism assigning non-uniform importance to alternative part-whole relationships and shows that the entropy of routing coefficients controls the ability to represent compositionality. This suggests that CAR should be understood not merely as a fusion operator, but as a structured part-whole selection mechanism specialized to multi-scale capsules (Venkatraman et al., 2020).

2. Architectural role inside MSPCaps

MSPCaps consists of three key components: a Multi-Scale ResNet Backbone (MSRB), a Patchify Capsule Layer (PatchifyCaps), and CAR blocks. The MSRB extracts diverse multi-scale feature representations from input images, preserving both fine-grained details and global contextual information. PatchifyCaps then partitions these multi-scale features into primary capsules using a uniform patch size, equipping the model with the ability to learn from diverse receptive fields. CAR operates on these patchified multi-scale capsules (Hu et al., 23 Aug 2025).

The routing scenario considered by CAR uses two capsule sets at layer $l$: a fine-scale set

$$u_1^l \in \mathbb{R}^{n_{\text{in}^{(1)}} \times d_{\text{in}^{(1)}}}$$

and a coarse-scale set

$$u_2^l \in \mathbb{R}^{n_{\text{in}^{(2)}} \times d_{\text{in}^{(2)}}}.$$

Here, $n_{\text{in}^{(2)}}$ is typically much smaller, as lower-resolution capsules cover more area. The design assumes a spatial correspondence between scales: each coarse capsule $u_{2,k}^l$ corresponds to

$$s = \frac{n_{\text{in}^{(1)}}}{n_{\text{in}^{(2)}}}$$

fine-scale capsules in its associated region (Hu et al., 23 Aug 2025).

This spatially grouped formulation is central. CAR does not compare all capsules across all scales indiscriminately; it restricts competition to fine-scale patches in the spatial region corresponding to each coarse-scale capsule. That locality is presented as reflecting part/whole structure and distinguishes CAR from blind concatenation of multi-scale capsules (Hu et al., 23 Aug 2025).

3. Mathematical formulation of cross-agreement routing

The first stage is linear transformation, or voting, into a shared output space. For each desired higher-layer output capsule $j = 1, \dots, n_{\text{out}}$, the fine-scale and coarse-scale capsules produce prediction vectors

$$\hat{u}_{j|i}^{(1)} = W_{j,i}^{(1)} u_{1,i}^l, \qquad \hat{u}_{j|k}^{(2)} = W_{j,k}^{(2)} u_{2,k}^l.$$

A parameter-efficient weight-sharing variant, used in MSPCaps-T, uses only $W^{(2)}$ for both scales:

$\hat{u}_{j|m,k}^{(1)} = W_{j,k}^{(2)} u_{1,m,k}^l.$

This associates the $$u_1^l \in \mathbb{R}^{n_{\text{in}^{(1)}} \times d_{\text{in}^{(1)}}}$$0-th fine-scale capsule in the spatial neighborhood of the $$u_1^l \in \mathbb{R}^{n_{\text{in}^{(1)}} \times d_{\text{in}^{(1)}}}$$1-th coarse-scale capsule with the same transformation family used for the coarse-scale prediction (Hu et al., 23 Aug 2025).

The defining CAR operation is cross-scale agreement computation. For each spatial group $$u_1^l \in \mathbb{R}^{n_{\text{in}^{(1)}} \times d_{\text{in}^{(1)}}}$$2 and output capsule $$u_1^l \in \mathbb{R}^{n_{\text{in}^{(1)}} \times d_{\text{in}^{(1)}}}$$3, the coupling coefficient is computed from the maximum similarity between the grouped fine-scale predictions and the corresponding coarse-scale prediction:

$$u_1^l \in \mathbb{R}^{n_{\text{in}^{(1)}} \times d_{\text{in}^{(1)}}}$$4

The similarity is typically a normalized dot product, and the softmax ensures that coupling coefficients over $$u_1^l \in \mathbb{R}^{n_{\text{in}^{(1)}} \times d_{\text{in}^{(1)}}}$$5 sum to $$u_1^l \in \mathbb{R}^{n_{\text{in}^{(1)}} \times d_{\text{in}^{(1)}}}$$6 for each $$u_1^l \in \mathbb{R}^{n_{\text{in}^{(1)}} \times d_{\text{in}^{(1)}}}$$7. The max operator enforces a “best aligned” selection rule: only the most coherent local part in each region is allowed to determine the cross-scale agreement score (Hu et al., 23 Aug 2025).

The output capsule computation then uses only the coarse-scale predictions, weighted by these cross-scale agreement coefficients:

$$u_1^l \in \mathbb{R}^{n_{\text{in}^{(1)}} \times d_{\text{in}^{(1)}}}$$8

The use of the standard capsule squash nonlinearity retains the capsule interpretation of vector length and direction while making the routing stage itself non-iterative (Hu et al., 23 Aug 2025).

The procedure is applied hierarchically. A representative pattern is to fuse fine-level $$u_1^l \in \mathbb{R}^{n_{\text{in}^{(1)}} \times d_{\text{in}^{(1)}}}$$9 and mid-level $$u_2^l \in \mathbb{R}^{n_{\text{in}^{(2)}} \times d_{\text{in}^{(2)}}}.$$0 capsules via CAR to obtain $$u_2^l \in \mathbb{R}^{n_{\text{in}^{(2)}} \times d_{\text{in}^{(2)}}}.$$1, then fuse $$u_2^l \in \mathbb{R}^{n_{\text{in}^{(2)}} \times d_{\text{in}^{(2)}}}.$$2 with the coarsest-level $$u_2^l \in \mathbb{R}^{n_{\text{in}^{(2)}} \times d_{\text{in}^{(2)}}}.$$3 in a second CAR block. This fine-to-coarse progression means that at each stage only cross-scale capsules in local agreement reinforce each other (Hu et al., 23 Aug 2025).

4. Relation to earlier routing and aggregation mechanisms

CAR is best understood by contrast with two earlier families of aggregation mechanisms: conventional feature fusion and iterative routing-by-agreement. Conventional multi-scale fusion strategies such as addition and concatenation merely aggregate multi-scale features without analyzing consistency or structure. In the multi-head attention setting, the corresponding baseline is to concatenate head outputs and apply a linear transformation,

$$u_2^l \in \mathbb{R}^{n_{\text{in}^{(2)}} \times d_{\text{in}^{(2)}}}.$$4

which assigns fixed, static importance to each head and combines them in a single-step, uniform way (Li et al., 2019).

Iterative routing-by-agreement methods, by contrast, update assignment proportions across several rounds. In the simple dynamic-routing variant, logits are initialized to zero, assignment weights are obtained by a softmax, output capsules are formed as assignment-weighted sums, and logits are incremented by the dot-product agreement between each vote and the current output capsule. EM routing casts the process as fitting a mixture of Gaussians and alternates between an M-step and an E-step (Li et al., 2019).

CAR differs from both. It is not blind concatenation, because it explicitly models and filters by agreement. It is also not an iterative dynamic-routing or EM-routing procedure, because its agreement mechanism is a max-selection across corresponding fine-scale and coarse-scale predictions followed by a single softmaxed weighting over coarse-scale votes. A plausible implication is that CAR trades the iterative refinement of generic routing algorithms for a more structured, localized, and computationally constrained cross-scale matching rule (Hu et al., 23 Aug 2025).

The distinction is especially important because recent work on compositionality argues that routing coefficients are meaningful precisely when they encode non-uniform part-whole assignments. CAR implements such selectivity through cross-scale matching rather than through repeated routing iterations. This suggests continuity in objective—selective part-whole assignment—but not identity of mechanism (Venkatraman et al., 2020).

5. Empirical performance, efficiency, and robustness

The empirical evidence reported for MSPCaps attributes substantial gains to CAR. The architecture is described as achieving configurations ranging from a highly efficient Tiny model with 344.3K parameters to a Large model with 10.9M parameters. For MSPCaps-T, the reported results are MNIST: 99.69% (best), SVHN: 95.79% (second only to OrthCaps-S), and CIFAR-10: 88.71% (best). For MSPCaps-L, the reported results are MNIST: 99.73% (best), FashionMNIST: 95.05% (best), and CIFAR-10: 92.88% (best, outperforming DeepCaps and RS-CapsNet) (Hu et al., 23 Aug 2025).

Ablation results isolate the contribution of CAR. Substituting CAR with dynamic routing reduces accuracy and increases parameter count; one explicit example is MSPCaps-T on CIFAR-10, where CAR obtains 88.71% and dynamic routing obtains 87.46%. The paper therefore characterizes CAR as more parameter efficient and more accurate than conventional deep multi-block routing in that setting (Hu et al., 23 Aug 2025).

The multi-scale design is also tested directly. Synthesizing features across scales using CAR gives a large boost, especially on complex datasets; on CIFAR-10, the full multi-scale CAR model reaches 88.71%, whereas single-scale variants can be as low as 74.81%. This result is consistent with the underlying claim that multi-scale capsules provide complementary information that should be fused through agreement rather than through addition or concatenation (Hu et al., 23 Aug 2025).

Robustness is another reported outcome. MSPCaps-L, using CAR, is described as significantly more robust to adversarial attacks (FGSM, BIM) than classical CapsNet and MSPCaps-T, maintaining higher accuracy at greater attack strengths. The available summary does not provide the exact attack-strength curves or numerical values, but it attributes the robustness comparison directly to the CAR-enabled MSPCaps-L configuration (Hu et al., 23 Aug 2025).

6. Interpretation, scope, and common points of confusion

One common confusion is to treat CAR as a generic synonym for routing-by-agreement. The term “Cross-Agreement Routing” refers specifically to the MSPCaps mechanism that adaptively routes multi-scale capsules by identifying cross-scale prediction pairs with maximum agreement. Earlier routing-by-agreement methods in capsule networks and multi-head attention use iterative updates of routing or assignment coefficients, whereas CAR uses cross-scale prediction matching with a max-selection step and softmaxed coupling over coarse-scale groups (Hu et al., 23 Aug 2025).

A second confusion is to equate CAR with simple stacking of multiple self-routing blocks. The MSPCaps description explicitly contrasts CAR with the simple concatenation of multiple self-routing blocks and states that CAR ensures that only the most coherent capsules contribute to the final voting. The key technical difference is the explicit search for cross-scale prediction pairs with maximum agreement in corresponding spatial neighborhoods (Hu et al., 23 Aug 2025).

A broader interpretive issue concerns compositionality. Work on routing entropy shows that capsule networks detect changes in compositionality more effectively when routing weights are low-entropy and sample-specific, and that without routing capsule networks perform similarly to convolutional neural networks in this respect (Venkatraman et al., 2020). CAR is not evaluated in those terms in the provided material. Nevertheless, this suggests that CAR may be viewed as one concrete way of imposing selective part-whole assignments in a multi-scale setting, with the max-agreement rule functioning as a hard preference for coherent cross-scale relations rather than as a diffuse fusion mechanism.

Within the available evidence, CAR is therefore most precisely characterized as a non-iterative, localized, cross-scale routing scheme for capsule fusion. Its significance lies in replacing blind multi-scale aggregation with an agreement-filtered voting process that is reported to improve classification accuracy, parameter efficiency, and robustness in MSPCaps (Hu et al., 23 Aug 2025).