Multi-scale Part-supervised Recognition Model

• The paper introduces MPM, which integrates multi-scale part supervision into standard ImageNet classifiers to enhance robustness and performance.
• It leverages the PIN++ dataset and pseudo-labeling to overcome annotation scarcity and instill fine-grained part awareness in deep models.
• Empirical results demonstrate improved clean accuracy, adversarial resistance, and out-of-distribution performance compared to baseline models.

Searching arXiv for the specified MPM/PIN++ papers to ground the article in current preprints. The Multi-scale Part-supervised recognition Model (MPM) is a training-time extension of a standard ImageNet classifier that uses part segmentation supervision at multiple intermediate feature stages to induce a part-aware representation for robust recognition. It was introduced together with PartImageNet++ (PIN++), a dataset providing high-quality part annotations for all 1,000 ImageNet-1K categories, so that part-based recognition could be studied directly in the standard large-scale setting rather than only on small or nonstandard subsets (Li et al., 2024). In the accompanying expanded presentation of PIN++, the dataset is described as containing 100 annotated images per category, for 100K annotated images, 3,308 part categories, and 406,364 part masks, and MPM is presented as the principal mechanism for converting that annotation scale into robustness gains without adding inference-time cost (Li et al., 4 Jan 2026).

1. Motivation and problem setting

MPM is motivated by the hypothesis that deep recognition systems are brittle in part because they lack a part-based inductive bias. The underlying contrast is between human recognition, described in the paper through the recognition-by-components perspective and broader human vision evidence, and standard deep classifiers, which often rely on shortcut cues such as texture or other non-human features. In the paper’s framing, this mismatch is one reason for vulnerability to adversarial perturbations, common corruptions, and out-of-distribution shifts (Li et al., 2024).

Earlier part-based robust recognition methods had already suggested that part information could improve robustness, but they were constrained by annotation scarcity. The prior systems highlighted in the paper, including ROCK and the part-based models of Sitawarin et al., were evaluated only on limited datasets such as PartImageNet or small subsets of it. This restricted both scale and comparability with standard ImageNet-trained baselines. PIN++ was created to remove that bottleneck by providing part segmentation annotations for the full ImageNet-1K taxonomy, making it possible to train a part-aware model directly in the canonical ImageNet regime (Li et al., 4 Jan 2026).

Within that setting, MPM addresses a specific engineering and methodological problem: how to inject part-level structure into a large-scale recognition model without paying an inference-time cost. The solution is neither a standalone segmentation model nor a permanent multi-branch test-time architecture. Instead, part supervision is attached only during training, so the deployed model remains the vanilla classifier.

2. Architectural formulation

MPM is defined over a conventional hierarchical classifier backbone,

$$\mathcal{F} := h_c \circ f_l \circ \cdots \circ f_1,$$

where $f_1,\dots,f_l$ are backbone blocks and $h_c$ is the classification head. The baseline backbone can be a model such as ResNet-50, ConvNeXt, or Swin, while the robustness experiments mainly use ResNet-50-GELU (Li et al., 4 Jan 2026).

The distinctive architectural element is a set of lightweight auxiliary bypass layers $g_i$ attached to intermediate features: $$\hat{\mathbf{M}}_i = g_i \circ f_i \circ \cdots \circ f_1(\mathbf{x}), \quad i \le l.$$ Here $\hat{\mathbf{M}}_i$ is a predicted part mask at stage $i$. These bypass layers are segmentation heads, but they are explicitly designed to improve recognition rather than to maximize segmentation quality. The final object prediction still comes from the standard classification backbone.

The model is called multi-scale because part supervision is applied to multiple intermediate feature maps rather than only to the deepest output. In the ResNet-50 configuration discussed in the paper, the last three blocks are supervised. For an input of $224 \times 224$, these correspond roughly to feature maps of $28 \times 28$, $14 \times 14$, and $f_1,\dots,f_l$0. Lower stages preserve finer spatial detail from the part annotations, while deeper stages encode more semantic abstraction. The reported rationale is that earlier part-based methods typically supervised only the final stage, forcing aggressive downsampling of part masks and discarding useful fine-grained structure (Li et al., 2024).

MPM also incorporates a lightweight top-down augmentation in the spirit of FPN-style designs. These top-down connections enrich lower-level features with higher-level context, but they remain auxiliary training structures rather than a full dense prediction subsystem. A central design point is that the auxiliary bypass layers are dropped at inference, so the test-time model is just the original classifier, with no extra inference cost.

3. Supervision pipeline and the role of PIN++

MPM is trained on samples of the form

$f_1,\dots,f_l$1

where $f_1,\dots,f_l$2 is the image, $f_1,\dots,f_l$3 is the object category label, and $f_1,\dots,f_l$4 is a composite segmentation mask with $f_1,\dots,f_l$5 channels. The extra channel is background, and the paper sets $f_1,\dots,f_l$6, the number of part categories in PIN++ (Li et al., 2024).

The pipeline does not rely on manual annotation for all of ImageNet. Instead, the paper specifies a two-stage scaling strategy. First, it manually annotates 100 images per class in ImageNet-1K. Second, it trains a Mask R-CNN with Swin-B backbone on those annotations. Third, that model generates pseudo part labels for the remaining unannotated ImageNet training images. Finally, the manually annotated masks and the pseudo labels are treated equally during MPM training (Li et al., 4 Jan 2026).

For an unlabeled image, the part segmentation model outputs

$f_1,\dots,f_l$7

where $f_1,\dots,f_l$8 is a predicted part mask and $f_1,\dots,f_l$9 is the predicted part-class distribution. To improve pseudo-label quality, the paper applies a Category Filter (CF) using the known image category $h_c$0. CF zeroes out probabilities for part categories that do not belong to the object category, producing $h_c$1, after which the pseudo part label is

$h_c$2

This filtering mechanism is reported to improve both pseudo-label quality and downstream robustness (Li et al., 2024).

The paper further states that pseudo labels are not merely supplementary. In the ablations, using only real part annotations is ineffective and may even hurt performance, whereas adding pseudo labels substantially improves robustness. This suggests that large-scale coverage over the full ImageNet training distribution is operationally important for MPM, not just the availability of a small set of high-quality manual labels.

4. Objective functions and optimization

MPM is trained with a joint recognition-and-segmentation objective,

$h_c$3

where $h_c$4 is the standard classification loss, $h_c$5 is the part segmentation supervision loss, and $h_c$6 controls the strength of part supervision (Li et al., 2024).

The segmentation component is computed as the average over the supervised scales: $h_c$7 where $h_c$8 is the downsampled part mask for the corresponding feature resolution. The appendix specifies that the segmentation loss uses Focal loss,

$h_c$9

described in the paper as a variant of cross-entropy used to accelerate part-segmentation convergence (Li et al., 4 Jan 2026).

When adversarial training is used, adversarial examples are generated by solving

$g_i$0

The adversary is therefore built against the classification objective, while parameter updates are performed using the joint loss $g_i$1. In the main robustness experiments, the training recipe uses standard adversarial training with $g_i$2, 2-step PGD inner maximization, and input size $g_i$3 (Li et al., 2024).

A further design observation reported in the paper is that $g_i$4 is not very sensitive as long as $g_i$5, while larger $g_i$6 tends to help clean accuracy. This indicates that the benefit is tied primarily to the presence of nonzero part supervision rather than to extremely delicate hyperparameter tuning.

5. Reported empirical behavior

The main ImageNet-1K robustness result compares MPM against a strong adversarially trained ResNet-50-GELU baseline. The paper reports gains in both clean accuracy and robustness across all tested threat models (Li et al., 4 Jan 2026).

Metric	Vanilla robust ResNet-50-GELU	MPM
clean	67.1	67.8
$g_i$7	38.1	39.1
$g_i$8	12.6	13.6
$g_i$9	5.0	6.2
$$\hat{\mathbf{M}}_i = g_i \circ f_i \circ \cdots \circ f_1(\mathbf{x}), \quad i \le l.$$0	21.6	24.3
average	19.3	20.8

The same section of the paper states that MPM outperforms several recent adversarially trained ResNet-50 baselines, including works by Salman et al. (transfer), EasyRobust, Liu et al., Bai et al., and Debenedetti et al. The strongest baseline in the reported table before MPM is the paper’s own improved vanilla adversarial training model, and MPM still improves over it (Li et al., 2024).

On common corruptions, the paper reports improved performance on nearly all corruption types. The average accuracy changes from 40.8% to 42.1% when part supervision is added without adversarial training, and from 34.9% to 36.4% when part supervision is added on top of adversarial training. On the OOD benchmarks ImageNet-A-Plus, ImageNet-Sketch, Stylized ImageNet, and DIN, the averaged result improves from 24.6% to 25.2% with adversarial training and from 23.3% to 23.6% without adversarial training (Li et al., 2024).

The paper also reports improved human alignment on distorted images. In the expanded presentation, this is summarized as lower accuracy difference, higher observed consistency, and higher error consistency. A plausible implication is that the combination of adversarial training and part supervision shifts the model toward decision patterns that are less dependent on fragile non-human cues (Li et al., 4 Jan 2026).

Finally, the representation learned by MPM transfers beyond classification. The paper states that using MPM to initialize a Faster R-CNN backbone improves downstream adversarially robust object detection (Li et al., 2024).

6. Ablations, distinctions from earlier models, and broader significance

A central distinction is that MPM is not a two-stage system. Earlier part-based approaches such as ROCK and the models of Sitawarin et al. are described as following a pipeline in which a segmenter predicts part masks and a classifier then predicts the object category from those masks. The paper lists several drawbacks of that pattern: extra inference parameters and compute, architectural complexity, reliance on low-resolution part labels, and in some prior methods the presence of non-differentiable components that complicate interpretation of robustness. MPM instead trains the standard classifier directly and uses part prediction only as an auxiliary regularizer on intermediate layers (Li et al., 2024).

The architectural ablation makes this contrast quantitative. A reported two-stage part-based model has 60.9M parameters at inference, with clean 67.3, $$\hat{\mathbf{M}}_i = g_i \circ f_i \circ \cdots \circ f_1(\mathbf{x}), \quad i \le l.$$1 38.7, and $$\hat{\mathbf{M}}_i = g_i \circ f_i \circ \cdots \circ f_1(\mathbf{x}), \quad i \le l.$$2 13.0. The corresponding MPM with multi-scale supervision has 25.6M inference parameters, with clean 67.8, $$\hat{\mathbf{M}}_i = g_i \circ f_i \circ \cdots \circ f_1(\mathbf{x}), \quad i \le l.$$3 39.1, and $$\hat{\mathbf{M}}_i = g_i \circ f_i \circ \cdots \circ f_1(\mathbf{x}), \quad i \le l.$$4 13.6. The paper therefore characterizes MPM as both stronger and substantially cheaper at inference (Li et al., 4 Jan 2026).

The same ablation isolates the effect of supervision depth through SV1, SV2, and SV3, where the model supervises only the final block, the last two blocks, or the last three blocks, respectively. The reported outcome is that supervising more scales helps, and removing the top-down connections reduces performance. This is consistent with the model’s stated rationale that intermediate representations benefit from simultaneous access to fine part boundaries and higher-level part-aware context.

Another notable ablation compares part supervision to object supervision. Replacing part masks with object masks weakens performance: object supervision yields clean 67.5, $$\hat{\mathbf{M}}_i = g_i \circ f_i \circ \cdots \circ f_1(\mathbf{x}), \quad i \le l.$$5 37.3, average 22.3, whereas part supervision yields clean 67.8, $$\hat{\mathbf{M}}_i = g_i \circ f_i \circ \cdots \circ f_1(\mathbf{x}), \quad i \le l.$$6 39.1, average 23.2 (Li et al., 4 Jan 2026). This directly addresses a common misconception that any spatial supervision should provide the same benefit. The reported evidence indicates that the useful signal lies in the fine-grained part structure, not merely in coarse foreground delineation.

In a broader PIN++ context, the same study shows that large-scale part annotation supports additional tasks such as part segmentation, object segmentation, and few-shot learning. Although those results are not properties of MPM alone, they situate the model within a larger research program in which part annotations act as a reusable supervision resource. Within that program, MPM is best understood as a training-time part-supervised regularizer for a standard recognition backbone: it uses PIN++ and pseudo labels to shape intermediate features, preserves the standard classifier at test time, and demonstrates reported gains in adversarial robustness, corruption robustness, OOD performance, human alignment, and downstream transfer (Li et al., 4 Jan 2026).