Multi-Scale Center Proposal Network

• The paper introduces a fully convolutional multi-scale center proposal network that generates class-specific object candidates using only image-level labels.
• It converts AlexNet into a FCN and employs LSE pooling and cascade refinement with VGG-16 to improve classification and localization performance on VOC and COCO.
• The approach leverages multi-scale processing, proposal sampling, and hard negative mining to achieve robust, efficient weakly supervised object detection.

A multi-scale center proposal network is a fully-convolutional architecture designed to generate class-specific object candidate regions without bounding box annotations. The approach enables efficient and accurate object classification and point-based localization, even in the presence of objects at different locations and scales. Each network stream generates dense spatial maps of objectness scores, corresponding to likely object centers at multiple image scales. This structure is central to the ProNet architecture, which cascades proposal generation with more powerful classifier networks for verification and refinement, achieving performance gains on PASCAL VOC and MS COCO benchmarks (Sun et al., 2015).

1. Network Structure and Multi-Scale Design

The network is based on converting AlexNet into a fully-convolutional network (FCN) by adapting all fully-connected layers ($fc6$, $fc7$, $fc8$) into $1 \times 1$ or “global” convolutional layers. The layer-wise architecture is:

Layer	Kernel / Stride / Padding	Output Channels
conv1 + maxpool1	$11\times 11$, $4$, $0$	$96$ + pool ($3\times 3$, $2$)
conv2 + maxpool2	$fc7$0, $fc7$1, $fc7$2	$fc7$3 + pool ($fc7$4, $fc7$5)
conv3	$fc7$6, $fc7$7, $fc7$8	$fc7$9
conv4	$fc8$0, $fc8$1, $fc8$2	$fc8$3
conv5 + maxpool5	$fc8$4, $fc8$5, $fc8$6	$fc8$7 + pool ($fc8$8, $fc8$9)
conv6 ($1 \times 1$0)	$1 \times 1$1, $1 \times 1$2, $1 \times 1$3	$1 \times 1$4
conv7 ($1 \times 1$5)	$1 \times 1$6, $1 \times 1$7, $1 \times 1$8	$1 \times 1$9
conv8 ($11\times 11$0)	$11\times 11$1, $11\times 11$2, $11\times 11$3	$11\times 11$4

Sigmoid non-linearity is applied after conv8, giving a $11\times 11$5 confidence tensor $11\times 11$6.

For multi-scale processing, $11\times 11$7 streams (typically $11\times 11$8) are instantiated, each operating on the input image resized such that the longer side is $11\times 11$9 pixels (shorter side scaled proportionally, mirror-filled as needed). All streams share weights. Each activation at location $4$0 and scale $4$1 corresponds to a center proposal in the original image, with receptive field size $4$2 pixels and stride $4$3 pixels.

2. Proposal Generation and Scoring

Each stream produces a spatial map, with each $4$4 activation interpreted as an object center proposal for a specific class $4$5. The proposal box is mapped to the original coordinates:

• Center: $4$6
• Box size: $4$7 of stream $4$8 (in scaled image), equivalent to an $4$9 box in the original.

The objectness score for class $0$0 at $0$1 is:

$0$2

where $0$3 is the feature vector from the last conv layer before conv8.

All $0$4 activations from all $0$5 scales are enumerated, resulting in a dense sampling of candidate proposals for each class.

3. Training with Image-Level Supervision

The network is trained exclusively with image-level class presence labels, without access to ground-truth bounding boxes. Aggregation of proposal confidences into an image-level score uses Log-Sum-Exp (LSE) pooling:

$0$6

with $0$7 controlling pooling sharpness ($0$8, $0$9).

The binary cross-entropy loss is computed as:

$96$0

Full back-propagation is applied, including through the LSE pooling, enabling end-to-end optimization. Training employs mini-batch SGD (learning rate $96$1, momentum $96$2, weight decay $96$3), with scales set to $96$4 and $96$5.

4. Cascaded and Tree-Structured Refinement

During inference, top-$96$6 ($96$7) proposals per class per stream are retained, subject to a threshold $96$8, resulting in approximately $96$9 candidate boxes per image. These are passed to a verification network:

• VGG-16 pretrained on ImageNet, with conv1–conv5 frozen and fc6–fc8 fine-tuned for the target dataset. The fc8 is replaced with a $3\times 3$0-way classifier.
• Each proposal $3\times 3$1 is cropped from the original image, resized to $3\times 3$2, and scored.
• For each class $3\times 3$3, if proposals are available the maximum refined score is used; otherwise the global proposal score is retained:

$3\times 3$4

A tree-structured cascade can be constructed by training multiple verification networks specialized on super-categories. Each candidate proposal traverses a unique root-to-leaf path, reducing overall computation.

5. Proposal Sampling and Hard Negative Mining

Cascade stage training requires robust sampling due to the absence of box-level supervision. The protocol is:

• For image $3\times 3$5 in a batch, retrieve all proposals with $3\times 3$6.
• If class $3\times 3$7 is present, sample one such proposal as a noisy positive; otherwise as a negative.
• Only the sampled class is active for loss calculation per instance, ignoring all others.
• This stochastic process (see Alg. 1 in (Sun et al., 2015)) increases diversity in training positives and negatives, mitigating overfitting to the highest-confidence boxes.

A plausible implication is that, by randomizing positives and negatives, the process yields a robust cascade even with limited and ambiguous supervision.

6. Empirical Performance and Analysis

Key empirical findings on VOC'12 and COCO show the influence of technical design choices:

Pooling Strategy	Classification mAP	Localization mAP
Max pooling	$3\times 3$8	$3\times 3$9
Average pooling	$2$0	$2$1
LSE pooling	$2$2	$2$3

Cascade refinement provides additive gains:

• Proposal only: $2$4 (cls), $2$5 (loc)
• • one VGG stage: $2$6 (cls), $2$7 (loc)
• • second cascade with added COCO data: $2$8 (cls), $2$9 (loc)

Number of top-$fc7$00 retained proposals per class/stream affects mAP; $fc7$01 ($fc7$02 boxes/image) yields $fc7$03 (cls) and $fc7$04 (loc).

Tree-structured cascades on COCO val outperformed chain-cascades ($fc7$05/$fc7$06 vs $fc7$07/$fc7$08).

7. Significance and Implementation

Every architectural, training, and inference hyper-parameter is fully specified for re-implementation: scales, layer configurations, proposal selection thresholds, cascade training, and pooling strategies. The multi-scale center proposal network framework enables accurate object classification and point-based localization with only image-level labels, without bounding box annotations. Its integration into a cascaded or tree-structured pipeline with strong classifier verifiers (e.g., VGG-16) establishes a template for robust, efficient weakly supervised object recognition (Sun et al., 2015).