Hierarchical Auto-Zoom Net (HAZN)

• The paper introduces a hierarchical approach that uses three FCNs to progressively refine object and part parsing through adaptive zooming.
• It integrates Scale Estimation Networks that predict bounding boxes and confidence scores to generate high-quality ROIs for detailed segmentation.
• Empirical results demonstrate significant improvements, with notable gains in mean IOU and instance AP metrics on benchmarks such as PASCAL-Person-Part and Horse-Cow.

Hierarchical Auto-Zoom Net (HAZN) is a model for object part parsing that employs a multi-stage, adaptive zooming approach to address challenging variability in scale and location of objects and their semantic parts in natural images. Built upon fully convolutional networks (FCNs), HAZN decomposes the parsing task hierarchically, enabling precise location estimation and boundary delineation, particularly for small instances and small parts. The method has demonstrated substantial improvements on standard benchmarks such as the PASCAL-Person-Part and Horse-Cow datasets, exhibiting state-of-the-art accuracy for articulated object and animal segmentation (Xia et al., 2015).

1. Architecture and Design Principles

HAZN comprises a hierarchical sequence of three FCNs, each adapted from the DeepLab-LargeFOV model and applied at progressively finer granularities: image-level, object-level, and part-level. At each stage $\ell$, the FCN is bifurcated into two specialized heads:

• a parsing head produces a pixel-level part score map ($P_{\ell}(l_j|\cdots)$),
• a Scale Estimation Network (SEN) head regresses bounding box parameters and their associated confidence scores.

The system pipelines data as follows:

1. The image-level FCN processes the entire image $I$, outputting initial part scores $P_1(l_j|I,j)$ per pixel and, via SEN$_1$, object bounding box hypotheses $b_j = (dx_j, dy_j, w_j, h_j)$ with confidence $c_j$ for each pixel $j$.
2. High-confidence object boxes are selected using a confidence threshold and non-maximum suppression (NMS), each then cropped and resized (zoomed) to a fixed standard size to form object-level ROIs.
3. Each zoomed object ROI is input to the object-level FCN, which predicts refined part scores within the ROI and, via SEN$_2$, part bounding boxes.
4. High-confidence part boxes again undergo NMS, are zoomed to a standard size, and then parsed by the part-level FCN to produce fine-grained, final per-pixel part segmentation scores.

The following table summarizes the hierarchical flow:

Level	Input	SEN Prediction	Zoom ROI	Parsing Output
Image-level	Image ($I$)	Object boxes ($b_k$)	$N(k)$	Initial part scores
Object-level	$N(k)$	Part boxes ($p_m$)	$M(m)$	Refined part scores
Part-level	$M(m)$	—	—	Final part scores

2. Scale and Location Prediction

At each relevant stage, the SEN is tasked with predicting bounding box parameters for either objects (SEN$_1$) or parts (SEN$_2$). Each SEN regresses, for a pixel $j$, a tuple $$\hat{t}_j = (\hat{x}_j, \hat{y}_j, \hat{w}_j, \hat{h}_j)$$ where $(\hat{x}_j, \hat{y}_j)$ specify the box center offsets and $(\hat{w}_j, \hat{h}_j)$ the box dimensions.

The SEN employs two loss terms per level:

• Regression loss:

$$L_{\mathrm{reg}} = \frac{1}{|\mathcal{S}|}\sum_{j\in\mathcal{S}} \|t_j^* - \hat{t}_j\|_2^2$$

where $\mathcal{S}$ is a seed region for supervision (e.g., the central $7 \times 7$ region of a ground-truth instance).

• Classification loss:

$$L_{\mathrm{cls}} = - \beta \sum_{j:l_j^*=1} \log c_j - (1-\beta) \sum_{j:l_j^*=0} \log(1 - c_j)$$

with $c_j$ as the sigmoid confidence and $\beta$ for class balance.

Total SEN loss: $$L_{\mathrm{SEN}} = L_{\mathrm{cls}} + \lambda L_{\mathrm{reg}}$$.

The zooming operation is formalized via:

$$N(k) = \mathrm{CropAndResize}(I, \hat{t}_k) = I\left[\hat{y}_k-\tfrac{\hat{h}_k}{2}:\hat{y}_k+\tfrac{\hat{h}_k}{2},\, \hat{x}_k-\tfrac{\hat{w}_k}{2}:\hat{x}_k+\tfrac{\hat{w}_k}{2}\right] \uparrow s_k$$

where $$s_k = \text{s}_{\mathrm{target}} / \max(\hat{w}_k, \hat{h}_k)$$, clipped to $[0.4,2.5]$.

3. Part Score Estimation and Loss Functions

The parsing head at each stage is a compact FCN subnetwork comprising convolutional layers and a softmax output, yielding pixel-wise part class distributions. Supervision is via standard cross-entropy segmentation loss:

$$L_{\mathrm{seg}} = -\sum_{j}\sum_{c} y_{j,c} \log p_{j,c}$$

where $y_{j,c} \in \{0,1\}$ indicates ground-truth class $c$ for pixel $j$, and $p_{j,c}$ is the predicted softmax output.

For all $\ell=1,2$, the full AZN loss is $$L_{\mathrm{AZN}_\ell} = L_{\mathrm{seg}} + L_{\mathrm{SEN}}$$, while the part-level FCN ($\ell = 3$) is trained only on segmentation loss.

4. Training and Inference Workflow

The model is initialized from ImageNet-pretrained VGG-16 backbones following DeepLab-LargeFOV protocols. Each FCN is trained in a multi-task setting relevant to its hierarchy level. The end-to-end inference pipeline proceeds as:

1. Image-level FCN infers pixelwise part scores and preliminary object box candidates.
2. Object ROI extraction via thresholding box confidences, NMS, followed by cropping and resizing.
3. Object-level FCN parses object ROIs and returns refined part scores with further part box localization.
4. Part ROI extraction (NMS/filtering), cropping, and zooming of part regions.
5. Part-level FCN parses the zoomed parts to yield the final part segmentation.

Overlapping ROI predictions are consolidated by a score-weighted sum:

$$P_{\ell+1}(l_j) = \sum_{N(k)\ni j} P_\ell(l_j|N(k))\, \frac{c_k}{\sum_{N(q)\ni j} c_q}$$

5. Empirical Results

Extensive evaluation was performed on both PASCAL-Person-Part (6 part classes + background) and Horse-Cow (4 part classes + bg) datasets. Notable quantitative performance includes:

• PASCAL-Person-Part (mean IOU):
  • Baseline DeepLab-LargeFOV: 51.78%
  • Multi-Scale Attention: 56.39%
  • HAZN: 57.54% (improvement of +5.8% over baseline, +1.15% over attention-based)
• Instance AP$^r_{\mathrm{part}}$ (IoU$\geq$0.5):
  • DeepLab: 31.3%
  • Multi-Scale Attention: 37.5%
  • HAZN: 43.7%
• Small-instance parsing (mIOU by size):
  • XS ($\sqrt{\text{area}}<80$): Baseline 32.5% $\to$ HAZN 47.1% (+14.6%)
  • S (80–140): 44.5% $\to$ 55.3% (+10.8%)
• Horse-Cow dataset:
  • Horse: JPO [Wang et al.’15] 67.0% $\to$ HAZN 72.4% (+5.4%)
  • Cow: 57.2% $\to$ 67.2% (+10.0%)

Qualitatively, HAZN achieves more accurate recovery of small object parts (e.g., limbs, tails), enhanced boundary adherence, and reduced background confusion (Xia et al., 2015).

6. Advantages, Limitations, and Prospective Directions

Advantages include:

• Adaptive zooming at region level minimizes computation compared with global multi-scale processing.
• Hierarchical refinement strategy results in sharper boundaries and improved recovery of fine parts.
• Unified detection and part parsing within a single end-to-end network architecture.

Limitations arise under severe occlusion or highly atypical object poses. The approach also incurs additional computational overhead due to successive ROI proposal and multi-stage FCN inference (approximately 1.3 seconds per image).

Future directions identified include:

• Extending the auto-zoom hierarchy to finer semantic parts (such as hands or eyes).
• Adapting the AZN approach for structure-related tasks such as pose estimation and fine-grained recognition.
• Development of more efficient ROI sampling strategies, e.g., reducing box proposals or reusing intermediate features (Xia et al., 2015).