---
title: 'PolyR-CNN: End-to-End Polygon Detection'
url: https://www.emergentmind.com/topics/polyr-cnn
type: topic
---

# PolyR-CNN: End-to-End Polygon Detection

PolyR-CNN is an end-to-end, polygon-capable Region-based Convolutional Neural Network (R-CNN) explicitly designed to predict both bounding boxes and vectorized polygonal building outlines directly from remotely sensed imagery. This method avoids the conventional multi-stage and complex specialized architectures typical of prior work, using a unified pipeline that leverages only Region of Interest (RoI) features and a novel vertex proposal feature scheme to directly regress ordered building polygons alongside standard detection outputs. PolyR-CNN demonstrates state-of-the-art efficiency and accuracy trade-offs in large-scale benchmarks, supports extraction of buildings with holes, and is agnostic to the use of semantic segmentation priors [2407.14912].

## 1. End-to-End Architecture and Workflow

PolyR-CNN processes an input image through a standard backbone (e.g., ResNet-50, Swin-Base) with a Feature Pyramid Network (FPN), producing multi-scale features $P_2,\ldots,P_5$. The method initializes $N$ proposal boxes (centered at $(0.5, 0.5)$, size $(1, 1)$) and corresponding $N$ proposal polygons ($M$ uniformly distributed vertices).

For $L=6$ sequential layers, the following steps are performed:
- **RoIAlign**: Extracts RoI feature $F_{\mathrm{roi}} \in \mathbb{R}^{C \times h \times w}$ for each proposal box.
- **Vertex Proposal Feature**: Computes a compact representation $f_\mathrm{vtx} \in \mathbb{R}^C$ from the current proposal polygon via a feedforward network (FFN) and a self-attention block.
- **Feature Guidance**: Dynamically modulates two $1\times1$ convolutional layers on $F_{\mathrm{roi}}$ using parameters from $f_\mathrm{vtx}$, followed by a self-attention block for interaction among all $N$ RoI features.
- **Prediction Heads (MLPs)**: Four parallel heads output the class score $s \in [0, 1]$, box refinement $\Delta b = (\Delta x, \Delta y, \Delta w, \Delta h)$, polygon coordinates $\{(x_i, y_i)\}_{i=1}^M$, and vertex-validity scores $\{c_i\}_{i=1}^M$.

After the final layer, proposals with $s < 0.05$ are discarded, and vertices with $c_i < 0.5$ are pruned to produce the output polygons.

**Process Flow Diagram (per layer):**
```
Proposal polygon {(x_i, y_i)} 
   → FFN + SA → f_vtx 
      → DynamicConv(F_roi; f_vtx) → SA 
          → {classification, box, polygon, vtx_cls}
```
This enables direct, recurrently refined regression of complete polygons from features localized to the proposal region.

## 2. Vertex Proposal Feature Construction

At each refinement layer $t$, the current polygon $V^{(t)} = \{v_i^{(t)} = (x_i, y_i)\}_{i=1}^M$ is flattened and mapped to a high-dimensional vector:
\[
z = \mathrm{GELU}(W_1 \, \mathrm{vec}(V^{(t)}) + b_1) \quad\in\mathbb{R}^{d_1}
\]
\[
f_{\rm vtx}^{(t)} = \mathrm{SA}(W_2 z + b_2) \quad\in\mathbb{R}^{C}
\]
with $d_1 = C = 256$. This vertex feature $f_{\rm vtx}$ generates dynamic kernel weights for $1\times1$ convolutions, adaptively conditioning the RoI feature on polygon geometry. The modulated feature is:
\[
\{K^{(t)}, b_K^{(t)}\} = \mathrm{Linear}(f_{\mathrm{vtx}}^{(t)}), \quad
F_{\rm guided}^{(t)} = \mathrm{SA}(\mathrm{Conv}_{1\times 1}(F_{\rm roi}; K^{(t)}, b_K^{(t)}))
\]
This guidance sharpens attention around the predicted polygon structure, empirically focusing features at corners and relevant vertices.

## 3. Training Losses and Hungarian Assignment

Training relies on a set-based Hungarian matching between $N$ predictions and $G$ ground-truth polygons, minimizing the following cost for each pair:
\[
C(\hat{s}, \hat{b}; s, b) = \lambda_{\rm cls} L_{\rm cls}(\hat{s}, s) + \lambda_{\rm box} L_1(\hat{b}, b) + \lambda_{\rm giou} L_{\rm GIoU}(\hat{b}, b)
\]
where $L_{\rm cls}$ is a focal loss for classification, $L_1$ is smooth-$L_1$ or $L_1$ box regression loss, and $L_{\rm GIoU}$ is the generalized IoU loss.

The full objective over all $N$ predictions is:
\[
\mathcal{L} = \lambda_{\rm cls} L_{\rm cls} + \lambda_{\rm box} L_1 + \lambda_{\rm giou} L_{\rm GIoU} +
\lambda_{\rm poly}\sum_{i=1}^M\lVert \hat v_i - v_i\rVert_1 + \lambda_{\rm vtx}\sum_{i=1}^M L_{\rm cls}(\hat c_i, c_i)
\]
with typical loss weights: $\lambda_{\rm cls}=2$, $\lambda_{\rm box}=5$, $\lambda_{\rm giou}=2$, $\lambda_{\rm poly}=5$, $\lambda_{\rm vtx}=1$.

Set-based matching is conducted as in DETR and SparseR-CNN, consistent with recent object detection literature.

## 4. Handling Structures With Holes

PolyR-CNN treats each valid polygon (outer or hole) as a separate instance. At inference, hole polygons are merged into their parent via spatial-join: for each predicted hole $H$, if a larger polygon $O$ exists such that the centroid of $H$ lies within $O$, $H$ is assigned as a hole of $O$. The final outline is given by set difference $O \setminus H$. Point-in-polygon tests use winding-number or ray-crossing methods.

## 5. Empirical Evaluation and Comparison

PolyR-CNN is benchmarked on the CrowdAI and Inria datasets:

### CrowdAI (300×300 patches, MS-COCO format)

| Method              | Backbone     | AP   | AP₅₀ | AP₇₅ | AP_boundary |
|---------------------|-------------|------|------|------|-------------|
| PolyBuilding '22    | ResNet-50   | 78.7 | 96.3 | 89.2 | –           |
| PolyWorld '22       | R2U-Net     | 63.3 | 88.6 | 70.5 | 50.0        |
| PolyR-CNN           | Swin-Base   | 79.2 | 97.4 | 90.0 | 63.3        |
| PolyR-CNN           | ResNet-50   | 71.1 | 93.8 | 82.9 | 50.0        |

| Method           | Backbone    | Epochs | FLOPs (G) | FPS    |
|------------------|-------------|--------|-----------|--------|
| PolyWorld '22    | –           | –      | 181.23    | 8.4    |
| PolyBuilding '22 | ResNet-50   | 200    | 21.45     | 14.3   |
| PolyR-CNN        | ResNet-50   | 100    | 21.91     | 32.7   |
| PolyR-CNN        | Swin-Base   | 100    | 46.55     | 20.5   |

With ResNet-50, PolyR-CNN achieves 71.1 AP at 32.7 FPS (4× faster than PolyWorld) in 100 epochs. Using Swin-Base, it reaches 79.2 AP, matching the most accurate methods while operating at more than twice their speed.

### Inria (512×512, mask-IoU evaluation)

| Method                     | Needs sem-seg? | IoU   | Acc.   | FPS    |
|----------------------------|---------------|-------|--------|--------|
| Zorzi et al. '19           | ✓             | 59.81 | 93.92  | –      |
| HiSup '22                  | ✓             | 75.53 | 96.27  | 16.4   |
| PolyR-CNN                  | –             | 68.35 | 95.09  | 35.7   |

PolyR-CNN, without any semantic segmentation prior, surpasses classical polygonization in accuracy and more than doubles the speed of the best segmentation-based pipeline.

## 6. Implementation Specifics

Key components and parameters:
- **Backbones**: ResNet-50 (ImageNet-1K), ResNeXt-101, Swin-Base (ImageNet-21K).
- **FPN**: C=256 channels, pyramid levels P2–P5.
- **Proposals**: $N=100, M=96$ (CrowdAI); $N=300, M=50$ (Inria).
- **Optimization**: AdamW, initial learning rate $2.5 \times 10^{-5}$, batch size 16 (2×A40 GPUs).
- **Training Schedules**: CrowdAI—100 epochs (ResNet-50/Swin), LR drops at epochs 54/83; ResNeXt-101—140 epochs. Inria—100 epochs, LR drop at epoch 89.
- **Data Augmentation**: Horizontal and vertical flips.
- **Hungarian matching**: As in DETR and SparseR-CNN.

PolyR-CNN thus integrates efficient architectural design, a principled geometric feature coupling, and streamlined training and inference procedures, establishing new speed-accuracy Pareto frontiers for end-to-end polygonal building outline extraction from high-resolution imagery [2407.14912].

Source: https://www.emergentmind.com/topics/polyr-cnn