---
title: Labeled Polygon Sequence Representation
url: https://www.emergentmind.com/topics/labeled-polygon-sequence-representation
type: topic
---

# Labeled Polygon Sequence Representation

A labeled polygon sequence representation encodes structured objects—such as floorplan regions, detected objects, or discrete code sequences—as an ordered sequence of geometric vertices, combined with semantic or categorical labels per vertex or polygon. This formalism underlies diverse recent advances in vector-graphics extraction from images, polygonal object detection, error-correcting code visualization, and sequence-to-sequence modeling of geometric shapes. The representation supports variable numbers of polygons, flexible vertex cardinality, semantic annotations, and efficient autoregressive or set-based generation mechanisms.

## 1. Formal Definition and Tokenization

For tasks such as floorplan vectorization or instance segmentation, a scene is decomposed into a sequence of labeled polygons, each represented as an ordered list of vertices. For example, consider \emph{Raster2Seq} for floorplan reconstruction: each corner token $c_i = (x_i, y_i, p_i)$ carries:

- $(x_i, y_i)$: continuous 2D coordinates of vertex $i$ (normalized to image or feature map dimensions)
- $p_i \in \Delta^C$: a one-hot or probabilistic $C$-dimensional semantic label (e.g., room type)

Tokenization involves linearizing the entire set of polygons (e.g., all rooms followed by windows and doors) into a single sequence, bracketed by symbolic tokens. The canonical structure is:

$S = [\texttt{<BOS>}, c^1_1, \ldots, c^1_{L_1}, \texttt{<SEP>}, \ldots, c^N_{L_N}, \texttt{<SEP>}, \ldots, \texttt{<EOS>}]$

where $L_k$ is the number of vertices in polygon $k$; \texttt{<SEP>} and \texttt{<EOS>} mark polygon boundaries and sequence termination [2602.09016], [2302.07387].

Analogous structures appear in object detection and instance segmentation methods, with coordinate tokens and separator tokens encoded consistently [2302.07387], [2304.02250].

## 2. Geometric Quantization, Embedding, and Representation

### Polygonal Geometry

Continuous $(x,y)$ vertex coordinates are typically embedded into a high-dimensional feature space via quantization and bilinear interpolation over a learnable codebook. For Raster2Seq and PolyFormer:

- Vertices are normalized to image bounds and mapped via bilinear interpolation among nearby codebook entries: 
  $$
  e_{x,y} = \sum_{i,j} w_{i,j} C[i,j]
  $$
  where $C \in \mathbb{R}^{H_b \times W_b \times D}$ is a codebook and $w_{i,j}$ are interpolation weights determined by the point's fractional coordinates [2602.09016], [2302.07387].

In DPPD, vertices are parameterized in polar coordinates relative to an inferred object center $o = (o_x, o_y)$, with each vertex as $(r_i, a_i)$. Angle tokens are produced via a cumulative-softmax, ensuring counter-clockwise and non-overlapping ordering [2304.02250].

### Semantic Labels

Semantic information is carried in the one-hot/probabilistic $p_i$ attached to each vertex or encoded separately for the polygon. At inference, semantic assignment is typically $\operatorname{argmax}(p_i)$; at training, a cross-entropy loss is applied for supervision [2602.09016].

### Special Tokens

Sequences always begin with \texttt{<BOS>} and end with \texttt{<EOS>}. \texttt{<SEP>} tokens split polygons. Token classes (per-step) are predicted via a classification head [2602.09016], [2302.07387].

## 3. Autoregressive and Set-Based Generation

State-of-the-art models employ autoregressive decoders (e.g., Transformers) to produce labeled polygon sequences, conditioned on past outputs and image features:

- At step $t$, the decoder predicts the next vertex coordinates $v_t = (x_t, y_t)$ and label $l_t$ from $(v_{<t}, l_{<t}, I)$:
  $$
  p(v_t, l_t \mid v_{<t}, l_{<t}, I)
  $$
- Causal masking ensures the decoder attends only to previously generated content.
- During inference, generation proceeds sequentially until an \texttt{<EOS>} token is produced [2602.09016], [2302.07387].

DPPD departs from strict sequence modeling, instead predicting a sparse, unordered set of vertices with explicit postprocessing to enforce consistent ordering [2304.02250].

## 4. Anchor Mechanisms and Attention Guidance

Attention-based decoders in labeled polygon sequence models utilize spatial anchor mechanisms to refine geometric precision and focus:

- Each token position $t$ is associated with a 2D anchor $a_t$, initialized randomly and learned during training.
- Rather than directly outputting the vertex, the model predicts an offset $\Delta_t$ from the anchor, so:
  $$
  (\hat{x}_t, \hat{y}_t) = \operatorname{Sigmoid}(a_t) + \Delta_t
  $$
- In transformer-based decoders, deformable attention is used: for query $q_t$, a small MLP predicts $K$ offsets $\{\delta_{t,k}\}$ around anchor $a_t$, and attention is restricted to $K$ spatial positions on the image feature map. This spatial focus yields more efficient and accurate processing of complex geometry [2602.09016].

## 5. Loss Functions and Training

Labeled polygon sequence models impose multiple supervisory losses:

- **Coordinate regression loss:** $L_{\text{coord}} = \frac{1}{L}\sum_t m_t \|(\hat{x}_t, \hat{y}_t) - (x_t, y_t)\|_1$ for vertex localization.
- **Semantic classification loss:** $L_{\text{sem}} = \frac{1}{L}\sum_{t=1}^L m_t \cdot \mathrm{CE}(\hat{p}_t, p_t)$ for semantic consistency, with $m_t$ masking special/padding tokens.
- **Token-type loss:** $L_{\text{token}} = \frac{1}{L}\sum_t m_t \cdot \mathrm{CE}(\hat{q}_t, q_t)$, where $q_t$ indicates the token class (\texttt{<CORNER>}, \texttt{<SEP>}, \texttt{<EOS>}).
- The total loss is a weighted sum: $L = \lambda_{\text{coord}} L_{\text{coord}} + \lambda_{\text{token}} L_{\text{token}} + \lambda_{\text{sem}} L_{\text{sem}}$ [2602.09016].

DPPD uses analogous losses in polar space, including polar-IoU, center regression, and smoothness regularization for ordered vertex subsequences [2304.02250].

## 6. Worked Examples and Application Domains

A general example (from Raster2Seq):

- Sequence: [\texttt{<BOS>}, (12.3, 45.1; $p=$bedroom), (32.7, 45.1; $p=$bedroom), (32.7,65.4; $p=$bedroom), (12.3,65.4; $p=$bedroom), \texttt{<SEP>}, (25.0,45.0; $p=$window), (35.0,45.0; $p=$window), \texttt{<EOS>}]
- Different semantic labels encode different object classes.
- At test time, the continuous geometry is reconstructed and polygons split at \texttt{<SEP>} boundaries [2602.09016].

Key application domains:
- Floorplan vectorization and semantic editing [2602.09016]
- Instance segmentation and referring image segmentation [2302.07387]
- Arbitrary-shape object detection in vision [2304.02250]
- Codeword visualization and finite-field transform analysis: mapping $x \in \mathrm{GF}(p)^N$ to N-gon “radar charts” yields geometric signatures invariant under certain number-theoretic transforms [2103.04832].

## 7. Extensions and Generalizations

The labeled polygon sequence formalism generalizes across object detection, segmentation, vector graphics extraction, and error-correcting code analysis:

- Sequences over arbitrary alphabets and dimensions: geometric mapping for codewords over $\mathrm{GF}(p)$ as $N$-gons in the complex plane [2103.04832].
- Differentiable resampling: DPPD’s polar parameterization enables loss computation and gradient flow for polygons of variable vertex number, with fully differentiable mapping between sparse and dense representations [2304.02250].
- Joint image–semantics modeling: Cross-modal conditioning (e.g., via language query) in PolyFormer tightly couples semantics, geometry, and input context [2302.07387].
- Applications to CAD, error-correcting coding, self-dual code symmetry analysis, and pedagogical tool development [2103.04832], [2602.09016].

The labeled polygon sequence paradigm provides a unifying structured representation bridging geometry, semantics, and sequence modeling for a wide class of modern computational perception and vector graphics tasks.

Source: https://www.emergentmind.com/topics/labeled-polygon-sequence-representation