Boundary-Aware Semantic Embedding

• Boundary-aware semantic embedding is a technique that constructs feature representations sensitive to semantic discontinuities, ensuring precise demarcation of objects and regions.
• This approach integrates various architectural strategies, such as multi-scale boundary detection and gated feature propagation, across visual, 3D, and language models.
• Empirical results demonstrate tangible gains, including up to +3.0% mIoU and improved boundary F-scores, validating its effectiveness in diverse applications.

Boundary-aware semantic embedding refers to the construction of feature representations that explicitly encode knowledge about semantic boundaries—object edges, word borders, or region transitions—within the embedding space. This approach is central in visual scene segmentation, 3D point cloud analysis, and, more recently, language modeling for tasks where demarcation of semantic units is crucial. The underlying principle is that embeddings should not only encode semantic category membership, but also respect the discontinuities present at true boundaries, thus improving precision at region borders and overall intra-class cohesion.

1. Architectural Paradigms for Boundary-Aware Semantic Embedding

Boundary-aware semantic embeddings are realized through diverse architectural strategies across modalities. In the SBCB framework for semantic segmentation, a model-agnostic backbone extracts hierarchical multi-scale features $$F_\ell\in \mathbb{R}^{C_\ell\times H_\ell\times W_\ell}$$; a Semantic Boundary Detection (SBD) head is attached to each scale, applying %%%%1%%%% convolution, upsampling, and predicting either binary or semantic class-specific boundaries. Side outputs $B_{\text{side }\ell}(x,y)$ are fused into a high-resolution boundary cue, which guides segmentation head learning. During training, the joint loss

$$L_{\text{total}} = L_{\text{seg}} + \lambda L_{\text{SBD}}$$

propagates boundary supervision into the backbone, enhancing spatial gradients and semantic discrimination at object edges. At inference, the SBD head is discarded, so downstream cost is unaffected (Ishikawa et al., 2023).

For context aggregation, BCANet employs a Multi-Scale Boundary extractor (MSB) to produce boundary maps at multiple scales, and a Boundary-guided Context Aggregation (BCA) module, which replaces standard non-local attention: boundary stream features serve as “keys,” ensuring that attention focuses on edge regions. This yields feature updates of the form

$$D_j = A_j + \sum_{i \in \text{Boundary}} F(i,j)\cdot A_{2i}$$

where $F(i,j)$ is a boundary-semantic affinity (Ma et al., 2021).

In 3D point cloud segmentation, boundary prediction (e.g., via a Boundary Prediction Module in (Gong et al., 2021) or a semantic classification head in (Li et al., 2023)) is used to mask, gate, or restructure local feature aggregation. The Geometric Encoding Module (GEM) uses the predicted boundary mask to selectively exclude points on object edges from neighborhood feature mixing, resulting in spatially sharp, boundary-sensitive embeddings.

In language modeling, boundary-aware embeddings are realized by constructing position-dependent vectors from boundary statistics (e.g., PMI, left/right entropy) and regressing Transformer representations toward these targets during pre-training, as in BABERT (Jiang et al., 2022, Zhang et al., 2024).

2. Loss Functions for Boundary Conditioning

Boundary-aware training uses composite objectives to enforce boundary sensitivity. Commonly, a cross-entropy segmentation loss is combined with boundary-prediction loss:

• Semantic Segmentation Loss (per-pixel cross-entropy):

$$L_{\text{seg}} = -\sum_{i\in\Omega} \sum_{k=1}^{C_\text{cat}} y_i^k \log p_i^k$$

• Semantic Boundary Loss (multilabel BCE):

$$L_{\text{SBD}} = - \sum_{i\in\Omega} \sum_{k=1}^{C_\text{cat}} [b_i^k \log \sigma(z_i^k) + (1-b_i^k)\log(1-\sigma(z_i^k))]$$

where $b_i^k$ is the binary boundary GT for class $k$ at pixel $i$ (Ishikawa et al., 2023).

In context-aware graph or attention modules, auxiliary losses may further penalize classification errors on predicted boundary sets (e.g., $L_{\text{att}}$ in (Ma et al., 2021)). For 3D, boundary detection, direction regression, and segmentation are combined: $$L_\text{total} = L_S + \lambda_1 L_B + \lambda_2 L_D$$ where $L_B$ is weighted BCE for boundary points and $L_D$ is regression to the interior direction (Du et al., 2022).

In LLMs, the auxiliary boundary regression loss supervises intermediate representations to match boundary-aware statistic vectors: $$\mathcal{L}_{\text{boundary}} = \sum_{i=1}^n \|\mathbf{W}_B \mathbf{h}^l_i - \mathbf{e}_i\|^2$$ where $\mathbf{e}_i$ concatenates PMI and entropy statistics at position $i$ (Jiang et al., 2022).

3. Boundary-Aware Feature Aggregation and Propagation

Boundary information is used to gate or modulate feature propagation both locally and globally:

• Gated Propagation: In BFP (Ding et al., 2019), boundaries are learned as a separate semantic class. Feature flow across pixels is controlled by a gate $$p_{x,y} = 1 - \beta \sigma(\alpha b_{x,y} - \gamma)$$, sharply reducing propagation across edges and encouraging intra-region consistency.
• Masked Aggregation: Boundary-aware GEMs (e.g., (Gong et al., 2021)) suppress contributions from boundary points in neighborhood aggregation:

$$f^{(\ell)}_i = \sigma\left( \sum_{p_j\in\mathcal{N}(p_i)} \phi(r_{ij}) [\mathcal{M}(\hat g_j f_{p_j}) ] \right)$$

with $\hat g_j \sim 0$ for boundary points (removing their influence).

• Graph/Attention Reweighting: In Graph-Segmenter (Wu et al., 2023) and BGR (Tang et al., 2021), boundary score maps are used to amplify graph connections, attention, or node affinities involving boundary regions, focusing long-range relational modeling on error-prone edges.
• 3D Feature Propagation: “Push-the-Boundary” utilizes per-point boundary and direction predictions to modulate decoder upsampling via

$$w_{ij} = \max\left( 0, w_s(x_j, x_i) + \alpha w_c(x_j, x_i) \right)$$

where $w_c$ encodes directionality and boundary probability (Du et al., 2022).

4. Ground Truth Construction and Supervision Strategies

Boundary labels generally require precise construction:

• Signed Distance Transform: Given a segmentation mask $M(x,y)$, semantic boundaries are defined as $|D_\ell(x,y)| \le r$ after computing the signed distance field per class; binarization yields pixelwise semantic boundary maps (Ishikawa et al., 2023).
• k-NN Heuristic and Geometric Filters: In 3D, boundary points are flagged if any k-NN neighbor holds a different semantic label (Gong et al., 2021); this is combined with plane fitting for clustering in (Li et al., 2023). For sequence tasks, boundaries are mined either statistically (PMI/entropy from n-gram counts (Jiang et al., 2022)) or supplied by a large external lexicon with positive-unlabeled learning for refinement (Zhang et al., 2024).

5. Application Domains and Quantitative Impact

Boundary-aware embeddings are validated across a range of visual and textual domains:

• Semantic Segmentation (2D): SBCB yields +0.5%–3.0% mean IoU gain and +1.6%–4.1% boundary F-score on Cityscapes, with benefits also on HRNet, SegFormer, BiSeNet, etc. (Ishikawa et al., 2023). BCANet achieves 80.92% mIoU on Cityscapes, and consistent boosts in both boundary and interior F-scores (Ma et al., 2021). Graph-based and attention-based approaches retain sharper edges with equivalent or improved mIoU (Wu et al., 2023, Tang et al., 2021, Fontinele et al., 2021).
• 3D Point Clouds: Dual-space clustering for roof planes yields coverage up to 93.2% on synthetic and 91.1% on real datasets, with near-perfect instance precision/recall by explicitly controlling boundary assignment (Li et al., 2023). Feature-masked architectures provide +2–3% mIoU and outperform standard PointNet++ and KP-Conv (Gong et al., 2021, Du et al., 2022).
• Zero-Shot Segmentation: Boundary-aware regression loss (BAR) in joint embedding spaces prevents center drift from hard label transitions, balancing accuracy on seen/unseen classes (Baek et al., 2021).
• Sequence Labeling and NLP: Injecting unsupervised or semi-supervised boundary information into BERT via BABERT or its semi-supervised variant improves average F1 by +0.67 to +0.9 absolute (Jiang et al., 2022, Zhang et al., 2024), with the highest Boundary Information Metric (BIM) among Chinese PLMs. The gain is especially pronounced in few-shot setups and on general NLU tasks.

6. Variants, Generalizations, and Limitations

Boundary-aware semantic embedding modules are plug-and-play across architectures as long as the backbone provides multi-scale or hierarchical features, as in SBCB (Ishikawa et al., 2023) and Graph-Segmenter (Wu et al., 2023). Lightweight boundary heads are discarded at inference, guaranteeing zero added run-time cost.

Alternative forms of supervision—fine-grained edge maps, statistical, or lexicon-derived cues—provide complementary information (Jiang et al., 2022, Zhang et al., 2024). Boundary masking and gating are consistently superior to simple up-weighting or late feature fusion (Gong et al., 2021, Du et al., 2022). Effective boundary information extraction and accurate boundary GT construction remain challenging for under-annotated or ambiguous semantic classes.

For multimodal transfer (e.g., language and vision), the central insight is generic: embeddings must encode not only category membership but also be sensitive to sub-category transitions. This sensitivity boosts both boundary accuracy and global consistency.

7. Representative Quantitative Results

Method/Domain	mIoU Gain	Boundary F-score Gain	Additional Metrics/Notes
SBCB (Cityscapes, DeepLabV3+)	+0.5%–3.0%	+1.6%–4.1%	Inference cost unaffected (Ishikawa et al., 2023)
BCANet (Cityscapes, ResNet101)	+4.5%	+2.45	Interior F-score +1.95 (Ma et al., 2021)
BEFBM (Mask2Former)	+2.8%	>3% boundary F1	Mask2Former mIoU: 78.5→81.3 (An et al., 28 Mar 2025)
Graph-Segmenter (Cityscapes)	+1.3%	—	Swin-L backbone (Wu et al., 2023)
Push-the-Boundary (S3DIS)	+1.6%	—	mIoU: 65.6→67.2 (Du et al., 2022)
BABERT (Chinese NLP)	+0.67 F1	—	Average across 10 datasets (Jiang et al., 2022)
Semi-BABERT	+0.7–0.9 F1	BIM: 15.2	Consistently best boundary metric (Zhang et al., 2024)

These empirical improvements validate the efficacy of boundary-aware embedding architectures across benchmarks and indicate that precise encoding of boundary structure in foundational representations substantially benefits recognition accuracy, intra-class consistency, and error localization, both in vision and language.

---

References:

• (Ishikawa et al., 2023) Boosting Semantic Segmentation with Semantic Boundaries
• (Ma et al., 2021) Boundary Guided Context Aggregation for Semantic Segmentation
• (An et al., 28 Mar 2025) A Deep Learning Framework for Boundary-Aware Semantic Segmentation
• (Li et al., 2023) A boundary-aware point clustering approach in Euclidean and embedding spaces for roof plane segmentation
• (Baek et al., 2021) Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentation
• (Gong et al., 2021) Boundary-Aware Geometric Encoding for Semantic Segmentation of Point Clouds
• (Ding et al., 2019) Boundary-Aware Feature Propagation for Scene Segmentation
• (Tang et al., 2021) Boundary-aware Graph Reasoning for Semantic Segmentation
• (Jiang et al., 2022) Unsupervised Boundary-Aware LLM Pretraining for Chinese Sequence Labeling
• (Wu et al., 2023) Graph-Segmenter: Graph Transformer with Boundary-aware Attention for Semantic Segmentation
• (Du et al., 2022) Push-the-Boundary: Boundary-aware Feature Propagation for Semantic Segmentation of 3D Point Clouds
• (Fontinele et al., 2021) Attention-based fusion of semantic boundary and non-boundary information to improve semantic segmentation
• (Zhang et al., 2024) Chinese Sequence Labeling with Semi-Supervised Boundary-Aware LLM Pre-training