TEFormer: Texture & Edge-Aware Transformer

• The paper introduces TEFormer, a Transformer architecture that integrates explicit statistical texture modeling and edge-aware decoding for precise semantic segmentation in urban imagery.
• Empirical evaluations on datasets like ISPRS Potsdam and Vaihingen show improved performance, with TEFormer achieving up to 88.57% mIoU and enhanced boundary precision.
• Ablation studies confirm that dedicated modules for texture discrimination and edge guidance significantly reduce class confusion and improve mask delineation.

A Texture-Aware and Edge-Guided Transformer (TEFormer) is a neural architecture designed for semantic segmentation in urban remote sensing imagery (URSI), addressing the challenges of ambiguous textures, blurred or irregular boundaries, and overlapping object distributions. TEFormer combines explicit statistical texture modeling with multi-scale edge preservation through a specialized hybrid Transformer framework that integrates novel modules for enhanced feature discrimination and precise mask delineation (Zhou et al., 8 Aug 2025).

1. Motivation and Challenges in Urban Remote Sensing Segmentation

Semantic segmentation of URSI encounters distinct obstacles:

• Texture ambiguity: Urban materials (e.g., pavement vs. rooftops, or dense vs. sparse vegetation) often exhibit globally similar visual patterns at the resolutions typical of modern sensors, resulting in class confusion.
• Blurred and irregular boundaries: Urban scenes contain objects with contiguous, overlapping, or highly intricate outlines, making conventional contour extraction unreliable.
• Multi-scale occlusion and object overlap: Phenomena such as vehicles under tree cover, shadows, and the coexistence of variously sized structures introduce complex spatial relationships and hinder pixel-precise labeling.

TEFormer addresses these via dual mechanisms: explicit texture encoding for global discrimination and edge-aware decoding to maintain boundary fidelity. This design is motivated by the limitations of standard Vision Transformers, which struggle with high-frequency details and local texture cues (Zhou et al., 8 Aug 2025, Azad et al., 2023).

2. Encoder Structure and the Texture-Aware Module (TaM)

The TEFormer encoder comprises four hierarchical Transformer stages. The first two integrate the Texture-Aware Module (TaM) in parallel with convolution-channel-attention (CCAB) and cross-shaped window self-attention (CWSA). Stages three and four employ standard dual-attention Transformers (from D2SFormer), deepening global and local feature reasoning.

TaM operates as follows:

1. Quantize-Count-Encode (QCO): Given $X\in \mathbb{R}^{H\times W\times C}$, QCO quantizes features into $N$ levels, counts occurrences, and produces a statistical histogram $A\in\mathbb{R}^{N\times C}$ and per-pixel indicator $B\in\mathbb{R}^{H\times W\times N}$.
2. Level-Attention Update: Attention weights over quantization bins refine the histogram encoding:

$$L' = \mathrm{Conv}_3(A) \odot \mathrm{Softmax}(\mathrm{Conv}_1(A)\cdot \mathrm{Conv}_2(A)).$$

1. Feature Concatenation: Fused histogram and indicator maps are concatenated with the original feature:

$X' = [\mathrm{Conv}_4(L')\odot B;\;X].$

1. Multi-scale Re-QCO: $X'$ is partitioned into four branches, each pooled at a distinct scale, passed through QCO, then upsampled and concatenated.
2. Final Texture Feature: Aggregated and fused to output $X_{\mathrm{TaM}}$ at full resolution.

TaM is injected into the low-level stages to amplify discrimination between visually similar classes via learned global texture signatures. Output is combined with representations from local attention and convolution for subsequent Transformers.

3. Edge-Guided Tri-Branch Decoder (Eg3Head)

The Eg3Head decoder preserves and integrates boundary and texture cues via three parallel branches:

• Edge Branch: Aggregates multi-scale features $E_1, E_2, E_3$ to predict edge probability maps through spatial downsampling and summation:

$P_e = \sum_{i=1}^3 \mathrm{Down}_i(E_i)$

followed by convolution to produce $N$0.

• Detail Branch (DAM): Employs the Detail Analysis Module from PIDNet, splitting features into high-frequency (edge/texture) and low-frequency (context) streams, fusing them via learned gates to yield $N$1.
• Context Branch (PASPPM): Applies parallel global pooling and atrous convolutions at varying dilation rates, concatenates outputs, and generates spatial attention maps modulating global context:

$N$2

with $N$3 produced by PASPPM attention.

Each branch isolates a complementary property: boundaries, fine local structure, and global semantic context.

4. Edge-Guided Feature Fusion Module (EgFFM)

EgFFM synthesizes the outputs of Eg3Head by:

1. Computing an edge-derived gating map:

$N$4

2. Fusing detail and context via gating:

$N$5

1. Adding a skip connection from the highest-resolution encoder feature $N$6.
2. Passing through an MLP and softmax for per-pixel semantic class prediction.

This procedure allows boundaries to modulate the contribution of fine details and global context, refining segmentation masks for both precision and smoothness.

5. Training Paradigm and Optimization

TEFormer employs the AdamW optimizer with parameters: learning rate $N$7, weight decay $N$8, batch size $N$9, trained for $A\in\mathbb{R}^{N\times C}$0 iterations on $A\in\mathbb{R}^{N\times C}$1 random crops.

The loss is a compound of standard multi-class cross-entropy for segmentation and binary cross-entropy for edge map prediction:

• Segmentation loss:

$A\in\mathbb{R}^{N\times C}$2

• Edge loss:

$A\in\mathbb{R}^{N\times C}$3

• Complete loss: $A\in\mathbb{R}^{N\times C}$4 (with $A\in\mathbb{R}^{N\times C}$5 not specified in the source, typically $A\in\mathbb{R}^{N\times C}$6).

6. Empirical Performance and Ablation Analysis

TEFormer is evaluated on:

• ISPRS Potsdam (RGB+NIR, $A\in\mathbb{R}^{N\times C}$7 tiles): mIoU $A\in\mathbb{R}^{N\times C}$8 (mF1 $A\in\mathbb{R}^{N\times C}$9), superior to D2SFormer by $B\in\mathbb{R}^{H\times W\times N}$0 mIoU.
• ISPRS Vaihingen (RGB+NIR, $B\in\mathbb{R}^{H\times W\times N}$1 tiles): mIoU $B\in\mathbb{R}^{H\times W\times N}$2, highest in comparative study.
• LoveDA ($B\in\mathbb{R}^{H\times W\times N}$3 patches): mIoU $B\in\mathbb{R}^{H\times W\times N}$4; especially robust in urban features (buildings, roads).

Ablation studies confirm:

• Full TaM (QCO + attention) significantly outperforms QCO alone and baseline (Potsdam: baseline $B\in\mathbb{R}^{H\times W\times N}$5, QCO $B\in\mathbb{R}^{H\times W\times N}$6, +TaM $B\in\mathbb{R}^{H\times W\times N}$7).
• Each Eg3Head component (PASPPM, DAM, EgFFM) incrementally adds performance, with the full Eg3Head yielding the highest mIoU.

Qualitative results demonstrate sharper object outlines, reduction in class confusion (especially between cars and trees), and better preservation of small objects.

7. Relation to Broader Context and Methodological Impact

TEFormer’s approach is distinct in combining explicit texture statistics and boundary-aware mechanisms within a unified Transformer framework. The methodology closely parallels observations in Laplacian-Former (Azad et al., 2023), which showed that Vision Transformer models benefit from frequency decomposition (Laplacian pyramids) and dual-attention mechanisms for texture and edge capture. While Laplacian-Former targets medical imaging and emphasizes frequency domain calibration, TEFormer adapts these conceptual underpinnings to urban scenes, demonstrating the versatility and generalizability of dual-path (texture/edge) attention architectures.

Core contributions are:

• The Texture-Aware Module (TaM), establishing global statistical descriptors for material discrimination.
• The Edge-Guided Tri-Branch Decoder, structurally segregating edge, detail, and context processing.
• Edge-Guided Feature Fusion, providing a principled, learnable strategy for integrating boundaries and global cues.

A plausible implication is that such architectural motifs—explicit global texture modeling and tight boundary integration—could be beneficial in other dense prediction tasks where edge preservation and fine-grained texture differentiation are critical.