SeeDiff: Attention-Driven Synthetic Segmentation

• SeeDiff is a training-free method that leverages cross- and self-attention maps from Stable Diffusion to produce pixel-level masks for semantic segmentation.
• It adopts a seeded segmentation approach where coarse class-localized seeds from cross-attention are iteratively expanded by self-attention across multiple scales.
• Experimental results on Pascal VOC and Cityscapes demonstrate significant performance improvements over previous diffusion-based segmentation techniques.

SeeDiff is a training-free method for generating pixel-level masks from the internal attention maps of an off-the-shelf text-to-image diffusion model, specifically Stable Diffusion 2-base. It is designed for synthetic semantic segmentation dataset generation, where the objective is to produce image-mask pairs without human pixel annotation. The method treats diffusion attention as a seeded segmentation system: cross-attention supplies coarse class-localized seeds, self-attention spreads those seeds across semantically corresponding regions, and a background-estimation stage suppresses leakage. In the formulation reported for the method, this is achieved without additional training procedure, prompt tuning, or a pre-trained segmentation network (Park et al., 26 Jul 2025).

1. Problem setting and scope

SeeDiff addresses the production of synthetic image-mask pairs $\{(x_n,m_n)\}_{n=1}^N$ for semantic segmentation. The motivating problem is that semantic segmentation networks require dense pixel-level annotation masks, and those masks are laborious to prepare. The method therefore targets automatic class-mask recovery from the same diffusion process that generates the image itself, with no prompt tuning, no learned refinement module, and no pretrained segmentation or detection model (Park et al., 26 Jul 2025).

The work is positioned against three strands of prior methods. Weakly supervised segmentation reduces annotation cost by using tags, points, scribbles, or boxes, but still often underperforms full supervision. GAN-based synthetic annotation methods usually require extra decoders and at least some annotated data. Diffusion-based synthetic annotation methods use text-guided generation but, according to the paper, often still depend on a pretrained segmentation model, extra trainable modules, prompt engineering, or refinement learning. SeeDiff’s stated objective is to avoid those additions and rely only on the attention mechanisms already present in Stable Diffusion (Park et al., 26 Jul 2025).

Within the broader landscape of “diff” systems, SeeDiff belongs to image synthesis and mask generation rather than program differencing or change detection. That distinction matters because similarly named systems in the literature solve substantially different problems: EC-Diff concerns Datalog-based semantic differencing of concurrent programs (Sung et al., 2018), ChangeDiff concerns semantic change detection data generation in remote sensing (Zang et al., 2024), and VisDiff concerns natural-language description of differences between sets of images (Dunlap et al., 2023).

2. Attention-based segmentation principle

The core conceptual contribution of SeeDiff is to reinterpret Stable Diffusion attention maps through the lens of classical seeded segmentation. In that analogy, cross-attention provides initial seeds, while self-attention performs region expansion. Cross-attention links image latent features to text tokens and is used to answer which image locations correspond to a class token. Self-attention links image latent features to one another and is used to answer which other locations are semantically similar to a seed location (Park et al., 26 Jul 2025).

The method explicitly distinguishes the roles of the two attentions. Cross-attention alone provides only coarse object localization. The reported observation is scale-dependent: low-resolution cross-attention preserves objectness better but is spatially coarse, whereas high-resolution cross-attention is more detailed but often diffuses into background. SeeDiff therefore treats cross-attention as a coarse object locator rather than a final segmenter, and selects the $16\times16$ cross-attention map as the seed source (Park et al., 26 Jul 2025).

Self-attention is interpreted as a semantic-correspondence map. If a seed falls on the object, then the self-attention map at that seed location often highlights other parts of the same object. Averaging self-attention maps from multiple seeds yields a broader soft object region. The paper further reports that low-resolution self-attention carries better semantic grouping, while high-resolution self-attention supplies finer detail. This motivates a multi-scale iterative propagation process from coarse to fine (Park et al., 26 Jul 2025).

A second empirical observation concerns the background structure of synthetic images. The paper states that simple text-guided synthetic images often have a relatively uniform background, and that self-attention can find correspondences in such backgrounds more reliably than across heterogeneous object regions. SeeDiff therefore performs a second seeded-growing process for the background and uses that background confidence to sharpen the foreground mask (Park et al., 26 Jul 2025).

3. Methodological pipeline

SeeDiff uses a pretrained Stable Diffusion 2-base model with denoising steps $T=50$. Attention maps are collected from the U-Net denoiser’s transformer blocks at four spatial resolutions,

$s_r \in \{8,16,32,64\}.$

The paper states that attention maps are aggregated over all relevant layers and all timesteps at each resolution, separately for cross-attention and self-attention. The class-specific cross-attention slice corresponding to the class token is then used for seed initialization (Park et al., 26 Jul 2025).

Seed initialization begins from the $16\times16$ class-token cross-attention map. The implementation uses a seed threshold $\alpha=0.5$. These seeds are intended to be reliable starting points rather than full object masks. Region expansion then proceeds by averaging the self-attention correspondence maps indexed by the current seed set, producing a soft mask at the current scale. The resulting mask is bilinearly upsampled to the next scale, thresholded again with $\alpha=0.5$ to form new seeds, and expanded using the self-attention maps at the higher resolution. This procedure is repeated until the final scale $64\times64$, where the method performs expansion without a further seed-extraction step (Park et al., 26 Jul 2025).

After the final object-region mask is obtained, the method applies background refinement. The final object mask is inverted, background seeds are extracted from the inverted mask by thresholding in the same style as object-seed extraction, and those background seeds are expanded with self-attention to obtain a background confidence map. The refined class mask is computed as

$$\mathcal{M}'_\text{final} = (1-\mathcal{M}'_\text{bg})\odot\mathcal{M}'_K.$$

The final soft mask is then upsampled to $512\times512$ and binarized using a threshold $16\times16$0 (Park et al., 26 Jul 2025).

The main method description does not include CRF, morphology, connected components, or learned postprocessing. However, the experimental prose states that no refinement steps are applied aside from the PAMR algorithm. The paper does not formalize PAMR in the method section or specify its parameters. Accordingly, the paper-faithful interpretation is that the core SeeDiff method is the attention-based seeded pipeline, while PAMR is mentioned as an additional refinement in experimental description without sufficient detail for faithful reconstruction from the paper text alone (Park et al., 26 Jul 2025).

4. Architectural and implementation specifics

The backbone is Stable Diffusion 2-base. The paper reports $16\times16$1 transformer blocks in the denoising U-Net, with attention maps appearing at resolutions $16\times16$2, $16\times16$3, $16\times16$4, and $16\times16$5. Final masks are upsampled to $16\times16$6 by bilinear interpolation. On Pascal VOC, the prompt template is “a photo of a <CLASS>”. On Cityscapes, the prompt template is “a photo of a <CLASS> in the city.” No prompt tuning or prompt search is used (Park et al., 26 Jul 2025).

The implementation details given explicitly are the denoising step count $16\times16$7, seed threshold $16\times16$8, binarization threshold $16\times16$9, cross-attention seed-map resolution $T=50$0, iterative self-attention expansion up to $T=50$1, bilinear interpolation, and final output resolution $T=50$2. These details are sufficient to reconstruct the high-level algorithmic procedure (Park et al., 26 Jul 2025).

The paper also identifies several omitted details. It does not provide exact attention-head aggregation, exact layer indices for each resolution group, or the explicit background-seed threshold formula beyond stating that it is obtained similarly to the seed-extraction equation. This suggests that the method is reproducible at the level of design and major hyperparameters, but not fully specified at the level of all implementation choices (Park et al., 26 Jul 2025).

5. Experimental evaluation

The evaluation uses Pascal VOC 2012 and Cityscapes, with indirect assessment through downstream semantic segmentation performance. The masks generated by SeeDiff are used to train Mask2Former, with ResNet-50 and Swin-B backbones and training settings following the original Mask2Former paper. Synthetic data scale is reported as $T=50$3 or $T=50$4 images on VOC and $T=50$5 synthetic images on Cityscapes. Data augmentation follows DiffuMask and includes splicing, Gaussian noise, perspective transformation, and occlusion. Hardware is reported as an NVIDIA RTX 4090 (Park et al., 26 Jul 2025).

On VOC, the strongest pure-synthetic result reported for SeeDiff is 79.6 mIoU on the validation set with 60k synthetic images and Mask2Former Swin-B. The paper reports 78.6 mIoU with 40k synthetic images and Swin-B. Under the ResNet-50 synthetic-only regime, it reports 61.2 mIoU with 40k images and 62.6 mIoU with 60k images. When synthetic data are combined with 5k real VOC images, SeeDiff reaches 88.8 mIoU with Swin-B. The comparison points reported in the paper include 70.6 for DiffuMask under the same 60k synthetic-image Swin-B regime, 84.9 for DiffuMask plus 5k real images, and 84.3 for the full real VOC train set with Swin-B (Park et al., 26 Jul 2025).

On Cityscapes, with 100k synthetic images and Swin-B, SeeDiff reports 74.3 for Human and 89.8 for Vehicle, compared with 72.1 and 87.0 respectively for DiffuMask. The paper also states that in the VOC zero-shot setting SeeDiff outperforms DiffuMask and several real-data-trained methods and shows similar performance on seen and unseen categories, but the exact zero-shot scores are not included in the provided content (Park et al., 26 Jul 2025).

The ablation study is central to the paper’s empirical argument. Cross-attention aggregation only (CAA) yields 68.8. Adding cross-attention-guided region expansion (CAA + CRE) yields 71.0. Adding iterative region expansion (+ IRE) yields 73.2. Adding background expansion (+ BE), corresponding to the full model, yields 79.6. A related scale study reports 65.9 when seeds are initialized from $T=50$6 cross-attention, 64.3 from $T=50$7, and 58.2 from $T=50$8, supporting the choice of $T=50$9 for seed initialization. A mask-generation-strategy comparison reports 68.8 for CA alone, 72.0 for CA $s_r \in \{8,16,32,64\}.$0 SA, and 79.6 for SeeDiff, supporting the seeded region-growing interpretation over direct multiplication of cross-attention and self-attention (Park et al., 26 Jul 2025).

6. Limitations, failure modes, and relation to adjacent work

The paper explicitly discusses two primary failure sources. First, cross-attention seeds may include background near object boundaries, especially for objects with complex structure; self-attention can then propagate that error, producing blurry boundaries. Second, cross-attention may concentrate on only a discriminative object part; for heterogeneous objects, self-attention may then fail to spread to all relevant regions. The proposed background refinement is intended to address these problems partially (Park et al., 26 Jul 2025).

The method also implicitly assumes that the prompt aligns reasonably well with the generated object, that the generated scene has a relatively simple or uniform background, and that the target class corresponds to a token whose cross-attention map is meaningful. The paper does not claim robust performance for cluttered or highly compositional scenes. A plausible implication is that SeeDiff is best suited to class-centric synthetic images where object-background separation is comparatively clean (Park et al., 26 Jul 2025).

In the broader literature, SeeDiff occupies a specific niche. It is not a direct change detector in remote sensing, unlike ChangeDiff, which is a diffusion-based semantic change data generator built around text-to-layout and layout-to-image synthesis for semantic change detection (Zang et al., 2024). It is not a program semantic differencer, unlike EC-Diff, which compares nearby versions of multithreaded programs by differentiating read-from and store-order relations (Sung et al., 2018). It is not a source-code diff optimizer, unlike the interactive edit-graph approach that studies corrective feedback on nonoptimal line diffs (Yagi et al., 2024), and it is not a benchmark for code-diff understanding such as Diff-XYZ (Glukhov et al., 14 Oct 2025). Nor is it a system for contrastive natural-language descriptions of differences between image sets, which is the focus of VisDiff and Set Difference Captioning (Dunlap et al., 2023).

Within diffusion-based synthetic annotation, SeeDiff’s defining feature is the claim that Stable Diffusion already contains enough localization and semantic-correspondence structure for mask generation if its attentions are used in a seeded segmentation framework. The empirical record reported in the paper suggests that the combination of coarse cross-attention seeding, multi-scale self-attention expansion, and background suppression is sufficient to outperform the cited diffusion-based baseline without prompt tuning, extra trainable modules, or a pretrained segmentation network (Park et al., 26 Jul 2025).