PC-SAM: Patch-Constrained Fine-Grained Interactive Road Segmentation in High-Resolution Remote Sensing Images

Abstract: Road masks obtained from remote sensing images effectively support a wide range of downstream tasks. In recent years, most studies have focused on improving the performance of fully automatic segmentation models for this task, achieving significant gains. However, current fully automatic methods are still insufficient for identifying certain challenging road segments and often produce false positive and false negative regions. Moreover, fully automatic segmentation does not support local segmentation of regions of interest or refinement of existing masks. Although the SAM model is widely used as an interactive segmentation model and performs well on natural images, it shows poor performance in remote sensing road segmentation and cannot support fine-grained local refinement. To address these limitations, we propose PC-SAM, which integrates fully automatic road segmentation and interactive segmentation within a unified framework. By carefully designing a fine-tuning strategy, the influence of point prompts is constrained to their corresponding patches, overcoming the inability of the original SAM to perform fine local corrections and enabling fine-grained interactive mask refinement. Extensive experiments on several representative remote sensing road segmentation datasets demonstrate that, when combined with point prompts, PC-SAM significantly outperforms state-of-the-art fully automatic models in road mask segmentation, while also providing flexible local mask refinement and local road segmentation. The code will be available at https://github.com/Cyber-CCOrange/PC-SAM.

• The paper introduces PC-SAM, a SAM-based model that restricts positive and negative point prompts to 32×32 patches for precise local road-mask refinement.
• PC-SAM combines automatic and prompt-driven decoders with dynamically generated supervision, achieving 75.27%, 65.37%, and 67.49% IoU on DeepGlobe, Massachusetts Roads, and CHN6-CUG.
• The results show prompt-based correction can improve recall by recovering missed roads and remove false positives, although performance depends on prompt quality and fixed patch size, especially for thin roads.

Motivation and problem statement

Road segmentation in high-resolution remote sensing imagery has been dominated by fully automatic segmentation networks, which have achieved steady accuracy gains but exhibit two persistent shortcomings. First, they produce false positive and false negative regions on challenging road segments, and the distribution shift between remote sensing and natural imagery makes these errors difficult to eliminate. Second, fully automatic pipelines offer no mechanism for user-driven correction: they cannot segment a region of interest selectively, exclude an undesired local region, or refine an existing mask interactively.

Interactive foundation models such as SAM [17] appear to address the second shortcoming, but the paper shows that vanilla SAM is poorly suited to this task: zero-shot SAM-H achieves only 21.72% IoU on DeepGlobe and 11.83% IoU on Massachusetts Roads, and even when adapted, SAM's point prompts exert an unpredictable, image-wide influence that precludes fine-grained local corrections. PC-SAM addresses both limitations by unifying fully automatic segmentation with patch-constrained interactive refinement in a single model.

Architecture

PC-SAM is built on the SAM-B architecture and preserves its "large encoder–small decoder" design, adding two mask decoders beyond the original one:

• High-Recall Mask Decoder (HRMD): produces a high-recall road mask that serves as a visual reference to help users place positive point prompts in missed road regions.
• Automatic Mask Decoder (AMD): performs fully automatic segmentation; it also implements mask removal under negative point prompts.
• Point-Prompt-Based Mask Decoder (PMD): performs patch-level segmentation guided by positive point prompts.

The ViT-B image encoder is frozen and adapted via LoRA ($r=8$, $\alpha=32$); the prompt encoder is frozen unchanged; only the lightweight decoders receive full-parameter fine-tuning. The final mask is obtained by merging AMD and PMD outputs, with an optional Mask Fusion Module (MFM) available when precision is prioritized.

The central design decision is the patch constraint: each point prompt's influence is restricted to a fixed $32 \times 32$ patch $P_{i,j}$ determined by a deterministic mapping $g(\cdot)$ from pixel coordinates to patches. Positive prompts direct PMD to segment roads only within their patches; negative prompts direct AMD to remove road predictions only within their patches. This converts SAM's ambiguous global prompt semantics into controllable, spatially bounded edits — the property the authors identify as essential for fine-grained refinement and absent from prior SAM-based road extraction methods such as Road-SAM, TPP-SAM, GeoSAM, and RemoteSAM, none of which support true local interactive control.

Fine-tuning strategy with dynamically generated supervision

Because no dataset provides labels for point-prompt-conditioned edits, the training strategy synthesizes them on the fly. For each batch, the total number of sampled points $N$ and positive ratio $R$ are randomized around base values ($N_B=20$, $R_F=0.5$) using scaling factors $\delta_N=1.3$ and $\alpha=32$0. Points are sampled from road foreground pixels, mapped to their patches, and used to construct two label masks: $\alpha=32$1, which retains ground-truth road labels only within positively prompted patches, and $\alpha=32$2, which removes all negatively prompted patches from the ground truth. PMD is supervised by $\alpha=32$3 and AMD by $\alpha=32$4, teaching the model exactly where each prompt should and should not take effect.

The loss combines Focal Loss and Dice Loss (weighted 0.7/0.3) for AMD, PMD, HRMD, and MFM outputs, plus two auxiliary terms addressing specific failure modes:

• $\alpha=32$5 targets removed regions explicitly by supervising the negated sigmoid of AMD logits against $\alpha=32$6. The paper notes that without this term, small removed patches contribute negligibly to the loss and negative prompts fail to delete masks reliably.
• $\alpha=32$7 biases HRMD toward recall by filtering false positives out of the loss computation before optimization, so HRMD learns to cover as much road as possible at the cost of precision.

At test time, an automated prompting procedure simulates user interaction: morphological opening (implemented as GPU-parallel min/max pooling) filters spurious edge errors from the false-positive and false-negative maps, then one point per affected patch is sampled. Ablations show kernel size 3 is optimal for both maps, and that denser sampling degrades quality — a single positive point per patch suffices, consistent with the training distribution.

Quantitative results

On DeepGlobe, Massachusetts Roads, and CHN6-CUG, PC-SAM with automated point prompts substantially exceeds state-of-the-art fully automatic methods:

Method	DeepGlobe IoU / F1 (%)	Massachusetts IoU / F1 (%)	CHN6-CUG IoU / F1 (%)
OARENet [26]	69.98 / 81.58	64.14 / 77.85	54.96 / 71.79
MADSNet [27]	67.25 / 79.60	61.22 / 75.71	55.72 / 72.78
CGCNet [38]	67.66 / 79.87	59.95 / 74.76	58.50 / 71.23
Zero-shot SAM-H [17]	21.72 / 31.54	11.83 / 20.12	29.10 / 40.87
PC-SAM w/ prompts	75.27 / 85.59	65.37 / 78.88	67.49 / 81.47

Relative to OARENet, prompting yields IoU gains of +5.29%, +1.23%, and +12.53% on the three datasets respectively. The gain is driven primarily by recall: on CHN6-CUG, recall rises from roughly 67% (automatic-only) to 91.61%, while precision remains competitive. Without prompts, PC-SAM's automatic mode performs comparably to but not above the strongest baselines (e.g., 52.68% IoU on CHN6-CUG versus CGCNet's 58.50%), so the headline results are conditional on prompt availability — a dependency stated plainly by the authors throughout.

The comparison also illustrates why prior-specific automatic models generalize unevenly: OARENet's atrous strip convolutions excel on thin, occluded roads (DeepGlobe, Massachusetts) but weaken contextual modeling on CHN6-CUG's wider roads, while CGCNet's compact global context favors CHN6-CUG but loses detail on Massachusetts' extremely thin roads. PC-SAM sidesteps this trade-off by correcting residual errors interactively rather than encoding structural priors.

For local refinement specifically, PMD recovers difficult missed segments with 68.14% IoU on DeepGlobe and 61.06% IoU on CHN6-CUG, though performance drops to 39.18% IoU on Massachusetts, which the authors attribute to that dataset's very thin roads occupying a small area fraction. Negative-prompt removal consistently improves precision across datasets (e.g., 70.06% → 74.81% on CHN6-CUG), but slightly reduces recall and IoU because the automated prompt generator sometimes samples negatives near correct boundaries — a limitation of the simulation strategy rather than the model, per the authors. The HRMD reference masks achieve 97.86%, 91.31%, and 93.22% recall on the three datasets at low precision (~39–48%), confirming their intended role as coverage-oriented guides.

Ablation over fusion strategies shows direct mask merging outperforms MFM everywhere (e.g., 75.27% vs. 73.97% IoU on DeepGlobe; 67.49% vs. 49.88% on CHN6-CUG), since prompted regions are generally reliable; MFM nonetheless raises precision and is retained as an option.

Limitations and open questions

Several constraints qualify the reported results. The strong benchmark numbers depend on automatically generated point prompts whose placement assumes the morphological-opening heuristic correctly identifies error regions; the observed recall loss under negative prompting shows this heuristic is imperfect. Local refinement quality degrades markedly on scenes with very thin roads (Massachusetts), suggesting the fixed $\alpha=32$8 patch granularity may be too coarse or ill-suited for low road-area-fraction imagery, and the paper does not explore adaptive patch sizing. The evaluation also relies on simulated rather than human-in-the-loop prompting, leaving actual interactive usability and prompt-effort curves with real users unmeasured. Finally, the framework is demonstrated only on road extraction; whether the patch-constrained fine-tuning recipe transfers to other linear structures in remote sensing imagery remains an open question.

Conclusion

PC-SAM demonstrates that constraining point-prompt influence to fixed image patches, enforced through dynamically synthesized supervision during fine-tuning, converts SAM into a controllable refinement tool for remote sensing road segmentation. Within a unified architecture, it matches fully automatic baselines without prompts and surpasses them decisively with prompts — up to +12.53% IoU on CHN6-CUG — while uniquely supporting local inclusion, exclusion, and refinement of road masks. Its main caveats are dependence on prompt availability and the fixed patch granularity, both of which bound its applicability to fine-structured scenes.