Inter2Former: Efficient Interactive Segmentation

• The paper introduces Inter2Former, which integrates dynamic prompt embedding, dynamic hybrid attention, hybrid MoE, and dynamic local upsampling to achieve state-of-the-art precision on commodity CPUs.
• It employs ROI-based computation to reduce processing latency to about 75 ms per click, offering a 55–85% speedup over standard models and efficient trade-offs between accuracy and speed.
• Empirical results on HQSeg44K and DAVIS benchmarks validate the model's superior segmentation fidelity and boundary preservation in interactive segmentation tasks.

Inter2Former is a high-precision interactive segmentation (IS) architecture optimized for efficient CPU execution, designed to address the trade-off between accuracy and speed seen in prior dense-token and sparse prompt-token methods. It introduces dynamic computation allocation across every decoder stage, comprising Dynamic Prompt Embedding (DPE), Dynamic Hybrid Attention (DHA), Hybrid Mixture-of-Experts (HMoE), and Dynamic Local Upsampling (DLU). The model leverages region-of-interest (ROI) cropping, hybrid attention routing, and expert-based feed-forward processing to achieve state-of-the-art segmentation fidelity at near real-time speeds on commodity CPUs (Huang et al., 13 Jul 2025).

1. Two-Stage Architecture and Data Flow

Inter2Former follows a two-stage IS paradigm featuring a heavyweight encoder and a dynamic, lightweight decoder. The high-level data flow is as follows:

1. Encoding: The input image $I$ is processed via a ViT-Base encoder (specifically, HRSAM++ without SSM), which produces image tokens $F \in \mathbb{R}^{d \times h \times w}$.
2. Prompt and Reference Update: At each interaction step $k$, user clicks $C_k$ and the previous mask $M_{k-1}$ update a reference map $M_{\rm ref}$.
3. DPE (see §2): A tight ROI box $B$ is computed around all user prompt and prediction regions. The local patch $M_{\rm ref}[B]$ is embedded and convolved to yield local prompt tensor $F_B$, which is then scattered to a full-sized tensor $P$ by filling background positions with learnable $F \in \mathbb{R}^{d \times h \times w}$0.
4. Prompt Fusion and Decoder: The image and prompt tokens are fused ($F \in \mathbb{R}^{d \times h \times w}$1) and processed by $F \in \mathbb{R}^{d \times h \times w}$2 alternating layers of DHA and HMoE (see §3, §4), operating on boundary vs. non-boundary tokens according to an edge map $F \in \mathbb{R}^{d \times h \times w}$3 derived from $F \in \mathbb{R}^{d \times h \times w}$4.
5. DLU (see §5): The final token map $F \in \mathbb{R}^{d \times h \times w}$5 is decoded to a segmentation mask via a local upsampling strategy, guided by edge features and restricted to detected ROIs.

This workflow enables spatially focused computation, drastically reducing FLOPs and latency relative to dense-token approaches.

2. Dynamic Prompt Embedding (DPE)

DPE is a localized prompt representation strategy that encodes user clicks and prior predictions only within a tight ROI, replacing traditional global click-embeddings and avoiding unnecessary background processing. The method proceeds as:

• Construct a reference mask $F \in \mathbb{R}^{d \times h \times w}$6, which encodes click and predicted-region semantics.
• Detect the minimal bounding box $F \in \mathbb{R}^{d \times h \times w}$7 covering all positive/negative clicks and predicted foreground pixels, with fixed padding.
• Embed the local ROI using a fixed embedding and four stride-2 convolutions:

$F \in \mathbb{R}^{d \times h \times w}$8

where $F \in \mathbb{R}^{d \times h \times w}$9, $k$0.

• The resulting $k$1 is scattered to the full $k$2, filling non-ROI positions with $k$3 (a learnable embedding).

DPE reduces computation to a fraction of the full image area, with latency scaling linearly with ROI area—approximately 25% of full cost for small objects (Huang et al., 13 Jul 2025).

3. Dynamic Hybrid Attention (DHA)

DHA introduces token-wise routing of attention mechanisms based on spatial boundary information. Specifically, an edge map $k$4 is derived from the previous mask $k$5:

$k$6

Tokens are partitioned into boundary ($k$7) and non-boundary ($k$8) sets.

• Boundary tokens undergo standard full attention (complexity $k$9, $C_k$0).
• Non-boundary tokens employ BSQ attention, which reduces complexity to $C_k$1 using quantization schemes on a hypersphere and factorized, codebook-driven affinity computations.

The two streams are merged post-attention. This selective routing allocates high model capacity only where fine boundary discrimination is critical.

4. Hybrid Mixture of Experts (HMoE)

HMoE implements adaptive, token-wise expert selection in the feed-forward network modules. Input tokens $C_k$2 are routed as follows:

• Non-boundary tokens always use the shared expert $C_k$3.
• Boundary tokens compute affinities $C_k$4 to $C_k$5 routed experts and the shared expert. Expert $C_k$6 is selected as $C_k$7.
• Each boundary token output is a weighted combination of shared and routed expert outputs:

$C_k$8

For CPU efficiency, tokens are batched by expert assignment, enabling large matrix multiplies and parallelized computation.

Empirically, HMoE achieves 55–85% speedup over a standard MoE when $C_k$9, with sublinear latency scaling as the number of experts increases (Huang et al., 13 Jul 2025).

5. Dynamic Local Upsampling (DLU)

DLU applies targeted mask upsampling restricted to localized ROIs, functioning as an inverse of DPE:

• Localization: A lightweight MLP decodes $M_{k-1}$0 to a coarse low-res mask $M_{k-1}$1. A new bounding box $M_{k-1}$2 is computed over the predicted positive region.
• Refinement: Features in $M_{k-1}$3 are progressively upsampled using deconvolution, with multi-scale edge features $M_{k-1}$4 (precomputed from a CannyNet) fused at each stage via addition and convolutional refinement.
• The final high-res mask is scattered back to the $M_{k-1}$5 canvas, with zero elsewhere.

This selective upsampling keeps computation proportional to the target object area, maintaining both efficiency and boundary precision.

6. Computational Complexity and Efficiency

Per decoder layer, the complexity and CPU latency behavior of key modules are summarized as follows:

Module	Algorithmic FLOPs	CPU Latency Trend
DPE	$M_{k-1}$6 (ROI)	$M_{k-1}$7 area ratio; ~25%
DHA–FA	$M_{k-1}$8	Minor, small edge sets
DHA–BSQA	$M_{k-1}$9	Linear in $M_{\rm ref}$0
HMoE	$M_{\rm ref}$1	55–85% faster than baseline
DLU	$M_{\rm ref}$2	$M_{\rm ref}$3

Inter2Former achieves per-click inference at $M_{\rm ref}$475 ms for $M_{\rm ref}$5 images (20 clicks per session) on CPUs, compared to 1020 ms for InterFormer and 900 ms for SegNext.

7. Empirical Performance and Significance

On high-precision IS benchmarks including HQSeg44K and DAVIS, Inter2Former achieves state-of-the-art quality and efficiency:

Model	CPU Time (20-SPC/online, ms)	HQSeg44K 5-mIoU (%)	DAVIS 5-mIoU (%)
Inter2Former (1024)	75 / 50	91.48	90.82
HRSAM++-ViT-B (1024)	65 / 40	90.32	90.40
SAM-ViT-B	142 / 40	86.16	90.95

Ablation studies reveal that dense computation in all attention (All FA) or all BSQ (All BSQA) degrades either efficiency or accuracy; the dynamic routing of DHA and DPE/DLU is essential for an optimal trade-off. Qualitative results demonstrate crisp mask boundaries and preservation of thin structures, even with 20-click interaction limits.

Training utilizes large-scale datasets (COCO, LVIS, HQSeg44K), click simulation, and NFL loss; inference is measured on 8-core CPUs, enabling real-world annotation deployment scenarios.

Overall, Inter2Former establishes a new empirical operating point for interactive segmentation: SOTA precision, efficient CPU execution, and dynamic, adaptive computation allocation (Huang et al., 13 Jul 2025).