---
title: 'Inter2Former: Efficient Interactive Segmentation'
url: https://www.emergentmind.com/topics/inter2former
type: topic
---

# Inter2Former: Efficient Interactive Segmentation

Inter2Former is a high-precision interactive segmentation (IS) architecture optimized for efficient CPU execution, designed to address the trade-off between accuracy and speed seen in prior dense-token and sparse prompt-token methods. It introduces dynamic computation allocation across every decoder stage, comprising Dynamic Prompt Embedding (DPE), Dynamic Hybrid Attention (DHA), Hybrid Mixture-of-Experts (HMoE), and Dynamic Local Upsampling (DLU). The model leverages region-of-interest (ROI) cropping, hybrid attention routing, and expert-based feed-forward processing to achieve state-of-the-art segmentation fidelity at near real-time speeds on commodity CPUs [2507.09612].

## 1. Two-Stage Architecture and Data Flow

Inter2Former follows a two-stage IS paradigm featuring a heavyweight encoder and a dynamic, lightweight decoder. The high-level data flow is as follows:

1. **Encoding**: The input image $I$ is processed via a ViT-Base encoder (specifically, HRSAM++ without SSM), which produces image tokens $F \in \mathbb{R}^{d \times h \times w}$.
2. **Prompt and Reference Update**: At each interaction step $k$, user clicks $C_k$ and the previous mask $M_{k-1}$ update a reference map $M_{\rm ref}$.
3. **DPE (see §2)**: A tight ROI box $B$ is computed around all user prompt and prediction regions. The local patch $M_{\rm ref}[B]$ is embedded and convolved to yield local prompt tensor $F_B$, which is then scattered to a full-sized tensor $P$ by filling background positions with learnable $e_{\rm bg}$.
4. **Prompt Fusion and Decoder**: The image and prompt tokens are fused ($F_P = F + P$) and processed by $L$ alternating layers of DHA and HMoE (see §3, §4), operating on boundary vs. non-boundary tokens according to an edge map $E$ derived from $M_{k-1}$.
5. **DLU (see §5)**: The final token map $F_{R}$ is decoded to a segmentation mask via a local upsampling strategy, guided by edge features and restricted to detected ROIs.

This workflow enables spatially focused computation, drastically reducing FLOPs and latency relative to dense-token approaches.

## 2. Dynamic Prompt Embedding (DPE)

DPE is a localized prompt representation strategy that encodes user clicks and prior predictions only within a tight ROI, replacing traditional global click-embeddings and avoiding unnecessary background processing. The method proceeds as:

- Construct a reference mask $M_{\rm ref} \in \{0,1,2,3,4\}^{H \times W}$, which encodes click and predicted-region semantics.
- Detect the minimal bounding box $B = [x_1:x_2, y_1:y_2]$ covering all positive/negative clicks and predicted foreground pixels, with fixed padding.
- Embed the local ROI using a fixed embedding and four stride-2 convolutions:
  $$
  E_{\rm init} = \mathrm{Embed}(M_{\rm ref}[y_1:y_2, x_1:x_2]) \in \mathbb{R}^{5 \times H_B \times W_B}, \quad
  F_B = \mathrm{Conv}_4(E_{\rm init}) \in \mathbb{R}^{d \times h_B \times w_B}
  $$
  where $h_B = H_B / 16$, $w_B = W_B / 16$.
- The resulting $F_B$ is scattered to the full $P \in \mathbb{R}^{d \times h \times w}$, filling non-ROI positions with $e_{\rm bg}$ (a learnable embedding).

DPE reduces computation to a fraction of the full image area, with latency scaling linearly with ROI area—approximately 25% of full cost for small objects [2507.09612].

## 3. Dynamic Hybrid Attention (DHA)

DHA introduces token-wise routing of attention mechanisms based on spatial boundary information. Specifically, an edge map $E_{k-1}$ is derived from the previous mask $M_{k-1}$:
$$
E_{k-1} = \mathrm{Pool} \left( \mathbbm{1} \{ \mathrm{Conv}(M^2_{k-1}) - \mathrm{Conv}(M_{k-1})^2 > 0 \} \right) \in \{0,1\}^{h \times w}
$$
Tokens are partitioned into boundary ($Q_{\rm FA}$) and non-boundary ($Q_{\rm BSQ}$) sets.

- **Boundary tokens** undergo standard full attention (complexity $\mathcal{O}(n_{\rm edge}^2)$, $n_{\rm edge} \ll N$).
- **Non-boundary tokens** employ BSQ attention, which reduces complexity to $\mathcal{O}(N)$ using quantization schemes on a hypersphere and factorized, codebook-driven affinity computations.

The two streams are merged post-attention. This selective routing allocates high model capacity only where fine boundary discrimination is critical.

## 4. Hybrid Mixture of Experts (HMoE)

HMoE implements adaptive, token-wise expert selection in the feed-forward network modules. Input tokens $X \in \mathbb{R}^{N \times d}$ are routed as follows:

- **Non-boundary tokens** always use the shared expert $\mathrm{FFN}_M$.
- **Boundary tokens** compute affinities $s_{i,t} = \sigma(\mathbf{x}_t^\top e_i)$ to $M$ routed experts and the shared expert. Expert $a_t$ is selected as $a_t = \arg\max_{i<M} s_{i,t}$.
- Each boundary token output is a weighted combination of shared and routed expert outputs:
  $$
  \mathbf{y}_t = \frac{e^{s_{M,t}} \mathrm{FFN}_M(\mathbf{x}_t) + e^{s_{a_t,t}} \mathrm{FFN}_{a_t}(\mathbf{x}_t)}{e^{s_{M,t}} + e^{s_{a_t,t}}}
  $$
For CPU efficiency, tokens are batched by expert assignment, enabling large matrix multiplies and parallelized computation.

Empirically, HMoE achieves 55–85% speedup over a standard MoE when $M=64$, with sublinear latency scaling as the number of experts increases [2507.09612].

## 5. Dynamic Local Upsampling (DLU)

DLU applies targeted mask upsampling restricted to localized ROIs, functioning as an inverse of DPE:

- **Localization**: A lightweight MLP decodes $F_R$ to a coarse low-res mask $M_{\rm low-res} \in \mathbb{R}^{1 \times h \times w}$. A new bounding box $B'$ is computed over the predicted positive region.
- **Refinement**: Features in $F_R[B']$ are progressively upsampled using deconvolution, with multi-scale edge features $F^e_i$ (precomputed from a CannyNet) fused at each stage via addition and convolutional refinement.
- The final high-res mask is scattered back to the $H \times W$ canvas, with zero elsewhere.

This selective upsampling keeps computation proportional to the target object area, maintaining both efficiency and boundary precision.

## 6. Computational Complexity and Efficiency

Per decoder layer, the complexity and CPU latency behavior of key modules are summarized as follows:

| Module      | Algorithmic FLOPs                             | CPU Latency Trend            |
|-------------|-----------------------------------------------|------------------------------|
| DPE         | $\mathcal{O}(A_B \cdot d \cdot k^2)$ (ROI)    | $\propto$ area ratio; ~25%   |
| DHA–FA      | $\mathcal{O}(n_{\rm edge}^2 \cdot C)$         | Minor, small edge sets       |
| DHA–BSQA    | $\mathcal{O}(N \cdot S \cdot C + N \cdot d)$  | Linear in $N$                |
| HMoE        | $\mathcal{O}(N \cdot d \cdot \text{exp})$     | 55–85% faster than baseline  |
| DLU         | $\mathcal{O}(A_{B'} \cdot d \cdot k^2)$       | $\propto A_{B'}$             |

Inter2Former achieves per-click inference at $\sim$75 ms for $1024\times1024$ images (20 clicks per session) on CPUs, compared to 1020 ms for InterFormer and 900 ms for SegNext.

## 7. Empirical Performance and Significance

On high-precision IS benchmarks including HQSeg44K and DAVIS, Inter2Former achieves state-of-the-art quality and efficiency:

| Model                    | CPU Time (20-SPC/online, ms) | HQSeg44K 5-mIoU (%) | DAVIS 5-mIoU (%) |
|--------------------------|-----------------------------|---------------------|------------------|
| Inter2Former (1024)      | 75 / 50                     | 91.48               | 90.82            |
| HRSAM++-ViT-B (1024)     | 65 / 40                     | 90.32               | 90.40            |
| SAM-ViT-B                | 142 / 40                    | 86.16               | 90.95            |

Ablation studies reveal that dense computation in all attention (All FA) or all BSQ (All BSQA) degrades either efficiency or accuracy; the dynamic routing of DHA and DPE/DLU is essential for an optimal trade-off. Qualitative results demonstrate crisp mask boundaries and preservation of thin structures, even with 20-click interaction limits.

Training utilizes large-scale datasets (COCO, LVIS, HQSeg44K), click simulation, and NFL loss; inference is measured on 8-core CPUs, enabling real-world annotation deployment scenarios.

Overall, Inter2Former establishes a new empirical operating point for interactive segmentation: SOTA precision, efficient CPU execution, and dynamic, adaptive computation allocation [2507.09612].

Source: https://www.emergentmind.com/topics/inter2former