SAM3-Agent: Iterative Seed-Mask Segmentation

• SAM3-Agent is a control layer that uses iterative prompt actions to produce a precise binary seed mask from a given key frame and text description.
• It leverages an external planner (Gemini-3.1 Pro) and a SAM3 toolset to execute point, box, and scribble prompts, ensuring optimal tracker initialization.
• Its bounded iterative process with strict stopping criteria and mask scoring is essential for achieving high segmentation performance in video analysis.

Searching arXiv for the cited SAM3-Agent and related SAM3 papers to ground the article in current literature. arXiv_search({"query":"id:(He et al., 1 Apr 2026) OR id:(Li et al., 4 Dec 2025) OR Segment Anything Model 3 promptable concept segmentation Carion", "max_results": 10, "sort_by": "submittedDate"}) SAM3-Agent denotes the Stage 2 component of a fully training-free pipeline for referring video object segmentation under motion-centric language expressions, introduced in the winning solution to the 5th PVUW MeViS-Text Challenge (He et al., 1 Apr 2026). Its operational role is narrowly defined but technically consequential: given a key frame $I_k$ and a short discriminative description $P_k$, it produces a single high-quality binary seed mask $M_k$ on that frame, after which the official SAM3 tracker propagates the mask bidirectionally through the video (He et al., 1 Apr 2026). In this formulation, SAM3-Agent is not a standalone segmentation backbone; it is an agentic control layer that sits on top of the SAM3 image segmentation model, using a planner-driven reasoning loop to choose prompts, inspect intermediate masks, and terminate when the mask is precise enough (He et al., 1 Apr 2026). The term also appears in later discussion of instruction-following extensions of SAM3, where external agent loops are contrasted with single-pass instruction-grounding models such as SAM3-I (Li et al., 4 Dec 2025).

1. Position within the three-stage RVOS pipeline

In the MeViS-Text challenge solution, the overall method is organized into three stages (He et al., 1 Apr 2026). Stage 1 uses Gemini-3.1 Pro to decompose each target event into instance-level grounding targets, select the frame where the target is most clearly visible, and generate a discriminative description. Stage 2 consists of SAM3-Agent together with the official SAM3 tracker. Stage 3 performs self-refinement with Qwen3.5-Plus and behavior-level verification, and may invoke SAM3-Agent recursively if semantic checks fail (He et al., 1 Apr 2026).

The formal interface of SAM3-Agent is simple. Its inputs are a single frame $I_k \in \mathbb{R}^{H\times W\times 3}$, a short discriminative text description $P_k$, and optionally previous intermediate masks or prompts if the system is engaged in a multi-round dialogue (He et al., 1 Apr 2026). Its output is a binary seed mask $M_k \in \{0,1\}^{H\times W}$ that ideally exactly covers the referent in frame $k$; this seed is then handed to SAM3’s official tracker for propagation to all other frames $t \neq k$ (He et al., 1 Apr 2026).

This placement clarifies an important conceptual boundary. SAM3-Agent does not itself perform long-range temporal propagation; instead, it solves the frame-local disambiguation problem that determines tracker initialization quality. A plausible implication is that the quality of the entire video segmentation sequence is strongly conditioned on the precision of the seed mask, because the official tracker is initialized directly from $M_k$.

2. Internal organization and promptable control loop

SAM3-Agent is described as sitting “on top” of the SAM3 image segmentation model and is conceptually divided into two sub-modules (He et al., 1 Apr 2026). The first is a Planner, implemented in practice with Gemini-3.1 Pro, which reasons about which SAM3 operation or prompt should be invoked next. The second is a SAM3 toolset exposing at least four operations: point-prompt segmentation, box-prompt segmentation, scribble or free-form prompts, and mask-refinement operations including erode, dilate, and split (He et al., 1 Apr 2026).

At each round $t=1\ldots T$, with $P_k$0–$P_k$1, the Planner takes as input the image $P_k$2, the original text $P_k$3, and any intermediate masks $P_k$4, and emits one prompt action $P_k$5 (He et al., 1 Apr 2026). The action can correspond, for example, to placing a positive point at coordinates $P_k$6 or proposing a bounding box around a discriminative region. That prompt is then executed through SAM3’s vision encoder, prompt encoder, and mask decoder, yielding a new candidate mask $P_k$7 (He et al., 1 Apr 2026). The Planner inspects $P_k$8—for example via visualization, area ratios, or internal LLM scoring against $P_k$9—and either terminates with a final mask $M_k$0 or issues another prompt (He et al., 1 Apr 2026).

This architecture is significant because it externalizes instance-specific reasoning rather than embedding it into the segmentation network. In the terminology of the source, SAM3-Agent implements “a small reasoning loop” that adaptively chooses point, box, or scribble prompts, scores each mask against the text description, and stops when the mask is precise enough (He et al., 1 Apr 2026). That design is consistent with the broader characterization of pre-SAM3-I systems in which SAM3 relies on external multimodal agents to convert complex instructions into noun phrases and then conduct iterative mask filtering (Li et al., 4 Dec 2025).

3. Seed-mask generation mechanics

The seed-mask procedure is specified algorithmically in the challenge report (He et al., 1 Apr 2026). The Planner state is reset, an optional previous mask is initialized as empty, and the system iterates up to a maximum number of rounds. At each round, the Planner calls plan_step with the image, text, and previous mask; SAM3 is invoked with the resulting action; mask logits are thresholded to produce a binary mask; and the Planner evaluates that mask against the text prompt using a scoring routine (He et al., 1 Apr 2026). If the score exceeds the early-accept criterion, the current mask is returned; otherwise the loop continues. If no mask satisfies the criterion, the system returns the best mask found so far (He et al., 1 Apr 2026).

The mathematical formulation given for mask binarization is

$M_k$1

where $M_k$2 denotes the mask logits at round $M_k$3 (He et al., 1 Apr 2026). If a text-mask matching score is used, it is written abstractly as

$M_k$4

where $M_k$5 and $M_k$6 are off-the-shelf encoders such as CLIP-style encoders or internal LLM visual encodings, and acceptance is defined by a threshold $M_k$7 such that

$M_k$8

(He et al., 1 Apr 2026). The report further gives Intersection-over-Union as

$M_k$9

for checking refinement or no-target detection (He et al., 1 Apr 2026).

The report also specifies the internal components of SAM3 as used by the agent: a ViT-based Vision Encoder $I_k \in \mathbb{R}^{H\times W\times 3}$0 that maps $I_k \in \mathbb{R}^{H\times W\times 3}$1 to dense image tokens, a lightweight Prompt Encoder $I_k \in \mathbb{R}^{H\times W\times 3}$2 for points, boxes, and scribbles, and a Mask Decoder $I_k \in \mathbb{R}^{H\times W\times 3}$3 taking $I_k \in \mathbb{R}^{H\times W\times 3}$4 to mask logits $I_k \in \mathbb{R}^{H\times W\times 3}$5 (He et al., 1 Apr 2026). In this sense, SAM3-Agent is an orchestration mechanism over standard promptable segmentation primitives rather than a modification of the SAM3 core architecture.

4. Tracker interface and multi-instance handling

Once the seed mask $I_k \in \mathbb{R}^{H\times W\times 3}$6 is finalized, the system invokes the SAM3 video-tracking API “exactly as in Carion et al.’s reference code” (He et al., 1 Apr 2026). Tracker initialization with frame $I_k \in \mathbb{R}^{H\times W\times 3}$7 and mask $I_k \in \mathbb{R}^{H\times W\times 3}$8 produces a hidden mask embedding state $I_k \in \mathbb{R}^{H\times W\times 3}$9. The tracker then propagates forward to frames $P_k$0 and backward to frames $P_k$1, using $P_k$2 and the new frame $P_k$3 to output $P_k$4 at each time step (He et al., 1 Apr 2026). The result is a full sequence $P_k$5 for that instance.

The pipeline also specifies how decomposed prompts with multiple instances are handled. If Stage 1 decomposes the instruction into $P_k$6 instances, the system repeats one SAM3-Agent call plus one SAM3 tracker run per instance, then merges the $P_k$7 video-long masks by simple per-pixel OR or by highest-score assignment (He et al., 1 Apr 2026). This reveals that the challenge solution treats instance decomposition upstream and propagation downstream, with SAM3-Agent acting as the bridge between the two.

A plausible implication is that the architecture favors modularity over end-to-end joint optimization. Because no task-specific fine-tuning is performed, the instance decomposition, seed-mask generation, tracking, and refinement stages remain decoupled operational modules (He et al., 1 Apr 2026).

5. Hyperparameters, heuristics, and system behavior

The challenge report provides several implementation details and heuristics for SAM3-Agent (He et al., 1 Apr 2026). These include max_rounds = 6–8, with diminishing returns reported beyond 6–7 SAM3 calls; a mask threshold $P_k$8 for binarization; and an acceptance threshold described in code as accept_iou = 0.85 for early stopping (He et al., 1 Apr 2026). The mathematical discussion separately notes that when a text-mask matching score $P_k$9 is used, the experiments set $M_k \in \{0,1\}^{H\times W}$0–$M_k \in \{0,1\}^{H\times W}$1 (He et al., 1 Apr 2026). The report also states that no learning or fine-tuning is performed on any RVOS data and that the system is fully training-free (He et al., 1 Apr 2026).

Operationally, the Planner is Gemini-3.1 Pro accessed via API, and all Planner-to-SAM3 calls happen on $M_k \in \{0,1\}^{H\times W}$2 GPUs and return in $M_k \in \{0,1\}^{H\times W}$3 per round (He et al., 1 Apr 2026). The system includes an early “no-target” detection heuristic: if after round $M_k \in \{0,1\}^{H\times W}$4 no single connected region exceeds $M_k \in \{0,1\}^{H\times W}$5 pixels with $M_k \in \{0,1\}^{H\times W}$6, the Planner may decide that the target is absent (He et al., 1 Apr 2026). Prompt-to-mask calls are cached to avoid redundant inference, and the best_mask_so_far heuristic selects the $M_k \in \{0,1\}^{H\times W}$7 with the largest $M_k \in \{0,1\}^{H\times W}$8 (He et al., 1 Apr 2026).

These details matter because they show that SAM3-Agent is not merely a conceptual agent wrapper; it is a bounded iterative procedure with explicit stopping rules, thresholding, and resource-aware heuristics. The bounded-round design also distinguishes it from unconstrained conversational agent frameworks.

6. Performance in the PVUW MeViS-Text Challenge

The enclosing three-stage method, of which SAM3-Agent is the Stage 2 core, ranked first on the PVUW 2026 MeViS-Text test set without task-specific fine-tuning (He et al., 1 Apr 2026). The reported scores are a Final score of 0.909064 and a $M_k \in \{0,1\}^{H\times W}$9 score of 0.7897 (He et al., 1 Apr 2026). Because the report attributes the pipeline’s success to the combination of event decomposition, SAM3-Agent seed-mask generation plus tracking, and self-refinement, those numbers should be interpreted as end-to-end pipeline results rather than isolated SAM3-Agent-only measurements.

Within that context, SAM3-Agent’s role is specifically to produce a precise seed mask from a motion-centric textual description on a selected key frame (He et al., 1 Apr 2026). The report’s emphasis on a “single, high-quality, pixel-wise ‘seed’ mask” suggests that the seed is treated as the principal control variable for downstream temporal propagation. This suggests that the challenge solution treats initialization quality as a more tractable lever than directly solving long-horizon joint language-video segmentation in a single monolithic model.

7. Relation to SAM3-I and the transition from agentic to single-pass instruction following

SAM3-Agent occupies a specific place in the evolution of SAM3-based language-grounded segmentation. In the SAM3-I paper, vanilla SAM3 is described as handling short noun phrases, while richer expressions involving attributes, spatial relations, functionalities, actions, states, and implicit reasoning typically require external multimodal agents that rewrite the instruction into a noun phrase and repeatedly filter masks (Li et al., 4 Dec 2025). In that framing, SAM3-Agent is exemplary of an external-agent approach: language-level disambiguation is performed outside the segmentation model through iterative prompt selection and filtering.

SAM3-I is presented as removing that external agent loop altogether by augmenting SAM3 with instruction-aware cascaded adapters that allow it to interpret concept instructions, simple referring instructions, and complex instructions within a single forward pass (Li et al., 4 Dec 2025). The paper states that SAM3-I is “the first extension of the SAM family that natively ingests free-form, multi-condition instructions and grounds them in a single-pass segmentation model, all while preserving SAM3’s original concept-prompting ability” (Li et al., 4 Dec 2025). On the PACO-LVIS-Instruct test split, SAM3-I is reported to preserve SAM3’s original concept-prompt scores exactly and to outperform SAM3+Agent baselines on simple and complex instructions, with simple-instruction gIoU 54.0 and complex-instruction gIoU 51.0, compared with 41.6 and 48.2 respectively for SAM3+Agent using Qwen3-VL 8B (Li et al., 4 Dec 2025).

This comparison clarifies a frequent misconception. “SAM3-Agent” is not a canonical built-in SAM3 module in the same sense as SAM3-I’s adapters; rather, in the challenge report it is an external planner-plus-toolset layer wrapped around SAM3 (He et al., 1 Apr 2026). Conversely, SAM3-I uses the term “SAM3-Agent” more loosely when discussing agentic baselines and the broader class of systems that rely on larger external LLMs for instruction decomposition and mask filtering (Li et al., 4 Dec 2025). The distinction is architectural: agentic SAM3 systems preserve the original segmentation model and add a reasoning loop outside it, whereas SAM3-I internalizes instruction following through adapter-based fine-tuning (Li et al., 4 Dec 2025).

From this contrast, a plausible implication is that SAM3-Agent is best understood as a pragmatic systems solution for complex instruction grounding under a training-free regime, while SAM3-I represents an attempt to absorb that capability into the model itself through supervised adaptation.