- The paper introduces VASA, a training-free vision harness that uses persistent working masks, Boolean editing, planning, constraint checking, and error recovery to construct task-specific segmentations.
- The paper shows that VASA outperforms agentic and open-vocabulary baselines on the PARS benchmark, improving ad-hoc mIoU by 13.5%, cIoU by 17.8%, and reducing xIoU by 25.3% versus SAM3 Agent.
- The paper demonstrates that persistent visual state makes long-form queries useful, with VASA improving gIoU by 5.9% while SAM3 Agent declines by 22.1%, though multi-round inference increases latency and remains domain-limited.
The problem: open ad-hoc segmentation
Segmentation of familiar visual wholes such as "cat" or "car" is now largely solved by text-conditioned foundation models, because these concepts are visually coherent and well represented in training data. Wang and Yu identify a harder setting they call open ad-hoc segmentation: segmenting open-ended, task-contingent concepts defined on demand through parts, relations, exclusions, and collections — for example, "the cat's head without the ears and eyes." In this setting, the target grounding may not exist as a single learned mask; it must be constructed from image evidence at inference time. This extends open ad-hoc categorization to pixel-level grounding, and it generalizes referring segmentation from grounding existing entities to composing arbitrary user-specified concepts.
The authors' diagnosis of existing agentic approaches is that they are language-led and visually stateless. SAM3 Agent, for instance, iteratively proposes revised text prompts to SAM3, but each trial is a fresh retrieval attempt: intermediate masks are neither preserved nor composed. Nothing visual persists across reasoning steps, so exclusion- and composition-defined targets cannot be built incrementally.
VASA: a vision harnessing agent
VASA (Vision harnessing Agent for open ad-hoc Segmentation) is a training-free framework coupling a VLM agent (Qwen3-VL 32B Thinking), SAM3 as the segmentation tool, and a vision harness workflow inspired by harness engineering in software engineering. Its central mechanism is a persistent working mask that records accumulated visual progress across interaction rounds.
The harness comprises six components:
- State management: persistent states include the image, query, current strategy, working mask, action history, and prior reasoning context.
- Tool calls and mask editing: beyond
segment_phrase and examine_each_mask, VASA adds update_working_mask, where the VLM selects candidate masks and an operation — Add, Remove, or Replace — while a deterministic program executes pixel-level Boolean updates.
- Long-horizon planning: the agent selects among strategies such as direct retrieval, undersegment-and-add, oversegment-and-remove, and coarse-to-fine refinement.
- Constraints and visual scrutiny: after each round, the working mask is checked against inclusion, exclusion, structural, and relational constraints in the query.
- Error recovery and stopping: retries and local reinitialization handle stalled progress and formatting errors; termination occurs on verification, stall, or budget exhaustion (20 rounds maximum).
The design claim is that the bottleneck for open ad-hoc segmentation lies not in the segmentation foundation model itself but in how it is harnessed — a claim the PARS results directly support, since raw SAM3 performs poorly there while VASA built atop it achieves strong results without any task-specific training.
The PARS benchmark
To evaluate open ad-hoc segmentation, the authors construct PARS from PartImageNet part-level annotations, focusing on the "body" class because its definition varies across object categories, making it inherently ad-hoc. Short category labels are replaced by long-form definition queries specifying what to include, what to exclude, and how the concept is structurally or contextually identified, generated semi-automatically via a VLM and manually verified. PARS contains 2,021 images split into 937 open ad-hoc and 1,084 common-concept examples.
Alongside gIoU and cIoU, the paper introduces xIoU, which measures the fraction of the prediction overlapping other annotated concept masks in the same image. xIoU captures fine-grained cross-concept confusion that causes only small IoU drops — such as failing to exclude ears from a head mask — though the authors note it can be trivially reduced by overly conservative masks and therefore use it only as a secondary diagnostic metric.
Results
On PARS, VASA outperforms all baselines across all metrics on both splits. On the ad-hoc split it improves over SAM3 Agent by 13.5% in mIoU and 17.8% in cIoU while reducing xIoU by 25.3%; overall gains over prior SOTA reach 25.1% in cIoU reduction of error. Notably, despite being training-free, VASA matches VLPart, a fully supervised closed-world method trained directly on PartImageNet — the same dataset from which PARS derives, so this comparison should be read as approximate rather than strictly fair.
On RefCOCOm, VASA again leads all subsets under both part-only and part-and-object settings, exceeding SAM3 Agent by 4.8–8.8% in gIoU and other agentic baselines by up to 20%. Gains are largest on testA part-level concepts (+8.8% gIoU), indicating that persistent visual state and iterative verification benefit conventional fine-grained referring segmentation as well.
Two analyses sharpen the picture. First, an ablation on query length shows a striking divergence: switching from short labels to long-form queries improves VASA by 5.9% gIoU but degrades SAM3 Agent by 22.1% gIoU. Long descriptions act as structured construction guidance for VASA but overwhelm prompt-retrieval agents. Second, larger numbers of additional reasoning steps relative to SAM3 Agent correlate positively with gIoU improvement, supporting the claim that extra long-horizon refinement is productive rather than wasteful for challenging ad-hoc concepts.
Limitations and open questions
The paper concedes three limitations. Iterative multi-round inference incurs latency incompatible with real-time use. Performance remains bounded by the underlying VLM and segmenter: if SAM3 cannot localize relevant primitives, or the VLM misreads mask overlays, iterative refinement may not recover. And PARS is derived solely from PartImageNet part concepts, leaving broader domains — object relations, functional concepts, human-centered instructions — untested. An additional open question concerns the correlation analysis: it shows association between more reasoning steps and better outcomes, but does not establish when additional rounds cease to help or actively harm, nor does it isolate which harness component contributes most.
Conclusion
VASA reframes open ad-hoc segmentation as iterative visual construction over a persistent working mask, coordinated by a vision harness that manages state, plans tool calls, enforces constraints, scrutinizes intermediate masks, and recovers from errors. Combined with the PARS benchmark and detailed long-form queries, it demonstrates substantial gains over open-vocabulary, reasoning-based, and agentic baselines, and matches supervised methods without task-specific training. The broader lesson the authors draw is that agent capability can be programmed through task knowledge, visual routines, working memory, and failure-aware workflows rather than through fine-tuning alone.