- The paper introduces Seeker, a lightweight action-supervised readout that uses gated attention over frozen DINOv3 features and diffusion-based action prediction to discover control-relevant regions without spatial annotations.
- Seeker raises simulated manipulation success from 42.6% to 62.6% and real-world in-domain success from 48.3% to 76.7%, while improving out-of-distribution performance from 20.0% to 60.0% under lighting and background changes.
- The learned ROI transfers across RGB crops, background augmentation, and point-cloud filtering, but requires separate pretraining, adds about 12 ms per step, and may struggle when actions are predictable from proprioception alone.
Seeker is a lightweight, action-supervised visual readout that learns where a manipulation policy should look without any spatial labels. The paper's central argument is that the observation–action stream in imitation data is an implicit but control-aligned source of supervision for localization: actions are generated from the cues the demonstrator used, so training a readout to predict actions forces it to surface control-relevant regions of interest (ROIs). Unlike action-derived heuristics such as Q-attention or RVT-2-style keyframe crops, Seeker does not hand-design event thresholds, proxy points, or crop scales; unlike gaze-, VLM-, or affordance-based interfaces, it requires no external annotations. The learned ROI is then reused as a spatial interface for RGB cropping, mask-guided background augmentation, and point-cloud filtering.
Method
Seeker operates over frozen DINOv3 patch features. A task- and state-conditioned query is formed by FiLM-modulating a CLIP task embedding with proprioception (end-effector translation and gripper state), then iteratively refined for T steps through gated multi-head cross-attention over patch keys and values. Query-dependent head gating scores each attention head against the mean query component and fuses per-head contexts and attention maps, allowing different heads to dominate as task stage and robot state change. After refinement, the fused context is passed to a diffusion action head trained with the standard noise-prediction objective; this head exists only to provide dense action supervision and is discarded after pretraining.
The final gated attention map is converted into a patch-level ROI via nucleus-style selection: the smallest token set whose cumulative mass exceeds top_p=0.8 yields a bounding box and coarse mask. Training uses two additional mechanisms: a coarse-to-fine staged scheme with attention trimming (a KL loss that suppresses out-of-box mass after a fine branch is introduced), and Random Overlay augmentation from the start of training, which the authors show is crucial to prevent early attention collapse onto spurious artifacts. A sequential view-training protocol—third-person first, then eye-in-hand with cached third-person context—keeps the standalone third-person ROI reliable despite cross-view complementarity.
The resulting frozen module serves three downstream purposes: RGB cropping with FiLM conditioning on box location/scale (eye-in-hand views are not cropped), point-cloud filtering by image-plane reprojection before DP3's farthest-point sampling, and mask-guided augmentation that preserves predicted control-critical regions while perturbing background appearance.
Simulation results
On six MimicGen tasks with 100 demonstrations per task, under a controlled ResNet-18 + Diffusion Policy stack, Seeker raises average success from 42.6% (best non-Seeker baseline, RVT2-Crop) to 62.6%, a mean task-wise relative gain of 52.8%. It also exceeds the external equivariant-policy reference RAVEN (62.6 vs. 52.1) and comes within 1.6 points of a privileged Oracle ROI built from stage-aware affordance pixels—despite using no scripted boxes, gaze, language grounding, or annotations. Gains are largest on contact-rich, spatially ambiguous tasks (Three-Piece Assembly: 58.7 vs. 26.7; Threading: 38.0 vs. 12.7), consistent with the claim that fixed keyframe/proxy heuristics fail when continuous motion lacks discrete events or when relevant evidence lies away from the TCP proxy.
Three further results support the reusability claim. First, frozen Seeker masks guide augmentation under shuffled tabletop textures and remain strongest on average even when combined with background-randomized training; notably, the mixed effect of randomized training across tasks suggests visual diversity alone does not guarantee background generalization. Second, without any retraining, Seeker's image-plane box filters point clouds on top of manual workspace cropping and improves DP3 by 30.3 absolute points on average across three tasks at 200 demonstrations. Third, ablations show that low-resolution cropping nearly matches full Seeker performance (61.2 vs. 62.6 average), whereas FiLM-only box conditioning drops to 39.4%—indicating the benefit comes primarily from exposing the progression-aware ROI as an explicit crop rather than merely informing the network where the box is. An objective ablation finds generative objectives essential: direct behavioral-cloning regression produces noisy ROIs, IMLE cleaner but diffuse ones, and flow matching the tightest masks (mean IoU to diffusion-trained Seeker of 0.67 vs. 0.50 and 0.30).
Real-world results
On a UFactory xArm7 with third-person and eye-in-hand cameras across three tasks (Coffee Transport, Table Cleanup, Board Assembly), Seeker improves average in-domain success from the best baseline's 48.3% to 76.7%, and average OOD success under lighting and background shifts from 20.0% to 60.0%, with a retention ratio of 78.2%. The paper attributes RVT2-Crop's failures to two concrete mechanisms: inconsistent demonstration velocity profiles creating false-positive low-speed keyframes ("ghost" targets), and continuous spoon motion in Coffee Transport providing no gripper-state anchor, so the heuristic locks onto rotation-only or final end-effector poses. Qualitatively, Seeker switches between object-centric and interaction-centric focus depending on the task—for example covering both spoon tip and cup during transport—which supports the interpretation that the bottleneck tracks control demands rather than semantic categories.
Limitations
The authors are explicit about several constraints. In spatially invariant phases where actions are predictable from proprioception, there is little incentive to localize visual evidence; new layouts varying those targets may require retraining or additional spatially varied demonstrations. A coverage diagnostic shows task-specific Seeker pretraining can shrink to 25 spatially diverse demonstrations within 0.4 points of full-100 performance, but reducing the multi-task setting additionally requires preserving task balance and mitigating cross-task interference, which remains unquantified. Seeker adds a separate pretraining stage (roughly 1.5–1.8× total training overhead in the conservative protocol) and about 12 ms per step at inference. Its pooled context is not a standalone controller—direct rollout achieves near-zero success (max 20% on Stack Three)—so it functions strictly as a localization bottleneck, discarding fine-grained geometric cues needed for control. Point-cloud policies still lag image-based policies overall, with residual failures in orientation inference and sparse task-relevant points in large workspaces.
Conclusion
Seeker demonstrates that action supervision alone can recover policy-useful, progression-aware visual bottlenecks that approach privileged oracle ROIs, transfer across input modalities, and substantially improve robustness under real appearance shifts. The results position action-grounded ROI discovery as a practical interface between perception and policy learning, while leaving open how spatial coverage, task balance, and cross-task interference jointly govern multi-task pretraining cost.