FARM: Find Anything using Relational Spatial Memory

Abstract: Robots operating in homes, warehouses, and other object-rich environments need memory systems that can find specific object instances on demand. Object-level memory alone is often insufficient: scenes contain many plausibly matching objects, and users refer to the target through relations to landmarks and surrounding objects (e.g. ``the tall lamp below the dartboard and to the left of the poster''), demanding a relational spatial memory that supports retrieval through semantic, appearance, and spatial predicates over objects. To achieve this, we present FARM (Find Anything using Relational Spatial Memory), which builds, in real time at 5-10 Hz, a compact, open-vocabulary, object-level memory with geometry, visual-language descriptors, and viewpoint evidence. At query time, FARM uses VLMs to parse the query and score visual evidence, while grounding spatial constraints explicitly through object symbols and relational predicates. This structured use of VLMs enables more accurate and robust retrieval than end-to-end reasoning over frame histories or scene-graph context. In experiments on 44k language queries spanning 67 indoor and outdoor scenes, ranging from 15 to 15,000 m^2, FARM improves Recall@5 and Recall@10 over prior methods by 164% and 224%, and a final VLM reranking stage improves Accuracy@1 by 35%, while running in real time. We further demonstrate closed-loop deployment on a quadrupedal robot using onboard sensors and compute.

• The paper introduces FARM, a system that enables the retrieval of specific objects from free-form language queries by constructing an online object-centric scene graph with relational spatial memory that scales seamlessly across environments, yielding 142% improved accuracy on target retrieval benchmarks relative to BBQ.
• FARM's success relies on key techniques: the online memory construction, a Gaussian representation, and decoupled retrieval stages, where the query is parsed into typed query graphs, candidate bindings are scored, and verification is performed.
• The research highlights the advantage of structuring VLMs and decoupling query processing over end-to-end reasoning, showcasing a 66% improvement in Accuracy@1 over DAAAM with GPT-5-mini, improving operational efficiency and robustness in real-world applications.

Overview and motivation

FARM (Find Anything using Relational Spatial Memory) addresses a specific gap in robot spatial perception: retrieving a particular object instance from free-form relational language queries, such as "the tall lamp below the dartboard and to the left of the poster," in environments containing many plausibly matching candidates. The authors argue that object-level memory alone is insufficient for this task because users specify targets compositionally through semantic, appearance, and spatial relations to anchors. FARM builds an open-vocabulary, object-centric scene graph online at 5–10 Hz from posed RGB-D streams, then grounds queries by compiling them into executable relational predicates over the memory rather than delegating reasoning end-to-end to a vision-LLM (VLM). The system is evaluated on 44k language queries across 67 indoor and outdoor scenes spanning 15 to 15,000 m², and deployed closed-loop on a Boston Dynamics Spot quadruped with onboard compute.

Method

Online memory construction

Each entity in the memory stores: (i) a single 3D Gaussian $\mathcal{N}(\mu^i, \Sigma^i)$ summarizing location and extent; (ii) a detection-time appearance feature; (iii) up to $k$ viewpoint-diverse RGB crops; (iv) an open-vocabulary caption; and (v) three retrieval embeddings (Qwen3-text caption embedding, SigLIP2 image embedding, Qwen3-VL image embedding). Relations are limited to cheaply derivable covisibility and adjacency edges; higher-order relations such as containment or left-of are deliberately excluded because their count grows combinatorially, and instead are recovered at query time from stored geometry.

The per-frame loop — detect, lift, associate, fuse — is fully GPU-vectorized, while VLM captioning runs asynchronously off the critical path via vLLM workers. A key design choice enabling cross-scale generalization is the Gaussian representation itself: it is resolution-free (no voxel size or density hyperparameter), admits $O(1)$ closed-form updates via sufficient statistics, and supports scale-invariant association through the Hellinger distance, which is bounded in $[0,1]$. This is what allows one fixed hyperparameter configuration to work across ScanNet rooms (~15 m²), HM3D apartments, and outdoor sites up to 15,000 m², without offline post-processing.

Relational retrieval

Retrieval proceeds in three decoupled stages. First, a Qwen3.5-9B parser compiles the query into a typed query graph $(x_\star, \{a_i\}, \mathcal{S}_q, \Phi_q)$ over a fixed vocabulary of sixteen spatial/semantic predicates (\textsf{Near}, \textsf{On}, \textsf{Above}, \textsf{Below}, \textsf{LeftOf}, \textsf{Between}, etc.) with open-vocabulary arguments. Second, candidate bindings are scored softly: unary description scores come from reciprocal-rank fusion over the three stored embeddings, and relational scores come from closed-form evaluators over entity Gaussians (e.g., Hellinger distance for \textsf{Near}, virtual-viewer frames for \textsf{LeftOf}/\textsf{RightOf}). Because typical referring expressions form star-shaped query graphs, conditioning on the target decomposes anchor binding, reducing cost from $O(K_t K_a^m)$ to $O(|\Phi_q| K_t K_a)$. Third, a Qwen3.5-9B verifier reranks the top-5 candidates using rendered projected views with colored mask overlays.

A notable design decision is that predicates act as soft scores, not hard filters: the composite score $s = s_{\mathrm{sem}}[(1-w) + w\bar{g}]$ with $w=0.5$ bounds the damage any noisy anchor or predicate can inflict at a factor of two, keeping anchor errors out of the recall term and confined to ranking.

Results

Across ScanNet-30, HM3D-30, and FARM-Scenes under visible-mask IoU at threshold 0.1, FARM reaches 35.9% Accuracy@1 on ScanNet-30 (23% relative over DAAAM+RynnBrain's 29.3%), 7.9% on HM3D-30, and 24.2% on FARM-Scenes. Relative to BBQ, it improves A@1, R@5, and R@10 by 142%, 164%, and 224%. Against the costlier DAAAM with GPT-5-mini (66 s/query versus FARM's ~2.6 s), evaluated on a 17-scene subset, gains are 66%, 65%, and 63%. Gains widen at stricter IoU thresholds: at $\tau=0.50$ on ScanNet, FARM obtains 13.0% A@1 versus 0.25% for BBQ and 1.4% for RynnBrain-30B, indicating the improvement reflects spatially precise grounding rather than approximate relevance. Mapping runs at roughly 8 Hz (~121 ms/frame on ScanNet/HM3D, 59 ms/frame on FARM-Scenes) with compact memory footprints (~23 MiB on ScanNet, ~125 MiB on HM3D).

Two ablation findings deserve emphasis. First, with merge and retrieval embeddings held fixed, switching the mapper from BBQ to FARM improves A@1 by 4.3× on ScanNet-5 (5.4 → 23.0), attributing most of the gap to memory construction quality rather than retrieval mechanics. Second, and more contrarily, end-to-end VLM reasoning can hurt: on ScanNet-5, VLM reranking degrades A@1 relative to locked retrieval alone (Qwen@5: 41.6 → 40.6; RynnBrain@10: 41.6 → 31.6), even though it adds 1.2–1.7 points on HM3D-5 and substantially more on FARM-Scenes (10.8 → 15.5). The authors accordingly treat reranking as a deployment knob useful only where retrieval needs a final tiebreaker. Embedding choice also matters sharply: pure SigLIP2 visual embeddings collapse on HM3D-5 (0.3 A@1), while caption-text encoders reach 3.3–3.6, and multi-embedding fusion gives the best result (4.1).

Failure analysis localizes bottlenecks differently per benchmark: on ScanNet, 82.0% of ground-truth objects enter the candidate set and 74.6% survive into the top-10, so the bottleneck is top-1 disambiguation; on HM3D, only 45.3% enter the candidate set, making memory construction and candidate recall the dominant failure mode. On NR3D human-written descriptions, FARM (0.252) slightly trails video-based baselines (RynnBrain 0.262, DAAAM+RynnBrain 0.271), a concession that the pipeline is strongest when queries carry explicit relational structure.

Real-world deployment

FARM runs fully onboard a Spot quadruped with a Manifold Tech Odin-1 sensor and NVIDIA Jetson Thor, including a local vLLM server hosting the captioner and embedding models. Memory is built online during teleoperation, after which the robot autonomously grounds relational queries (e.g., "cardboard package next to a humanoid robot," where several same-category distractors exist) and navigates to saved viewpoints via a simple Dijkstra graph planner. The trial is qualitative rather than a quantitative benchmark, and the navigation stack is deliberately minimal.

Limitations and open questions

The paper concedes two main limitations. Spatial predicates are manually specified and uncalibrated: \textsf{Near} can decay too quickly with distance, and uniform weighting between semantic and spatial scores lets semantically similar distractors dominate when spatial constraints favor the true target. Learning calibrated predicate functions and query-dependent weights remains open. Second, the implementation handles only target-to-anchor relations; queries requiring reasoning among anchors themselves (e.g., disambiguating "a sofa that a humanoid robot sits on" before grounding the target) are unsupported by both the system and existing benchmarks. Additional open issues evident from the results include the candidate-recall bottleneck in large scenes, degraded performance on free-form human language, and the fact that HM3D evaluation relies on rendered trajectories whose lower image quality confounds scale-related conclusions.

Conclusion

FARM demonstrates that structuring VLM use — parsing queries symbolically, scoring predicates explicitly over memory, and reserving VLM verification for a small candidate set — yields more accurate and robust relational retrieval than end-to-end reasoning over frame histories or scene-graph context, while maintaining real-time, single-configuration operation from indoor rooms to large outdoor environments. Its state-of-the-art results on ScanNet and HM3D referring-expression benchmarks, the release of the FARM-Scenes benchmark for large-scale relational grounding, and the fully onboard quadrupedal deployment collectively establish that persistent, queryable object-level memory is a viable alternative to implicit video-context grounding for embodied agents operating at scale.