SOCRATES: Robot Dog Human Search Framework

• SOCRATES is a robotics framework that interprets free-text descriptions to locate and approach humans using language-mediated and vision-based techniques.
• The system employs a two-stage strategy, using an LLM-derived search prior and VLM verification to efficiently navigate and minimize false positives.
• Its approach module integrates imitation learning with language distillation to generate socially acceptable motions, validated in simulation and real-world trials.

SOCRATES, short for “Text-based Human Search and Approach using a Robot Dog,” is a robotics framework for locating a target person from a free-form textual description and then approaching that person in a socially acceptable manner. The system takes text that combines appearance and location clues, searches for the described individual with a language-mediated zero-shot pipeline, and, after user confirmation, switches to a hybrid learning-based approach planner designed to produce target-cordial robotic motion. The framework was validated in simulation, in a real-world lab, and with the Boston Dynamics Spot robot (Park et al., 2023).

1. Problem formulation and operational setting

SOCRATES models person-finding as a two-stage task. Its input is a free-form textual description $t=(t_1,t_2)$, where $t_1$ is an appearance description such as “wearing a white shirt with black hair,” and $t_2$ is a location clue such as “is a student who works with robots” (Park et al., 2023). The robot is assumed to have an annotated 2D floorplan partitioned into $K$ labeled regions $\{a_i\}_{i=1\dots K}$, its own pose $p_r=(x,y,\theta)$ on the map, an onboard RGB-D camera, and a “general human” detector implemented with YOLOv5.

The outputs are differentiated by stage. For search, SOCRATES produces a sequence of waypoints $p=(x,y,\theta)$ that drives the robot until the target is detected and user-confirmed within a $5\,\mathrm m$ radius. For approach, it produces a dynamically generated velocity-level motion that brings the robot to approximately $0.6\,\mathrm m$ from the target in a socially acceptable way (Park et al., 2023).

This decomposition is central to the framework. Search is formulated as zero-shot, language-mediated person localization over a mapped environment, while approach is formulated as socially aware local motion generation after the robot is already within interaction range. A common misunderstanding is to treat SOCRATES as a monolithic end-to-end policy; the system instead composes a search module and an approach module with an explicit confirmation handoff.

2. Human Search Socratic Model

The Human Search Socratic Model casts “find the person described by $t$” as a zero-shot, language-mediated search problem by composing three off-the-shelf models: an LLM-derived search prior, a VLM-based text-guided localizer, and a waypoint generator (Park et al., 2023).

For the search prior, GPT-3 is prompted with the floorplan labels $t_1$0 and the location clue $t_1$1. From $t_1$2 sentence-level answers $t_1$3, SOCRATES computes an occurrence likelihood for each region:

$t_1$4

where $t_1$5 is a pretrained word2vec embedding and $t_1$6 is the $t_1$7-th word of $t_1$8 (Park et al., 2023). This converts a free-form semantic clue into a region-wise prior over the annotated map.

For text-guided localization, SOCRATES uses a zero-shot VQA model, PNP-VQA, together with Grad-CAM. The robot asks the question “Is a person $t_1$9?.” If the VQA answer is “yes,” the Grad-CAM activation map $t_2$0 over image $t_2$1 is thresholded at $t_2$2 to form a bounding box $t_2$3 (Park et al., 2023). Appearance matching is therefore not trained specifically for the downstream task; it is induced from the composition of a VQA model and saliency extraction.

The waypoint generator combines global and local search. Globally, the cost of visiting region $t_2$4 from the current robot pose is

$t_2$5

where $t_2$6 is the closest reachable pose in region $t_2$7 and $t_2$8 is a tunable weight (Park et al., 2023). The robot repeatedly chooses the unvisited label with minimum cost, navigates there, and switches to local search.

Local search is indirect and human-centric. The robot first detects any person bounding box with YOLOv5. If a person is found, it moves to maintain a $t_2$9 distance and re-centers the person in view, then runs the text-based VLM check; if a match is found, it asks the user “Is this X?” If no person is in view, it performs frontier-based exploration within that region, excluding waypoints that are less than $K$0 away from previously visited waypoints (Park et al., 2023). This local procedure is designed to reduce false user queries while preserving search efficiency.

3. Hybrid learning-based framework for approach

Once the target is identified and the robot is within $K$1, SOCRATES invokes a learned cordial approach planner. The state is

$K$2

where $K$3 is the robot pose in human-centric coordinates, $K$4 indicates whether the person is gazing at the robot, and $K$5 is the robot’s current speed. The system learns a state-only reward $K$6 (Park et al., 2023).

Input processing combines perception and scene summarization. The person’s 3D orientation is estimated with a pre-trained face mesh network, and gaze is reduced to a binary variable using a $K$7 threshold. The current image is captioned with BLIP, producing a set of scene captions $K$8 (Park et al., 2023). These cues condition both the imitation-based reward and the language-distilled reward.

The learning-from-demonstration component uses KDMRL. Expert approach trajectories $K$9 are collected with human feedback, and the empirical state density is estimated by kernel density estimation:

$\{a_i\}_{i=1\dots K}$0

with $\{a_i\}_{i=1\dots K}$1, $\{a_i\}_{i=1\dots K}$2 the trajectory length, $\{a_i\}_{i=1\dots K}$3, and $\{a_i\}_{i=1\dots K}$4 an RBF kernel (Park et al., 2023). SOCRATES then solves a density-matching inverse-RL problem in an RKHS and represents the resulting reward as

$\{a_i\}_{i=1\dots K}$5

The knowledge-distillation component queries GPT-3 for each caption $\{a_i\}_{i=1\dots K}$6 and gaze bit $\{a_i\}_{i=1\dots K}$7, asking what trajectory the robot should take “to gently approach a person.” The responses are reduced to keywords such as “straight,” “curve,” “slowly,” and “front,” which are mapped by a small dictionary $\{a_i\}_{i=1\dots K}$8 to short $\{a_i\}_{i=1\dots K}$9-pose segments $p_r=(x,y,\theta)$0. These segments are stitched into sample trajectories, and the visited RKHS states accumulate a reward signal $p_r=(x,y,\theta)$1, which is then normalized and smoothed with an RBF (Park et al., 2023). The resulting planner is therefore neither purely demonstrative nor purely language-specified.

The final reward combines the two sources:

$p_r=(x,y,\theta)$2

Motion planning is performed with FMT* in OMPL on the local egocentric map, using the cost

$p_r=(x,y,\theta)$3

where

$p_r=(x,y,\theta)$4

with $p_r=(x,y,\theta)$5, $p_r=(x,y,\theta)$6, and $p_r=(x,y,\theta)$7 (Park et al., 2023). The approach module thus optimizes a reward-shaped geometric motion cost rather than directly imitating raw trajectories.

4. Search-to-approach integration

SOCRATES explicitly couples the two stages at the moment of human confirmation. As soon as the user answers “Yes, that is X,” the current robot pose $p_r=(x,y,\theta)$8 and the last camera frame are passed to the approach module (Park et al., 2023). The approach planner then constructs the initial human-centric state $p_r=(x,y,\theta)$9 from $p=(x,y,\theta)$0 and the human pose $p=(x,y,\theta)$1, obtains the gaze bit from face mesh, generates captions with BLIP, and solves planning under the combined reward $p=(x,y,\theta)$2.

The resulting velocity commands are sent to the Spot robot until the final goal—approximately $p=(x,y,\theta)$3 in front of the person—is reached (Park et al., 2023). This handoff is operationally significant because search uses region-level map priors and image-based re-identification, whereas approach uses human-centric state variables, gaze, speed, and local trajectory optimization. The framework therefore switches representational regimes rather than extending a single global planner all the way through person approach.

A second common misconception is that the search component alone defines SOCRATES. In fact, the paper treats human search and socially acceptable approach as a single integrated system, and the social-navigation component is not a post hoc add-on but one of the core contributions (Park et al., 2023).

5. Experimental validation

The search module was evaluated in a real-world lab with two floors and five area labels, using $p=(x,y,\theta)$4 people and $p=(x,y,\theta)$5 start positions for a total of $p=(x,y,\theta)$6 trials. Simulation was performed in Gazebo in two buildings: a household environment with $p=(x,y,\theta)$7 people and $p=(x,y,\theta)$8 starts, and an office environment with $p=(x,y,\theta)$9 people and $5\,\mathrm m$0 starts. The comparison methods were CoW, CoW + indirect, and a knowledge-prior baseline (Park et al., 2023).

Search performance was measured with Success Rate (SR), Success Path Length (SPL), and False Detections (#FD). In the real-world results, SOCRATES achieved the highest SR and SPL and the lowest #FD among the reported methods (Park et al., 2023).

Method	SPL	SR	#FD
CoW (direct)	0.314	0.40	0.80
CoW+indir.	0.295	0.60	0.30
Knowledge prior	0.636	0.90	0.80
SOCRATES (ours)	0.647	1.00	0.20

The simulation results showed the same ordering, with SOCRATES achieving approximately $5\,\mathrm m$1 SR and $5\,\mathrm m$2 SPL (Park et al., 2023). This suggests that the combination of an LLM-derived regional prior and indirect human-centered local search produced both better path efficiency and fewer unnecessary user confirmations than the comparison conditions.

The real-world approach study involved $5\,\mathrm m$3 human participants, split into robot-experienced ($5\,\mathrm m$4) and inexperienced ($5\,\mathrm m$5) groups. The experiment used a Latin-square balanced design with $5\,\mathrm m$6 methods, $5\,\mathrm m$7 gaze states, and $5\,\mathrm m$8 total trajectories per participant, giving $5\,\mathrm m$9 trials. The four methods were a direct position-control baseline at constant speed, KD only, LfD only, and the hybrid SOCRATES approach planner (Park et al., 2023).

Evaluation used the Robotic Social Attributes Scale (RoSAS), a $0.6\,\mathrm m$0-point Likert instrument over $0.6\,\mathrm m$1 adjectives summarized into Competence, Warmth, and Discomfort. Aggregate results are as follows (Park et al., 2023).

Method	Competence	Warmth	Discomfort
Baseline	4.41 (1.08)	3.34 (1.23)	3.23 (1.23)
KD only	3.96 (0.82)	3.28 (1.03)	3.59 (1.08)
LfD only	4.14 (1.01)	3.28 (1.29)	3.30 (1.54)
Ours	4.56 (1.28)	3.63 (1.28)	2.96 (1.28)

The hybrid method scored highest in Competence and Warmth and lowest in Discomfort. The experienced group preferred the hybrid method by approximately $0.6\,\mathrm m$2 points in Competence, whereas the inexperienced group showed a smaller absolute preference gap (Park et al., 2023).

6. Findings, limitations, and prospective extensions

The reported findings are threefold. First, incorporating LLM-derived priors improved search efficiency, with higher SPL and fewer false user queries. Second, indirect human-centered local search using YOLOv5 followed by VLM verification further reduced false positives without hurting SPL. Third, hybridizing data-driven learning from demonstration with commonsense LLM distillation yielded more human-friendly motions, reflected in higher Warmth and Competence and lower Discomfort in the participant study (Park et al., 2023).

The paper also identifies several limitations. Ambiguous or out-of-distribution textual queries, such as “faculty members having lunch,” can mislead the LLM prior. Grad-CAM-based vision-language localization may fail in crowded scenes or poor lighting and may therefore require user feedback. The LfD data were collected from robot-experienced experts, which biases the learned reward toward their preferences over those of lay users. The human-participant study is described as a pilot with limited statistical power for a full MANOVA (Park et al., 2023).

The proposed extensions are correspondingly concrete: interactive clarification dialogue when the search prior or VLM is unsure, online personalization of the approach policy to individual comfort preferences, adoption of stronger open-vocabulary detectors such as OWL-ViT, extension from single static targets to multiple moving or interacting humans, and integration of 3D LIDAR semantics for more robust people search in larger maps (Park et al., 2023). A plausible implication is that the framework is best understood as a compositional robotics system: it uses pre-trained language and vision models to bootstrap search from natural-language descriptions, but reserves socially situated motion generation for a separate hybrid reward-learning stage rather than collapsing the full problem into a single end-to-end controller.