---
title: 'SOCRATES: Robot Dog Human Search Framework'
url: https://www.emergentmind.com/topics/socrates-da3616c1-e05c-49ea-9b42-281772925c46
type: topic
---

# SOCRATES: Robot Dog Human Search Framework

SOCRATES, short for “Text-based Human Search and Approach using a Robot Dog,” is a robotics framework for locating a target person from a free-form textual description and then approaching that person in a socially acceptable manner. The system takes text that combines appearance and location clues, searches for the described individual with a language-mediated zero-shot pipeline, and, after user confirmation, switches to a hybrid learning-based approach planner designed to produce target-cordial robotic motion. The framework was validated in simulation, in a real-world lab, and with the Boston Dynamics Spot robot [2302.05324].

## 1. Problem formulation and operational setting

SOCRATES models person-finding as a two-stage task. Its input is a free-form textual description $t=(t_1,t_2)$, where $t_1$ is an appearance description such as “wearing a white shirt with black hair,” and $t_2$ is a location clue such as “is a student who works with robots” [2302.05324]. The robot is assumed to have an annotated 2D floorplan partitioned into $K$ labeled regions $\{a_i\}_{i=1\dots K}$, its own pose $p_r=(x,y,\theta)$ on the map, an onboard RGB-D camera, and a “general human” detector implemented with YOLOv5.

The outputs are differentiated by stage. For search, SOCRATES produces a sequence of waypoints $p=(x,y,\theta)$ that drives the robot until the target is detected and user-confirmed within a $5\,\mathrm m$ radius. For approach, it produces a dynamically generated velocity-level motion that brings the robot to approximately $0.6\,\mathrm m$ from the target in a socially acceptable way [2302.05324].

This decomposition is central to the framework. Search is formulated as zero-shot, language-mediated person localization over a mapped environment, while approach is formulated as socially aware local motion generation after the robot is already within interaction range. A common misunderstanding is to treat SOCRATES as a monolithic end-to-end policy; the system instead composes a search module and an approach module with an explicit confirmation handoff.

## 2. Human Search Socratic Model

The Human Search Socratic Model casts “find the person described by $t$” as a zero-shot, language-mediated search problem by composing three off-the-shelf models: an LLM-derived search prior, a VLM-based text-guided localizer, and a waypoint generator [2302.05324].

For the search prior, GPT-3 is prompted with the floorplan labels $\{a_1,\dots,a_K\}$ and the location clue $t_2$. From $M$ sentence-level answers $s_1,\dots,s_M=f_1(t_2,\{a_i\})$, SOCRATES computes an occurrence likelihood for each region:
$$
p(a_i \mid t_2)=\frac{1}{M}\sum_{k=1}^{M}\max_l \cos\bigl(w(a_i),w(s_k^l)\bigr),
$$
where $w(\cdot)$ is a pretrained word2vec embedding and $s_k^l$ is the $l$-th word of $s_k$ [2302.05324]. This converts a free-form semantic clue into a region-wise prior over the annotated map.

For text-guided localization, SOCRATES uses a zero-shot VQA model, PNP-VQA, together with Grad-CAM. The robot asks the question “Is a person $[t_1]$?.” If the VQA answer is “yes,” the Grad-CAM activation map $A(x,y)$ over image $I$ is thresholded at $t_c$ to form a bounding box $b=(x_1,y_1,x_2,y_2)$ [2302.05324]. Appearance matching is therefore not trained specifically for the downstream task; it is induced from the composition of a VQA model and saliency extraction.

The waypoint generator combines global and local search. Globally, the cost of visiting region $a_i$ from the current robot pose is
$$
c_l(a_i)=\|p_r-p_{a_i}\|_2+w_e\cdot(1-p(a_i\mid t_2)),
$$
where $p_{a_i}$ is the closest reachable pose in region $a_i$ and $w_e$ is a tunable weight [2302.05324]. The robot repeatedly chooses the unvisited label with minimum cost, navigates there, and switches to local search.

Local search is indirect and human-centric. The robot first detects any person bounding box with YOLOv5. If a person is found, it moves to maintain a $5\,\mathrm m$ distance and re-centers the person in view, then runs the text-based VLM check; if a match is found, it asks the user “Is this X?” If no person is in view, it performs frontier-based exploration within that region, excluding waypoints that are less than $t_g$ away from previously visited waypoints [2302.05324]. This local procedure is designed to reduce false user queries while preserving search efficiency.

## 3. Hybrid learning-based framework for approach

Once the target is identified and the robot is within $5\,\mathrm m$, SOCRATES invokes a learned cordial approach planner. The state is
$$
x=[x_r^h,y_r^h,\theta_r^h,g,v],
$$
where $(x_r^h,y_r^h,\theta_r^h)$ is the robot pose in human-centric coordinates, $g\in\{0,1\}$ indicates whether the person is gazing at the robot, and $v\in\{0.15,0.4,0.65\}$ is the robot’s current speed. The system learns a state-only reward $R(x)\in[0,1]$ [2302.05324].

Input processing combines perception and scene summarization. The person’s 3D orientation is estimated with a pre-trained face mesh network, and gaze is reduced to a binary variable using a $40^\circ$ threshold. The current image is captioned with BLIP, producing a set of scene captions $\{c_i\}$ [2302.05324]. These cues condition both the imitation-based reward and the language-distilled reward.

The learning-from-demonstration component uses KDMRL. Expert approach trajectories $\{x_k^D\}$ are collected with human feedback, and the empirical state density is estimated by kernel density estimation:
$$
\hat\mu(x)=\frac{1}{Z}\sum_{k=1}^{N_D}\gamma_k\,k_\mu(x,x_k^D),
$$
with $\gamma_k=\delta^{T-t}$, $T$ the trajectory length, $\delta\in[0,1]$, and $k_\mu$ an RBF kernel [2302.05324]. SOCRATES then solves a density-matching inverse-RL problem in an RKHS and represents the resulting reward as
$$
R(x)=\sum_{i=1}^{N_U}\alpha_i\,k(x,x_i^U).
$$

The knowledge-distillation component queries GPT-3 for each caption $c_i$ and gaze bit $g$, asking what trajectory the robot should take “to gently approach a person.” The responses are reduced to keywords such as “straight,” “curve,” “slowly,” and “front,” which are mapped by a small dictionary $f_d(w)$ to short $\Delta$-pose segments $(\Delta x,\Delta y,\Delta\theta)$. These segments are stitched into sample trajectories, and the visited RKHS states accumulate a reward signal $R_L(x)$, which is then normalized and smoothed with an RBF [2302.05324]. The resulting planner is therefore neither purely demonstrative nor purely language-specified.

The final reward combines the two sources:
$$
R_T(x)=w_r\cdot R_I(x)+(1-w_r)\cdot R_L(x), \qquad w_r=0.2.
$$
Motion planning is performed with FMT* in OMPL on the local egocentric map, using the cost
$$
c_m(\bar x_1,\bar x_2)=\zeta\cdot[1-R_T(\bar x_2)]\cdot \mathrm{dist}(\bar x_1,\bar x_2),
$$
where
$$
\mathrm{dist}(\bar x_1,\bar x_2)=w_p\|q_2-q_1\|+w_o|\theta_2-\theta_1|,
$$
with $\zeta=1.5$, $w_p=1$, and $w_o=0.5$ [2302.05324]. The approach module thus optimizes a reward-shaped geometric motion cost rather than directly imitating raw trajectories.

## 4. Search-to-approach integration

SOCRATES explicitly couples the two stages at the moment of human confirmation. As soon as the user answers “Yes, that is X,” the current robot pose $p_r^K$ and the last camera frame are passed to the approach module [2302.05324]. The approach planner then constructs the initial human-centric state $x^0$ from $p_r^K$ and the human pose $p_h$, obtains the gaze bit from face mesh, generates captions with BLIP, and solves planning under the combined reward $R_T$.

The resulting velocity commands are sent to the Spot robot until the final goal—approximately $0.6\,\mathrm m$ in front of the person—is reached [2302.05324]. This handoff is operationally significant because search uses region-level map priors and image-based re-identification, whereas approach uses human-centric state variables, gaze, speed, and local trajectory optimization. The framework therefore switches representational regimes rather than extending a single global planner all the way through person approach.

A second common misconception is that the search component alone defines SOCRATES. In fact, the paper treats human search and socially acceptable approach as a single integrated system, and the social-navigation component is not a post hoc add-on but one of the core contributions [2302.05324].

## 5. Experimental validation

The search module was evaluated in a real-world lab with two floors and five area labels, using $5$ people and $2$ start positions for a total of $10$ trials. Simulation was performed in Gazebo in two buildings: a household environment with $4$ people and $2$ starts, and an office environment with $8$ people and $2$ starts. The comparison methods were CoW, CoW + indirect, and a knowledge-prior baseline [2302.05324].

Search performance was measured with Success Rate (SR), Success Path Length (SPL), and False Detections (\#FD). In the real-world results, SOCRATES achieved the highest SR and SPL and the lowest \#FD among the reported methods [2302.05324].

| Method | SPL | SR | #FD |
|---|---:|---:|---:|
| CoW (direct) | 0.314 | 0.40 | 0.80 |
| CoW+indir. | 0.295 | 0.60 | 0.30 |
| Knowledge prior | 0.636 | 0.90 | 0.80 |
| SOCRATES (ours) | 0.647 | 1.00 | 0.20 |

The simulation results showed the same ordering, with SOCRATES achieving approximately $0.956$ SR and $0.709$ SPL [2302.05324]. This suggests that the combination of an LLM-derived regional prior and indirect human-centered local search produced both better path efficiency and fewer unnecessary user confirmations than the comparison conditions.

The real-world approach study involved $16$ human participants, split into robot-experienced ($n=8$) and inexperienced ($n=8$) groups. The experiment used a Latin-square balanced design with $4$ methods, $2$ gaze states, and $8$ total trajectories per participant, giving $128$ trials. The four methods were a direct position-control baseline at constant speed, KD only, LfD only, and the hybrid SOCRATES approach planner [2302.05324].

Evaluation used the Robotic Social Attributes Scale (RoSAS), a $7$-point Likert instrument over $18$ adjectives summarized into Competence, Warmth, and Discomfort. Aggregate results are as follows [2302.05324].

| Method | Competence | Warmth | Discomfort |
|---|---:|---:|---:|
| Baseline | 4.41 (1.08) | 3.34 (1.23) | 3.23 (1.23) |
| KD only | 3.96 (0.82) | 3.28 (1.03) | 3.59 (1.08) |
| LfD only | 4.14 (1.01) | 3.28 (1.29) | 3.30 (1.54) |
| Ours | 4.56 (1.28) | 3.63 (1.28) | 2.96 (1.28) |

The hybrid method scored highest in Competence and Warmth and lowest in Discomfort. The experienced group preferred the hybrid method by approximately $0.48$ points in Competence, whereas the inexperienced group showed a smaller absolute preference gap [2302.05324].

## 6. Findings, limitations, and prospective extensions

The reported findings are threefold. First, incorporating LLM-derived priors improved search efficiency, with higher SPL and fewer false user queries. Second, indirect human-centered local search using YOLOv5 followed by VLM verification further reduced false positives without hurting SPL. Third, hybridizing data-driven learning from demonstration with commonsense LLM distillation yielded more human-friendly motions, reflected in higher Warmth and Competence and lower Discomfort in the participant study [2302.05324].

The paper also identifies several limitations. Ambiguous or out-of-distribution textual queries, such as “faculty members having lunch,” can mislead the LLM prior. Grad-CAM-based vision-language localization may fail in crowded scenes or poor lighting and may therefore require user feedback. The LfD data were collected from robot-experienced experts, which biases the learned reward toward their preferences over those of lay users. The human-participant study is described as a pilot with limited statistical power for a full MANOVA [2302.05324].

The proposed extensions are correspondingly concrete: interactive clarification dialogue when the search prior or VLM is unsure, online personalization of the approach policy to individual comfort preferences, adoption of stronger open-vocabulary detectors such as OWL-ViT, extension from single static targets to multiple moving or interacting humans, and integration of 3D LIDAR semantics for more robust people search in larger maps [2302.05324]. A plausible implication is that the framework is best understood as a compositional robotics system: it uses pre-trained language and vision models to bootstrap search from natural-language descriptions, but reserves socially situated motion generation for a separate hybrid reward-learning stage rather than collapsing the full problem into a single end-to-end controller.

Source: https://www.emergentmind.com/topics/socrates-da3616c1-e05c-49ea-9b42-281772925c46