---
title: Speech-Guided Collaborative Perception (SCOPE)
url: https://www.emergentmind.com/topics/speech-guided-collaborative-perception-scope
type: topic
---

# Speech-Guided Collaborative Perception (SCOPE)

Speech-Guided Collaborative Perception (SCOPE) denotes a family of systems in which spoken commands are coupled to live visual perception, language-mediated task selection, and spatially grounded assistance. In the recent literature, this design pattern appears in at least three closely related forms: a referent-disambiguation framework that converts verbal remote-assistance instructions into spatially grounded AR guidance on a live shared view; a surgical scene segmentation framework that combines speech guidance, large language model (LLM) reasoning, and open-set vision foundation models (VFMs); and a speech-guided embodied agent for video-guided skull-base surgery that dynamically executes segmentation, tracking, registration, pose estimation, and navigation overlays on intraoperative video [2602.03059] [2509.10748] [2603.16024]. Taken together, these systems define SCOPE as a collaborative perception paradigm in which speech serves as the operative control channel for invoking and constraining downstream perception.

## 1. Conceptual scope and application domains

Across the cited systems, SCOPE operates on live visual streams and continuous audio, then converts transcribed speech into structured intents, referent descriptions, or task calls. The resulting outputs are not limited to textual responses: they include persistent AR guidance anchored at a resolved referent, top-\(K\) mask proposals for clinician selection, propagated instrument and anatomy masks, tracked tool tips, rigid registrations, and depth-aware navigation overlays [2602.03059] [2509.10748] [2603.16024].

| Instantiation | Primary setting | Core output |
|---|---|---|
| "From Speech-to-Spatial: Grounding Utterances on A Live Shared View with Augmented Reality" [2602.03059] | Remote guided assistance and intent disambiguation | Arrow widget and concise text panel at \(v^*\) |
| "SCOPE: Speech-guided COllaborative PErception Framework for Surgical Scene Segmentation" [2509.10748] | Endoscopic and microscopy video | Top-\(K\) masks, labeled instrument mask, tip trajectory, anatomy mask |
| "Speak, Segment, Track, Navigate: An Interactive System for Video-Guided Skull-Base Surgery" [2603.16024] | Video-guided skull-base surgery | Segmentation, tracking, registration, pose, navigation overlay |

In remote assistance, the central problem is referent disambiguation from speech input alone. In the surgical variants, the problem is broader: real-time, on-the-fly segmentation and tracking of surgical instruments and relevant anatomy, together with downstream image-guidance functions. This suggests that SCOPE is better understood as an architectural pattern—speech-mediated orchestration of perception—than as a single fixed algorithm.

## 2. System architecture and execution model

The architectural commonality is a staged pipeline linking speech capture, language interpretation, perception modules, and visual output. In the AR system, Speech-to-Spatial ingests a live AR video feed and audio stream, transcribes incoming audio via Whisper and LLM (GPT-4.1) into a structured utterance AST with fields \(\{\text{TargetLabel}, \text{AnchorLabels}, \text{Attributes}, \text{RelationalPhrases}, \text{TemporalCues}, \text{Actions}\}\), constructs or updates an object-centric relational graph \(G=(V,E)\), resolves the top referent \(v^*\), and anchors an arrow widget and concise text panel at the selected object [2602.03059].

The surgical scene segmentation framework is organized around a Video Input Module, Speech-Processing Unit, LLM Interface (GPT-4.1 mini), Collaborative Perception Agent, Vision Foundation Models, and Tracking Module. The LLM outputs an action plan \(\mathcal{R}_i = \{\text{Action}, \text{Text Response}\}\), where \(\text{Action} \in \{\text{segment}, \text{select\_mask}, \text{track}, \text{display}\}\), and the Collaborative Perception Agent expands the speech query, calls GSAM and LISA++, ranks candidate masks, and returns top-\(K\) masks together with JSON for display [2509.10748].

The skull-base surgery system makes the orchestration layer more explicit. Its SCOPE framework is organized into three subsystems: Speech Interface, Language Reasoning & Task Manager, and Perception & Geometry Pipeline. The textual block diagram is: surgeon’s voice \(\rightarrow\) microphone \(\rightarrow\) ASR \(\rightarrow\) Language Parser (LLM) \(\rightarrow\) Task Manager \(\rightarrow\) \(\{\)Segmentation Module, Tracking Module, Anatomy Segmentation, Registration Module, Pose Estimator, Navigation Overlay\(\}\) \(\rightarrow\) Shared Memory \(\rightarrow\) Video Renderer \(\rightarrow\) surgeon. Intermediate results such as binary masks \(M_{\text{tool}}\), 2D tip \(\mathbf p\), and 3D pose \(T=(R,t)\) are written into a shared “streaming memory,” and subsequent modules read from that memory to avoid recomputation [2603.16024].

A recurrent property is modularity. The surgical papers explicitly describe decoupled LLM reasoning with specialized VFMs, while the AR paper separates speech parsing, graph construction, grounding, and rendering. A plausible implication is that SCOPE systems are designed to preserve both interpretability of task flow and interchangeability of downstream perception modules.

## 3. Speech parsing, intent representation, and referent semantics

The language layer in SCOPE systems is structured rather than purely conversational. In the AR referent-grounding formulation, referent cue extraction classifies each utterance against four patterns: Direct-Attribute, Relational, Remembrance, and Chained. These derive from a formative study of speech referencing patterns, which characterized recurring ways people specify targets as Direct Attribute, Relational, Remembrance, and Chained. The LLM is prompted with a grammar specification to extract fields
\[
\text{Utterance} \rightarrow \{\text{TARGET}, \text{ANCHORS}[], \text{ATTRIBUTES}[], \text{RELATION}[], \text{TEMPORAL}[], \text{ACTION}[]\},
\]
as in the example parse of “Move the black thermos above the Quest3 box we discussed,” which yields \(\text{TARGET}=\) “thermos”, \(\text{ATTRIBUTES}=\{\text{color}=\text{black}\}\), \(\text{RELATION}=\{\text{above}\rightarrow \text{“Quest3 box”}\}\), and \(\text{TEMPORAL}=\{\text{“we discussed”}\}\) [2602.03059].

In the surgical scene segmentation framework, speech processing is modeled as a sequence
\[
Q_i = \mathrm{Whisper}(\text{audio}_i), \qquad
\mathcal{R}_i = \mathrm{LLM}(Q_i,\mathbb{S}_i,\mathcal{P},\mathcal{H}_{(<i)}).
\]
The system prompt \(\mathcal{P} = \{\mathcal{M},\mathcal{T},\mathcal{E},\mathcal{R}\}\) defines workflow modules, tool calls, in-context examples, and output formatting rules. If the LLM selects \(\text{"select\_mask"}\), it emits JSON such as `{"action":"select_mask","mask_index":3,"label":"forceps"}`, after which the agent applies the selected mask and label [2509.10748].

In skull-base surgery, the parsing layer is lighter-weight and more task-API oriented. Surgeon utterances such as “Segment drill,” “Track tool,” “Segment anatomy,” “Register model,” “Show facial nerve overlay,” and “Done” are transcribed by a commercial ASR engine and passed to a lightweight LLM prompt specialized with surgical grammar. The output is a JSON-style intent object, for example `{ "action": "segment", "object": "tool" }` or `{ "action": "register", "object": "anatomy", "landmarks": 4 }`, and a rule-based mapper converts each intent into function calls such as `segmentTool()`, `trackToolTip()`, `segmentAnatomy()`, `registerAnatomy()`, `estimatePose()`, and `overlayStructure(name)` [2603.16024].

One common misconception is that SCOPE reduces speech to free-form transcription. The cited systems instead impose explicit semantic interfaces: utterance ASTs, JSON-style intents, and rule-based function calls. Speech is therefore not merely an input modality; it is the entry point to a structured control language for collaborative perception.

## 4. Perception, grounding, tracking, and geometry

The perception layer differs by domain, but all instances bind language-derived constraints to live visual state. In the AR system, the state representation is an object-centric relational graph
\[
G=(V,E),
\]
where each node \(v \in V\) represents a detected object with intra-object attributes—color, class, 6DoF transform, and interaction history—and each edge \(e \in E\) encodes one of \(\{\text{left}, \text{right}, \text{above}, \text{below}, \text{in-front-of}, \text{behind-of}\}\). Node attributes are represented as
\[
a_v = [p_v; c_v; s_v; h_v],
\]
with \(p_v \in \mathbb{R}^3\) the 3D center coordinate in world frame, \(c_v\) a color histogram vector, \(s_v\) a class-label embedding, and \(h_v\) an interaction-history list of \((\text{action}, \text{actor}, \text{timestamp})\). Edge weights satisfy \(w_{ij}=1\) if \(\|p_i-p_j\|\le 0.5\,\text{m}\) and the spatial relation holds, else \(0\) [2602.03059].

Semantic grounding in that system is computed by embedding each attribute phrase \(\phi\) as \(e_\phi = \mathrm{Embed}(\phi)\in\mathbb{R}^d\), then scoring nodes with
\[
s_v = \cos(e_\phi, a_v) = \frac{e_\phi \cdot a_v}{\|e_\phi\|\,\|a_v\|}.
\]
Relational constraints are enforced through adjacency, and memory cues are matched through timestamps. After view-frustum and occlusion filtering, candidate scores are
\[
\mathrm{score}(v_i)=\lambda_1 s_{v_i} + \lambda_2 \sum_{u\in Anchors}\mathbf 1_{(u,v_i)\in E} + \lambda_3 m_{v_i},
\]
with \(v^*=\arg\max_{v_i\in C}\mathrm{score}(v_i)\). The paper also states the corresponding probabilistic referent model
\[
p(v\mid U)\propto \exp\!\left(\lambda_1\cdot s_v+\lambda_2\cdot \sum_{u\in Anchors}\mathbf 1_{(u,v)\in E}+\lambda_3\cdot m_v\right)
\]
[2602.03059].

In the surgical scene segmentation framework, perception is collaborative in a different sense. For each expanded query \(Q^k\), a VFM returns masks \(\{M_{k,j}\}_{j=1}^{N}\), and the ranking score is
\[
S_{k,j} = \alpha\,\mathrm{conf}_{\mathrm{VFM}}(M_{k,j}) + \beta\,\mathrm{IoU}\bigl(\mathrm{BBox}(M_{k,j}),\mathrm{BBox}(M_{\text{prev}})\bigr).
\]
Sorting by \(S_{k,j}\) yields top-\(K\) non-overlapping masks. The framework uses GSAM and LISA++ for open-set segmentation and CUTIE or SAM2 for video object segmentation. No per-procedure fine-tuning is performed; all models are used zero-shot, with lightweight prompt-tuning via the system prompt and few-shot examples [2509.10748].

The surgical papers also converge on an “instrument-as-pointer” mechanism. In the scene segmentation framework, the principal axis \(v_1\) of the selected instrument mask \(M_{\text{instr}}\) is computed via PCA on boundary points, and the tip is localized by
\[
p_{\mathrm{tip}} = \arg\max_{x\in \partial M}\bigl(v_1^\top(x-\bar{x})\bigr).
\]
DepthAnything provides per-pixel depth \(d(x)\); when a tip-depth condition is met, the system infers a virtual cursor
\[
p_{\mathrm{cursor}} = p_{\mathrm{tip}} + \gamma v_1,
\]
feeds \(p_{\mathrm{cursor}}\) as a positive point prompt to SAM, and propagates the resulting anatomy mask with CUTIE [2509.10748].

The skull-base surgery system extends this geometric layer substantially. It computes 2D PCA on tool mask pixels to obtain the principal axis direction \(\hat d_{\rm 2D}\) and two extrema, defines the tip \(\mathbf p_t\) with temporal consistency, tracks two pose hypotheses per frame, compares rendered silhouettes to the observed mask using
\[
\mathrm{F1}(\hat M,M) = \frac{2\,|\hat M\cap M|}{|\hat M| + |M|},
\]
and retains the higher-scoring hypothesis. Anatomy registration is formulated as a PnP problem over \(N\ge 4\) 2D landmarks and corresponding 3D mesh points, solved with EPnP followed by non-linear Gauss–Newton refinement to produce \(T_{C\leftarrow A}=(R,t)\in SE(3)\). Monocular pose estimation then combines DepthAnything v2, anatomy-derived metric depth, back-projection, axis priors, Rodrigues’ formula, and frame-to-frame refinement [2603.16024].

## 5. Collaborative workflows and visual guidance

The defining feature of SCOPE is not only perception accuracy but the organization of human-machine collaboration. In the AR framework, once the referent is selected, the AR Rendering Module anchors an arrow widget using Unity AR Foundation’s `ARAnchor` at \(p_{\mathrm{ar}}\), instantiates a 2D Canvas billboarded above the arrow with concise LLM-summarized instruction such as “Tighten bolt C at rear,” and persists the panel until the action recorded in \(h_{v^*}\) updates to include the specified intent. If the resolution is ambiguous—specifically, if \(p(v^*|U)\) falls below a confidence threshold—the fail-safe behavior is to display raw transcription only, with no arrow, so that the user can choose manually [2602.03059].

In the surgical segmentation framework, collaboration is iterative at the mask level. The Collaborative Perception Agent generates top candidates of VFM-generated segmentation and incorporates intuitive speech feedback from clinicians to guide the segmentation of surgical instruments. Afterwards, instruments themselves serve as interactive pointers to label additional elements of the surgical scene. This workflow is intended to support on-the-fly segmentation, labeling, and tracking of surgical instruments and anatomy in intraoperative video streams [2509.10748].

The skull-base surgery system makes the task choreography explicit. Representative call sequences include: “Segment tool.” \(\rightarrow\) `segmentTool()` \(\rightarrow\) returns \(M_{\text{tool}}\) and tip \(\mathbf p\); “Segment cochlear nerve.” \(\rightarrow\) `segmentAnatomy()` using the stored tip trajectory \(\rightarrow\) returns \(M_{\text{anat}}\); “Register anatomy.” \(\rightarrow\) `registerAnatomy()` \(\rightarrow\) returns \(T_{C\leftarrow A}\); and “Show facial nerve overlay.” \(\rightarrow\) `overlayStructure("facial nerve")`, which reads \(T_{C\leftarrow A}\) and \(M_{\text{anat}}\) and renders a depth-aware overlay. Shared memory holds all masks, poses, and transforms; modules never recompute masks or reprobe the user. This enables rapid task-switching, including automatic reporting of tool tip location in the anatomy frame after registration [2603.16024].

A second common misconception is that SCOPE is equivalent to speech-only assistance. The AR system explicitly transforms disembodied verbal instruction into visually explainable, actionable guidance on a live shared view, while the surgical systems use speech to trigger perception workflows whose outputs are rendered as masks, tracked tips, or anatomy-aware overlays. Speech is therefore the control channel, not the sole representational medium.

## 6. Empirical performance and reported outcomes

The evaluation protocols vary by application, but each paper reports measurable improvements or competitive performance under live or mock-operating conditions. In the AR referent-grounding study, a user study with \(N=18\) on Meta Quest 3 compared Audio-only (baseline), Full (verbatim + arrow), and Summary (condensed + arrow) under Locate and Move tasks. Metrics were Task Completion Time, Accuracy, Cognitive Load (RTLX 0–100), and Confidence (0–100), analyzed with Friedman \(\chi^2\), RM-ANOVA, and post-hoc Bonferroni tests. Reported findings were: Locate Task Time, Summary \(3.25\,\text{s} <\) Audio \(4.33\,\text{s}\) \((\chi^2=21.0,p<.001)\); Move Task Time, Summary \(6.33\,\text{s} <\) Full \(7.94\,\text{s} <\) Audio \(9.31\,\text{s}\) \((\chi^2=20.3,p<.001)\); Move Task Accuracy, Summary \(73.1\% >\) Audio \(64.4\%\) \((\chi^2=8.04,p<.02)\); Cognitive Load (Move), Full \(32.2 <\) Audio \(47.2\) \((F(2,34)=7.8,p<.01)\); and Confidence (Move), Summary \(88.9 >\) Audio \(77.6\) \((F(2,34)=5.6,p<.01)\). Across referencing patterns—Direct, Relational, Memory, Chained—Summary outperformed Audio in time and accuracy, especially for Memory and Chained cues. In open-ended usage, out of 81 real queries, \(77.8\%\) were correctly grounded, \(13.6\%\) fell back to Full transcription, and \(8.6\%\) were speech-recognition errors [2602.03059].

The surgical scene segmentation paper evaluates on a Cataract1k subset and an in-house ex-vivo skull-base dataset using DSC and ASD for initial segmentation, and mDSC and mASD for mask propagation and tracking. For initial segmentation, GSAM outperformed LISA++ in the reported table: Eye, GSAM DSC \(0.82\), ASD \(2.83\) px, \(1.3\) iterations, \(1.16\) s per iteration; Skull Base, GSAM DSC \(0.93\), ASD \(5.52\) px, \(1.0\) iteration, \(1.28\) s per iteration. For propagation and tracking, CUTIE achieved Eye mDSC \(0.840\), mASD \(2.736\), and Skull Base mDSC \(0.973\), mASD \(2.540\). The live mock experiment reported segmentation first-iteration success in \(100\%\) of trials with average iteration time \(1.2\,\text{s}\), tip localization mean error \(\sim 2\) px, tracking stable over 200 frames with mDSC \(>0.95\), real-time frame rate \(\sim 25\) fps, and automatic anatomy segmentation upon tool–tissue contact with \(<0.5\,\text{s}\) latency [2509.10748].

The skull-base surgery system benchmarks tracking performance against a commercially available optical tracking system. It reports tip RMSE with mean \(\|\Delta \mathbf p\|=2.83\pm1.64\,\text{mm}\) over three trials, roll error \(0.16\pm0.21^\circ\), pitch error \(0.17\pm0.22^\circ\), interactive tool segmentation plus selection in \(27\pm2\,\text{s}\), anatomy registration with four landmarks in \(1\,\text{min }15\pm15\,\text{s}\), complete setup in \(1\,\text{min }42\pm12\,\text{s}\) versus \(>10\,\text{min}\) for optical tracker, and real-time tracking and overlay at \(\sim 30\,\text{Hz}\) on an NVIDIA A100 [2603.16024].

Collectively, these results indicate that SCOPE can be evaluated along at least three axes: referent-resolution efficiency and usability, segmentation and propagation quality, and geometry-aware navigation accuracy. The papers do not reduce these axes to a single unified benchmark, which suggests that SCOPE remains application-driven rather than standardized.

## 7. Limitations, misconceptions, and research directions

The limitations are domain-specific but structurally related. In the AR system, language coverage is currently object-centered only; planned extensions include viewer- and environment-centered frames such as “to my left” and “near the wall.” Complex references such as ordinal expressions (“second from left”) and “between A and B” require multi-hop graph traversal. The visual design space remains open, with future exploration of 2D halos, 3D reticles, and icon overlays, together with legibility and fatigue evaluation. Additional future directions include optional gaze and gesture cues for adaptive cue fusion, on-the-fly frame alignment for asynchronous AR sessions and unsynchronized coordinate frames, and user-facing reasoning breakdown and adjustable summarization verbosity [2602.03059].

In the surgical scene segmentation framework, end-to-end latency is dominated by cloud calls to the LLM and VFMs, motivating on-device deployment. The mock procedures also lack full operating-room complexity, including blood, smoke, and multi-agent workflow. Proposed future directions include model optimization and quantization for on-premise GPU or embedded hardware, extension to laparoscopic, robotic, and open surgery, integration of haptics and tool-force sensing, and formal user studies with surgeons to quantify usability and decision-making impact [2509.10748].

In the skull-base surgery setting, limitations include depth sensitivity when anatomy masks are small, partial occlusion and specularities that can bias PCA tip estimation, ASR errors in a noisy OR environment, and brief tracking loss under large sudden tool motions. Reported mitigations are scale anchoring with anatomy registration, fusion of anatomical and tool depth, boundary cropping, dual-hypothesis gating, DA2-based fallback, limited grammar, confirmation prompts such as “Did you mean segment tool?”, and re-initialization through the command “Re-segment tool” buffered across frames [2603.16024].

A final misconception is that SCOPE implies a fixed multimodal policy. The AR paper explicitly distinguishes its contribution from prior systems that rely on additional cues such as gesture or gaze, because it infers the intended target solely from spoken references. By contrast, the surgical literature treats speech as the primary sterile-compatible interface while leaving room for future multimodal feedback and adaptive cue fusion. This suggests that SCOPE is defined less by any single sensor configuration than by a collaboration principle: spoken intent is translated into executable perception grounded in live visual state.

Source: https://www.emergentmind.com/topics/speech-guided-collaborative-perception-scope