Speech-Guided Collaborative Perception (SCOPE)

• SCOPE is a collaborative perception paradigm that fuses spoken commands with live visual streams to generate spatially grounded task outputs.
• It employs a modular pipeline—capturing speech, parsing intent with LLMs, and executing vision tasks—for applications in remote assistance and surgery.
• Empirical evaluations show improved task accuracy, reduced completion times, and enhanced usability in both AR and surgical systems.

Speech-Guided Collaborative Perception (SCOPE) denotes a family of systems in which spoken commands are coupled to live visual perception, language-mediated task selection, and spatially grounded assistance. In the recent literature, this design pattern appears in at least three closely related forms: a referent-disambiguation framework that converts verbal remote-assistance instructions into spatially grounded AR guidance on a live shared view; a surgical scene segmentation framework that combines speech guidance, LLM reasoning, and open-set vision foundation models (VFMs); and a speech-guided embodied agent for video-guided skull-base surgery that dynamically executes segmentation, tracking, registration, pose estimation, and navigation overlays on intraoperative video (Kim et al., 3 Feb 2026, Mao et al., 12 Sep 2025, Mao et al., 17 Mar 2026). Taken together, these systems define SCOPE as a collaborative perception paradigm in which speech serves as the operative control channel for invoking and constraining downstream perception.

1. Conceptual scope and application domains

Across the cited systems, SCOPE operates on live visual streams and continuous audio, then converts transcribed speech into structured intents, referent descriptions, or task calls. The resulting outputs are not limited to textual responses: they include persistent AR guidance anchored at a resolved referent, top-$K$ mask proposals for clinician selection, propagated instrument and anatomy masks, tracked tool tips, rigid registrations, and depth-aware navigation overlays (Kim et al., 3 Feb 2026, Mao et al., 12 Sep 2025, Mao et al., 17 Mar 2026).

Instantiation	Primary setting	Core output
"From Speech-to-Spatial: Grounding Utterances on A Live Shared View with Augmented Reality" (Kim et al., 3 Feb 2026)	Remote guided assistance and intent disambiguation	Arrow widget and concise text panel at $v^*$
"SCOPE: Speech-guided COllaborative PErception Framework for Surgical Scene Segmentation" (Mao et al., 12 Sep 2025)	Endoscopic and microscopy video	Top-$K$ masks, labeled instrument mask, tip trajectory, anatomy mask
"Speak, Segment, Track, Navigate: An Interactive System for Video-Guided Skull-Base Surgery" (Mao et al., 17 Mar 2026)	Video-guided skull-base surgery	Segmentation, tracking, registration, pose, navigation overlay

In remote assistance, the central problem is referent disambiguation from speech input alone. In the surgical variants, the problem is broader: real-time, on-the-fly segmentation and tracking of surgical instruments and relevant anatomy, together with downstream image-guidance functions. This suggests that SCOPE is better understood as an architectural pattern—speech-mediated orchestration of perception—than as a single fixed algorithm.

2. System architecture and execution model

The architectural commonality is a staged pipeline linking speech capture, language interpretation, perception modules, and visual output. In the AR system, Speech-to-Spatial ingests a live AR video feed and audio stream, transcribes incoming audio via Whisper and LLM (GPT-4.1) into a structured utterance AST with fields $$\{\text{TargetLabel}, \text{AnchorLabels}, \text{Attributes}, \text{RelationalPhrases}, \text{TemporalCues}, \text{Actions}\}$$, constructs or updates an object-centric relational graph $G=(V,E)$, resolves the top referent $v^*$, and anchors an arrow widget and concise text panel at the selected object (Kim et al., 3 Feb 2026).

The surgical scene segmentation framework is organized around a Video Input Module, Speech-Processing Unit, LLM Interface (GPT-4.1 mini), Collaborative Perception Agent, Vision Foundation Models, and Tracking Module. The LLM outputs an action plan $$\mathcal{R}_i = \{\text{Action}, \text{Text Response}\}$$, where $$\text{Action} \in \{\text{segment}, \text{select\_mask}, \text{track}, \text{display}\}$$, and the Collaborative Perception Agent expands the speech query, calls GSAM and LISA++, ranks candidate masks, and returns top-$K$ masks together with JSON for display (Mao et al., 12 Sep 2025).

The skull-base surgery system makes the orchestration layer more explicit. Its SCOPE framework is organized into three subsystems: Speech Interface, Language Reasoning & Task Manager, and Perception & Geometry Pipeline. The textual block diagram is: surgeon’s voice $\rightarrow$ microphone $v^*$0 ASR $v^*$1 Language Parser (LLM) $v^*$2 Task Manager $v^*$3 $v^*$4Segmentation Module, Tracking Module, Anatomy Segmentation, Registration Module, Pose Estimator, Navigation Overlay$v^*$5 $v^*$6 Shared Memory $v^*$7 Video Renderer $v^*$8 surgeon. Intermediate results such as binary masks $v^*$9, 2D tip $K$0, and 3D pose $K$1 are written into a shared “streaming memory,” and subsequent modules read from that memory to avoid recomputation (Mao et al., 17 Mar 2026).

A recurrent property is modularity. The surgical papers explicitly describe decoupled LLM reasoning with specialized VFMs, while the AR paper separates speech parsing, graph construction, grounding, and rendering. A plausible implication is that SCOPE systems are designed to preserve both interpretability of task flow and interchangeability of downstream perception modules.

3. Speech parsing, intent representation, and referent semantics

The language layer in SCOPE systems is structured rather than purely conversational. In the AR referent-grounding formulation, referent cue extraction classifies each utterance against four patterns: Direct-Attribute, Relational, Remembrance, and Chained. These derive from a formative study of speech referencing patterns, which characterized recurring ways people specify targets as Direct Attribute, Relational, Remembrance, and Chained. The LLM is prompted with a grammar specification to extract fields

$K$2

as in the example parse of “Move the black thermos above the Quest3 box we discussed,” which yields $K$3 “thermos”, $K$4, $K$5, and $K$6 (Kim et al., 3 Feb 2026).

In the surgical scene segmentation framework, speech processing is modeled as a sequence

$K$7

The system prompt $K$8 defines workflow modules, tool calls, in-context examples, and output formatting rules. If the LLM selects $K$9, it emits JSON such as {"action":"select_mask","mask_index":3,"label":"forceps"}, after which the agent applies the selected mask and label (Mao et al., 12 Sep 2025).

In skull-base surgery, the parsing layer is lighter-weight and more task-API oriented. Surgeon utterances such as “Segment drill,” “Track tool,” “Segment anatomy,” “Register model,” “Show facial nerve overlay,” and “Done” are transcribed by a commercial ASR engine and passed to a lightweight LLM prompt specialized with surgical grammar. The output is a JSON-style intent object, for example { "action": "segment", "object": "tool" } or { "action": "register", "object": "anatomy", "landmarks": 4 }, and a rule-based mapper converts each intent into function calls such as segmentTool(), trackToolTip(), segmentAnatomy(), registerAnatomy(), estimatePose(), and overlayStructure(name) (Mao et al., 17 Mar 2026).

One common misconception is that SCOPE reduces speech to free-form transcription. The cited systems instead impose explicit semantic interfaces: utterance ASTs, JSON-style intents, and rule-based function calls. Speech is therefore not merely an input modality; it is the entry point to a structured control language for collaborative perception.

4. Perception, grounding, tracking, and geometry

The perception layer differs by domain, but all instances bind language-derived constraints to live visual state. In the AR system, the state representation is an object-centric relational graph

$$\{\text{TargetLabel}, \text{AnchorLabels}, \text{Attributes}, \text{RelationalPhrases}, \text{TemporalCues}, \text{Actions}\}$$0

where each node $$\{\text{TargetLabel}, \text{AnchorLabels}, \text{Attributes}, \text{RelationalPhrases}, \text{TemporalCues}, \text{Actions}\}$$1 represents a detected object with intra-object attributes—color, class, 6DoF transform, and interaction history—and each edge $$\{\text{TargetLabel}, \text{AnchorLabels}, \text{Attributes}, \text{RelationalPhrases}, \text{TemporalCues}, \text{Actions}\}$$2 encodes one of $$\{\text{TargetLabel}, \text{AnchorLabels}, \text{Attributes}, \text{RelationalPhrases}, \text{TemporalCues}, \text{Actions}\}$$3. Node attributes are represented as

$$\{\text{TargetLabel}, \text{AnchorLabels}, \text{Attributes}, \text{RelationalPhrases}, \text{TemporalCues}, \text{Actions}\}$$4

with $$\{\text{TargetLabel}, \text{AnchorLabels}, \text{Attributes}, \text{RelationalPhrases}, \text{TemporalCues}, \text{Actions}\}$$5 the 3D center coordinate in world frame, $$\{\text{TargetLabel}, \text{AnchorLabels}, \text{Attributes}, \text{RelationalPhrases}, \text{TemporalCues}, \text{Actions}\}$$6 a color histogram vector, $$\{\text{TargetLabel}, \text{AnchorLabels}, \text{Attributes}, \text{RelationalPhrases}, \text{TemporalCues}, \text{Actions}\}$$7 a class-label embedding, and $$\{\text{TargetLabel}, \text{AnchorLabels}, \text{Attributes}, \text{RelationalPhrases}, \text{TemporalCues}, \text{Actions}\}$$8 an interaction-history list of $$\{\text{TargetLabel}, \text{AnchorLabels}, \text{Attributes}, \text{RelationalPhrases}, \text{TemporalCues}, \text{Actions}\}$$9. Edge weights satisfy $G=(V,E)$0 if $G=(V,E)$1 and the spatial relation holds, else $G=(V,E)$2 (Kim et al., 3 Feb 2026).

Semantic grounding in that system is computed by embedding each attribute phrase $G=(V,E)$3 as $G=(V,E)$4, then scoring nodes with

$G=(V,E)$5

Relational constraints are enforced through adjacency, and memory cues are matched through timestamps. After view-frustum and occlusion filtering, candidate scores are

$G=(V,E)$6

with $G=(V,E)$7. The paper also states the corresponding probabilistic referent model

$G=(V,E)$8

(Kim et al., 3 Feb 2026).

In the surgical scene segmentation framework, perception is collaborative in a different sense. For each expanded query $G=(V,E)$9, a VFM returns masks $v^*$0, and the ranking score is

$v^*$1

Sorting by $v^*$2 yields top-$v^*$3 non-overlapping masks. The framework uses GSAM and LISA++ for open-set segmentation and CUTIE or SAM2 for video object segmentation. No per-procedure fine-tuning is performed; all models are used zero-shot, with lightweight prompt-tuning via the system prompt and few-shot examples (Mao et al., 12 Sep 2025).

The surgical papers also converge on an “instrument-as-pointer” mechanism. In the scene segmentation framework, the principal axis $v^*$4 of the selected instrument mask $v^*$5 is computed via PCA on boundary points, and the tip is localized by

$v^*$6

DepthAnything provides per-pixel depth $v^*$7; when a tip-depth condition is met, the system infers a virtual cursor

$v^*$8

feeds $v^*$9 as a positive point prompt to SAM, and propagates the resulting anatomy mask with CUTIE (Mao et al., 12 Sep 2025).

The skull-base surgery system extends this geometric layer substantially. It computes 2D PCA on tool mask pixels to obtain the principal axis direction $$\mathcal{R}_i = \{\text{Action}, \text{Text Response}\}$$0 and two extrema, defines the tip $$\mathcal{R}_i = \{\text{Action}, \text{Text Response}\}$$1 with temporal consistency, tracks two pose hypotheses per frame, compares rendered silhouettes to the observed mask using

$$\mathcal{R}_i = \{\text{Action}, \text{Text Response}\}$$2

and retains the higher-scoring hypothesis. Anatomy registration is formulated as a PnP problem over $$\mathcal{R}_i = \{\text{Action}, \text{Text Response}\}$$3 2D landmarks and corresponding 3D mesh points, solved with EPnP followed by non-linear Gauss–Newton refinement to produce $$\mathcal{R}_i = \{\text{Action}, \text{Text Response}\}$$4. Monocular pose estimation then combines DepthAnything v2, anatomy-derived metric depth, back-projection, axis priors, Rodrigues’ formula, and frame-to-frame refinement (Mao et al., 17 Mar 2026).

5. Collaborative workflows and visual guidance

The defining feature of SCOPE is not only perception accuracy but the organization of human-machine collaboration. In the AR framework, once the referent is selected, the AR Rendering Module anchors an arrow widget using Unity AR Foundation’s ARAnchor at $$\mathcal{R}_i = \{\text{Action}, \text{Text Response}\}$$5, instantiates a 2D Canvas billboarded above the arrow with concise LLM-summarized instruction such as “Tighten bolt C at rear,” and persists the panel until the action recorded in $$\mathcal{R}_i = \{\text{Action}, \text{Text Response}\}$$6 updates to include the specified intent. If the resolution is ambiguous—specifically, if $$\mathcal{R}_i = \{\text{Action}, \text{Text Response}\}$$7 falls below a confidence threshold—the fail-safe behavior is to display raw transcription only, with no arrow, so that the user can choose manually (Kim et al., 3 Feb 2026).

In the surgical segmentation framework, collaboration is iterative at the mask level. The Collaborative Perception Agent generates top candidates of VFM-generated segmentation and incorporates intuitive speech feedback from clinicians to guide the segmentation of surgical instruments. Afterwards, instruments themselves serve as interactive pointers to label additional elements of the surgical scene. This workflow is intended to support on-the-fly segmentation, labeling, and tracking of surgical instruments and anatomy in intraoperative video streams (Mao et al., 12 Sep 2025).

The skull-base surgery system makes the task choreography explicit. Representative call sequences include: “Segment tool.” $$\mathcal{R}_i = \{\text{Action}, \text{Text Response}\}$$8 segmentTool() $$\mathcal{R}_i = \{\text{Action}, \text{Text Response}\}$$9 returns $$\text{Action} \in \{\text{segment}, \text{select\_mask}, \text{track}, \text{display}\}$$0 and tip $$\text{Action} \in \{\text{segment}, \text{select\_mask}, \text{track}, \text{display}\}$$1; “Segment cochlear nerve.” $$\text{Action} \in \{\text{segment}, \text{select\_mask}, \text{track}, \text{display}\}$$2 segmentAnatomy() using the stored tip trajectory $$\text{Action} \in \{\text{segment}, \text{select\_mask}, \text{track}, \text{display}\}$$3 returns $$\text{Action} \in \{\text{segment}, \text{select\_mask}, \text{track}, \text{display}\}$$4; “Register anatomy.” $$\text{Action} \in \{\text{segment}, \text{select\_mask}, \text{track}, \text{display}\}$$5 registerAnatomy() $$\text{Action} \in \{\text{segment}, \text{select\_mask}, \text{track}, \text{display}\}$$6 returns $$\text{Action} \in \{\text{segment}, \text{select\_mask}, \text{track}, \text{display}\}$$7; and “Show facial nerve overlay.” $$\text{Action} \in \{\text{segment}, \text{select\_mask}, \text{track}, \text{display}\}$$8 overlayStructure("facial nerve"), which reads $$\text{Action} \in \{\text{segment}, \text{select\_mask}, \text{track}, \text{display}\}$$9 and $K$0 and renders a depth-aware overlay. Shared memory holds all masks, poses, and transforms; modules never recompute masks or reprobe the user. This enables rapid task-switching, including automatic reporting of tool tip location in the anatomy frame after registration (Mao et al., 17 Mar 2026).

A second common misconception is that SCOPE is equivalent to speech-only assistance. The AR system explicitly transforms disembodied verbal instruction into visually explainable, actionable guidance on a live shared view, while the surgical systems use speech to trigger perception workflows whose outputs are rendered as masks, tracked tips, or anatomy-aware overlays. Speech is therefore the control channel, not the sole representational medium.

6. Empirical performance and reported outcomes

The evaluation protocols vary by application, but each paper reports measurable improvements or competitive performance under live or mock-operating conditions. In the AR referent-grounding study, a user study with $K$1 on Meta Quest 3 compared Audio-only (baseline), Full (verbatim + arrow), and Summary (condensed + arrow) under Locate and Move tasks. Metrics were Task Completion Time, Accuracy, Cognitive Load (RTLX 0–100), and Confidence (0–100), analyzed with Friedman $K$2, RM-ANOVA, and post-hoc Bonferroni tests. Reported findings were: Locate Task Time, Summary $K$3 Audio $K$4 $K$5; Move Task Time, Summary $K$6 Full $K$7 Audio $K$8 $K$9; Move Task Accuracy, Summary $\rightarrow$0 Audio $\rightarrow$1 $\rightarrow$2; Cognitive Load (Move), Full $\rightarrow$3 Audio $\rightarrow$4 $\rightarrow$5; and Confidence (Move), Summary $\rightarrow$6 Audio $\rightarrow$7 $\rightarrow$8. Across referencing patterns—Direct, Relational, Memory, Chained—Summary outperformed Audio in time and accuracy, especially for Memory and Chained cues. In open-ended usage, out of 81 real queries, $\rightarrow$9 were correctly grounded, $v^*$00 fell back to Full transcription, and $v^*$01 were speech-recognition errors (Kim et al., 3 Feb 2026).

The surgical scene segmentation paper evaluates on a Cataract1k subset and an in-house ex-vivo skull-base dataset using DSC and ASD for initial segmentation, and mDSC and mASD for mask propagation and tracking. For initial segmentation, GSAM outperformed LISA++ in the reported table: Eye, GSAM DSC $v^*$02, ASD $v^*$03 px, $v^*$04 iterations, $v^*$05 s per iteration; Skull Base, GSAM DSC $v^*$06, ASD $v^*$07 px, $v^*$08 iteration, $v^*$09 s per iteration. For propagation and tracking, CUTIE achieved Eye mDSC $v^*$10, mASD $v^*$11, and Skull Base mDSC $v^*$12, mASD $v^*$13. The live mock experiment reported segmentation first-iteration success in $v^*$14 of trials with average iteration time $v^*$15, tip localization mean error $v^*$16 px, tracking stable over 200 frames with mDSC $v^*$17, real-time frame rate $v^*$18 fps, and automatic anatomy segmentation upon tool–tissue contact with $v^*$19 latency (Mao et al., 12 Sep 2025).

The skull-base surgery system benchmarks tracking performance against a commercially available optical tracking system. It reports tip RMSE with mean $v^*$20 over three trials, roll error $v^*$21, pitch error $v^*$22, interactive tool segmentation plus selection in $v^*$23, anatomy registration with four landmarks in $v^*$24, complete setup in $v^*$25 versus $v^*$26 for optical tracker, and real-time tracking and overlay at $v^*$27 on an NVIDIA A100 (Mao et al., 17 Mar 2026).

Collectively, these results indicate that SCOPE can be evaluated along at least three axes: referent-resolution efficiency and usability, segmentation and propagation quality, and geometry-aware navigation accuracy. The papers do not reduce these axes to a single unified benchmark, which suggests that SCOPE remains application-driven rather than standardized.

7. Limitations, misconceptions, and research directions

The limitations are domain-specific but structurally related. In the AR system, language coverage is currently object-centered only; planned extensions include viewer- and environment-centered frames such as “to my left” and “near the wall.” Complex references such as ordinal expressions (“second from left”) and “between A and B” require multi-hop graph traversal. The visual design space remains open, with future exploration of 2D halos, 3D reticles, and icon overlays, together with legibility and fatigue evaluation. Additional future directions include optional gaze and gesture cues for adaptive cue fusion, on-the-fly frame alignment for asynchronous AR sessions and unsynchronized coordinate frames, and user-facing reasoning breakdown and adjustable summarization verbosity (Kim et al., 3 Feb 2026).

In the surgical scene segmentation framework, end-to-end latency is dominated by cloud calls to the LLM and VFMs, motivating on-device deployment. The mock procedures also lack full operating-room complexity, including blood, smoke, and multi-agent workflow. Proposed future directions include model optimization and quantization for on-premise GPU or embedded hardware, extension to laparoscopic, robotic, and open surgery, integration of haptics and tool-force sensing, and formal user studies with surgeons to quantify usability and decision-making impact (Mao et al., 12 Sep 2025).

In the skull-base surgery setting, limitations include depth sensitivity when anatomy masks are small, partial occlusion and specularities that can bias PCA tip estimation, ASR errors in a noisy OR environment, and brief tracking loss under large sudden tool motions. Reported mitigations are scale anchoring with anatomy registration, fusion of anatomical and tool depth, boundary cropping, dual-hypothesis gating, DA2-based fallback, limited grammar, confirmation prompts such as “Did you mean segment tool?”, and re-initialization through the command “Re-segment tool” buffered across frames (Mao et al., 17 Mar 2026).

A final misconception is that SCOPE implies a fixed multimodal policy. The AR paper explicitly distinguishes its contribution from prior systems that rely on additional cues such as gesture or gaze, because it infers the intended target solely from spoken references. By contrast, the surgical literature treats speech as the primary sterile-compatible interface while leaving room for future multimodal feedback and adaptive cue fusion. This suggests that SCOPE is defined less by any single sensor configuration than by a collaboration principle: spoken intent is translated into executable perception grounded in live visual state.