---
title: 'Speech2See: Speech-Driven Visual Grounding'
url: https://www.emergentmind.com/topics/speech-to-see-speech2see
type: topic
---

# Speech2See: Speech-Driven Visual Grounding

Searching arXiv for the cited Speech-to-See and closely related multimodal speech–vision papers.
Speech-to-See (Speech2See) designates a family of multimodal systems in which spoken input serves as the control signal for visual understanding, visual grounding, or visual generation, rather than text. In the most specific computer-vision usage, it denotes audio grounding or speech-driven open-set object detection: localizing and identifying objects directly from speech in open-vocabulary settings [2409.18372], [2509.16670]. In a broader assistive and omni-modal usage, it also covers systems that answer spoken questions about the environment from current or remembered visual context, as well as models that generate images directly from spoken descriptions [2410.03771], [2409.18042], [2511.03423].

## 1. Definition, scope, and lineage

A useful way to delimit Speech2See is by the direction of conditioning. The canonical setting is **speech $\rightarrow$ vision-grounded output**: a spoken command, caption, or question is mapped to a localized object, a scene-grounded answer, or a generated image. This distinguishes Speech2See from text-guided grounding, where language is already symbolic, and from inverse audiovisual tasks such as lip reading or silent-video-to-speech reconstruction.

Representative formulations in the cited literature include YOSS, which introduces **Audio Grounding** as localizing objects in images given spoken queries or descriptions; Speech-to-See, which frames the problem as **speech-driven open-set object detection**; SeeSay, which treats spoken questions as the front end of an assistive visual-query system; EMOVA, which treats speech, text, and vision as a unified token space for spoken multimodal dialogue; and VoxStudio, which extends the idea to direct speech-to-image generation [2409.18372], [2509.16670], [2410.03771], [2409.18042], [2511.03423].

| Formulation | Representative system | Output |
|---|---|---|
| Audio grounding | YOSS | Object localization in images |
| Speech-driven open-set detection | Speech-to-See | Bounding boxes and class labels |
| Assistive spoken visual query | SeeSay | Audio answers grounded in scene memory |
| Omni-modal spoken visual QA | EMOVA | Text and/or speech responses |
| Expressive speech-to-image | VoxStudio | Generated images |

A precursor to this line of work is **visually grounded speech**, in which paired speech and image data are embedded into a shared space for retrieval. “Learning to Recognise Words using Visually Grounded Speech” reports that such a model can retrieve images from spoken captions and can recognise isolated words from partial input, indicating that speech-only representations can already acquire visually grounded lexical structure without an intermediate transcript [2006.00512].

## 2. Canonical formulation: speech-driven object grounding

The first explicit object-localization formulation appears in YOSS, “You Only Speak Once to See,” which defines **Audio Grounding** as localizing objects in images from spoken queries or descriptions. YOSS is described as the first framework specifically designed to directly map speech to visual object localization in open-vocabulary settings. Its architecture combines a frozen CLIP Vision Transformer for image embeddings, the final three feature map levels from the YOLOv8-CSPDarknet backbone for multi-scale cues, and a HuBERT-based audio encoder with an Aggregation Branch that converts frame-level speech features into a fixed-size utterance embedding [2409.18372].

YOSS then performs **audio-image contrastive learning** in a shared semantic space and adds **audio-text alignment** with paired loss and CORAL to lock audio, text, and image embeddings into the CLIP space. Detection is performed with a YOLOv8 backbone and a NAS-FPN detection head. Visual features and the audio embedding are fused to produce **audio-guided queries**, which steer the detector toward regions relevant to the spoken input. Training proceeds in two stages: contrastive pretraining on paired image-audio-caption data, followed by detection finetuning with fixed backbone and fine-tuned audio-visual query layers.

The later “Speech-to-See: End-to-End Speech-Driven Open-Set Object Detection” reformulates the same problem more aggressively as a direct end-to-end alternative to indirect, text-mediated pipelines. It identifies three bottlenecks: scarcity of large-scale paired audio-image data, representation mismatch between speech and vision, and the loss of rich acoustic cues in ASR- or text-aligned systems. Its architecture uses a Swin Transformer visual backbone, a HuBERT speech backbone, a Grounding DINO-inspired fusion/decoder stack, a **Query-Guided Semantic Aggregation** module to condense redundant speech embeddings, and a **Mixture-of-LoRA-Experts** mechanism for parameter-efficient cross-modal adaptation during fine-tuning [2509.16670].

The Query-Guided Semantic Aggregation mechanism is explicitly defined as cross-attention from $K$ learnable queries to the speech sequence:
$$
O_i = \mathrm{Attn}(q_i, X_A) = \sum_{j=1}^{N_t} \mathrm{softmax}\left( \frac{q_i \cdot x_j}{\sqrt{d}} \right) x_j
$$
with $K \ll N_t$. In this formulation, the aggregated outputs become compact speech-semantic tokens for downstream fusion.

## 3. Alignment strategies and training regimes

The central technical problem in Speech2See is multimodal alignment: speech is temporally dense, acoustically variable, and non-symbolic, whereas modern visual grounding pipelines were largely built around text-image supervision. YOSS addresses this by projecting audio and image features into a shared semantic space and then explicitly aligning audio to CLIP text embeddings. Its full pretraining objective combines contrastive audio-image learning with audio-text alignment,
$$
L_1(x_i, x_a, x_t) = L_{\mathrm{Con}(x_i, x_a)} + \eta L_{\mathrm{Align}(x_t, x_a)},
$$
followed by a detection objective
$$
L_2(x_i, x_a) = L_{\mathrm{cls}(x_i, x_a)} + L_{\mathrm{loc}(x_i, x_a)}.
$$
This design uses text as a semantic bridge while still grounding from speech at inference time [2409.18372].

Speech-to-See replaces this bridge-heavy strategy with a progressive pre-training and fine-tuning paradigm. During pre-training, visual, audio, and decoder networks are initialized from large pretrained models, while the QSA and fusion modules are optimized to align modalities. During fine-tuning, each decoder layer receives multiple LoRA experts, and a router selects the expert with maximal routing score for the incoming query. This is intended to adapt text-pretrained decoder layers to speech-conditioned grounding without retraining the full network [2509.16670].

A related alignment strategy appears in EMOVA, which treats speech, text, and visual tokens as a single token sequence inside an omni-modal large language model. EMOVA uses a semantic-acoustic disentangled speech tokenizer, discretizes only the semantic component into speech units, and uses text as a bridge for omni-modal alignment. The model then jointly predicts output text tokens and output speech units conditioned on text tokens, speech units, and visual tokens:
$$
\mathbb{P}(U_T^{o},U_S^{o} \mid U_T,U_S,H_V) = \prod_{i=1}^L \mathbb{P}(\boldsymbol{x}_i \mid U_{T,<i}^{o},U_{S,<i}^{o},U_T,U_S,H_V).
$$
This formulation is not a detector, but it generalizes Speech2See from grounding to spoken multimodal reasoning over images, charts, PDFs, and other visual inputs [2409.18042].

These three strategies expose a methodological spectrum. YOSS uses CLIP-space locking and detection finetuning; Speech-to-See uses compact semantic aggregation plus parameter-efficient decoder adaptation; EMOVA uses a unified token sequence and omni-modal instruction tuning. A plausible implication is that the field is moving from bridge-based alignment toward tighter end-to-end modeling, while still relying on pretrained cross-modal priors.

## 4. Benchmarks and empirical performance

The empirical record for speech-driven grounding is now sufficiently concrete to separate feasibility from remaining gaps. YOSS evaluates audio-image retrieval and object detection on Flickr8k, Flickr30k, SpokenCOCO, GQA, and COCO images, using both human and TTS audio. On Flickr8k validation, its ablation study reports **Images $\rightarrow$ Audio: R@10 = 90.5\%** and **Audio $\rightarrow$ Images: R@10 = 78.7\%** when both pair and CORAL alignment are used. On COCO detection, **YOSS-base** reports **AP: 34.00, AP50: 47.20, AP75: 36.90**, while **YOSS-large** reports **AP: 39.20, AP50: 53.30, AP75: 42.60**. On LVIS zero-shot detection, AP ranges from **6.80** to **16.30** depending on alignment and finetuning. The paper explicitly notes that fully zero-shot audio-driven object localization is feasible, although audio still lags text [2409.18372].

The 2025 Speech-to-See system reports substantially higher closed-set COCO performance. Without MoLE it reaches **AP 54.1, AP50 70.2, AP75 59.6**; with MoLE it reaches **AP 56.2, AP50 71.3, AP75 60.7**. In zero-shot COCO evaluation, training on Objects365 with MoLE yields **AP 39.8, AP50 50.3, AP75 41.9**, while adding Flickr and GQA raises this to **AP 42.7, AP50 55.7, AP75 46.8**. On LVIS zero-shot detection, the same paper reports **AP 19.9, APr 7.9, APc 13.3, APf 18.9** for its MoLE model, exceeding the YOSS numbers but still trailing text-guided Grounding DINO. The paper also reports an efficiency comparison against a cascaded Whisper + Grounding DINO pipeline: **197.8M vs 266.7M parameters** and **RTF: 0.35 vs 0.41** [2509.16670].

A related but more general omni-modal benchmark picture comes from EMOVA. For custom Speech2See tasks, it reports **Speech-Image-EN: End-to-end 7.21, Text resp. unit in 7.43, ASR WER 2.45, TTS-WER 4.60**; **Speech-Image-ZH: End-to-end 5.87, Text resp. unit in 6.51, ASR WER 1.85, TTS-WER 13.95**. These results situate spoken visual reasoning within a broader any-to-any multimodal capability rather than object detection alone [2409.18042].

Taken together, these results establish three points. First, direct speech-conditioned localization is no longer merely conceptual. Second, explicit alignment modules materially improve retrieval and detection. Third, the performance gap to text-guided open-vocabulary detectors remains a defining empirical constraint.

## 5. Assistive, interactive, and generative extensions

Speech2See has been strongly motivated by accessibility and human-robot interaction. The YOSS paper explicitly frames spoken commands as more intuitive in real-world scenarios and argues that speech-conditioned localization can improve robotic systems and computer-vision applications. The 2025 Speech-to-See paper similarly emphasizes human-robot interaction and voice navigation as settings in which textual input is not available or practical [2409.18372], [2509.16670].

SeeSay provides a concrete assistive instantiation. It consists of a 3D-printed glasses attachment containing an ESP32 board, camera, and battery, plus a Raspberry Pi 5B processing unit. The glasses capture images every 30 seconds, while user speech is transcribed with Whisper. Scene descriptions are generated via ChatGPT4, stored as embeddings, and retrieved by cosine similarity inside a retrieval-augmented generation pipeline orchestrated through MQTT. Answers are generated locally with Phi-2 when possible, escalated to ChatGPT when necessary, and spoken back with Piper TTS. The system also supports commands to “remember” entities or people, thereby augmenting a persistent visual memory store [2410.03771].

Its reported outcomes show a split between routine environmental querying and harder navigation. **Simple Question** and **Scene Description** each report **86% accuracy**, **4.5 usability**, and **9.2 s** or **10.5 s** response time, respectively. **Recognize Person** reports **85% accuracy** and **16.5 s** response time; **Item Locator** reports **80% accuracy** and **18.8 s** response time. By contrast, **Indoor Navigation** and **Street Navigation** both report **70% accuracy**, with usability dropping to **3.5** and **3.0**, and response times rising to **18.5 s** and **16.5 s**. The paper attributes part of this degradation to cloud dependence, computational limits on Raspberry Pi, and the difficulty of dynamic spatial reasoning [2410.03771].

EMOVA extends the assistive and interactive conception into an end-to-end omni-modal assistant with no external ASR or TTS modules. A spoken request can be converted into semantic speech units, fused with image tokens, and answered in text, speech, or both. The examples explicitly include speech-driven OCR of charts and documents and conversational interaction with visual data. This suggests a convergence between Speech2See and omni-modal dialogue systems, with the distinction increasingly defined by task output rather than by architecture alone [2409.18042].

A generative extension appears in VoxStudio, which proposes the first unified and end-to-end speech-to-image model that generates expressive images directly from spoken descriptions. Its **speech information bottleneck** compresses dense speech embeddings into compact semantic tokens while preserving prosody and emotional nuance, and a Stable Diffusion-based latent diffusion model conditions on those tokens via cross-attention. The release of **VoxEmoset**, a paired emotional speech-image dataset, reframes Speech2See not only as grounding or querying but also as direct expressive image synthesis from speech [2511.03423].

## 6. Adjacent directions, misconceptions, and open problems

Speech2See is often conflated with any system involving both speech and vision, but the cited literature draws a sharper distinction. **Visual Speech Recognition** and **Vid2speech** operate in the reverse direction: they infer spoken content or reconstruct speech from visual input alone. VSR defines its main aim as recognizing spoken words by using only the visual signal produced during speech, while Vid2speech reconstructs an intelligible acoustic speech signal from silent video frames with an end-to-end CNN [1409.1411], [1701.00495]. These are better understood as adjacent **see-to-speech** or **vision-to-speech** tasks, not Speech2See in the strict speech-conditioned sense.

The same boundary applies to newer silent-video speech generation systems. ImaginTalk generates faithful speech from silent face video using a discrete lip aligner, an error detector, a face-style adapter, and a style diffusion transformer. AV-TranSpeech uses lip movements to improve direct speech-to-speech translation in noise. Both are multimodal speech systems, but their conditioning direction is visual or audiovisual rather than speech-to-visual [2503.14928], [2305.15403].

A second misconception is that Speech2See is simply ASR plus a text-guided vision model. The 2025 Speech-to-See paper explicitly treats such text-mediated pipelines as a bottleneck because they lose speech’s rich cues, introduce cascaded errors, and limit end-to-end optimization. YOSS already acknowledged a persistent **gap vs. text-based grounding**, attributing it to speech variability, ASR errors, and less training data, while the later Speech-to-See paper reports that the gap to text-guided detectors remains especially visible in long-tail zero-shot LVIS settings [2409.18372], [2509.16670].

Data scarcity is the most recurrent structural limitation. YOSS uses paired image-audio-caption data and pseudo-labels; Speech-to-See uses synthetic multi-speaker speech via edge-TTS because COCO, Objects365, and Flickr30k do not natively contain speech; EMOVA relies on joint image-text and speech-text corpora with text as semantic bridge; SeeSay depends on cloud vision-language models because edge hardware cannot run the strongest visual systems locally. This suggests that progress is currently coupled to borrowed supervision from text or synthetic speech, even when inference is speech-native [2409.18372], [2509.16670], [2409.18042], [2410.03771].

Finally, evaluation itself remains an open issue. VASR shows that richer visual context can improve speech recognition through explicit Audio-Visual Chain-of-Thought reasoning and that models suffer from **single-modality dominance** when they either over-rely on visual context or fail to use it. AV-SpeakerBench shows that even strong multimodal large language models still struggle to align **who speaks**, **what is said**, and **when it occurs**, with **Gemini 2.5 Pro** at **73.04%** overall accuracy versus a **93.74%** human upper bound. Although these works target audiovisual speech understanding rather than speech-driven visual grounding, they diagnose the same underlying problem: robust fusion of temporally precise speech and visual evidence remains difficult even for large multimodal models [2603.07263], [2512.02231].

In the cited literature, the main forward directions are consistent: better audio-visual pretraining corpora, improved speech representations, integrated end-to-end transformers, improved robustness to noisy and diverse speech, and stronger handling of long-tail open-set categories. Speech2See is therefore best understood not as a single model family but as a research program aimed at making spoken language a first-class interface to visual perception, grounding, memory, and generation.

Source: https://www.emergentmind.com/topics/speech-to-see-speech2see