Speech2See: Speech-Driven Visual Grounding

• Speech2See is a family of multimodal systems where spoken input is directly mapped to visual outputs like object detection and image generation.
• The approach employs techniques such as audio-CLIP locking, contrastive learning, and parameter-efficient decoding to align speech with visual features.
• It supports applications in assistive technology, robotics, and generative tasks, while addressing challenges like data scarcity and long-tail detection.

Searching arXiv for the cited Speech-to-See and closely related multimodal speech–vision papers. Speech-to-See (Speech2See) designates a family of multimodal systems in which spoken input serves as the control signal for visual understanding, visual grounding, or visual generation, rather than text. In the most specific computer-vision usage, it denotes audio grounding or speech-driven open-set object detection: localizing and identifying objects directly from speech in open-vocabulary settings (Yang et al., 2024, Lu et al., 20 Sep 2025). In a broader assistive and omni-modal usage, it also covers systems that answer spoken questions about the environment from current or remembered visual context, as well as models that generate images directly from spoken descriptions (Yu, 2024, Chen et al., 2024, Lee et al., 5 Nov 2025).

1. Definition, scope, and lineage

A useful way to delimit Speech2See is by the direction of conditioning. The canonical setting is speech $\rightarrow$ vision-grounded output: a spoken command, caption, or question is mapped to a localized object, a scene-grounded answer, or a generated image. This distinguishes Speech2See from text-guided grounding, where language is already symbolic, and from inverse audiovisual tasks such as lip reading or silent-video-to-speech reconstruction.

Representative formulations in the cited literature include YOSS, which introduces Audio Grounding as localizing objects in images given spoken queries or descriptions; Speech-to-See, which frames the problem as speech-driven open-set object detection; SeeSay, which treats spoken questions as the front end of an assistive visual-query system; EMOVA, which treats speech, text, and vision as a unified token space for spoken multimodal dialogue; and VoxStudio, which extends the idea to direct speech-to-image generation (Yang et al., 2024, Lu et al., 20 Sep 2025, Yu, 2024, Chen et al., 2024, Lee et al., 5 Nov 2025).

Formulation	Representative system	Output
Audio grounding	YOSS	Object localization in images
Speech-driven open-set detection	Speech-to-See	Bounding boxes and class labels
Assistive spoken visual query	SeeSay	Audio answers grounded in scene memory
Omni-modal spoken visual QA	EMOVA	Text and/or speech responses
Expressive speech-to-image	VoxStudio	Generated images

A precursor to this line of work is visually grounded speech, in which paired speech and image data are embedded into a shared space for retrieval. “Learning to Recognise Words using Visually Grounded Speech” reports that such a model can retrieve images from spoken captions and can recognise isolated words from partial input, indicating that speech-only representations can already acquire visually grounded lexical structure without an intermediate transcript (Scholten et al., 2020).

2. Canonical formulation: speech-driven object grounding

The first explicit object-localization formulation appears in YOSS, “You Only Speak Once to See,” which defines Audio Grounding as localizing objects in images from spoken queries or descriptions. YOSS is described as the first framework specifically designed to directly map speech to visual object localization in open-vocabulary settings. Its architecture combines a frozen CLIP Vision Transformer for image embeddings, the final three feature map levels from the YOLOv8-CSPDarknet backbone for multi-scale cues, and a HuBERT-based audio encoder with an Aggregation Branch that converts frame-level speech features into a fixed-size utterance embedding (Yang et al., 2024).

YOSS then performs audio-image contrastive learning in a shared semantic space and adds audio-text alignment with paired loss and CORAL to lock audio, text, and image embeddings into the CLIP space. Detection is performed with a YOLOv8 backbone and a NAS-FPN detection head. Visual features and the audio embedding are fused to produce audio-guided queries, which steer the detector toward regions relevant to the spoken input. Training proceeds in two stages: contrastive pretraining on paired image-audio-caption data, followed by detection finetuning with fixed backbone and fine-tuned audio-visual query layers.

The later “Speech-to-See: End-to-End Speech-Driven Open-Set Object Detection” reformulates the same problem more aggressively as a direct end-to-end alternative to indirect, text-mediated pipelines. It identifies three bottlenecks: scarcity of large-scale paired audio-image data, representation mismatch between speech and vision, and the loss of rich acoustic cues in ASR- or text-aligned systems. Its architecture uses a Swin Transformer visual backbone, a HuBERT speech backbone, a Grounding DINO-inspired fusion/decoder stack, a Query-Guided Semantic Aggregation module to condense redundant speech embeddings, and a Mixture-of-LoRA-Experts mechanism for parameter-efficient cross-modal adaptation during fine-tuning (Lu et al., 20 Sep 2025).

The Query-Guided Semantic Aggregation mechanism is explicitly defined as cross-attention from $K$ learnable queries to the speech sequence:

$$O_i = \mathrm{Attn}(q_i, X_A) = \sum_{j=1}^{N_t} \mathrm{softmax}\left( \frac{q_i \cdot x_j}{\sqrt{d}} \right) x_j$$

with $K \ll N_t$. In this formulation, the aggregated outputs become compact speech-semantic tokens for downstream fusion.

3. Alignment strategies and training regimes

The central technical problem in Speech2See is multimodal alignment: speech is temporally dense, acoustically variable, and non-symbolic, whereas modern visual grounding pipelines were largely built around text-image supervision. YOSS addresses this by projecting audio and image features into a shared semantic space and then explicitly aligning audio to CLIP text embeddings. Its full pretraining objective combines contrastive audio-image learning with audio-text alignment,

$$L_1(x_i, x_a, x_t) = L_{\mathrm{Con}(x_i, x_a)} + \eta L_{\mathrm{Align}(x_t, x_a)},$$

followed by a detection objective

$$L_2(x_i, x_a) = L_{\mathrm{cls}(x_i, x_a)} + L_{\mathrm{loc}(x_i, x_a)}.$$

This design uses text as a semantic bridge while still grounding from speech at inference time (Yang et al., 2024).

Speech-to-See replaces this bridge-heavy strategy with a progressive pre-training and fine-tuning paradigm. During pre-training, visual, audio, and decoder networks are initialized from large pretrained models, while the QSA and fusion modules are optimized to align modalities. During fine-tuning, each decoder layer receives multiple LoRA experts, and a router selects the expert with maximal routing score for the incoming query. This is intended to adapt text-pretrained decoder layers to speech-conditioned grounding without retraining the full network (Lu et al., 20 Sep 2025).

A related alignment strategy appears in EMOVA, which treats speech, text, and visual tokens as a single token sequence inside an omni-modal LLM. EMOVA uses a semantic-acoustic disentangled speech tokenizer, discretizes only the semantic component into speech units, and uses text as a bridge for omni-modal alignment. The model then jointly predicts output text tokens and output speech units conditioned on text tokens, speech units, and visual tokens:

$$\mathbb{P}(U_T^{o},U_S^{o} \mid U_T,U_S,H_V) = \prod_{i=1}^L \mathbb{P}(\boldsymbol{x}_i \mid U_{T,<i}^{o},U_{S,<i}^{o},U_T,U_S,H_V).$$

This formulation is not a detector, but it generalizes Speech2See from grounding to spoken multimodal reasoning over images, charts, PDFs, and other visual inputs (Chen et al., 2024).

These three strategies expose a methodological spectrum. YOSS uses CLIP-space locking and detection finetuning; Speech-to-See uses compact semantic aggregation plus parameter-efficient decoder adaptation; EMOVA uses a unified token sequence and omni-modal instruction tuning. A plausible implication is that the field is moving from bridge-based alignment toward tighter end-to-end modeling, while still relying on pretrained cross-modal priors.

4. Benchmarks and empirical performance

The empirical record for speech-driven grounding is now sufficiently concrete to separate feasibility from remaining gaps. YOSS evaluates audio-image retrieval and object detection on Flickr8k, Flickr30k, SpokenCOCO, GQA, and COCO images, using both human and TTS audio. On Flickr8k validation, its ablation study reports Images $\rightarrow$ Audio: R@10 = 90.5\% and Audio $\rightarrow$ Images: R@10 = 78.7\% when both pair and CORAL alignment are used. On COCO detection, YOSS-base reports AP: 34.00, AP50: 47.20, AP75: 36.90, while YOSS-large reports AP: 39.20, AP50: 53.30, AP75: 42.60. On LVIS zero-shot detection, AP ranges from 6.80 to 16.30 depending on alignment and finetuning. The paper explicitly notes that fully zero-shot audio-driven object localization is feasible, although audio still lags text (Yang et al., 2024).

The 2025 Speech-to-See system reports substantially higher closed-set COCO performance. Without MoLE it reaches AP 54.1, AP50 70.2, AP75 59.6; with MoLE it reaches AP 56.2, AP50 71.3, AP75 60.7. In zero-shot COCO evaluation, training on Objects365 with MoLE yields AP 39.8, AP50 50.3, AP75 41.9, while adding Flickr and GQA raises this to AP 42.7, AP50 55.7, AP75 46.8. On LVIS zero-shot detection, the same paper reports AP 19.9, APr 7.9, APc 13.3, APf 18.9 for its MoLE model, exceeding the YOSS numbers but still trailing text-guided Grounding DINO. The paper also reports an efficiency comparison against a cascaded Whisper + Grounding DINO pipeline: 197.8M vs 266.7M parameters and RTF: 0.35 vs 0.41 (Lu et al., 20 Sep 2025).

A related but more general omni-modal benchmark picture comes from EMOVA. For custom Speech2See tasks, it reports Speech-Image-EN: End-to-end 7.21, Text resp. unit in 7.43, ASR WER 2.45, TTS-WER 4.60; Speech-Image-ZH: End-to-end 5.87, Text resp. unit in 6.51, ASR WER 1.85, TTS-WER 13.95. These results situate spoken visual reasoning within a broader any-to-any multimodal capability rather than object detection alone (Chen et al., 2024).

Taken together, these results establish three points. First, direct speech-conditioned localization is no longer merely conceptual. Second, explicit alignment modules materially improve retrieval and detection. Third, the performance gap to text-guided open-vocabulary detectors remains a defining empirical constraint.

5. Assistive, interactive, and generative extensions

Speech2See has been strongly motivated by accessibility and human-robot interaction. The YOSS paper explicitly frames spoken commands as more intuitive in real-world scenarios and argues that speech-conditioned localization can improve robotic systems and computer-vision applications. The 2025 Speech-to-See paper similarly emphasizes human-robot interaction and voice navigation as settings in which textual input is not available or practical (Yang et al., 2024, Lu et al., 20 Sep 2025).

SeeSay provides a concrete assistive instantiation. It consists of a 3D-printed glasses attachment containing an ESP32 board, camera, and battery, plus a Raspberry Pi 5B processing unit. The glasses capture images every 30 seconds, while user speech is transcribed with Whisper. Scene descriptions are generated via ChatGPT4, stored as embeddings, and retrieved by cosine similarity inside a retrieval-augmented generation pipeline orchestrated through MQTT. Answers are generated locally with Phi-2 when possible, escalated to ChatGPT when necessary, and spoken back with Piper TTS. The system also supports commands to “remember” entities or people, thereby augmenting a persistent visual memory store (Yu, 2024).

Its reported outcomes show a split between routine environmental querying and harder navigation. Simple Question and Scene Description each report 86% accuracy, 4.5 usability, and 9.2 s or 10.5 s response time, respectively. Recognize Person reports 85% accuracy and 16.5 s response time; Item Locator reports 80% accuracy and 18.8 s response time. By contrast, Indoor Navigation and Street Navigation both report 70% accuracy, with usability dropping to 3.5 and 3.0, and response times rising to 18.5 s and 16.5 s. The paper attributes part of this degradation to cloud dependence, computational limits on Raspberry Pi, and the difficulty of dynamic spatial reasoning (Yu, 2024).

EMOVA extends the assistive and interactive conception into an end-to-end omni-modal assistant with no external ASR or TTS modules. A spoken request can be converted into semantic speech units, fused with image tokens, and answered in text, speech, or both. The examples explicitly include speech-driven OCR of charts and documents and conversational interaction with visual data. This suggests a convergence between Speech2See and omni-modal dialogue systems, with the distinction increasingly defined by task output rather than by architecture alone (Chen et al., 2024).

A generative extension appears in VoxStudio, which proposes the first unified and end-to-end speech-to-image model that generates expressive images directly from spoken descriptions. Its speech information bottleneck compresses dense speech embeddings into compact semantic tokens while preserving prosody and emotional nuance, and a Stable Diffusion-based latent diffusion model conditions on those tokens via cross-attention. The release of VoxEmoset, a paired emotional speech-image dataset, reframes Speech2See not only as grounding or querying but also as direct expressive image synthesis from speech (Lee et al., 5 Nov 2025).

6. Adjacent directions, misconceptions, and open problems

Speech2See is often conflated with any system involving both speech and vision, but the cited literature draws a sharper distinction. Visual Speech Recognition and Vid2speech operate in the reverse direction: they infer spoken content or reconstruct speech from visual input alone. VSR defines its main aim as recognizing spoken words by using only the visual signal produced during speech, while Vid2speech reconstructs an intelligible acoustic speech signal from silent video frames with an end-to-end CNN (Hassanat, 2014, Ephrat et al., 2017). These are better understood as adjacent see-to-speech or vision-to-speech tasks, not Speech2See in the strict speech-conditioned sense.

The same boundary applies to newer silent-video speech generation systems. ImaginTalk generates faithful speech from silent face video using a discrete lip aligner, an error detector, a face-style adapter, and a style diffusion transformer. AV-TranSpeech uses lip movements to improve direct speech-to-speech translation in noise. Both are multimodal speech systems, but their conditioning direction is visual or audiovisual rather than speech-to-visual (Ye et al., 19 Mar 2025, Huang et al., 2023).

A second misconception is that Speech2See is simply ASR plus a text-guided vision model. The 2025 Speech-to-See paper explicitly treats such text-mediated pipelines as a bottleneck because they lose speech’s rich cues, introduce cascaded errors, and limit end-to-end optimization. YOSS already acknowledged a persistent gap vs. text-based grounding, attributing it to speech variability, ASR errors, and less training data, while the later Speech-to-See paper reports that the gap to text-guided detectors remains especially visible in long-tail zero-shot LVIS settings (Yang et al., 2024, Lu et al., 20 Sep 2025).

Data scarcity is the most recurrent structural limitation. YOSS uses paired image-audio-caption data and pseudo-labels; Speech-to-See uses synthetic multi-speaker speech via edge-TTS because COCO, Objects365, and Flickr30k do not natively contain speech; EMOVA relies on joint image-text and speech-text corpora with text as semantic bridge; SeeSay depends on cloud vision-LLMs because edge hardware cannot run the strongest visual systems locally. This suggests that progress is currently coupled to borrowed supervision from text or synthetic speech, even when inference is speech-native (Yang et al., 2024, Lu et al., 20 Sep 2025, Chen et al., 2024, Yu, 2024).

Finally, evaluation itself remains an open issue. VASR shows that richer visual context can improve speech recognition through explicit Audio-Visual Chain-of-Thought reasoning and that models suffer from single-modality dominance when they either over-rely on visual context or fail to use it. AV-SpeakerBench shows that even strong multimodal LLMs still struggle to align who speaks, what is said, and when it occurs, with Gemini 2.5 Pro at 73.04% overall accuracy versus a 93.74% human upper bound. Although these works target audiovisual speech understanding rather than speech-driven visual grounding, they diagnose the same underlying problem: robust fusion of temporally precise speech and visual evidence remains difficult even for large multimodal models (Tian et al., 7 Mar 2026, Nguyen et al., 1 Dec 2025).

In the cited literature, the main forward directions are consistent: better audio-visual pretraining corpora, improved speech representations, integrated end-to-end transformers, improved robustness to noisy and diverse speech, and stronger handling of long-tail open-set categories. Speech2See is therefore best understood not as a single model family but as a research program aimed at making spoken language a first-class interface to visual perception, grounding, memory, and generation.