--- title: 'TextlessRAG: Speech-Based Document QA' url: https://www.emergentmind.com/topics/textlessrag type: topic --- # TextlessRAG: Speech-Based Document QA Searching arXiv for the specified paper and closely related visual/pixel-space RAG work to ground the article. TextlessRAG is a fully “textless” speech-based Retrieval–Augmented Generation (RAG) pipeline over a large collection of document images. It is presented as the first end-to-end framework for speech-based question answering over large-scale document images, with the central design choice of eliminating ASR, TTS and OCR from the retrieval pipeline and instead directly interpreting speech, retrieving relevant visual knowledge, and generating answers over multimodal document content [2509.07538]. The system is paired with SV-DOC, described as the first bilingual speech–document RAG dataset featuring Chinese and English voice queries aligned with document images, and it is evaluated as a document-image QA system rather than a conventional text-centric RAG stack. ## 1. Definition and research scope TextlessRAG addresses a problem formulation that had not previously been explored in the cited work: knowledge base question answering over visual document images with queries provided directly in speech [2509.07538]. The knowledge base consists of page images rather than extracted text, and the query modality is raw audio rather than typed text. The intended effect is to preserve document layout, charts, tables, pictures, and other visually grounded structures that are typically degraded by OCR or linearization. The framework is explicitly positioned against prior methods that depend on cascades such as ASR \(\rightarrow\) text retrieval, OCR \(\rightarrow\) text indexing, or text-conditioned multimodal answer generation. In contrast, TextlessRAG maps speech and document images into a shared embedding space, retrieves relevant pages, refines them with layout-aware reranking, and answers with a multimodal decoder [2509.07538]. A plausible implication is that TextlessRAG belongs to a broader family of text-abstraction-free RAG systems. In the same data block, PixelRAG is described as performing retrieval and reading entirely in pixel space over web screenshots rather than document images [2606.28344]. Read together, these systems suggest a common research direction in which the native visual form of the knowledge source is retained rather than converted into text. ## 2. End-to-end architecture The pipeline is organized into three stages: a speech encoder, a visual document retrieval and layout reranking stage, and a generator module [2509.07538]. In the first stage, raw user query audio \(q\) is mapped to a fixed-length speech embedding \(e_q\). The retrieval encoder is described as reusing “ColQwen-Omni,” a multimodal encoder capable of ingesting both document screenshot images and raw audio. Each page image \(P_i\) in the document knowledge base \(\mathcal{I}=\{P_1 \ldots P_n\}\) is pre-encoded into an image embedding \(e_i\). Retrieval is based on MaxSim scoring in a shared \(d\)-dimensional embedding space, where the reported dimensionality is \(d=768\). The page relevance score is written as $$ s_i \;=\; \mathrm{Score}(P_i, q) \;=\;\mathrm{MaxSim}(e_q, e_i) \;=\;\sum_{u \in e_q}\max_{v \in e_i} \langle u,\,v\rangle . $$ Top-\(k\) retrieval selects a candidate set \(\mathcal{T}_k\) of pages for subsequent refinement. The page embeddings are pre-computed and stored in an approximate nearest-neighbor index, with FAISS IVFFlat given as the example index structure, and the query-time probe is described as taking \(O(\log n)\) time [2509.07538]. The generator stage then consumes the refined visual evidence \(\{P_t'\}_{t=1}^k\) together with the speech representation \(e_q\). The reported model is a transformer-based decoder, “Qwen2.5-Omni,” which accepts image inputs and a speech prefix and outputs an answer. ## 3. Retrieval refinement and multimodal decoding A distinctive component of TextlessRAG is the layout-aware reranking mechanism. After the top-\(k\) pages are retrieved, each page \(P_t\) is decomposed into fine-grained blocks of four types, \(C=\{\text{chart, table, text, image}\}\), using DocLayout-YOLO [2509.07538]. The decomposition is written as $$ \mathrm{YOLO}(P_t) = \bigcup_{c\in C} \mathrm{Block}^c(P_t). $$ For each block \(b\), the system computes a block-level speech–image similarity score \(s_b=\mathrm{MaxSim}(e_q,\mathrm{Enc\_image}(b))\), then keeps only the blocks that exceed a threshold \(\theta\): $$ \mathrm{Block}^c_\theta = \{\,b\in\mathrm{Block}^c(P_t)\mid s_b\ge\theta\}. $$ The surviving blocks are sorted in descending order of \(s_b\) and packed into a refined page representation \(P_t'\). The stated rationale is that block-level reranking allows the model to focus on the most relevant sub-regions, such as a specific table or chart, rather than passing an entire page to the generator. The generator module is described as a transformer-based multimodal decoder that takes a linearized sequence of image patch embeddings from the reranked blocks and a prefix embedding derived directly from the raw speech query. Internally, the decoder cross-attends to both audio tokens and image patch tokens. The token prediction distribution is given as $$ y_t = \mathrm{softmax}\bigl(W_o\,[h_t;\,\mathrm{Attn}(h_t,\,E_I\cup E_Q)]\bigr), $$ where \(h_t\) is the decoder hidden state, \(E_I\) is the set of image patch embeddings, and \(E_Q\) is the set of audio token embeddings [2509.07538]. A common point of confusion concerns the term “textless.” The abstract states that the framework eliminates ASR, TTS and OCR in the pipeline, while the generator description states that the decoder produces a text answer, which is then vocoded back into speech, and that this TTS step can be bypassed if a text response is sufficient [2509.07538]. This suggests that “textless” is used primarily to describe the internal interpretation, retrieval, and evidence-processing pipeline, rather than to prohibit textual answer output. ## 4. Training objectives TextlessRAG is trained end-to-end with three loss components: retrieval loss, reranking loss, and generation loss [2509.07538]. The retrieval objective is a contrastive loss over a training pair \((q, P^+)\) and sampled negatives \(P^-\), encouraging the positive page to score higher than negative pages under the MaxSim retrieval score. The reranking objective applies a similar contrastive principle at the block level, promoting relevant blocks of the positive page over blocks from negatives or low-score blocks below \(\theta\). The generation component is standard cross-entropy over the ground-truth answer sequence \(y_{1:|T|}\): $$ \mathcal{L}_{\mathrm{gen}} = -\sum_{t=1}^{|T|} \log p_\theta(y_t\mid y_{