DocsRay: Zero-Shot Document Understanding

• DocsRay is a zero-shot, training-free system that uses pseudo-TOC-guided hierarchical retrieval to efficiently analyze complex multimodal documents.
• It segments documents into semantically coherent sections and employs a two-stage retrieval process that reduces query latency by 45.4%.
• Empirical results show DocsRay-Pro achieves 64.7% accuracy on MMLongBench-Doc, outperforming prior LVLM benchmarks and traditional OCR pipelines.

DocsRay is a zero-shot, training-free document understanding system that integrates pseudo Table of Contents (TOC) generation with hierarchical Retrieval-Augmented Generation (RAG) for complex multimodal documents (Jeong et al., 31 Jul 2025). It is designed for PDFs and structured documents containing text, images, charts, tables, diagrams, and multi-column layouts, and it relies on multimodal LLMs’ native capabilities rather than task-specific fine-tuning, separate OCR pipelines, or specialized layout models. The system combines prompt-based semantic structuring, zero-shot multimodal analysis, and an efficient two-stage hierarchical retrieval procedure. On MMLongBench-Doc, whose documents average 49.4 pages and 20,971 textual tokens, DocsRay-Pro attains 64.7% accuracy; the same study reports a reduction in query latency from 3.89 to 2.12 seconds, corresponding to a 45.4% speedup (Jeong et al., 31 Jul 2025).

1. Conceptual basis and architectural organization

DocsRay is defined by four design principles: it is zero-shot and training-free; it unifies heterogeneous content; it targets semantic coherence and efficiency; and it is modular but synergistic (Jeong et al., 31 Jul 2025). In the formulation given for the system, zero-shot deployment means that it requires no task-specific fine-tuning or additional training data and can be applied immediately to new PDFs or structured documents. Unification of heterogeneous content means that the same multimodal LLM processes text, images, tables, charts, diagrams, and multi-column layouts, avoiding separate OCR or layout modules.

The architectural workflow is divided into two stages. Stage 1 performs document processing: PDF parsing, multimodal analysis, pseudo-TOC generation, chunking, and embedding construction. Stage 2 performs query processing: coarse search over sections, fine search over chunks, optional query refinement, and RAG answer generation (Jeong et al., 31 Jul 2025). A plausible implication is that DocsRay treats document understanding not as a single-pass extraction problem but as a structured retrieval problem over semantically induced document units.

The system’s central claim to efficiency is that pseudo-TOC-guided hierarchy changes retrieval from flat chunk search over the entire document to section-first retrieval followed by localized chunk search. This preserves document semantics while constraining the search space. The same description also emphasizes that the three major components outperform any component in isolation, framing DocsRay as an orchestrated pipeline rather than a collection of independent utilities (Jeong et al., 31 Jul 2025).

2. Pseudo-TOC generation and semantic structuring

DocsRay organizes an unstructured document into a hierarchical pseudo-TOC in three phases: initial segmentation by semantic boundary detection, size-constrained merging, and title generation (Jeong et al., 31 Jul 2025). In the first phase, document pages $P=\{p_1,\dots,p_n\}$ are grouped into fixed-size chunks of $k$ pages, with $k=5$ given as an example. For each adjacent pair of chunks, the system extracts the last 500 characters of the first chunk and the first 500 characters of the next chunk. A multimodal LLM is then prompted to decide whether the two excerpts discuss the same topic or whether the second excerpt introduces a new topic; a reply of 1 marks a section boundary.

In the second phase, any section $s$ smaller than a minimum size $m$ is merged with its semantically closest neighbor, with $m=3$ pages given as an example. Semantic closeness is measured by cosine similarity of section-level content embeddings. This step constrains fragmentation and enforces minimum section granularity. In the third phase, representative text from each final section, typically the first few paragraphs, is passed to a title-generation prompt that returns one concise title capturing the section’s main topic. The resulting output is a human-readable pseudo-TOC of the form “Section 1: Introduction,” “Section 2: Data Collection and Methodology,” and so forth (Jeong et al., 31 Jul 2025).

The pseudo-TOC mechanism is not restricted to a single level. DocsRay can be applied recursively to long sections to create multi-level hierarchies, such as chapters and subsections. The associated pseudocode is correspondingly simple: split pages into chunks of size $k$; use the boundary-detection prompt on adjacent chunk pairs; merge under-sized sections by semantic similarity; and generate a concise title for each section. This suggests that semantic structuring is implemented as prompt-mediated segmentation rather than as supervised layout analysis or structural parsing.

3. Zero-shot multimodal analysis of heterogeneous document elements

DocsRay’s multimodal analysis is explicitly designed to avoid separate OCR, table-parsing, or image-analysis pipelines (Jeong et al., 31 Jul 2025). For textual content, raw text is extracted via PyMuPDF. Multi-column layouts are detected by K-means clustering on text-bounding-box x-coordinates, after which columns are reassembled in natural reading order. This procedure allows the text stream to be reconstructed before indexing.

For tables, the system identifies rows of text aligned at consistent x-coordinates. Detected tables are rendered at 2× zoom as images so that the LLM’s vision model preserves structure, and the model is prompted to describe the table semantically. For charts and diagrams above a size threshold, with 100×100px given as an example, DocsRay uses a visual-content prompt that instructs the LLM to explain what a chart shows or to describe a photo or illustration. The returned text is then indexed alongside ordinary text (Jeong et al., 31 Jul 2025).

Vector graphics are filtered with heuristics: small or noise-like elements are discarded when size is below 50×50px, when aspect ratio is extreme, or when color diversity is low. When text extraction fails, as in scanned images, the system uses adaptive OCR via LLM vision with a prompt requesting readable paragraphs. All extracted and generated text is concatenated in page order, preserving semantic coherence, and then split into chunks aligned to the pseudo-TOC. A plausible implication is that DocsRay converts a multimodal document into a unified text-centric substrate while preserving the document’s higher-level section structure.

4. Hierarchical retrieval, embedding design, and retrieval complexity

DocsRay’s retrieval system is a two-stage hierarchical procedure whose stated complexity changes from flat $O(N)$ retrieval to $O(S + k_1 N_s)$ (Jeong et al., 31 Jul 2025). Here $N$ is the total number of chunks in the document, $k$0 is the number of sections in the pseudo-TOC, $k$1 is the average number of chunks per section, and $k$2 is the number of top sections retrieved in the coarse stage. The flat retrieval cost is

$k$3

whereas the hierarchical retrieval cost is

$k$4

The coarse stage computes similarity against section embeddings, and the fine stage restricts chunk-level comparison to the top $k$5 sections.

Each section has two precomputed embeddings: a title embedding,

$k$6

and a content embedding,

$k$7

where $k$8 denotes the chunks in section $k$9. Given a query embedding $k=5$0, the section score is

$k=5$1

with $k=5$2 controlling the balance between title-based and content-based matching (Jeong et al., 31 Jul 2025). Fine search then compares the query embedding against chunk embeddings inside the top-ranked sections.

Embedding construction uses two 1024-dimensional encoders. DocsRay computes $k=5$3 and $k=5$4 for each snippet, concatenates them, and applies $k=5$5 normalization to produce a combined 2048-dimensional vector:

$k=5$6

The system also includes optional iterative query refinement: after initial retrieval $k=5$7, the LLM inspects the retrieved context, formulates a refined query $k=5$8, and repeats retrieval. The retrieval flow is illustrated in the supplied example query “Revenue growth in Asia,” where coarse search selects Sections 3, 7, and 9, and fine search then selects Chunks 7.3, 9.1, and 9.4 for answer generation and possible refinement (Jeong et al., 31 Jul 2025).

5. Empirical results, ablations, and operational parameters

The reported evaluation uses MMLongBench-Doc, where documents average 49.4 pages and 20,971 textual tokens (Jeong et al., 31 Jul 2025). The latency reduction attributed to pseudo-TOC guidance occurs in Stage 2 chunk scoring: without pseudo-TOC, chunk scoring takes 3.89 s; with pseudo-TOC, it takes 2.12 s. The efficiency gain is reported as

$k=5$9

that is, a 45.4% speedup.

On the same benchmark, DocsRay-Pro with 27B parameters attains 64.7% accuracy, DocsRay-Base with 12B attains 62.8%, and DocsRay-Lite with 4B attains 31.8% (Jeong et al., 31 Jul 2025). The best prior LVLM result listed is GPT-4.1 at 49.7%, so DocsRay-Pro is reported as +15.0 percentage points higher. The best OCR+LLM pipeline listed is Gemini-1.5-Pro+OCR at 31.2%. The ablation of pseudo-TOC reports 62.8% at 2.12 s with pseudo-TOC and 63.5% at 3.89 s without pseudo-TOC, indicating a trade of 0.7 percentage points in accuracy for a 45.4% latency reduction. The same summary states that DocsRay achieves near–human-level performance of 64.7% versus 65.8% expert (Jeong et al., 31 Jul 2025).

The implementation details are concrete. Chunking uses 550 tokens per chunk with 25-token overlap. Coarse search uses $s$0 and top-5 sections, so $s$1. Fine search returns the top-10 chunks from the selected sections. Answer generation uses temperature $s$2, top-$s$3, and repeat-penalty $s$4. Hardware requirements are given as at least 16 GB VRAM for Lite, at least 32 GB VRAM for Base, and at least 80 GB VRAM for Pro, with NVIDIA H100 given as an example for the latter. The software stack includes Python 3.11, PyTorch $s$5, llama-cpp-python, PyMuPDF, tiktoken, and numpy. All components—Gemma-3 models, BGE-M3, and E5-Large embeddings—are described as publicly available on HuggingFace, and complete code, prompts, and evaluation scripts are stated to be released under the MIT license (Jeong et al., 31 Jul 2025).

6. Practical considerations, scope, and nomenclature

DocsRay is described as deployable “out of the box” on new document collections because it is training-free (Jeong et al., 31 Jul 2025). The pseudo-TOC mechanism is reported as robust to document style and language, although prompts may require minor tuning for non-English scripts. Embedding fusion by concatenation is said to incur modest memory overhead because of the 2048-dimensional vectors, in exchange for an 8–9 percentage point retrieval accuracy gain. These points situate DocsRay as a practical zero-shot system rather than a benchmark-only construction.

A common source of confusion is the similarity between the names DocsRay and DART-Ray. They refer to different systems. DART-Ray is “Deterministic 3D Adaptive Ray-Tracing,” a purely ray-tracing code for 3D dust radiative transfer on an adaptive Cartesian grid; it computes the radiation field energy density, the scattered and thermal dust source functions, and images for arbitrary observer positions (Natale et al., 2017). By contrast, DocsRay addresses document understanding through pseudo-TOC-guided hierarchical RAG rather than radiative transfer. The naming resemblance is therefore superficial rather than methodological.

Within its stated scope, DocsRay is oriented toward long-document multimodal understanding where structural inconsistencies and limited training data availability are significant constraints (Jeong et al., 31 Jul 2025). This suggests that its main contribution lies not in a new trained backbone, but in a training-free orchestration of semantic structuring, multimodal conversion, and hierarchical retrieval over long, heterogeneous documents.