Deep-DxSearch: Retrieval Diagnostics

• Deep-DxSearch is a retrieval-oriented diagnostic paradigm that indexes medical evidence using deep embeddings to support differential diagnosis.
• The system employs methods such as Euclidean k-NN, majority vote, and compressed latent features to integrate imaging and textual data in pathology and radiology.
• Advanced approaches extend the paradigm with agentic retrieval-augmented reasoning via LLMs for traceable, explainable, and real-time clinical decision support.

Deep-DxSearch denotes a retrieval-centered family of diagnostic systems in which medical evidence is indexed in a learned representation space and then searched to support discrimination, triage, subtype assignment, or differential diagnosis. In computational pathology and radiology, the approach is instantiated as a searchable digital atlas or image archive queried by deep embeddings and nearest-neighbour matching, with majority voting or top-$n$ recall providing a computational or virtual second opinion. In later large-language-model work, the same name refers to an end-to-end agentic retrieval-augmented generation system trained with reinforcement learning for traceable diagnostic reasoning over patient records and medical knowledge corpora. Taken together, these works suggest that Deep-DxSearch is best understood as a retrieval-oriented diagnostic paradigm rather than a single invariant architecture (Shafique et al., 2023, Sze-To et al., 2021, Shafique et al., 2023, Zheng et al., 21 Aug 2025).

1. Retrieval-centered formulation

Across the cited literature, Deep-DxSearch is organized around three recurring components: a searchable repository of diagnostically confirmed material, a deep-feature representation used for indexing and similarity computation, and a retrieval-driven decision rule that converts nearest matches into diagnostic support. In the breast-tumour study, these components are explicitly described as “(1) a searchable digital atlas, (2) deep-feature indexing and search, and (3) matching for diagnostic support.” In the chest-radiograph and histopathology variants, the indexed objects are images or patches. In the agentic RAG variant, the indexed objects include disease profiles, patient records, and clinical knowledge chunks, and retrieval is embedded inside a multi-step reasoning loop (Shafique et al., 2023, Sze-To et al., 2021, Shafique et al., 2023, Zheng et al., 21 Aug 2025).

Setting	Indexed objects	Decision mechanism
Breast tumour discrimination	21 847 atlas patches from 35 tumour types	Euclidean $k$-NN; majority vote; top-$n$ tumour types
Pneumothorax search	256-D encodings for 551,383 chest X-ray images	Euclidean nearest neighbours; majority vote among top $K$
IHC-guided histopathology	1024-D KimiaNet embeddings from targeted H&E patches	standard $k$-NN; majority-$k$ subtype prediction
Agentic diagnostic reasoning	disease guidelines, matched patient records, PubMed/Wiki/textbook chunks	tool-augmented reasoning with $\langle\mathrm{lookup}\rangle$, $\langle\mathrm{match}\rangle$, $\langle\mathrm{search}\rangle$, $\langle\mathrm{diagnose}\rangle$

This organization distinguishes Deep-DxSearch from a conventional end-to-end classifier. The retrieved neighbours or retrieved knowledge are part of the output interface, not merely an internal latent computation. That design is central to the repeated characterization of search as a “virtual second opinion” or “computational second opinion” in the imaging literature, and as “traceable diagnostic reasoning” in the agentic RAG formulation (Sze-To et al., 2021, Shafique et al., 2023, Zheng et al., 21 Aug 2025).

2. Chest-radiograph realization: AutoThorax-Net and large-scale pneumothorax search

In chest radiography, Deep-DxSearch was instantiated through the Autoencoding Thorax Net, or AutoThorax-Net, for pneumothorax search. The system preprocesses a frontal chest X-ray by splitting it vertically at the midsagittal plane into left and right halves, horizontally flipping the right half so that both halves have the same view, and retaining the entire image. Each of the three images is resized to $k$0 and passed through a DenseNet121 pretrained on ImageNet. The output of the final global average-pooling layer from each branch is a 1024-D feature vector, and concatenation yields a 3072-D representation. An autoencoder then compresses this representation through fully connected layers $k$1, with ReLU activations and dropout$k$2, producing a 256-D latent feature vector used for search (Sze-To et al., 2021).

After training, every archived chest X-ray is encoded into $k$3 and stored in an index such as a brute-force array, KD-tree, or FAISS. Retrieval uses Euclidean distance,

$k$4

and the top-$k$5 nearest neighbours are converted into a diagnosis by majority vote,

$k$6

The archive was assembled from ChestX-ray14, CheXpert, and MIMIC-CXR frontal views, yielding 551,383 images, of which 34,605 were pneumothorax and 516,778 were non-pneumothorax. In the semi-automated setting restricted to normal and pneumothorax, 194,608 images were used with 10-fold cross-validation, and the best result for $k$7 was AUC = 92 %, sensitivity $k$8 %, and specificity $k$9 %. In the fully automated setting over all 551,383 images, the best result for $n$0 was AUC = 82 %, sensitivity $n$1–73 %, and specificity $n$2 %. The paper also reports that encoding a single image takes $n$3 ms on a GPU, brute-force distance to $n$4K$n$5 floats requires $n$6–$n$7 ms on a modern GPU, sorting top-$n$8 adds $n$9 ms, and end-to-end query time is $K$0 ms, supporting the claim that real-time triage is feasible in a clinical PACS (Sze-To et al., 2021).

Two features of this realization are methodologically important. First, search is used as the diagnostic primitive rather than as a post hoc explanation layer over a classifier. Second, the representation is intentionally compressed, which makes large-archive retrieval operationally plausible. The paper frames this as clinically relevant because matched archived cases can provide an explainable output through their labels and associated metadata (Sze-To et al., 2021).

3. Histopathology atlases: breast WHO taxonomy and IHC-guided targeted search

In computational pathology, Deep-DxSearch has been developed around digital atlases of tumour morphology. In the breast-tumour study, Shafique et al. indexed the WHO breast taxonomy, spanning 35 primary breast-tumour types as defined in the WHO Classification of Tumours, 5th Ed. The dataset comprised 38 archival whole-slide images, each containing one tumour type; 35 WSIs formed the atlas and 3 WSIs served as held-out test cases. From tumour regions delineated by an expert pathologist in ASAP, 512$K$1512 patches were densely sampled at high resolution with overlap varying 20–80 %. A high-cellularity filter based on H&E colour deconvolution, a fixed threshold on the hematoxylin channel, and a cellularity ratio threshold of $K$2 % retained “abnormal” patches while discarding low-cellularity background. This yielded 32 530 extracted patches in total, with 21 847 atlas patches and 2 406 test patches (Shafique et al., 2023).

Feature extraction used KimiaNet, a DenseNet-121 fine-tuned on 240 k histopathology patches from TCGA to discriminate 30 classes of tumours and normal tissue. The final classification layer was removed, and the 1024-dimensional output of the global average-pooling layer was used as the patch embedding,

$K$3

For visualization and clustering, PCA could be applied to the 1024-D vectors with the top 50 principal components retained. Retrieval used Euclidean distance over a flat table of $K$4 pairs for $K$5, with brute-force $K$6-NN on GPU and no tree or hashing. On the held-out WSIs—Encapsulated Papillary Carcinoma, Intraductal Papilloma, and Solid Papillary Carcinoma—the system achieved Top-1 (majority-1) accuracy = 88 %, Top-3 $K$7 %, Top-5 $K$8 %, and Top-7 $K$9 %. For majority-1, Class 11 had Precision 0.96, Recall 0.95, F$k$0 0.95; Class 12 had Precision 1.00, Recall 1.00, F$k$1 1.00; and Class 32 had Precision 0.98, Recall 0.47, F$k$2 0.64 (Shafique et al., 2023).

A second histopathology realization incorporated immunohistochemistry guidance. In Hodgkin-lymphoma experiments, two co-registered IHC biomarker images, CD30 and PAX5, were combined into a Composite Biomarker Image,

$k$3

with $k$4 and $k$5. Before summation, each biomarker image was passed through an ANFIS filter and morphological clean-up. A normalized attention map derived from the CBI was then used to sample only targeted $k$6 H&E patches at 20$k$7 for which the patch-mean attention exceeded $k$8. The dataset contained 23 Hodgkin-lymphoma cases, including 15 classical (CHL) and 8 nodular (NLPHL), 69 whole-slide images including H&E plus 2 IHC slides per case, and 9 296 H&E patches. KimiaNet was again used as a fixed feature extractor, producing 1024-D embeddings without fine-tuning (Shafique et al., 2023).

Retrieval in the IHC-guided system also used Euclidean distance with standard $k$9-NN search, and subtype prediction used a majority-$k$0 rule evaluated at $k$1. Validation was leave-one-patient-out. Relative to H&E-only random-patch search, the paper reports CHL top-1 precision 75.0% $k$2 84.6%, recall 80.0% $k$3 73.3%, and F1 77.4% $k$4 78.6%; CHL majority-5 precision 83.3% $k$5 100.0% and F1 74.1% $k$6 88.9%; NLPHL top-1 F1 53.3% $k$7 66.7%; and majority-5 F1 63.2% $k$8 84.2%. The stated interpretation is that IHC-guided search retrieves relevant data more accurately than a conventional H&E-only search engine and can accurately conclude subtypes through majority votes (Shafique et al., 2023).

4. Similarity, voting, and differential narrowing

Despite differences in modality, the image-based Deep-DxSearch systems rely on closely related similarity and decision rules. For image patches or radiographs denoted by $k$9 and $\langle\mathrm{lookup}\rangle$0 with embeddings $\langle\mathrm{lookup}\rangle$1 and $\langle\mathrm{lookup}\rangle$2, Euclidean distance is the principal retrieval metric,

$\langle\mathrm{lookup}\rangle$3

Cosine similarity also appears as a common alternative in the literature,

$\langle\mathrm{lookup}\rangle$4

but the breast-tumour and IHC-guided studies report Euclidean distance in practice, and the pneumothorax system likewise uses Euclidean distance over its 256-D latent space (Shafique et al., 2023, Shafique et al., 2023, Sze-To et al., 2021).

The canonical classifier induced by retrieval is majority vote over the nearest neighbours. In the breast-tumour formulation, for the neighbourhood $\langle\mathrm{lookup}\rangle$5 and class counts

$\langle\mathrm{lookup}\rangle$6

the predicted label is

$\langle\mathrm{lookup}\rangle$7

If $\langle\mathrm{lookup}\rangle$8, the majority-1 vote is considered confident. The same general principle appears in the chest X-ray and Hodgkin-lymphoma systems, where the output is the mode of the retrieved labels or the subtype with the largest vote count (Shafique et al., 2023, Sze-To et al., 2021, Shafique et al., 2023).

A further diagnostic role of retrieval is differential narrowing rather than single-label commitment. The breast-tumour study defines “Top-$\langle\mathrm{lookup}\rangle$9 accuracy” as correct when the true label appears at least once among the top $\langle\mathrm{match}\rangle$0 retrieved neighbours, and “majority-$\langle\mathrm{match}\rangle$1 accuracy” as the stricter requirement that a majority among the top $\langle\mathrm{match}\rangle$2 neighbours carry the true label. It also reports “Top-3@Top-$\langle\mathrm{match}\rangle$3,” namely the three most frequently retrieved atlas labels among the top-$\langle\mathrm{match}\rangle$4 neighbours and their percentages for each test WSI. This is presented as a way to narrow a broad differential, especially for rare entities such as solid papillary carcinoma (Shafique et al., 2023).

These formulations clarify a common misconception that retrieval merely duplicates classification. In the cited systems, retrieval changes the diagnostic object: instead of only estimating a posterior label, the system returns matched cases, retrieved label distributions, or a shortlist of likely tumour types. The intended significance is interpretive rather than purely discriminative, since pathologists or radiologists can inspect the retrieved exemplars or associated metadata as part of the decision process (Sze-To et al., 2021, Shafique et al., 2023).

5. Agentic RAG reformulation for traceable diagnostic reasoning

The 2025 formulation generalizes Deep-DxSearch from image retrieval to retrieval-augmented reasoning over heterogeneous medical corpora. Its retrieval environment $\langle\mathrm{match}\rangle$5 contains three sub-corpora: a Disease Information Guideline built from the ICD-10-CM website, Orphanet, and public health sites such as NIH, Mayo, WebMD, and NCBI; a Patient Record Database containing 177 029 curated records from 6 public datasets plus in-house Xinhua records, with 52 000+ cases retained for retrieval after preprocessing; and a Clinical Knowledge Collection containing 3.31 M Wikipedia medical pages, 23.9 M PubMed abstracts, and 18 standard textbooks split into 125 847 chunks. The guideline corpus contains 16 371 diseases and 257 022 disease–phenotype pairs, with 100% ICD coverage, 38.7% Orpha coverage, and $\langle\mathrm{match}\rangle$650% HPO term coverage, while indexing combines BM25, FAISS, BioLORD embeddings, Pyserini, and LLM-based summarization (Zheng et al., 21 Aug 2025).

This system casts diagnosis as an episodic RL problem in which the LLM is the agent and the retrieval corpus is the environment. The active action space is

$\langle\mathrm{match}\rangle$7

and passive observations are returned as $\langle\mathrm{match}\rangle$8, $\langle\mathrm{match}\rangle$9, and $\langle\mathrm{search}\rangle$0. The trajectory ends when $\langle\mathrm{search}\rangle$1. The objective is

$\langle\mathrm{search}\rangle$2

and the policy is optimized with Group-Relative PPO. Reward design is central: a strict format coefficient $\langle\mathrm{search}\rangle$3 gates the final reward, which then combines matching reward $\langle\mathrm{search}\rangle$4, search reward $\langle\mathrm{search}\rangle$5, and diagnosis reward $\langle\mathrm{search}\rangle$6 through

$\langle\mathrm{search}\rangle$7

The reward weights are annealed over four rounds: R1 emphasizes $\langle\mathrm{search}\rangle$8, R2 emphasizes $\langle\mathrm{search}\rangle$9, R3 emphasizes $\langle\mathrm{diagnose}\rangle$0, and R4 uses $\langle\mathrm{diagnose}\rangle$1 (Zheng et al., 21 Aug 2025).

The core agent is an instruction-tuned LLM such as Qwen-2.5 or Llama-3, wrapped in an interleaved tool-use loop. At generation time, the system detects tool-invocation tokens, pauses decoding, issues the query to the environment, appends returned evidence, and resumes generation. Training uses HybridFlow + vLLM, a batch size of 256 trajectories per gradient step, 800 training steps, BM25 hyperparameters $\langle\mathrm{diagnose}\rangle$2 and $\langle\mathrm{diagnose}\rangle$3 for guideline lookup, and four retrieval servers for Wiki, PubMed, textbooks, and summarization (Zheng et al., 21 Aug 2025).

Empirically, the paper reports that Deep-DxSearch consistently outperforms prompt-engineering and training-free RAG approaches across multiple data centers. With a Qwen-14B backbone on six in-distribution datasets, performance reaches up to 35.22% Acc@1 / 46.83% Acc@5 on MIMIC-C and 70.48% / 82.96% on RareBench, compared with Vanilla LLM at 8.80% / 12.40% and 18.07% / 31.38%, and Training-free RAG at 13.22% / 15.91% and 34.70% / 59.20%. In out-of-distribution evaluation, Qwen-14B Deep-DxSearch attains 31.09% / 42.70% on Mendeley and 35.13% / 45.77% on Xinhua rare cases. Against broader diagnostic baselines, its ID averages are 43.04% Acc@1 and 53.30% Acc@5 on common diseases, and 49.25% and 61.02% on rare diseases, exceeding DeepSeek-R1 and GPT-4o+Env. Ablations report that removing policy reward causes drops of –16.68% Acc@1 on common disease and –22.14% on rare disease, while removing patient records causes drops of –11.78% and –17.46%, and quantitative interpretability shows Symptom Association (Hit@20) from 25.79%$\langle\mathrm{diagnose}\rangle$460.39%, Differential Diagnosis (Acc@5) up about 30 pp from 41.70%$\langle\mathrm{diagnose}\rangle$571.07%, and an Irrelevance Exclusion gain of +10% under noisy retrieval (Zheng et al., 21 Aug 2025).

6. Relation to differential-diagnosis generation, limitations, and open problems

Deep-DxSearch occupies a design space adjacent to, but not identical with, generative differential-diagnosis models. "DDxT: Deep Generative Transformer Models for Differential Diagnosis" formulates DDx list generation as an encoder–decoder sequence-to-sequence problem with autoregressive likelihood

$\langle\mathrm{diagnose}\rangle$6

trained with a generator loss and a classification loss. DDxT uses a 6-layer encoder, 6-layer decoder, 4 attention heads, model dimension $\langle\mathrm{diagnose}\rangle$7, feed-forward dimension $\langle\mathrm{diagnose}\rangle$8, dropout rate $\langle\mathrm{diagnose}\rangle$9, Adam with initial learning rate $k$00 and decay $k$01 per epoch, and 20 epochs on DDXPlus, a dataset of 1.3 million synthetic patient cases with 49 possible pathologies. It reports mean accuracy 99.82% and mean F1 0.9472 for DDx list generation, and mean accuracy 99.98% with mean F1 0.9949 for ground-truth pathology classification, outperforming the RL baselines AARLC and BASD on the benchmark (Alam et al., 2023).

A plausible implication is that Deep-DxSearch and DDxT represent two complementary responses to the same diagnostic problem. DDxT emphasizes direct generative mapping from patient record to DDx list, whereas Deep-DxSearch emphasizes retrieval-aware reasoning or retrieval-grounded case matching. The agentic RAG formulation of Deep-DxSearch narrows this gap by making retrieval part of an explicit reasoning policy rather than a separate postprocessing stage (Alam et al., 2023, Zheng et al., 21 Aug 2025).

The literature also defines several limitations with unusual clarity. The breast-tumour atlas used only 38 WSIs from a single institution, relied on a brute-force index that becomes a scalability bottleneck, and remained patch-level rather than whole-slide-level; the authors identify multi-centre expansion, approximate-NN engines such as FAISS, slide-level aggregation, end-to-end fine-tuning on the 35 WHO classes, multi-scale or graph-based indices, and a pathologist-friendly GUI as future work (Shafique et al., 2023). The agentic RAG system similarly notes that clinical deployment and prospective validation are still pending, that corpus customization for local practice patterns remains undeveloped, and that extension beyond diagnosis toward treatment planning or monitoring is open (Zheng et al., 21 Aug 2025).

These limitations matter because they mark the boundary between proof-of-concept retrieval systems and clinically integrated diagnostic infrastructure. The image-search variants demonstrate that indexed archives can support rare-subtype discrimination and second-opinion workflows. The agentic RAG variant extends the same retrieval principle to textual reasoning and rare-disease diagnosis. The cited record therefore presents Deep-DxSearch as an evolving retrieval paradigm whose central question is not simply whether a model can classify, but how retrieved cases or retrieved evidence can be made diagnostically useful, traceable, and operational at scale (Sze-To et al., 2021, Shafique et al., 2023, Zheng et al., 21 Aug 2025).