- The paper introduces a curated, multi-domain corpus and benchmark for scientific code search to address sparse documentation and domain-specific challenges.
- It employs LLM-driven domain classification and context enrichment to enhance retrieval accuracy across diverse scientific repositories.
- Benchmark evaluations reveal that domain adaptation and improved documentation are crucial for advancing scientific tool discovery and search performance.
Scientific Code Search at Scale: Corpus, Benchmarks, and Baseline Evaluation
Motivation and Contributions
The paper addresses the challenge of scientific software discoverability within the context of open science and the proliferation of public code repositories (600 million+ on GitHub). Existing code search benchmarks (e.g., CodeSearchNet, CoSQA, AdvTest) are focused on general software engineering, with query vocabulary and granularity mismatched to scientific workflows and data-centric tasks. Researchers need tools for specific scientific data formats, instrument processing, and mission-driven analysis, and current search engines (including those on GitHub) are fundamentally designed for lexical matching, making them ineffective for domain-specific retrieval.
The paper introduces a suite of resources:
- A curated corpus of 5,264 high-quality, domain-classified GitHub repositories spanning five NASA Science Mission Directorate (SMD) divisions.
- Two benchmarks: (1) repository search (219 expert-curated queries across Earth Science, Astrophysics, Planetary Science); (2) code snippet retrieval (117,950 snippets, 119,720 queries across seven programming languages).
- Structured annotation, LLM-driven domain classification, and extensive context enrichment for each repository to address documentation sparsity.
- Baseline evaluations using lexical, general semantic, domain-specific, and code-specialized retrieval models, with multi-faceted analysis of retrieval scores across domains, query types, and programming languages.
All datasets and evaluation scripts are released on HuggingFace for reproducibility and continued research development.
Dataset Construction and Preprocessing
The repository corpus is assembled via a multi-channel pipeline spanning: literature mining (NASA EO-KG), institutional organization enumeration (53 GitHub orgs), registry extraction (ASCL for astrophysics), and governmental search (NASA Science Discovery Engine). Deduplication and quality filtering (active/public, non-empty README) are applied, with LLM-based domain classification using GPT-4.1-mini for SMD division assignment (F1 scores: Astrophysics 0.93, Earth Science 0.85).
Documentation enrichment is critical, given the limitations of project-level READMEs. The pipeline leverages LLMs for README cleaning, topic extraction, and contextual expansion via crawling external links (averaging 4.7 high-signal links/repository, predominantly from scientific publishers and data repositories). Each repository record includes cleaned documentation, domain reasoning, extracted topics, context from crawled links, and source provenance.
Benchmark Design and Query Annotation
Repository Search
Queries are formulated and annotated by subject matter experts, reflecting authentic scientific information needs. Queries are varied: tool discovery, workflow, data access, and analysis method, matched to the real vocabulary and concepts used within scientific communities. An average of 2.3 ground-truth repositories per query is achieved, capturing the multi-relevance property inherent in scientific tool ecosystems.
Code Snippet Retrieval
Snippets are extracted using tree-sitter parsers from TheVault, across Python, C, C++, Java, JavaScript, Fortran, Matlab. Queries are constructed both from semantic docstrings and function/class identifiers (masked in code bodies to avoid trivial matching). Function-level queries comprise 78% of the dataset, with 62% drawn from docstrings and 38% from identifiers, reflecting documentation realities in scientific codebases.
Baseline Retrieval Evaluation
Repository Search
Lexical (BM25), general semantic (all-MiniLM-L6-v2), and domain-specific semantic (INDUS-Retriever) models are evaluated on varying repository representations (raw README, cleaned README, topics, context-enriched, combined). The highest scores are achieved by INDUS-Retriever on combined representations, with substantial improvement from context enrichment—demonstrating the retrieval bottleneck imposed by sparse documentation. Notably, there is marked variance across domains: Astrophysics achieves MRR@10 = 0.87, Earth Science MRR@10 = 0.52, Planetary Science MRR@10 = 0.22, reflecting maturity and documentation standards in each community.
Hybrid retrieval architectures (Hybrid-RRF, Hybrid-Rerank) further boost performance (Recall@10 improvement >40% over BM25), with cross-encoder reranking yielding the highest aggregate metrics. Domain-difficulty remains dominant in performance variation, underscoring ecosystem and documentation as limiting factors.
Code Snippet Retrieval
Five approaches are benchmarked: BM25, two INDUS domain-specific models, Qwen3-Embedding-0.6B (general-purpose LLM), and SFR-Embedding-Code-400M_R (code-specialized embeddings). Qwen3-Embedding-0.6B achieves top scores (MRR@10 = 0.54, Recall@10 = 0.68, NDCG@10 = 0.58), outperforming both domain-specific and code-specialized models. There is pronounced disparity between docstring-based queries (MRR@10 up to 0.76) and identifier-based queries (MRR@10 drops to 0.25). Lexical methods effectively fail on identifier retrieval (MRR@10 = 0.01), revealing the depth of semantic complexity in scientific code naming conventions.
Cross-domain and language-level analysis finds that documentation quality—not language or corpus size—trumps performance. Fortran and C/C++ perform surprisingly well (MRR@10 ∼0.63–0.66), while Python results are middle-ranked despite being the most represented.
Practical and Theoretical Implications
The results demonstrate that scientific code retrieval is fundamentally a documentation-driven problem. Documentation sparsity and domain-specific conventions are the critical obstacles; even state-of-the-art models cannot compensate for semantically impoverished project or code-level documentation. General-purpose LLM-derived embeddings transfer well to code retrieval, but domain adaptation is essential for document-level search.
The benchmarks establish a modular foundation for scientific RAG pipelines and agentic retrieval systems, enabling evaluation beyond simple one-pass IR. The pronounced difficulty in identifier-based retrieval suggests future work in iterative retrieval, semantic query reformulation, and naming convention modeling. Domain expansion (Heliophysics, Biological/Physical Sciences), multilingual retrieval, temporal update pipelines, and cross-domain transfer represent actionable directions.
Limitations and Future Directions
Current coverage is skewed (Astrophysics, Earth Science dominate) due to registry and organization maturity; underrepresentation in Heliophysics and Biological/Physical Sciences persists. All queries are English-only; the benchmarks do not yet enable multilingual or translation-driven search evaluation. Temporal dynamics—rapid repository evolution—demand continual upkeep for corpus relevance. Expert annotations are concentrated in three domains; broadening SME participation is essential.
Future research should focus on:
- Expansion to underrepresented domains using dedicated registries and community outreach.
- Benchmarking agentic retrieval systems (multi-step RAG, query refinement).
- Domain-adaptive fine-tuning and transferability studies.
- Documentation augmentation strategies as complementary to model improvements.
Conclusion
The paper presents a rigorous suite of resources and benchmarks for scientific code discovery, anchored in authentic domain queries and real-world software artifacts. Strong numerical findings (e.g., MRR@10 = 0.87 for astrophysics repository search, 0.54 for code snippet retrieval via general LLM embeddings) highlight domain and documentation-driven performance bottlenecks. The implication is clear: retrieval advances in scientific computing must pair modeling innovations with documentation enhancement and domain-aware pipeline design. The public release of these datasets provides a foundation for continued progress in scientific information retrieval and AI-augmented research tool discovery.