INQUIRE-Search: A Framework for Interactive Discovery in Large-Scale Biodiversity Databases

Abstract: Large community science platforms such as iNaturalist contain hundreds of millions of biodiversity images that often capture ecological context on behaviors, interactions, phenology, and habitat. Yet most ecological workflows rely on metadata filtering or manual inspection, leaving this secondary information inaccessible at scale. We introduce INQUIRE-Search, an open-source system that enables scientists to rapidly and interactively search within an ecological image database for specific concepts using natural language, verify and export relevant observations, and utilize this discovered data for novel scientific analysis. Compared to traditional methods, INQUIRE-Search takes a fraction of the time, opening up new possibilities for scientific questions that can be explored. Through five case studies, we show the diversity of scientific applications that a tool like INQUIRE-Search can support, from seasonal variation in behavior across species to forest regrowth after wildfires. These examples demonstrate a new paradigm for interactive, efficient, and scalable scientific discovery that can begin to unlock previously inaccessible scientific value in large-scale biodiversity datasets. Finally, we emphasize using such AI-enabled discovery tools for science call for experts to reframe the priorities of the scientific process and develop novel methods for experiment design, data collection, survey effort, and uncertainty analysis.

• The paper introduces an interactive VLM-based framework that enables rapid semantic retrieval and expert verification from large biodiversity image databases.
• It employs a joint image-text embedding model with FAISS for sub-second search and context-specific filtering across diverse ecological use cases.
• Empirical results validate the system’s capacity to analyze seasonal, post-fire, and mortality patterns, extending remote-sensing and species re-identification studies.

INQUIRE-Search: A Framework for Interactive Discovery in Large-Scale Biodiversity Databases

Overview and Motivation

INQUIRE-Search introduces an interactive system for semantic retrieval and analysis of secondary ecological information from massive-scale image datasets, with a principal demonstration on the iNaturalist database. The motivation stems from the exponential growth of community science repositories containing petabyte-scale biodiversity images. However, traditional ecological workflows leveraging only metadata or manual inspection are ill-equipped to flexibly extract contextual signals such as organismal behavior, interactions, phenology, and event-related details, and supervised ML pipelines break down in open-ended data regimes due to sparse annotation of secondary events. INQUIRE-Search is explicitly designed to operationalize modern VLM-backed retrieval—in particular, open-ended, text-driven search—within large, heterogeneous databases for ecological research.

Figure 1: High-level overview of the iterative INQUIRE-Search pipeline, supporting natural language query, user-driven verification, and downstream analysis of retrieved imagery.

System Architecture and Retrieval Pipeline

The system leverages a joint image-text embedding model (SigLIP-So400m) to create a shared semantic space where arbitrary natural language prompts and unlabeled images can be indexed. At inference, the user provides a text query, which is encoded into a vector; the system then compares this vector against precomputed image embeddings utilizing the FAISS ANN vector database for sub-second large-scale retrieval. Optional taxonomic, temporal, and spatial filters further focus the candidate set. Retrieved images are exposed to the ecologist via a web-based interface for expert verification and annotation, with data export for downstream analysis.

Figure 2: Schematic of the VLM embedding space and retrieval procedure using metric learning and FAISS-based semantic search.

This iterative, human-in-the-loop workflow is optimized to substantially decrease labor relative to conventional annotation, integrating expert domain refinement and flexible prompt engineering as core design parameters.

Empirical Applications and Case Studies

The authors empirically validate INQUIRE-Search via five representative ecological use-cases, encompassing both routine and rare signal discovery across trophic, phenological, disturbance-response, and population monitoring scenarios.

Seasonal Variation in Avian Diet

By formulating compositional text queries (e.g., "American Robin with invertebrate in its mouth") and constraining over explicit geographic and temporal axes, the tool systematically recovered seasonal shifts in avian diet for taxa well-studied in the literature (high concordance with the SAviTraits reference dataset), but was empirically constrained by the underlying distribution of images for less-commonly photographed species. Quantitative alignment between tool outputs and existing trait databases provides strong support for the semantic fidelity of retrieval in data-rich regimes.

Figure 3: Heterogeneous avian diets by species and season extracted via semantic search, capturing ecological context typically missed by metadata.

Figure 4: Verified INQUIRE-Search returns for "American Robin" dietary queries, with accuracy validated against curated ornithological databases.

Post-fire Forest Regrowth

INQUIRE-Search enables rapid extraction of site- and context-specific signals (young conifers/deciduous tree regeneration within fire perimeters, parsed by burn severity gradients). Results revealed marked structuring of observed regeneration by fire severity, with conifer recovery far more circumscribed to sites with low-severity impact. These findings replicate prior regional-scale field studies, validating the tool's capacity to generalize remote-sensing limitations and facilitate event-driven ecological inference at landscape scale.

Figure 5: Task setup highlighting burn severity classification and spatial constraints in post-fire regrowth study.

Figure 6: Examples of correct retrieval/verification for regenerating vegetation and the burn severity dependence of observed tree regrowth extracted from user-queried INQUIRE-Search.

Wildlife Mortality Patterns

The pipeline successfully surfaced and temporally resolved avian mortality signals at both urban and rural scales (inferred from "dead bird" queries), allowing normalization for observer effort and evidence-based comparisons of mortality indices across biogeographic gradients and seasonality. The workflow sheds light on detection bias, observer effects, and the potential for rapid context-dependent hypothesis generation, but also exposes the limitations of image provenance and event rarity in sampling underrepresented signals.

Figure 7: Contrasting urban and rural site setups and the retrieval of event-based evidence (e.g., avian mortality) using semantic search.

Figure 8: Normalized, temporally explicit indices of avian mortality rates, revealing migration-driven peaks and urban/rural differences.

Resolving Fine-Scale Plant Phenology

The use of compositional queries over specific phenological stages ("milkweed flowering", "milkweed senescence") enabled the assembly of high-quality, stage-differentiated observational datasets at spatial and temporal resolutions unattainable through manual survey alone. While certain stages (flowering/seeding) were heavily overrepresented due to both observation and model biases, the pipeline demonstrates the feasibility of macrophenological studies from opportunistic datasets.

Figure 9: Retrieval of images within distinct milkweed phenological stages, supporting fine-grained phenological inference.

Figure 10: Human-verified retrieval for flowering stages and statistical summaries showing clear seasonal staging differences detected by INQUIRE-Search.

Humpback Whale Individual Re-Identification

INQUIRE-Search was used to mine iNaturalist for re-ID-suitable images ("white underside of humpback whale fluke") and align them with curated individuals from the HappyWhale dataset via deep metric learning. Of 153 high-quality images identified under expert verification, 57 matched known individuals, immediately extending the spatiotemporal coverage of these population monitoring datasets. This use-case highlights large-scale, cross-repository entity linking powered by the pipeline.

Figure 11: The fluke—critical for photo-ID of humpback individuals, and illustration of the challenge in existing repositories.

Figure 12: Deployment of an automated cropping and deep re-ID pipeline on INQUIRE-Search retrievals, with mapping of new matches.

Technical, Theoretical, and Practical Implications

INQUIRE-Search transforms unstructured, opportunistic biodiversity image datasets into analyzable ecological evidence. The work demonstrates that VLM-based retrieval can be systematized for real-world scientific workflows under expert supervision, resulting in orders of magnitude speedup over manual annotation pipelines and enabling new frontiers in data-driven hypothesis testing. The pipeline exposes both the strengths (rapid data surfacing, compositional prompt agility, scaling to hundreds of millions of records) and core limitations (under-observation of rare events, inherent community and observer bias, VLM compositional and calibration failures, reliance on image verifiability) that require critical consideration as these tools are operationalized. The need for robust uncertainty quantification and bias correction remains paramount; integration with field data, additional repositories, and ongoing advances in open-set VLMs will increase both rigor and coverage.

Future Directions

Potential avenues for extension include:

• Domain-generalization and dataset expansion: Adapting the methodology and system for other large-scale community science or remote camera trap datasets.
• Automated uncertainty quantification: Incorporation of conformal prediction, IPW, and selective classification tools to surface retrieval reliability, especially for rare class or ambiguous queries.
• Iterative scientist-AI co-design: Formalizing prompt programming as part of ecological experimental design, and integrating uncertainty diagnostics to optimize user annotation effort.
• Bias correction: Addressing observer and context bias via structured causal inference techniques and harmonization with ground-truth or structured sampling data.
• Multimodal integration and downstream inference: Fusing retrieved image-based data with genomic, acoustic, or environmental measurements for comprehensive ecological modeling.

Conclusion

INQUIRE-Search provides a robust framework for semantic, expert-in-the-loop mining of secondary ecological information from petascale image databases. It establishes benchmarks for interactive discovery, reframes the design of ecological experiments in the age of AI-enabled retrieval, and surfaces both the opportunities and challenges posed by open-world, community-sourced data and model-based scientific workflows. These capabilities promise to alter the scale and pace of data-driven discovery across biodiversity research, but require careful design to ensure rigor, bias-awareness, and theoretical generalizability.