Automatic Image-Level Morphological Trait Annotation for Organismal Images

Abstract: Morphological traits are physical characteristics of biological organisms that provide vital clues on how organisms interact with their environment. Yet extracting these traits remains a slow, expert-driven process, limiting their use in large-scale ecological studies. A major bottleneck is the absence of high-quality datasets linking biological images to trait-level annotations. In this work, we demonstrate that sparse autoencoders trained on foundation-model features yield monosemantic, spatially grounded neurons that consistently activate on meaningful morphological parts. Leveraging this property, we introduce a trait annotation pipeline that localizes salient regions and uses vision-language prompting to generate interpretable trait descriptions. Using this approach, we construct Bioscan-Traits, a dataset of 80K trait annotations spanning 19K insect images from BIOSCAN-5M. Human evaluation confirms the biological plausibility of the generated morphological descriptions. We assess design sensitivity through a comprehensive ablation study, systematically varying key design choices and measuring their impact on the quality of the resulting trait descriptions. By annotating traits with a modular pipeline rather than prohibitively expensive manual efforts, we offer a scalable way to inject biologically meaningful supervision into foundation models, enable large-scale morphological analyses, and bridge the gap between ecological relevance and machine-learning practicality.

• The paper introduces an automated pipeline that leverages sparse autoencoders to localize and extract precise morphological traits from organismal images.
• It utilizes a frozen DINOv2 backbone and SAE-based decomposition to outperform gradient-based methods by providing spatially tight, taxonomically informative trait annotations.
• Empirical results on BIOSCAN-5M show improved human ratings and enhanced zero-shot classification performance using multi-image consensus prompts.

Automatic Image-Level Morphological Trait Annotation for Organismal Images

Motivation and Problem Statement

Linking biological images to morphological trait annotations enables mechanistic insight into ecological and evolutionary processes, but current trait acquisition is deeply manual, labor-intensive, and non-scalable. The paper "Automatic Image-Level Morphological Trait Annotation for Organismal Images" (2604.01619) addresses the lack of large, standardized, image-trait datasets, introduces a pipeline to automate trait-level annotation, and demonstrates utility for downstream ecological analysis.

Methodology: Sparse Autoencoder-Guided Annotation

The central technical contribution is a modular pipeline coupling sparse autoencoder (SAE)-based part discovery with multimodal LLMs (MLLMs) for trait description generation. The process can be summarized as follows:

1. Dense Feature Extraction: Each specimen image is encoded using a frozen vision transformer backbone (DINOv2) to produce patch-level feature maps.
2. Sparse Autoencoder (SAE) Decomposition: These dense representations are fed through a pre-trained SAE, which enforces both non-negativity and sparsity constraints. Each neuron in the SAE efficiently localizes to semantically meaningful regions, often corresponding to biological parts (Figure 1).

   Figure 1: The pipeline computes dense DINOv2 features, then localizes monosemantic, high-activation SAE neurons on input images for trait-relevant ROI extraction and prompts an MLLM for trait description generation.

3. Salient Trait Selection: Activations are filtered and ranked using a species-contrastive criterion, selecting latents with strong activation in a focal species and weak activation among congeners, ensuring alignment with taxonomically diagnostic traits.
4. Spatial Localization and Prompt Construction: For each salient latent, spatial masks (bounding boxes) are extracted, and the corresponding regions are cropped for MLLM input.
5. Trait Description Generation: Cropped regions—optionally across multiple images per species—are input to an MLLM (e.g., Qwen2.5-VL-72B), which is prompted using a lightweight, template-driven system to generate interpretable, fine-grained morphological captions.

By design, this pipeline provides local grounding for trait extraction, substantially reducing background influence and hallucinations typically encountered in end-to-end MLLM pipelines.

Empirical Results

Trait Quality and Human Evaluation

The system is deployed on BIOSCAN-5M, a large insect image corpus, to construct Bioscan-Traits: 80K trait annotations over 19K images, averaging 4.2 high-quality traits per image. Extensive expert evaluation demonstrates that adding SAE-based spatial grounding boosts average human ratings from 3.15 (MLLM-only) to 3.91 (MLLM+SAE) for three-image prompts, indicating significant improvement in trait plausibility and taxonomic informativeness.

Figure 2: MLLM-only prompts (left) produce generic, sometimes spurious trait descriptions, while SAE-guided region prompts (right) yield focused, accurate, and interpretable trait statements.

Comparison with Gradient-Based Methods

Contrast with Grad-CAM reveals fundamental differences: Grad-CAM heatmaps remain spatially diffuse, overfit to coarse anatomical regions, and lack species granularity, while SAE neurons yield spatially tight, disentangled, and monosemantic regions that correspond to functional morphological units.

Figure 3: Bioscan-Traits (left) outputs detailed traits for specific structures; Grad-CAM (center) highlights diffuse, non-specific regions, lacking species-level focus.

Multiple-Image Trait Consensus

Trait description quality is further enhanced when using multiple images per latent, enforcing morphological consistency and reducing overfitting to individual instances. Qualitative analysis shows improved trait localization and greater inter-image agreement when three-image consensus prompts are used.

Figure 4: SAE with multi-image input produces concise, taxonomically relevant traits by filtering out sample-specific noise, compared to single-image prompts.

SAE Neuron-Dimension Analysis

Neuron-level inspection confirms that individual SAE latents consistently fire for interpretable, anatomical subregions (e.g., wings, antennae), displaying a high degree of semantic alignment and supporting the use of SAEs as unsupervised morphological part proposers.

Figure 5: Neurons 4852 and 13860 in the SAE consistently activate on wings and antennae, despite broad visual diversity and intra-species variation.

MLLM Backbone Effects

Larger MLLM backbones (Qwen2.5-VL-72B) yield superior trait plausibility and spatial grounding versus smaller variants (Qwen2.5-VL-7B), substantiating the dependence of verbalizer quality on model scale.

Figure 6: Qwen2.5-VL-72B correctly refrains from hallucinating in background regions (right), while Qwen2.5-VL-7B and GPT-5 mini produce erroneous attributions.

Robustness, Throughput, and Cost

Runtime profiling indicates that the SAE-guided annotation pipeline adds minimal overhead (sub-10ms/image) compared to MLLM inference (multi-second per annotation), and can scale efficiently on modern hardware. Trait annotation can be conducted via API for large datasets, with cost analyses suggesting the feasibility of large-scale deployment, especially with in-house hosting of open MLLMs.

Downstream Effects: Enabling Biologically Grounded Foundation Models

Trait-level supervision provided by the pipeline demonstrably improves fine-grained zero-shot classification performance on in-the-wild benchmarks. For instance, fine-tuning BioCLIP on Bioscan-Traits raises zero-shot accuracy on the Insects dataset from 34.8% (baseline) to 39.9% (trait-level fine-tuning) for BioCLIP and from 55.3% to 56.2% for BioCLIP 2, highlighting the practical value of precise, interpretable trait labels for foundation model transfer and robust ecological inference.

Scalability and Generalization

Because the pipeline only requires images with taxonomic labels for latent ranking, it is broadly portable across taxonomic datasets (e.g., iNaturalist, Caltech CUB-200-2011), and can be leveraged to generate rich trait annotations for under-studied taxa beyond insects, extending its impact to a diversity of organismal image repositories.

Limitations

The paper recognizes that not all SAE latents are monosemantic and that some may encode conjunctive/composite traits. Trait descriptions from small MLLMs remain susceptible to hallucination, and absence of ground-truth trait annotations complicates large-scale evaluation. Nonetheless, rigorous multi-image filtering and human evaluation mitigate many known limitations of SAE-based interpretability approaches.

Implications and Future Directions

The introduced pipeline enables automated, interpretable morphological trait annotation at unprecedented scale, facilitating ecological research, database expansion, and trait-based species identification. Theoretical advances lie in robust unsupervised part discovery with constrained autoencoders and in compositional trait verbalization via grounded MLLMs. Practically, the pipeline lowers annotation cost and democratizes trait database construction, empowering less-resourced research groups and accelerating trait ecology studies.

The authors speculate that future work will extend the approach to finer trait granularity, broader taxon coverage, and integration of multi-modal evidence (e.g., combining DNA and image-based traits), as well as feedback integration from domain experts in the annotation loop.

Conclusion

The paper presents a general, modular, and empirically validated pipeline using SAEs and MLLMs for automated, spatially grounded, and interpretable morphological trait annotation in organismal images. Bioscan-Traits, the resulting dataset, substantially advances the state of accessible trait-level supervision, and pipeline design choices are rigorously justified by human and downstream task evaluation. The approach constitutes a scalable mechanism to bridge digitized specimen archives and biological understanding, with direct utility for large-scale, interpretable machine learning in organismal biology and ecology.