BioTrove-CLIP

Intended use. This model supports zero-shot classification of organism images by comparing images with candidate species names in a shared semantic space. It also provides a useful baseline for further training for insect applications.

Architecture and training

The model pairs a Vision Transformer image encoder with a text encoder using the CLIP contrastive learning objective. Training uses BioTrove-Train, a curated subset containing approximately 40 million images covering 33,000 species, across birds, arachnids, insects, plants, fungi, molluscs, and reptiles, sourced from citizen science repositories.

Inputs and outputs

The models accept RGB images and text. Use the OpenCLIP preprocessing and tokenizer appropriate to the selected architecture, including image resizing/cropping to 224 × 224 pixels and CLIP normalization.

The encoders return image and text embeddings. For zero-shot classification, compare normalized image embeddings with normalized embeddings of candidate labels and select the highest-scoring label.

Performance

The model achives 16.9% zero-shot top-1 accuracy on the “insects-2” benchmark which contains 4000 images across 102 categories of insect pests.

Limitations

Zero shot performance is shown to vary substantially across benchmarks. This model is similar to later editions which include a greater training scope and further benchmarks on insects.

May require further fine tuning for use in insect classification applications.

How to obtain the weights

All three checkpoints are publicly available from BGLab/BioTrove-CLIP

Citation

@inproceedings{NEURIPS2024_b92854f8, title = {{BioTrove}: A Large Curated Image Dataset Enabling AI for Biodiversity}, author = {Yang, Chih-Hsuan and Feuer, Ben and Jubery, Zaki and Deng, Zi K. and Nakkab, Andre and Hasan, Zahid and Chiranjeevi, Shivani and Marshall, Kelly and Baishnab, Nirmal and Singh, Asheesh K and Singh, Arti and Sarkar, Soumik and Merchant, Nirav and Hegde, Chinmay and Ganapathysubramanian, Baskar}, booktitle = {Advances in Neural Information Processing Systems}, volume = {37}, pages = {102101–102120}, publisher = {Curran Associates, Inc.}, year = {2024}, doi = {10.52202/079017-3241}, url = {https://proceedings.neurips.cc/paper_files/paper/2024/hash/b92854f80ba4feefb973959b259dbc2c-Abstract-Datasets_and_Benchmarks_Track.html} }