TaxaBind

Intended use. Model provides representations for classification and retrieval which can be informed by multiple modalities of training information. Can be used zero shot. Not all modalities have to be used at for inference.

Architecture and training

The image-text representation starts from that provided by BioCLIP. TaxaBind then aligns additional modalities with ground-level species images using contrastive learning using: iSatNat, which contains 2.7 million species-image and species-satelite image pairs, iSoundNat which contains 88,130 image-audio pairs. Geographic coordinates come from iNaturalist 2021, environmental variables come from WorldClim 2.1.

Inputs and outputs

The model accepts RGB images and text. It also accepts information from extra modalities including: location; environmental features; satelite images and audio. Use the evaluation preprocessing and tokeniser supplied on HuggingFace. Images are preprocessed to 224x224 pixels using bicubic resizing and the checkpoint’s normalisation settings. The text encoder has a context length of 77 tokens. Both encoders produce a 512-D embedding vectors.

Performance

One table in the paper reports zero-shot classification accuracy using image inputs matched against taxanomic text strings. These include evaluations which contain insect classses, although no insect-only evaluation is provided.

Limitations

Reported benchmarks do not establish performance for a given insect monitoring workflow.

Classification zero shot is only possible depending on the supplied candidate taxa names. Further fine tuning may be required for your task.

How to obtain the weights

Download the image–text checkpoint from MVRL/taxabind-vit-b-16.

Citation

@inproceedings{sastry2025taxabind, title = {{TaxaBind}: A Unified Embedding Space for Ecological Applications}, author = {Sastry, Srikumar and Khanal, Subash and Dhakal, Aayush and Ahmad, Adeel and Jacobs, Nathan}, booktitle = {Winter Conference on Applications of Computer Vision}, year = {2025}, organization = {IEEE/CVF}, url = {https://arxiv.org/abs/2411.00683} }