BioCAP

Intended use. BioCAP supports zero-shot species classification, biological image-text retrieval, and image feature extraction. Its scope includes animals, plants, fungi, insects, etc.

Architecture and training

This model further trained OpenAI’s CLIP ViT-B/16 checkpoint on a dataset of 10 million biology images paired with descriptive captions of thier appearance.

The training architecture uses shared image and text encoders to align the descriptions and images into a shared space.

It produces output vectors in a 512-D embedding space.

Inputs and outputs

Accepts RGB images and text, which can be encoded separately. Images are processed at 224 × 224 pixels using the checkpoint’s CLIP preprocessing transforms. Pixel values are scaled to [0, 1] and normalized using channel means (0.48145466, 0.4578275, 0.40821073) and standard deviations (0.26862954, 0.26130258, 0.27577711).

Performance

Reported top-1 classification accuracy on Meta-Album Insects: 41.9%.

Reported top-1 classification accuracy on Meta-Album Insects 2: 23.7%

Limitations

It is not yet well understood how applicable these large scale models are on field-imagery.

Training data includes classes across the tree-of-life. Insects may not be eqivialently represented alongside other animal taxa.

How to obtain the weights

Download open_clip_model.safetensors and the matching supporting files from the pinned Hugging Face revision.

Citation

@inproceedings{zhang2026biocap, title = {{BioCAP}: Exploiting Synthetic Captions Beyond Labels in Biological Foundation Models}, author = {Zhang, Ziheng and Ma, Xinyue and Chowdhury, Arpita and Campolongo, Elizabeth G. and Thompson, Matthew J. and Zhang, Net and Stevens, Samuel and Lapp, Hilmar and Berger-Wolf, Tanya and Su, Yu and Chao, Wei-Lun and Gu, Jianyang}, booktitle = {The Fourteenth International Conference on Learning Representations}, year = {2026}, url = {https://openreview.net/forum?id=SCKLkfgevy} }