SAM 3

Segments every instance of a concept, given as a phrase such as “moth” or “bee”, or as one example box.

Intended use. Open-vocabulary detection, segmentation and tracking without training. For insect work this means:

• Annotation assist: prompt “insect” or “moth” on trap images to get masks you correct, rather than drawing them.
• Zero-shot baselines for new trap designs.
• Video tracking of prompted individuals.

It returns where the instances are, not which species. Pair it with a classifier such as lepinet or BioCLIP 2 for identification. Unlike SAM 1 and SAM 2, which segment the one object you click, SAM 3 finds all matching instances from a single prompt.

Architecture and training

• Model: 848M parameters in total. A detector and a tracker share one vision encoder, a ViT with patch size 14, 32 layers and width 1024, run at 1008 px.
• Detector: DETR-based, conditioned on a text phrase, geometry (points, boxes, masks) and image exemplars.
• Tracker: inherits the SAM 2 transformer encoder-decoder, for video and interactive refinement.
• Training data: SA-Co, built by a data engine with humans and AI in the loop. It annotated over 4 million unique concepts, which the authors call the largest open-vocabulary segmentation dataset to date.
• SAM 3.1 (March 2026) adds Object Multiplex, a shared memory for tracking many objects at once.

Inputs and outputs

• Input: an RGB image or video (a JPEG folder or MP4), resized to 1008x1008. Prompts can be a short noun phrase, positive or negative example boxes, points or masks.
• Output: one mask, box and score per instance. Video also gives per-frame masks with a persistent object ID.
• Loading: the sam3 package (build_sam3_image_model, Sam3Processor) or transformers.
• Requirements: Python 3.12 or later, PyTorch 2.7 or later, and a CUDA GPU.

Prompting for insects

The text prompt is a short noun phrase, not a sentence and not a class list. SAM 3 returns every instance matching that one concept, so run one prompt per concept you want.

• Broad: insect or arthropod to find every specimen on a trap sheet, sticky plate or flower photo. This is a good first pass for counting or for making crops for a classifier.
• Groups at roughly the level of common names: moth, butterfly, bee, wasp, fly, beetle, ladybug, ant, dragonfly, grasshopper, spider, caterpillar.
• Parts and states: insect wing, antenna, insect leg, pupa, egg cluster. These help with trait measurements or life-stage screening.
• Context: flower, leaf, sticky trap segment the surroundings, for example to measure flower visits.
• Visual exemplars instead of text: draw one box around a specimen you want, and SAM 3 finds the others that look like it. Add a negative box on debris, a shadow or a trap edge to suppress that kind of false positive. This is the practical route for taxa that have no common name.
• Video: give a prompt on one frame, and the tracker keeps the same ID for each individual through the clip. You can add or remove objects with clicks.

Scientific names (for example “Bombus terrestris”) and fine distinctions between species are outside what SAM 3 was trained or evaluated for. Use the prompt to find the specimens, then identify them with a dedicated classifier.

Performance

All numbers are as reported by the authors, on general-domain benchmarks. There is no insect-specific evaluation.

Source: facebookresearch/sam3 README and Carion et al. (2025). cgF1 is the paper’s concept-level F1 for promptable concept segmentation.

Limitations

• Concepts, not species. Prompts are short noun phrases. “Insect”, “moth” or “bee” work at that level. Latin binomials and fine distinctions between similar taxa are not what it was trained or evaluated for.
• No insect benchmark. Expect misses on very small insects in high-resolution trap images at 1008 px (tile large images), and false positives on debris, shadows and trap texture.
• Heavy. 848M parameters and about 3.4 GB per checkpoint; a CUDA GPU is required by the reference code. Not an edge model.
• Gated. Every user must request access on Hugging Face and authenticate before downloading.

How to obtain the weights

Both repositories are gated: request access on facebook/sam3 or facebook/sam3.1, then log in (hf auth login) before downloading. The asset links above are pinned to the curated commits.

License: the SAM License (Meta, November 2025), which covers both code and weights. It is a royalty-free license that permits use, modification, redistribution and commercial use. It requires passing the license on with any redistribution and complying with trade controls, and it excludes ITAR and other prohibited end uses. This is not an OSI license, so it is recorded here as custom. The full text is the LICENSE file upstream.

Citation

@misc{carion2025sam3segmentconcepts,
  title         = {SAM 3: Segment Anything with Concepts},
  author        = {Nicolas Carion and Laura Gustafson and Yuan-Ting Hu and Shoubhik Debnath and Ronghang Hu and
                   Didac Suris and Chaitanya Ryali and Kalyan Vasudev Alwala and Haitham Khedr and Andrew Huang and
                   Jie Lei and Tengyu Ma and Baishan Guo and Arpit Kalla and Markus Marks and Joseph Greer and
                   Meng Wang and Peize Sun and Roman Rädle and Triantafyllos Afouras and Effrosyni Mavroudi and
                   Katherine Xu and Tsung-Han Wu and Yu Zhou and Liliane Momeni and Rishi Hazra and Shuangrui Ding and
                   Sagar Vaze and Francois Porcher and Feng Li and Siyuan Li and Aishwarya Kamath and Ho Kei Cheng and
                   Piotr Dollár and Nikhila Ravi and Kate Saenko and Pengchuan Zhang and Christoph Feichtenhofer},
  year          = {2025},
  eprint        = {2511.16719},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2511.16719}
}