Grounding DINO

Intended use. Zero-shot object detection for scenarios where target classes are unknown. Ideal for annotation assist tools, and open-set detection tasks where fixed-class detectors fail. Also supports expression comprehension (detecting objects via complex descriptions like “the damaged leaf near the stem”).

Architecture and training

Grounding DINO combines a DINO-style DETR detector with grounded pre-training that fuses vision and text modalities. The architecture uses:

• Visual backbone: Swin Transformer (Tiny or Base variants; later versions use ViT-L or EfficientViT-L1)
• Text encoder: BERT-based transformer
• Fusion module: Feature enhancer with cross-modality attention and language-guided query selection
• Decoder: Cross-modality decoder predicting bounding boxes and text alignment scores

Training data: O365 (Objects365), GoldG (Grounding Data), Cap4M (caption data), and for later versions: COCO, OpenImage, ODinW-35, RefCOCO. The model learns object-level representations without explicit class labels through contrastive image-text grounding.

Inputs and outputs

Inputs:

• Image (RGB, any resolution; internally resized to 800×1333 pixels)
• Text prompt (lowercase, phrases must end with period—e.g., “a butterfly.” “damaged wing.”)

Outputs:

• Bounding boxes (xyxy format) with confidence scores
• Per-box text similarity scores
• Maximum ~900 object proposals (reduced to 300 in 1.6+)

Preprocessing requirement: No class list needed.

The documented Hugging Face results profile describes post_process_grounded_object_detection in Transformers 4.40.2, matching the author model-card example. It returns one dictionary per image with scores and boxes tensors plus decoded phrase labels. Passing target_sizes produces pixel corners; omitting it produces normalized corners. This profile is source-inspected only and deliberately version-specific: newer Transformers APIs can change keys and signatures.

Performance

Sources: Liu et al. 2023 (ECCV 2024), Grounding DINO 1.5/1.5 API technical report (arXiv:2405.10300).

Limitations

Critical constraints for insect monitoring:

• Small objects: Performance degrades on very small insects (<1% image area); post-processing size filtering recommended
• Occlusion: Limited robustness to heavily occluded specimens; partial visibility reduces detection confidence
• Fine-grained distinctions: Struggles with morphologically similar taxa (e.g., closely related species without distinctive markings)
• Life stage variation: Not evaluated on larval/pupal stages; training data is dominated by adult forms
• Background complexity: Dense vegetation or cluttered substrates increase false positives
• Speed: ~3 FPS on V100 GPU (base variant); Edge variants reach 14 FPS but with accuracy trade-offs
• Hallucinations: Earlier versions produced spurious detections; 1.5+ variants show reduced hallucination rates but still occur with ambiguous prompts

The model cannot perform segmentation (unlike Grounded SAM) and requires post-processing (e.g., with SAM) for instance masks.

How to obtain the weights

Hugging Face (recommended for prototyping):

• Tiny variant: https://huggingface.co/IDEA-Research/grounding-dino-tiny
• Base variant: https://huggingface.co/IDEA-Research/grounding-dino-base

Latest versions (1.5/1.6 Pro & Edge):

• GitHub: https://github.com/IDEA-Research/Grounding-DINO-1.5-API
• Manual checkpoint download required; not accessible via transformers library

License: Apache-2.0 permits commercial use and modification. No gated access required.

Usage note: When loading via Hugging Face Transformers, ensure transformers>=4.38.0 for AutoModelForZeroShotObjectDetection support. Text tokens must end with periods and be lowercased.

Citation

@misc{liu2023grounding,
      title={Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection}, 
      author={Shilong Liu and Zhaoyang Zeng and Tianhe Ren and Feng Li and Hao Zhang and Jie Yang and Chunyuan Li and Jianwei Yang and Hang Su and Jun Zhu and Lei Zhang},
      year={2023},
      eprint={2303.05499},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}