SAM 3: The Eyes for AI — Nikhila & Pengchuan (Meta Superintelligence), ft. Joseph Nelson (Roboflow)

Introduction to SAM 3's core capability: concept-based prompting for detection, segmentation, and tracking. Live demo shows how text prompts like 'watering can' or 'red jersey' can find all instances in images/video, with visual exemplars for refinement. Demonstrates real-world applications including video editing effects and sports player tracking.

• •SAM 3 enables concept prompts (short text phrases) to find all instances without manual clicking on each object
• •Works on both images and video with real-time tracking and detection of new instances throughout video
• •Runs at 30ms on single image with 100 objects on H200; video scales with parallel GPU inference
• •Demo shows practical applications: player tracking in sports, video effects, and exhaustive object detection
• •Can refine results with visual exemplars (positive/negative boxes) when model misses instances

Technical discussion of SAM 3's novel architecture including the presence token for separating recognition from localization, decoupled detector and tracker to handle identity-agnostic detection vs. identity-preserving tracking, and integration of components from Meta's ecosystem (Perception Encoder, DETR, SAM 2, Llama).

• •Presence token explicitly separates recognition (is object in image?) from localization (where is it?), simplifying the task
• •Detector must be identity-agnostic (all dogs same) while tracker preserves individual identities - fundamental task conflict
• •Training data includes 70%+ negative phrases to teach model not to detect absent objects
• •Unified architecture handles interactive segmentation, text prompting, open vocabulary detection, and tracking without task-specific models
• •Shares Perception Encoder (PE) as visual backbone aligned with text and image representations

Detailed walkthrough of SAM 3's automated data engine that reduced annotation time from 2+ minutes to 25 seconds per data point. Uses model-in-the-loop for candidate generation, AI-powered verification for mask quality and exhaustivity checks, with humans only correcting missing instances. Created SAICO benchmark with 200K+ unique concepts vs. previous 1.2K.

• •Data engine pipeline: source images → generate noun phrases → model proposes candidates → AI verification → human correction only for missing instances
• •Fine-tuned Llama 3.2 achieves superhuman performance on mask verification and exhaustivity checking tasks
• •Exhaustivity is central design principle - finding every instance, not just some instances
• •SAICO benchmark has 200K+ unique concepts vs. 1.2K in previous benchmarks, matching real language diversity
• •Intentionally excluded OCR-heavy images during data collection, yet model still performs surprisingly well on text/numbers

Discussion of SAM 3 Agent approach where SAM provides 'eyes' for large language models to handle complex visual grounding tasks beyond atomic concepts. Shows performance gains when combining SAM 3 with Gemini/Llama for tasks requiring advanced reasoning, with synergy where LLMs can correct SAM errors and vice versa.

• •SAM 3 focuses on atomic visual concepts (e.g., 'yellow school bus'), but users want complex natural language queries
• •SAM 3 Agent uses LLMs for language understanding/reasoning while SAM provides precise visual grounding
• •Table 8 shows significant performance gains on complex reasoning tasks when combining SAM 3 with Gemini 2.5
• •Bidirectional synergy: LLMs provide reasoning brain, SAM provides accurate eyes, and they correct each other's errors
• •Example: LLMs struggle with counting fingers (often say 5 for 6-finger hand), but SAM 3 detects 6 fingers accurately

Joseph Nelson from Roboflow shares adoption metrics showing SAM's massive real-world impact across diverse fields: 106M smart poly annotations created, saving ~130 years of human time. Use cases span cancer research (neutrophil counting), underwater trash cleanup robots, aquarium species tracking, EV manufacturing, and medical imaging.

• •106M SAM-powered annotations created on Roboflow, saving estimated 130 years of human labeling time
• •2+ research papers published daily citing SAM/Roboflow work in Nature, Science Direct, and other prestigious journals
• •8M+ inferences in first 5 days of SAM 3 release, with continued acceleration post-Thanksgiving
• •Diverse applications: medical labs (cancer research), aerial imagery (solar panels, insurance), robotics (underwater cleanup)
• •CZI Imaging Institute uses SAM for segmenting individual organelles in human cell imaging

Discussion of video-specific challenges including the masculine detection score for temporal smoothing within windows, trade-offs between streaming latency and accuracy, and why video is still far from human performance. Explains why compute scales linearly with number of tracked objects.

• •Masculine detection score gathers information across temporal windows for more robust concept detection
• •Trade-off between streaming (immediate decisions) vs. accuracy (waiting for full context)
• •Humans make same mistakes - can't identify gender from partial hand at video edge, need full context
• •Video tracking scales linearly with number of objects because each instance needs independent tracking
• •Video still has significant gap from human performance, needs end-to-end training and better AI annotators

Practical discussion of fine-tuning SAM 3 with minimal data (as few as 10 examples) for domain adaptation. Covers the surprising effectiveness of small numbers of negative examples (3-5) and how SAM adapts to user-specific concept definitions (e.g., whether 'hand' includes forearm).

• •Can fine-tune effectively with just 10 data points for domain-specific adaptation
• •Single negative example goes a long way; 3-5 negative examples significantly update model priors
• •Example: Waymo detection - SAM labels as 'vehicle' but fine-tuning teaches specific 'Waymo' class from short video clip
• •Different users have different concept definitions (e.g., hand = palm only vs. palm + forearm)
• •Fine-tuning adapts to user's specific definition rather than imposing single 'correct' interpretation

Pengchuan Zhang discusses the future path beyond human-level performance in computer vision, drawing parallels to language models' evolution from SFT to RLHF. Argues that verification (which is better?) is easier than generation (create from scratch), enabling superhuman capabilities.

• •Current approach (human annotation → SFT) is bounded by human performance ceiling
• •RLHF paradigm for vision: humans indicate preferences (which is better?) rather than create labels from scratch
• •Verification is fundamentally easier than generation, enabling models to exceed human performance
• •SAM 3 approaches human performance on images, but new learning paradigms needed to surpass it
• •Video domain still needs significant work before reaching human parity, let alone superhuman levels

Team outlines SAM 3.x roadmap including efficient small models for edge deployment, end-to-end video training (currently decoupled), and debate over whether visual grounding should be native LLM capability (system 1) vs. tool call (system 2) for complex tasks.

• •SAM 3.x priorities: small/efficient models for edge cases, improved video throughput without multi-GPU requirement
• •Video needs end-to-end training (currently decoupled approach) and scalable AI annotators for data engine
• •Simple visual tasks (counting <20 objects) should be native 'system 1' capability in frontier models
• •Complex tasks (counting thousands of objects) may still need specialized tool calls
• •Robotics companies immediately adopting SAM 3; explicit world models vs. implicit representations remains open question

Discussion of open source's critical role in SAM's development, with community contributions (datasets, benchmarks, optimizations) feeding back into SAM 3. Roboflow demonstrates auto-labeling, fine-tuning, and deployment infrastructure, highlighting challenges of human intention vs. deterministic model behavior.

• •SAM 3 leveraged community contributions from SAM 2: new datasets, benchmarks, inference optimizations
• •Roboflow built complete pipeline: auto-labeling with SAM 3, fine-tuning for domains, scalable deployment infrastructure
• •Challenge: concept ambiguity (e.g., 'vehicle' includes reflections?) requires human-in-loop for intention, not just knowledge
• •Different users have different definitions even for same concepts - fine-tuning adapts to user preferences
• •SAICO benchmark designed to outlast individual models, guide community toward and beyond human performance