Skip to content

README.md

Latest commit

 

History

History
64 lines (38 loc) · 4.4 KB

README.md

File metadata and controls

64 lines (38 loc) · 4.4 KB
Raw

SAM3 Interactive Video Object Segmentation Example

Introduction

SAM3 is a unified foundation model for promptable segmentation in images and videos. It can detect, segment, and track objects using text or visual prompts such as points, boxes, and masks. Compared to its predecessor SAM2, SAM3 introduces the ability to exhaustively segment all instances of an open-vocabulary concept specified by a short text phrase. SAM3 Video performs Promptable Concept Segmentation (PCS) on videos, taking text as prompts and automatically detecting and tracking all matching object instances across video frames.

In this tutorial, you'll learn how to leverage the video tracking feature of SAM3 on X-AnyLabeling to accomplish iVOS tasks. Let's get started!

Installation

You'll need to get X-AnyLabeling-Server up and running first. Check out the installation guide for the details. Make sure you're running at least v0.0.4 of the server and v3.3.4 of the X-AnyLabeling client, otherwise you might run into compatibility issues.

Once that's done, head over to configs/models.yaml and enable segment_anything_3_video. There's an example config you can reference if you're not sure how to set it up.

Download the SAM3 model checkpoint from ModelScope, then update model_path in configs/auto_labeling/segment_anything_3_video.yaml to point to the downloaded file.

You can tweak the settings in segment_anything_3_video.yaml to fit your needs. By default, detection boxes are shown, which works fine for most cases. If you want to see the actual masks instead, flip show_masks to true. Just keep in mind that rendering masks can slow things down a bit, so boxes are usually the safer bet.

Usage

Launch the X-AnyLabeling client, press Ctrl+A or click the AI button in the left menu bar to open the auto-labeling panel. In the model dropdown list, select Remote-Server, then choose Segment Anything 3 Video.

Text Prompting

text-prompt.mp4
  1. Enter object names in the text field (e.g., person, car, bicycle)
  2. Click Send to initiate detection
  3. After verification, click Auto Run on the left toolbar or press Ctrl+B to start forward propagation.

Note

Each session supports one category only. Propagation starts from the current frame and runs to the end of the video, with a brief warm-up period of approximately 15 frames before processing begins.

Warning

If you cancel the propagation task midway, all processed frame results will be lost. Please wait for the task to complete or ensure you have saved the results before canceling.

Visual Prompting

visual.prompt.mp4

With visual prompting, use points to mark what you want to track.

  1. Click Point (Q) to add positive points or Point (E) to add negative points. Add multiple points if needed.
  2. Click Finish (F) to complete the prompt and prefill the target label and track_id. Click Clear (B) to start over.
  3. After confirming the selection, click Auto Run on the left toolbar or press Ctrl+B to start forward propagation.

Tip

The letters in parentheses are keyboard shortcuts: Q and E switch point modes, F finishes, and B clears the prompts.

Note

A few things to keep in mind: you can start tracking from any frame in the video, but each session only supports tracking a single target. Also, visual prompting uses a non-overwrite mode by default, so it won't replace existing annotations.

We've also included a couple of handy tools to make your life easier. The Label Manager and Group ID Manager let you quickly modify label names or track_ids either locally or globally. Check out the demo video below to see how they work.

sam3-video-feature.mp4

See User Guide for more details.