NOTICE

idtrackerai-detectron2
======================

This repository is a modified version of idtracker.ai.

  Upstream project : https://gitlab.com/polavieja_lab/idtrackerai
  Upstream website : https://idtracker.ai
  Fork point       : commit 14ff552d209d743e514e8acdeb1590777b9fef98
                     (master, 2026-08-04), idtracker.ai version 6.0.15a0

idtracker.ai is developed by the de Polavieja Lab at the Champalimaud
Foundation and is distributed under the GNU General Public License v3.0.
Copyright in the original work belongs to its authors: Jordi Torrents,
Francisco Romero-Ferrero, Mattia G. Bergomi, Francisco J. H. Heras,
Robert C. Hinz, Tiago Costa and Gonzalo G. de Polavieja, among other
contributors listed in the upstream repository.

This derivative work is distributed under the same licence. See LICENSE
for the full GPL-3.0 text. The licence is unchanged and the original text
is included verbatim.

  Copyright (C) 2026 Martina Bellio
    for the modifications described below.

THIS PROJECT IS NOT AFFILIATED WITH, ENDORSED BY, OR SUPPORTED BY THE
DE POLAVIEJA LAB OR THE CHAMPALIMAUD FOUNDATION. Please do not report
problems with this fork to the upstream project. Issues belong here:
https://github.com/mbel-tech/idtrackerai-detectron2/issues


Statement of modifications (GPL-3.0 section 5a)
-----------------------------------------------

Modified by Martina Bellio, from 2026-09-20.

The modifications have three purposes.

The first is to let animal outlines produced by an external
instance-segmentation model (a fine-tuned Detectron2 Mask R-CNN) enter
idtracker.ai directly, instead of being rendered back into a video for
idtracker.ai to re-threshold. Identity tracking is deliberately left as it
was: crossing detection, fragmentation, identification and gap closing are
unmodified. Alongside it, frame enhancement (CLAHE and correction for uneven
lighting) became a control of the Segmentation App in its own right, applied
to thresholding as much as to the Detectron2 route.

The second is to offer Meta's SAM 3 as a second such model, prompted with a
word for the animal rather than trained on annotated frames. It writes the
same contour files the Detectron2 route does, so nothing downstream of
segmentation distinguishes them, and it also drafts LabelMe annotations for
correction, which is a way of preparing a Detectron2 dataset rather than an
alternative to one.

The third is to make the Validator, the tool for correcting trajectories
by hand after tracking, usable on long recordings: an undo stack, a save
that survives an unwritable session folder, an autosave, three further
editing operations, and filters that keep the list of reported errors
workable. These are described in VALIDATOR.md.

Files changed relative to the fork point:

  src/idtrackerai/base/animals_detection/segmentation.py
      process_frame() takes an optional external contour source -- one
      file or one per video path -- and a frame number, and short-circuits
      to contours_from_external() instead of
      thresholding. New function contours_from_external(). It also takes an
      optional enhancement, applied before thresholding in place of the plain
      greyscale conversion; generate_frame_stack() and compute_background()
      take the same, so the background model is built from the same kind of
      image the frames will be. New function apply_enhancement(). With no
      enhancement given, the behaviour is unchanged.

  src/idtrackerai/base/animals_detection/animals_detection.py
      Validates the contour file against the video before segmenting,
      disables background subtraction when external contours are in use, and
      passes the session's enhancement to both the background computation and
      the segmentation.

  src/idtrackerai/session.py
      New enhancement parameter carrying the frame enhancement chosen in the
      Segmentation App, so it round-trips through the session .toml and
      reaches segmentation. New external_contours parameter, which takes one
      contour file per video path so the clips of one recording can be
      tracked as a single session, resolved and checked in
      prepare_tracking(). New presence_intervals parameter recording, per
      identity, the frames in which that animal is present; the Validator
      stops reporting an identity as missing outside them. Fixed
      prepare_tracking() raising NameError because idtrackerai_version()
      was never imported.

  src/idtrackerai/segmentation_app/main.py
  src/idtrackerai/segmentation_app/widgets/__init__.py
  src/idtrackerai/segmentation_app/tooltips.toml
      The left column is inside a scroll area. A QVBoxLayout answers a
      column that wants more height than the screen by compressing every
      child below its natural size, which left a panel step too short to
      show its own button; scrolling the column lets each widget be the
      size it asked for, in every mode.
      Segmentation App gains a segmentation source selector with four
      modes (thresholding, external contours, Detectron2 preparation,
      SAM 3) and hosts the panels for the last two. The enhancement
      control sits above the
      source selector and applies in every mode, with its live preview; a
      caution appears beside it when enhancement is on in thresholding mode,
      because it shifts the brightness range the intensity thresholds were
      chosen against.

  src/idtrackerai/segmentation_app/widgets/frame_analyzer.py
      New set_suspended(), so the blob overlay stands down in the
      Detectron2 and SAM 3 modes, where thresholding means nothing. With suspended
      False the behaviour is unchanged. It also segments the enhanced frame
      when an enhancement is set, so the blob outlines match the picture
      they are drawn on.

  src/idtrackerai/segmentation_app/widgets/bkg_widget.py
      The background computation thread carries the enhancement, and the
      cached background model is discarded when the enhancement changes.

  src/idtrackerai/segmentation_app/widgets/open_video_widget.py
      Selected files that are filesystem metadata rather than videos are
      dropped, with a dialog naming the first. macOS writes a "._name.mp4"
      companion beside each file it copies onto a non-Apple drive, and
      OpenCV reported the first of those as an unreadable video.

  src/idtrackerai/utils/py_utils.py
  src/idtrackerai/utils/__init__.py
      New is_metadata_sidecar(), drop_metadata_sidecars() and SIDECAR_NAMES,
      backing the above.

  src/idtrackerai/__init__.py
  src/idtrackerai/utils/py_utils.py
  src/idtrackerai/utils/__init__.py
  src/idtrackerai/utils/logging_utils.py
  src/idtrackerai/utils/telemetry.py
  src/idtrackerai/start/__main__.py
      New idtrackerai_version(), which asks importlib.metadata which
      distribution provides this package instead of assuming it is called
      "idtrackerai". Required by the rename below: without it the package
      raises PackageNotFoundError at import time once installed.

  src/idtrackerai/extra_tools/validator/validation_GUI.py
      Undo stack bound to Ctrl+Z; Swap, Extend and Merge editing
      operations; a save that falls back to a local backup when the
      session folder cannot be written, instead of failing silently on a
      worker thread; a five-minute autosave; the presence-interval dialog;
      the Video Info tab.

  src/idtrackerai/extra_tools/validator/widgets/errors_explorer.py
      Minimum-duration and identity-pair filters, a next-error button, and
      masking of missing-identity errors outside an identity's declared
      presence interval.

  src/idtrackerai/extra_tools/validator/widgets/interpolator.py
      PCHIP interpolation mode, which cannot overshoot between fixed
      points; linear default; Enter applies; clicking a fixed point jumps
      to its frame; undo integration.

  src/idtrackerai/extra_tools/validator/widgets/length_calibrator.py
  src/idtrackerai/extra_tools/validator/widgets/__init__.py
      The calibration line now follows the cursor. Calibrations are deleted
      by row rather than by matching their text, which removed the wrong
      one when two read the same. The distance is entered numerically.

  src/idtrackerai/utils/py_utils.py
      LengthCalibration.from_dict() now fills the color slot, and
      json_default() no longer uses asdict() on it. Being a slots
      dataclass, a calibration restored from disk previously raised
      AttributeError on both painting and the next save.

  src/idtrackerai/blob.py
      New swap_identity() and propagate_swap_identity(), which exchange two
      identities along a fragment rather than renaming one.

  src/idtrackerai/list_of_blobs.py
      New merge_blobs(), which rebuilds one blob from the convex hull of
      several, for an animal that segmentation split in two. save() takes a
      verbose argument so the Validator's autosave writes nothing to the
      terminal.

  src/idtrackerai/GUI_tools/widgets_utils/canvas.py
      New move_event signal and mouse tracking, for tools that draw a
      preview following the cursor.

  src/idtrackerai/GUI_tools/widgets_utils/custom_list.py
      New removedItemIndex signal, which identifies a removed row when two
      rows read the same. The existing removedItem signal is unchanged.

  src/idtrackerai/GUI_tools/widgets_utils/id_labels.py
      Context-menu entry for setting an identity's presence interval.

  src/idtrackerai/GUI_tools/widgets_utils/video_player.py
      Key auto-repeat is allowed through for the frame-step keys, so
      holding one scrubs through the video instead of stepping once.

  docs/source/user_guide/changelog.rst
      A section for this fork's changes, above upstream's own entries.

  pyproject.toml
      Distribution renamed to idtrackerai-detectron2 to distinguish it
      from the upstream release. The import name idtrackerai is unchanged,
      so the two cannot share an environment. LabelMe added as a
      dependency, since annotation is a required stage of this pipeline.

  README.md
      Replaced. The upstream README is preserved as README.idtrackerai.md.

  CONTRIBUTING.md
      Prefaced with a notice redirecting reports about this fork here
      rather than to the upstream issue tracker. Otherwise unchanged.

Files added:

  src/idtrackerai/base/animals_detection/external_contours.py
      Sidecar contour format and reader.

  src/idtrackerai/extra_tools/detectron2_pipeline/
      The whole pipeline that produces the contour files: frame
      enhancement, stratified frame sampling for annotation, LabelMe to
      COCO conversion, the state that lets a preparation be resumed, Mask
      R-CNN fine-tuning, batch inference, video preflight, the Colab
      bundle builder and the notebook itself. Lives in the package so that
      it installs with it, and so the GUI and the command line run the
      same code rather than two copies that drift. Exposed as the
      idtrackerai_d2_* commands declared in pyproject.toml.

  src/idtrackerai/segmentation_app/widgets/segmentation_source.py
      The segmentation source selector widget.

  src/idtrackerai/segmentation_app/widgets/detectron2_panel.py
  src/idtrackerai/segmentation_app/widgets/pipeline_threads.py
      The guided preparation panel and its background workers. Six steps:
      the four local ones, plus training, which runs here as a managed
      subprocess when the machine has a CUDA GPU, and the contour export,
      handed over as a command because it takes days over a collection.
      Colab is the fallback for machines without a GPU rather than the
      route, and a button installs PyTorch and Detectron2 for the machine
      when they are missing. The panel
      keeps its own editable list of videos to draw annotation frames
      from, independent of the tracking session, so one model can be
      trained across every recording from a setup while each recording is
      still tracked separately. It previews the per-video allocation
      before sampling, counting frames on a background thread, and shows
      the train/validation grouping it inferred before the dataset is
      built. A button on the annotate step writes the Colab bundle and
      names what else to upload, so the handover to a hosted GPU starts
      where the last hands-on step ends.

  src/idtrackerai/segmentation_app/widgets/enhancement_widget.py
  src/idtrackerai/segmentation_app/widgets/enhancement_preview.py
      Enhancement controls, and the live preview that overdraws the
      enhanced frame on the video player.

  tests/external_contours_test.py
      Tests for the sidecar format and the segmentation path.

  src/idtrackerai/extra_tools/validator/widgets/video_info.py
      Read-only summary of the session being validated: frame rate,
      resolution, frame and animal counts, and the video files.

  docs/detectron2-pipeline.md
      Documentation for the pipeline above.

  src/idtrackerai/extra_tools/detectron2_pipeline/gpu.py
      The capability probe also reports whether the sam3 package is present,
      with sam3_usable and sam3_missing beside the existing usable and
      missing. SAM 3 needs a CUDA GPU and its own package but not
      Detectron2, so one notion of "usable" would refuse it on a machine
      that can run it.

  src/idtrackerai/extra_tools/detectron2_pipeline/sam3_predictor.py
      SAM 3 behind the same predict(frame) -> (masks, scores) interface the
      contour exporter uses for Detectron2. Prompts each frame independently
      rather than using SAM 3's video tracking, because the sidecar carries
      no identity and idtracker.ai resolves identity itself.

  src/idtrackerai/extra_tools/detectron2_pipeline/prelabel.py
      Drafts LabelMe annotations from SAM 3 for correction by hand. Never
      overwrites an existing annotation.

  src/idtrackerai/segmentation_app/widgets/sam3_panel.py
      The SAM 3 mode of the Segmentation App: drafting annotations, and
      exporting contours when a CUDA GPU is present.

  tests/contours_export_test.py
      Tests for the contour exporter, driven by a stand-in model so they
      need no GPU.

  tests/sam3_prelabel_test.py
      Tests that drafted annotations build a COCO dataset unchanged.

  tests/sam3_panel_test.py, tests/bundle_test.py
      Tests for the SAM 3 panel's wiring and for the Colab bundle's layout.

  docs/sam3-pipeline.md
      Documentation for the SAM 3 route.

  VALIDATOR.md
      What this fork changes in the Validator, and why.

  USER_GUIDE.md
      An illustrated guide for someone who has not tracked an animal
      before, covering both segmentation routes end to end.

  docs/make_screenshots.py, docs/images/
      The screenshots used by that guide, and the script that drives the
      Segmentation App offscreen to regenerate them.

  NOTICE, CITATION.cff
      This file and citation metadata.


Third-party components
----------------------

The pipeline depends on, but does not include, the following.
Each carries its own licence:

  Detectron2   Apache-2.0   https://github.com/facebookresearch/detectron2
  PyTorch      BSD-3-Clause https://pytorch.org
  OpenCV       Apache-2.0   https://opencv.org
  LabelMe      GPL-3.0      https://github.com/wkentaro/labelme
  SAM 3        SAM License  https://github.com/facebookresearch/sam3

None of them is distributed with this software. Detectron2 and SAM 3 are
both optional and are installed separately by whoever runs the GPU stages.

The SAM 3 licence deserves a note. It is Meta's own "SAM License", not an
OSI-approved licence, and it governs both the sam3 package and the sam3.pt
weights. Its terms, including its acceptable use policy, are neither
granted nor restricted by this project's GPL-3.0, and anyone using the
SAM 3 route should read it. The ultralytics package can load the same
weights and was not used, because it is AGPL-3.0.

See CITATION.cff and README.md for the papers to cite.


Acknowledgement
---------------

The SAM 3 route began as an independent implementation by Talha
(github.com/laiquet) in idtrackerai-enhanced, which integrates SAM 3 and
Detectron2 into idtracker.ai by a different route: running the models
in-process during tracking rather than exporting contours beforehand. No
code was copied. What was taken is the approach -- segmenting from a text
prompt, and falling back on threshold-derived points to recover animals a
text prompt misses -- and the demonstration that it was worth doing at all.