Skip to content

Commit 658e514

Browse files
authored
Merge pull request #152 from mbel-tech/fatina-del-codice/sp3-vertical-provenance
feat(fusion): carry the side view's reconstructed-depth flags into the export (SP3)
2 parents 9c64fed + 2e0796d commit 658e514

8 files changed

Lines changed: 161 additions & 0 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
99

1010
### Added
1111

12+
- **Fused 3-D projects: the depth says which values were reconstructed.** A fused session's `was_interpolated` and `jump_replaced` describe the top view only, so a depth that rests on a side-view position the pipeline gap-filled or replaced was exported as if measured. The fused session now carries the side view's own flags, `depth_interpolated` and `depth_jump_replaced`, and the per-frame table (long CSV, feather, Excel) writes them beside `depth_fraction` and `depth_cm`. A flag is True only where the depth is a number; it is False where the depth is missing, on separator rows, and (like the depth) cleared by the ID-quality threshold. The depth values themselves are unchanged. Time windows slice the flags with the depth.
1213
- **Native 3-D files: the vertical coordinate is exported.** The per-frame table (`master_fish_by_frame` in the long CSV, feather and Excel exports) gains `z_cm`, the session's vertical coordinate in the project's length unit, whenever a native 3-D file is imported, with or without a water column; it is the same preprocessed value IL-16/IL-17 use, so IL-16 can be reproduced from `x_cm`, `y_cm` and `z_cm`. It is the file's own coordinate (source sign and origin), not a depth; `depth_fraction` and `depth_cm` still need the water column. NaN where the position is NaN, and masked by the ID-quality threshold like the other position columns. A metadata column named `z_cm` is ignored with a warning, as for `depth_cm`. 2-D and fused exports are unchanged. **The wide CSV export** also writes `master_fish_by_frame.csv` (byte-identical to the long export's) when the per-frame table has `z_cm` or `depth_*`, so a wide-format 3-D run no longer loses its vertical data; a 2-D wide export still writes only `trial_summary_wide.csv`. IL-15 on a native file with no water column now reports `depth_extent_source = none:no_water_column` instead of `none:not_derived`.
1314
- **Native 3-D files: Anipose and DeepLabCut 3-D.** Two new readers (`anipose` for `pose-3d` CSV, `deeplabcut_3d` for the triangulated tables; both built from the published layouts and tested on synthetic files only, so labelled *synthetic only*) give sessions with an absolute z in the project's length unit. `fps` and `unit` (mm, cm, m, in) are required options and never guessed (Anipose proposes `mm`, DeepLabCut 3-D proposes nothing); `target_unit` (default `cm`), `keypoint`, `likelihood_cutoff` (Anipose only), `vertical_axis` (`x`, `y`, `z`; default `z`), `flip` and `keep_skeleton` are optional. Units are converted when the file is read. IL-16 and IL-17 run on them with no water column; the 2-D metrics measure on the horizontal plane. **IL-15** runs once you give the **water column** (Surface z and Floor z on the Calibration screen); positions outside it have no depth and are counted in `frac_outside_extent`, never clipped. A native 3-D project is a 2-D project (leave Analysis type on 2D) with no Views or Fusion step; adding the first file sets Session calibration, the unit label and its confirmation, and later files must use the same target unit (`NATIVE_3D_UNIT_MISMATCH`); native and pixel sessions cannot share a project (`NATIVE_3D_MIXED`); other calibration modes are blocked (`NATIVE_3D_NEEDS_SESSION_CALIBRATION`). Not offered for native files (greyed out with a reason): every group metric, GL-16 included (a file holds one animal), and IL-3, IL-9, IL-10 and IL-14 (pixel-sized defaults that would change with the project unit). The water column is entered on the file's own vertical axis; each session's flip is applied for you. Limits: one animal per file, no z column in the per-frame table, no zones or background image, no refraction correction, a multi-animal DeepLabCut 3-D table (or one without full x, y, z triples) is still named by the scan as one it cannot read yet. The run `README.md` gains a **Native 3-D source** section. See the user guide and decision D-043.
1415
- **3-D metrics (fused 3-D projects).** Three new metrics next to the 2-D ones, which are unchanged: **IL-16 3-D Distance Travelled** (`path_length_3d_cm`, `path_length_3d_bl`, `n_valid_steps`), **IL-17 3-D Speed** (`mean_speed_3d_cm_s`, `median_speed_3d_cm_s`, `max_speed_3d_cm_s`, `mean_speed_3d_bl_s`) and **GL-16 3-D Nearest-Neighbour Distance** (`mean_nnd_3d_cm`, `median_nnd_3d_cm`, `mean_nnd_3d_bl`, `n_skipped_frames_3d`). Positions are in cm: x and y from the top view divided by its scale, depth fraction times the tank height from the fusion settings. IL-16 and IL-17 need identities, GL-16 does not; all three compute per time bin like their 2-D counterparts. They need the top view's cm scale (scalar or session calibration): in a 2-D project they are greyed out ("needs a fused 3-D session"), under body-length calibration they are greyed out ("needs a cm scale for the top view (use scalar or session calibration)"), and a unit whose top view has no scale is skipped at run time ("needs a cm scale for the top view"), with the reason in the run README. Track2Data does not estimate the scale from the fusion. IL-17 uses the project's kinematics estimator for the vertical speed, and its horizontal part is IL-2's speed. A step, frame or speed needs a valid x, y and depth, otherwise the value is NaN, never 0. Not included: vertical speed as its own metric, 3-D inter-individual distance and cohesion. The Metrics screen presets (for example *All metrics*) no longer tick any row that is greyed out for the project (also, for instance, identity-needing metrics when every session is identity-free). `codebook.csv` lists the whole registry, so it gains rows for the three metrics in 2-D runs too; no other 2-D output changes. `track2data list-metrics` shows "fused 3-D + cm scale" for them.

‎tests/test_exporters/test_depth_columns.py‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -333,3 +333,21 @@ def test_single_video_layout_records_offset_zero(tmp_path: Path) -> None:
333333
assert meta["run_metadata"]["fusion"]["fusion_frame_offset"] == 0
334334
sessions = pd.read_csv(out / "sessions.csv")
335335
assert set(sessions.fusion_frame_offset) == {0}
336+
337+
338+
def test_quality_threshold_clears_the_depth_provenance_flags_with_the_depth(tmp_path: Path) -> None:
339+
manifest = build_scene(tmp_path / "data")
340+
manifest.metrics.quality_threshold = 0.5
341+
engine = Engine(manifest)
342+
psess = engine.fuse_pair(manifest.view_pairs[0]).psess
343+
psess.depth_interpolated = np.ones(psess.depth.shape, dtype=bool)
344+
psess.depth_jump_replaced = np.ones(psess.depth.shape, dtype=bool)
345+
prob = np.ones((psess.n_frames, psess.n_animals))
346+
prob[30, 0] = 0.1
347+
psess.id_probabilities_rows = prob
348+
df = engine.build_fish_by_frame(psess)
349+
low = df[(df.individual_id == 0) & (df.frame == 30)].iloc[0]
350+
assert np.isnan(low.depth_fraction)
351+
assert not low.depth_interpolated and not low.depth_jump_replaced
352+
assert df.depth_interpolated.dtype == bool
353+
assert df.depth_interpolated.sum() == len(df) - 1
Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
"""A fused depth carries the side view's own status: which values were reconstructed."""
2+
3+
from __future__ import annotations
4+
5+
import numpy as np
6+
7+
from track2data.exporters.schema import depth_columns
8+
from track2data.fusion.fuse import fuse
9+
from track2data.metrics.binning import slice_psess
10+
11+
from .builders import make_pair, settings
12+
13+
14+
def _fused(**side_updates):
15+
top, side, pair = make_pair(fusion=side_updates.pop("fusion", None))
16+
n = side.xy.shape[0]
17+
# the tracker's own side positions; NaN where the preprocessing later filled a gap
18+
side.raw_xy_rows = side.xy.copy()
19+
for rows, animal in side_updates.pop("interpolated", []):
20+
side.raw_xy_rows[rows, animal] = np.nan
21+
jumped = side_updates.pop("jumped", [])
22+
if jumped:
23+
side.jump_replaced = np.zeros((n, 3), dtype=bool)
24+
for rows, animal in jumped:
25+
side.jump_replaced[rows, animal] = True
26+
return top, side, pair, fuse(top, side, pair, same_video=False)
27+
28+
29+
def test_depth_interpolated_marks_the_sides_filled_frames_for_that_fish():
30+
_, _, _, fused = _fused(interpolated=[(slice(10, 20), 0)])
31+
flag = fused.psess.depth_interpolated
32+
assert flag.shape == fused.psess.depth.shape and flag.dtype == bool
33+
expected = np.zeros_like(flag)
34+
expected[10:20, 0] = True
35+
np.testing.assert_array_equal(flag, expected)
36+
assert np.isfinite(fused.psess.depth[10:20, 0]).all() # still a number, now marked
37+
38+
39+
def test_depth_jump_replaced_marks_the_sides_replaced_frames():
40+
_, _, _, fused = _fused(jumped=[(slice(30, 35), 1)])
41+
flag = fused.psess.depth_jump_replaced
42+
expected = np.zeros_like(flag)
43+
expected[30:35, 1] = True
44+
np.testing.assert_array_equal(flag, expected)
45+
46+
47+
def test_a_side_without_a_jump_mask_gives_an_all_false_flag_not_none():
48+
_, _, _, fused = _fused()
49+
assert fused.psess.depth_jump_replaced is not None
50+
assert not fused.psess.depth_jump_replaced.any()
51+
assert not fused.psess.depth_interpolated.any()
52+
53+
54+
def test_the_flags_follow_the_frame_offset():
55+
top, side, pair = make_pair(fusion=settings(frame_offset=5))
56+
side.raw_xy_rows = side.xy.copy()
57+
side.raw_xy_rows[15, 0] = np.nan # side row 15 is top frame 10
58+
fused = fuse(top, side, pair, same_video=False)
59+
assert fused.psess.depth_interpolated[:, 0].nonzero()[0].tolist() == [10]
60+
61+
62+
def test_a_flag_is_false_where_the_depth_is_missing():
63+
top, side, pair = make_pair()
64+
side.raw_xy_rows = side.xy.copy()
65+
side.raw_xy_rows[3, 0] = np.nan
66+
side.xy[3, 0, 1] = 50.0 # filled, but outside the water column, so no depth
67+
fused = fuse(top, side, pair, same_video=False)
68+
assert np.isnan(fused.psess.depth[3, 0])
69+
assert not fused.psess.depth_interpolated[3, 0]
70+
71+
72+
def test_separator_rows_are_false():
73+
top, side, pair = make_pair()
74+
tracked = np.ones(100, dtype=bool)
75+
tracked[30:70] = False
76+
side.tracked_mask = tracked
77+
side.raw_xy_rows = side.xy.copy()
78+
side.raw_xy_rows[:, :] = np.nan # every tracked side frame counts as filled
79+
fused = fuse(top, side, pair, same_video=False)
80+
sep = fused.psess.separator_mask
81+
assert sep is not None and sep.any()
82+
assert not fused.psess.depth_interpolated[sep].any()
83+
assert fused.psess.depth_interpolated[~sep].all()
84+
85+
86+
def test_slice_psess_slices_the_flags():
87+
_, _, _, fused = _fused(interpolated=[(slice(10, 20), 0)], jumped=[(slice(30, 35), 1)])
88+
window = slice_psess(fused.psess, 8, 22)
89+
assert window.depth_interpolated.shape == (14, 3)
90+
assert window.depth_interpolated[2:12, 0].all() and not window.depth_interpolated[:2].any()
91+
assert window.depth_jump_replaced.shape == (14, 3) and not window.depth_jump_replaced.any()
92+
93+
94+
def test_the_export_columns_carry_the_flags_beside_depth():
95+
_, _, _, fused = _fused(interpolated=[(slice(10, 20), 0)], jumped=[(slice(30, 35), 1)])
96+
cols = depth_columns(fused.psess)
97+
assert list(cols) == [
98+
"depth_fraction", "depth_cm", "depth_interpolated", "depth_jump_replaced"
99+
]
100+
assert cols["depth_interpolated"].dtype == bool
101+
flat = fused.psess.depth_interpolated.reshape(-1)
102+
np.testing.assert_array_equal(cols["depth_interpolated"], flat)
103+
assert cols["depth_jump_replaced"].sum() == 5
104+
105+
106+
def test_a_session_without_a_depth_exports_no_flag_columns():
107+
top, _, _ = make_pair()
108+
assert depth_columns(top) == {}

‎track2data/api.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1703,6 +1703,10 @@ def build_fish_by_frame(
17031703
if c in df.columns
17041704
]
17051705
df.loc[below, masked_cols] = np.nan
1706+
# flags describe a depth that is now blank; a boolean cannot be NaN
1707+
for flag in ("depth_interpolated", "depth_jump_replaced"):
1708+
if flag in df.columns:
1709+
df.loc[below, flag] = False
17061710

17071711
self._attach_metadata(
17081712
df, psess, self.identity_free_for(psess.session, identity_free)

‎track2data/core/models.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -913,6 +913,14 @@ class PreprocessedSession:
913913
# shape as ``depth`` and sliced with it, so a window can count its own outside samples;
914914
# ``depth_outside`` equals its sum over frames.
915915
depth_outside_mask: np.ndarray | None = None
916+
# (n_frames, n_animals) bool, same shape as ``depth``: True where the side view's position
917+
# behind that depth was reconstructed rather than measured -- gap-filled
918+
# (``depth_interpolated``) or a replaced jump (``depth_jump_replaced``). False where the depth
919+
# is missing and on separator rows. A fused session's ``was_interpolated`` / ``jump_replaced``
920+
# describe the top view only, so these are the vertical axis's own. None for a session
921+
# without a fused depth (a native 3-D session marks its reconstructed z through xy).
922+
depth_interpolated: np.ndarray | None = None
923+
depth_jump_replaced: np.ndarray | None = None
916924
# (n_rows, n_animals) absolute vertical coordinate of a native 3-D session, in the
917925
# session's coordinate unit, laid out on the same rows as ``xy`` (NaN on inserted rows).
918926
# None for every other session. It is the file's coordinate cleaned like ``xy``: blanked

‎track2data/exporters/schema.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,8 @@ def depth_columns(psess: PreprocessedSession) -> dict[str, np.ndarray]:
4949
* ``depth_fraction`` / ``depth_cm`` -- depth below the surface, when a depth exists (a fused
5050
session, or a native one with a water column). ``depth_cm`` is NaN when the tank height
5151
is unknown.
52+
* ``depth_interpolated`` / ``depth_jump_replaced`` -- booleans beside a fused depth, True
53+
where it rests on a side-view position the pipeline gap-filled or replaced.
5254
5355
Row order is the table's (frame-major, animal-minor). Every value is NaN where the animal's
5456
position is NaN. Empty for a 2-D session.
@@ -68,6 +70,12 @@ def depth_columns(psess: PreprocessedSession) -> dict[str, np.ndarray]:
6870
cm = fraction * height if height is not None else np.full_like(fraction, np.nan)
6971
out["depth_fraction"] = fraction
7072
out["depth_cm"] = cm
73+
for name, flag in (
74+
("depth_interpolated", psess.depth_interpolated),
75+
("depth_jump_replaced", psess.depth_jump_replaced),
76+
):
77+
if flag is not None:
78+
out[name] = np.asarray(flag, dtype=bool).reshape(-1)
7179
return out
7280

7381

‎track2data/fusion/fuse.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -137,6 +137,12 @@ def fuse(
137137
# counted on the side y alone; depth is NaN wherever either view lacks a position
138138
outside = np.isfinite(y) & ~((depth >= 0) & (depth <= 1))
139139
depth = np.where(valid & ~outside, depth, np.nan)
140+
# which of those depths rest on a reconstructed side position (the side view's own flags)
141+
has_depth = np.isfinite(depth)
142+
depth_interpolated = side.was_interpolated[srows][:, side_cols] & has_depth
143+
depth_jump_replaced = np.zeros_like(has_depth)
144+
if side.jump_replaced is not None:
145+
depth_jump_replaced = side.jump_replaced[srows][:, side_cols] & has_depth
140146

141147
kin = top.kinematics
142148
n = top.n_animals
@@ -186,6 +192,8 @@ def per_row_optional(a: np.ndarray | None) -> np.ndarray | None:
186192
depth_height_cm=fs.tank_height_cm,
187193
depth_outside=outside.sum(axis=0).astype(int),
188194
depth_outside_mask=_separate(outside, at),
195+
depth_interpolated=_separate(depth_interpolated, at),
196+
depth_jump_replaced=_separate(depth_jump_replaced, at),
189197
source_animal_index=np.asarray(keep, dtype=np.int64),
190198
)
191199
rms: float | None = None

‎track2data/metrics/binning.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,12 @@ def slice_psess(psess: PreprocessedSession, start: int, stop: int) -> Preprocess
8383
depth_outside_mask=(
8484
None if psess.depth_outside_mask is None else psess.depth_outside_mask[start:stop]
8585
),
86+
depth_interpolated=(
87+
None if psess.depth_interpolated is None else psess.depth_interpolated[start:stop]
88+
),
89+
depth_jump_replaced=(
90+
None if psess.depth_jump_replaced is None else psess.depth_jump_replaced[start:stop]
91+
),
8692
z=None if psess.z is None else psess.z[start:stop],
8793
kinematics=KinematicsArrays(
8894
speed_px_s=kin.speed_px_s[start:stop],

0 commit comments

Comments
 (0)