Skip to content

Commit 3a0d4b2

Browse files
3d frame adjustments and demo (#231)
* 3d frame adjustments and demo * snapshots * color distinctions and anisotopy * notebook updates and testing * readd tests, address coderabbit * [pre-commit.ci lite] apply automatic fixes * fixes * Fix CI segfault: remove 2D screen-space marker overlay The vtkActor2D-based "always visible" center marker overlay (_add_center_marker_2d_overlay_to_plotter) segfaulted under headless Linux CI rendering, in its own regression test (test_center_marker_2d_overlay_stays_visible_over_dense_volume), never reproduced locally. This is the second "always visible" marker approach to crash CI this way (the first was a pv.Renderer/ SetNumberOfLayers render-layer trick); reverting to the plain 3D mesh marker rather than risk a third untested VTK technique. Also fixes .pre-commit-config.yaml's ruff-pre-commit pin, which had drifted back to v0.16.8 via a merge, out of sync with pyproject.toml's ruff==0.16.5 dependency pin. * Fix CI segfault: close off-screen plotter after each snapshot _pyvista_volume_snapshot_html created a fresh pv.Plotter(off_screen=True) per cell but never closed it. Leaving many off-screen VTK render windows/GL contexts open across a test run (each test/cell renders its own throwaway plotter) eventually segfaulted under headless Mesa software rendering in Linux CI, at a different, nondeterministic test each run -- most recently in test_generate_trame_snapshot_html_skips_eager_full_volume_read, a pre-existing test unrelated to the marker feature, confirming this is a general resource-leak issue in the snapshot path rather than anything specific to the marker code removed in the prior commit. Closing the plotter in a finally block after the screenshot is taken (it's never reused after that point) releases the VTK resources immediately instead of accumulating them for the life of the process. * CI: use Xvfb for headless pyvista rendering on Linux VTK/pyvista's built-in headless (EGL/OSMesa) off-screen rendering path has segfaulted intermittently in plotter.render() partway through the test suite on ubuntu-24.04 runners (never reproduced on macOS runners or locally), even after closing each throwaway snapshot plotter. A real Xvfb virtual framebuffer is pyvista's own documented, more heavily tested recommendation for headless Linux CI rendering, so use the pyvista/setup-headless-display-action for the Linux jobs that run the test suite. --------- Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com>
1 parent 9a28fde commit 3a0d4b2

9 files changed

Lines changed: 3686 additions & 105 deletions

File tree

‎.github/dependabot.yml‎

Lines changed: 37 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@
66

77
version: 2
88
updates:
9+
# Cooldown docs:
10+
# https://docs.github.com/en/code-security/dependabot/working-with-dependabot/dependabot-options-reference
911
# GitHub Actions checks
1012
# See here for more information:
1113
# https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuring-dependabot-version-updates
@@ -14,16 +16,48 @@ updates:
1416
schedule:
1517
# Check for updates to GitHub Actions every month
1618
interval: "monthly"
19+
cooldown:
20+
default-days: 7
21+
groups:
22+
github-actions:
23+
patterns:
24+
- "*"
1725

18-
# Perform checks and updates for python poetry environment.
26+
# Perform checks and updates for python uv environment.
27+
# This depends on changes to dependabot via:
28+
# https://github.com/dependabot/dependabot-core/pull/10040
1929
# See here for more information:
2030
# https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#pip-and-pip-compile
21-
- package-ecosystem: "pip"
31+
- package-ecosystem: "uv"
2232
directory: "/"
2333
schedule:
24-
# Check for updates to poetry lockfile every month
34+
# Check for updates to uv environment files every month
2535
interval: "monthly"
36+
cooldown:
37+
default-days: 7
38+
semver-major-days: 14
39+
semver-minor-days: 7
40+
semver-patch-days: 3
2641
groups:
2742
python-packages:
2843
patterns:
2944
- "*"
45+
46+
# Perform checks and updates for pre-commit hook versions.
47+
# Replaces the pre-commit-update pre-commit hook (removed): that hook
48+
# rewrote .pre-commit-config.yaml mid-run inside every prek/pre-commit
49+
# invocation (including in CI), so a newer, stricter tool version could
50+
# fail lint on code nobody had touched, in the same run that just bumped
51+
# it -- and CI would then auto-commit that bump back to the PR branch.
52+
# Dependabot opens a normal, reviewable PR for hook version bumps instead.
53+
- package-ecosystem: "pre-commit"
54+
directory: "/"
55+
schedule:
56+
# Check for updates to pre-commit hooks every month
57+
interval: "monthly"
58+
cooldown:
59+
default-days: 7
60+
groups:
61+
pre-commit-hooks:
62+
patterns:
63+
- "*"

‎.github/workflows/run-tests.yml‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,9 @@ name: tests
44

55
"on":
66
push:
7-
branches: [main]
7+
branches: [main, 0.3.x]
88
pull_request:
9-
branches: [main]
9+
branches: [main, 0.3.x]
1010

1111
jobs:
1212
pre_commit_checks:
@@ -23,6 +23,14 @@ jobs:
2323
python-version: "3.11"
2424
- name: Install the latest version of uv
2525
uses: astral-sh/setup-uv@v7
26+
# VTK/pyvista's built-in headless (EGL/OSMesa) off-screen rendering
27+
# path has been unreliable on ubuntu-24.04 runners here (intermittent
28+
# segfaults inside plotter.render() partway through the suite, never
29+
# reproduced on macOS runners or locally). A real Xvfb virtual
30+
# framebuffer is pyvista's own documented, far more heavily tested
31+
# recommendation for headless Linux CI rendering.
32+
- name: Setup headless display
33+
uses: pyvista/setup-headless-display-action@v5
2634
# run pre-commit
2735
- uses: pre-commit/action@v3.0.1
2836
id: pre_commit
@@ -46,6 +54,12 @@ jobs:
4654
python-version: ${{ matrix.python_version }}
4755
- name: Install the latest version of uv
4856
uses: astral-sh/setup-uv@v7
57+
# See the same step in pre_commit_checks above for why: VTK/pyvista's
58+
# built-in headless off-screen rendering has segfaulted intermittently
59+
# on ubuntu-24.04 runners; a real Xvfb display is the documented fix.
60+
- name: Setup headless display
61+
if: ${{ runner.os == 'Linux' }}
62+
uses: pyvista/setup-headless-display-action@v5
4963
- name: Run pytest (non-3.13)
5064
if: ${{ matrix.python_version != '3.13' }}
5165
# run all tests

‎.pre-commit-config.yaml‎

Lines changed: 1 addition & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ repos:
1515
- id: check-yaml
1616
- id: detect-private-key
1717
- repo: https://github.com/tox-dev/pyproject-fmt
18-
rev: "v2.29.3"
18+
rev: "v2.29.4"
1919
hooks:
2020
- id: pyproject-fmt
2121
- repo: https://github.com/codespell-project/codespell
@@ -58,11 +58,6 @@ repos:
5858
rev: v0.1.17
5959
hooks:
6060
- id: almanack-check
61-
- repo: https://gitlab.com/vojko.pribudic.foss/pre-commit-update
62-
rev: v0.6.0
63-
hooks:
64-
- id: pre-commit-update
65-
args: ["--keep", "mdformat", "--keep", "pre-commit-update", "--keep", "cffconvert"]
6661
- repo: local
6762
hooks:
6863
- id: code-cov-gen

‎docs/src/examples/nf1_3d_pilot_cytodataframe_verification.ipynb‎

Lines changed: 1707 additions & 0 deletions
Large diffs are not rendered by default.
Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
# ---
2+
# jupyter:
3+
# jupytext:
4+
# formats: ipynb,py:light
5+
# text_representation:
6+
# extension: .py
7+
# format_name: light
8+
# format_version: '1.5'
9+
# jupytext_version: 1.19.1
10+
# kernelspec:
11+
# display_name: CytoDataFrame (3.13.5.final.0)
12+
# language: python
13+
# name: python3
14+
# ---
15+
16+
# # NF1 3D pilot cropped voxel view
17+
#
18+
# View real 3D single-cell crops (with mask overlays) from two NF1 wells in one `CytoDataFrame`.
19+
#
20+
# Most of the setup below isn't about `CytoDataFrame` -- it's working around this
21+
# data's layout, which doesn't look like typical CellProfiler output:
22+
#
23+
# - There are no `Image_FileName_*`/`Image_PathName_*` columns, so each row's raw
24+
# image path is built from the patient/well/field metadata (`load_image_set`).
25+
# - Every well's mask file has the exact same generic name (`nuclei_mask.tiff`),
26+
# so combining two wells into one table needs `stage_mask` to tell them apart
27+
# -- a single-well notebook wouldn't need this at all.
28+
#
29+
# The actual `CytoDataFrame(...)` call at the end is just a handful of arguments.
30+
31+
# +
32+
import os
33+
import pathlib
34+
import re
35+
import tempfile
36+
37+
import pandas as pd
38+
39+
from cytodataframe import CytoDataFrame
40+
41+
# Local mount point for bandicoot. Set BANDICOOT_LOCAL_BASE if it is mounted
42+
# somewhere else on your machine.
43+
BANDICOOT_LOCAL_BASE = pathlib.Path(
44+
os.environ.get("BANDICOOT_LOCAL_BASE", "~/mnt/bandicoot")
45+
).expanduser()
46+
DATA_DIR = BANDICOOT_LOCAL_BASE / "NF1_organoid_data/data"
47+
PROFILES_DIR = (
48+
DATA_DIR / "image_based_profiles_production_zedprofiler/ibp/sc_profiles_related"
49+
)
50+
if not PROFILES_DIR.is_dir():
51+
raise FileNotFoundError(
52+
f"Profiles not found at {PROFILES_DIR}. Set the BANDICOOT_LOCAL_BASE "
53+
"environment variable to wherever bandicoot is mounted on this machine."
54+
)
55+
56+
# An "image set" here is one well + field-of-view (one 3D image). Add every
57+
# image set you want to view to this list -- e.g. all wells/fields in a
58+
# plate for per-plate QC -- not just the two shown here.
59+
IMAGE_SETS = [
60+
"NF0055_T1__NF0055_T1__B10__F1",
61+
"NF0014_T1__NF0014_T1__C4__F2",
62+
]
63+
CHANNEL = "DNA"
64+
CHANNEL_CODE = "405"
65+
COMPARTMENT = "Nuclei"
66+
67+
# Every well's mask file has the same generic name (e.g. "nuclei_mask.tiff"),
68+
# but CytoDataFrame's data_mask_context_dir only matches masks by filename
69+
# pattern within one shared directory -- it can't tell two identically-named
70+
# files in different wells apart. MASK_LINK_DIR is a scratch directory (not
71+
# part of the repo) that `stage_mask` below fills with per-well symlinks to
72+
# the real mask files, renamed to embed each well's own identifier, so that
73+
# matching works.
74+
MASK_LINK_DIR = pathlib.Path(tempfile.gettempdir()) / "cytodataframe_nf1_3d_mask_links"
75+
MASK_LINK_DIR.mkdir(exist_ok=True)
76+
77+
78+
def load_image_set(image_set: str) -> tuple[pd.DataFrame, pathlib.Path, pathlib.Path]:
79+
profiles = pd.read_parquet(PROFILES_DIR / f"{image_set}.parquet").head(1)
80+
row = profiles.iloc[0]
81+
patient = row["Metadata_Biology_PatientTumor"]
82+
# e.g. B10-1
83+
well_field = (
84+
f"{row['Metadata_Experiment_WellID']}-{row['Metadata_Imaging_FieldID']}"
85+
)
86+
87+
channel_path = (
88+
DATA_DIR
89+
/ patient
90+
/ "zstack_images"
91+
/ well_field
92+
/ f"{well_field}_{CHANNEL_CODE}.tif"
93+
)
94+
mask_path = (
95+
DATA_DIR
96+
/ patient
97+
/ "segmentation_masks"
98+
/ well_field
99+
/ f"{COMPARTMENT.lower()}_mask.tiff"
100+
)
101+
profiles[f"Image_FileName_{CHANNEL}"] = str(channel_path)
102+
return profiles, channel_path, mask_path
103+
104+
105+
def stage_mask(channel_path: pathlib.Path, mask_path: pathlib.Path) -> pathlib.Path:
106+
# See MASK_LINK_DIR above for why this exists.
107+
link = MASK_LINK_DIR / f"{channel_path.stem}__{mask_path.name}"
108+
if not link.exists():
109+
link.symlink_to(mask_path)
110+
return link
111+
112+
113+
# -
114+
115+
# Load one object per well into a single two-row table.
116+
117+
# +
118+
profile_rows = []
119+
for image_set in IMAGE_SETS:
120+
profiles, channel_path, mask_path = load_image_set(image_set)
121+
stage_mask(channel_path, mask_path)
122+
profile_rows.append(profiles)
123+
124+
profiles = pd.concat(profile_rows, ignore_index=True)
125+
mask_name = mask_path.name
126+
127+
# +
128+
bbox_column_map = {
129+
"x_min": f"{COMPARTMENT}_NoChannel_VolumeSizeShape_MinX",
130+
"x_max": f"{COMPARTMENT}_NoChannel_VolumeSizeShape_MaxX",
131+
"y_min": f"{COMPARTMENT}_NoChannel_VolumeSizeShape_MinY",
132+
"y_max": f"{COMPARTMENT}_NoChannel_VolumeSizeShape_MaxY",
133+
"z_min": f"{COMPARTMENT}_NoChannel_VolumeSizeShape_MinZ",
134+
"z_max": f"{COMPARTMENT}_NoChannel_VolumeSizeShape_MaxZ",
135+
}
136+
137+
center_columns = [
138+
f"{COMPARTMENT}_NoChannel_VolumeSizeShape_Center{axis}" for axis in "XYZ"
139+
]
140+
141+
voxel_view = CytoDataFrame(
142+
data=profiles[[f"Image_FileName_{CHANNEL}"]],
143+
data_bounding_box=profiles[list(bbox_column_map.values())],
144+
compartment_center_xy=profiles[center_columns],
145+
data_mask_context_dir=str(MASK_LINK_DIR),
146+
segmentation_file_regex={rf"__{re.escape(mask_name)}$": r"_\d+\.tif$"},
147+
display_options={
148+
"width": 260,
149+
"height": 260,
150+
"table_max_height": "580px",
151+
"label_overlay_mode": "filled",
152+
# Voxel size (x, y, z) in um. These TIFFs carry no voxel-size
153+
# metadata, and z is coarser than xy, so without this the objects are
154+
# drawn too flat. xy_spacing (0.1) is the NF1 3D pipeline manifests'
155+
# recorded value; z_spacing (0.5) was chosen visually against these
156+
# two objects rather than read from the manifests (which record 1.0).
157+
"volume_spacing": (0.1, 0.1, 0.5),
158+
"volume_bbox_column_map": bbox_column_map,
159+
},
160+
)
161+
# The rendered widget includes a "Mask" checkbox to toggle the overlay on/off,
162+
# and a red dot marking each object's center.
163+
voxel_view

‎pyproject.toml‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,9 +34,23 @@ dependencies = [
3434
"pyvista>=0.46.4",
3535
"pywavelets>1.4.1",
3636
"scikit-image>0.19.3",
37+
# tifffile+zarr enable a lazy, windowed read of a per-object 3D bounding
38+
# box (see `_try_lazy_windowed_tiff_crop`): only the object's own
39+
# z-planes/window are read from a plain TIFF z-stack, instead of decoding
40+
# the whole (often hundreds-of-MB) volume just to crop it down afterward.
41+
# Without these, 3D volumes still load correctly via the existing eager
42+
# `imageio` read -- just slower for large field-of-view TIFFs. tifffile
43+
# dropped Zarr 2 support for its zarr-store integration in 2025.5.21 (an
44+
# older zarr paired with a newer tifffile breaks the lazy path entirely,
45+
# rather than just falling back to the eager read), and 2026.4.11 was its
46+
# last release still supporting Python 3.11 (required below); a later
47+
# tifffile also starts requiring zarr>=3.2.0, which itself needs
48+
# Python>=3.12, so staying under that keeps 3.11 support intact.
49+
"tifffile>=2025.5.21,<2026.4.11",
3750
"trame>=3.12",
3851
"trame-vtk>=2.10",
3952
"trame-vuetify>=3.1",
53+
"zarr>=3",
4054
]
4155

4256
[dependency-groups]
@@ -53,6 +67,12 @@ dev = [
5367
"poethepoet>=0.37,<0.49",
5468
"pytest>=8.3.3,<10",
5569
"pytest-cov>=5,<8",
70+
# Pinned to match .pre-commit-config.yaml's ruff-pre-commit rev. ruff was
71+
# not previously a declared project dependency at all, so `uv run ruff`
72+
# silently fell through to an unrelated, unpinned `ruff` found elsewhere
73+
# on PATH (a pyenv shim, in practice) -- giving different lint results
74+
# locally than prek/CI's pinned hook used, depending on which one you ran.
75+
"ruff==0.16.5",
5676
"sqlalchemy>=1.3.6,<3",
5777
]
5878
docs = [
@@ -114,7 +134,7 @@ lint.per-file-ignores."docs/src/examples/*.py" = [ "E402", "E501" ]
114134
lint.per-file-ignores."src/cytodataframe/*.py" = [ "ANN401", "PLC0415" ]
115135
lint.per-file-ignores."src/cytodataframe/image.py" = [ "PLR2004" ]
116136
# ignore typing rules for tests
117-
lint.per-file-ignores."tests/*" = [ "ANN201", "PLR0913", "PLR2004", "SIM105" ]
137+
lint.per-file-ignores."tests/*" = [ "ANN201", "PLR0913", "PLR0917", "PLR2004", "SIM105" ]
118138

119139
# specify where version replacement is performed
120140
[tool.bandit]

0 commit comments

Comments
 (0)