Characterization of Training Data for Downstream Segmentation Model Training

This repository packages tooling for characterizing training data used in downstream segmentation model development.

The checked-in example figures and wrapper script come from the 802449 carveout workflow, but the repository is intended to support the more general question:

• given partitioned GT reconstructions (correct, omit, split)
• given per-point intensity arrays
• given sparse carveout volumes on GCS
• given merge/self-merge detections

which 3D blocks are enriched for omits, merges, or unusual intensity structure, and how can those blocks be visualized on top of the full neuron geometry before they are used for downstream model training or review?

What The Repository Does

The scripts in scripts/ support:

• sparse carveout block accounting from remote OME-Zarr chunk occupancy
• 512-cube analysis with omit/correct/split length and intensity summaries
• block catalog export and metric distribution plots
• ROI-based block selection
• neuron-level component overlays
• merge/self-merge explanatory figures
• rerendering of block plots with lighter visual styling

This repository does not rerun GT partitioning itself. It expects an analysis workspace that already contains:

• partitioned-swcs/
• intensity-npys/
• predicted-components/
• optional merge-detection exports under carveout-analysis/merge-detection/errors/

Repository Layout

.
├── docs/
│   └── images/
├── scripts/
├── requirements.txt
└── README.md

Example Visuals

Single 512 block with high detected merge signal:

Whole-neuron cube visualization:

Merge-detection process figure:

Main Scripts

• analyze_carveout_cubes_512.py
  • Main 512-cube analysis entrypoint.
  • Produces cube_summary_512.csv, carveout inventories, and 3D block plots.
• export_block_catalog_distributions.py
  • Builds a block catalog and per-carveout metric distribution plots.
• render_neuron_component_overlays.py
  • Renders GT vs predicted-component overlay panels for each neuron.
• select_roi_40_blocks.py
  • Selects and plots ROI-constrained 512 blocks from the analyzed catalog.
• visualize_merge_detection_process.py
  • Creates explanatory merge/self-merge process figures from merge-detection outputs.
• rerender_carveout_analysis_512_light_boxes.py
  • Re-renders 512-cube plots with lighter box styling for presentation.
• run_802449_seedfix_overlapfix.sh
  • Example orchestration script for the 802449_raw_unet_r3_ckpt_425250_seedfix_overlapfix workspace.
  • Included as a concrete sample configuration, not as the repository scope.

Inputs

The core workflow expects:

• a local workspace directory with partitioned SWCs, intensity arrays, and predicted components
• access to a carveout bucket on GCS
• gsutil on PATH
• optionally, a checkout of the merge-detection repository for process-figure generation

Example carveout base used for the included figures:

gs://allen-nd-goog/from_aind/agrim-experimental/image-carveouts/802449

Installation

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install -r requirements.txt

System dependency:

• gsutil

Example Workflow

Run the end-to-end example wrapper:

export WORKSPACE_DIR=/abs/path/to/802449_workspace
export OUTPUT_ROOT=/abs/path/to/output_root
export EVENTS_CSV=/abs/path/to/events.csv
export MERGE_REPO_DIR=/abs/path/to/neuron-segmentation-merge-detection-main

bash scripts/run_802449_seedfix_overlapfix.sh

This wrapper runs:

1. 512-cube carveout analysis
2. block catalog export
3. neuron component overlays
4. ROI block selection
5. merge-detection process visualizations

Direct Script Usage

512-cube analysis

python3 scripts/analyze_carveout_cubes_512.py \
  --workspace-dir /abs/path/to/workspace \
  --carveout-base-gcs gs://allen-nd-goog/from_aind/agrim-experimental/image-carveouts/802449 \
  --events-csv /abs/path/to/events.csv \
  --output-dir /abs/path/to/output_root/carveout-analysis-512-full-chunks-batched

Block catalog export

python3 scripts/export_block_catalog_distributions.py \
  --source-csv /abs/path/to/cube_summary_512.csv \
  --output-dir /abs/path/to/output_root/block-catalog-distributions

Merge-detection process figures

python3 scripts/visualize_merge_detection_process.py \
  --results-dir /abs/path/to/workspace \
  --errors-dir /abs/path/to/errors \
  --repo-dir /abs/path/to/neuron-segmentation-merge-detection-main \
  --output-dir /abs/path/to/output_root/merge-detection-process-figures

Output Structure

A typical output tree looks like:

outputs/sample_name/
├── block-catalog-distributions/
├── carveout-analysis-512-full-chunks-batched/
├── merge-detection-process-figures/
├── neuron-component-overlays/
└── roi-40-block-selection/

Production Notes

The source in this repository has been cleaned up from the original ad hoc analysis directory by:

• removing machine-specific hardcoded paths from the public CLIs
• separating source from generated outputs
• preserving only a few documentation figures under docs/images
• documenting the expected workspace layout and command-line entrypoints

The scripts are still intentionally file-based and analysis-oriented rather than packaged as a Python library. The goal here is reproducible block analysis, not a generalized SDK.