Repository navigation
Drop the run-level cell count and read acquisitions from the registry - #32
Merged
Merged
Conversation
build_packaging_process was called before package_cells with cell_count=0, and the record was embedded verbatim in every cell, so every processing.json claimed zero cells packaged. Its end_date_time was also stamped before any packaging work ran. package_cells now resolves every cell before writing any, then builds the record from the resulting count and the actual finish time. It takes a factory rather than a record because neither value is known at the call site.
Every asset is a single cell, so recording how many cells the run produced says nothing about the asset it sits in. build_packaging_process no longer takes cell_count and package_cells' factory needs only the finish time, which remains necessary so the record does not claim to have finished before the work ran. The acquisition now resolves DocDB v2 -> v1 -> S3, matching default_sources, with a staged copy only as a last resort. The registry holds what the indexer published; a mounted bundle may carry a stale copy or none. check_orientation reads the axes from it, so a wrong one mis-registers every cell without failing. Both fetchers are generalised to any core field.
peter-grotz
changed the title
Packaging hardlinks reconstructions and writes metadata; the SWCs that come out are byte-identical to the ones that go in. A DataProcess for it describes assembling the asset rather than producing the data, in a document meant for the latter. Where the parent metadata came from stays on DataDescription.tags, which is indexed and so searchable, unlike output_parameters. Removing the step also removes the need for the finish-time factory, so package_cells returns to a single pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No run-level cell count. Every asset is a single cell, so
cells_packagedsaid nothing about the asset it sat in.build_packaging_processno longer takescell_count.package_cellsstill takes a factory, nowCallable[[datetime], DataProcess], because the second defect stands on its own:end_date_timewas stamped before any packaging ran, so the recorded interval excluded the work. Cells are resolved first, then the record is built with the real finish time.Acquisitions resolve from the registry. New
acquisition.pywalks DocDB v2 → v1 → S3, matchingdefault_sources, and writes the document to a file for the ANTs API.docdb_fetcherands3_fetcherare generalised fromdata_descriptionto any core field.Verified live: 826509 and 794492 both resolve from DocDB v1 with
axespresent; neither is in v2.169 tests, 100% coverage.