CHANGELOG.md

Changelog

0.5.0

Added, at the suggestion of a BIL maintainer: BIL's BrAinPI service (brainapi.brainimagelibrary.org), which serves Imaris, .omehans, TeraFly and OME-Zarr files as OME-Zarr pyramids and Neuroglancer precomputed on the fly.

• DatasetDetail.brainpiroot / .brainpidata / .has_brainpi: BIL's list of files it serves this way (28 percent of datasets in a 1,500 dataset sample; mostly Imaris).
• brainpi_links(path): every view BrAinPI offers for one file or store, from the service's own answer. The URLs are not a prefix of the disk path (/bil/data/ becomes bil_data/), so they are never assembled here. brainpi_views(dataset), brainpi_paths(detail), first_omezarr(dataset), disk_path(url).
• open_zarr() and thumbnail() fall back to the first BrAinPI OME-Zarr view when a dataset has no TIFF, JPEG 2000 or zarr on the download server. An Imaris-only MORF dataset previews from a few chunks of the coarsest level instead of "download it and open with h5py".
• zarr_levels() no longer fails on a store with no directory listing (a BrAinPI view); it probes the declared levels instead.
• CLI: scigantic-bil views <id | /bil/path | url>.

Measured 2026-10-02: a 3-channel 12000 x 16000 x 140 .omehans opens as 13 levels in 20 s, a 64 x 64 read takes 0.3 s; an .ims opens in 2.6 s. A .jp2 view answered neither in 120 s nor 410 s, so JPEG 2000 views are returned as links but never opened automatically.

0.4.1

The 2026-09-26 CI run failed on two tests. One found a real defect in how timeouts were retried; the other asserted a fixed total on a number the Brain Image Library does not keep stable between inventory snapshots.

Fixed:

• A read timeout was retried four times with the same timeout, so a directory BIL's nginx could not render within 120 s (a 44,744-entry one took 120 s for 5.3 MB on 2026-09-30) cost ten minutes per call before failing. send() now raises BilError on the first read timeout, with the timeout in the message: the server is still building the response, and asking again is no faster. Connection errors, 429 and 5xx are still retried.
• list_files() waits LISTING_TIMEOUT (300 s, was 120 s) for a listing and takes a timeout argument; download() takes one too.
• BilCatalog.summary() reports datasets_without_size: the inventory's size column is blank or zero for hundreds of rows on some days (492 of 14,233 on 2026-09-24, 208 on 2026-09-27), which moved the archive total between 4,577 and 6,007 TB across three days and is why a fixed threshold on it failed.
• Dataset.from_row() no longer raises on a short inventory row (csv.DictReader fills missing columns with None).

Tests: the encoding test reads the 6-entry parent of the # directory instead of the directory itself; the crawl-budget and preview-size tests count requests and bytes instead of seconds. CI runs all five Python legs to completion (fail-fast: false), caps each at 45 minutes, prints the 15 slowest tests, and publishes with --skip-existing.

0.4.0

Fixes from a second run against the live archive on 2026-09-23 (stratified listing sweep over 253 datasets, 30 thumbnails across every layout, odd API inputs, offline edge cases).

Fixed:

• walk() (and so find(), slices(), read_stack()) crawled without limit when a dataset's manifest is over MANIFEST_MAX_BYTES: on ace-boo-sag (1.8 million files, 232 MB manifest) it had yielded 51 entries after 90 s and was still going. The crawl now lists at most max_dirs directories (WALK_MAX_DIRS, 512) and raises BilError pointing at manifest(bildid, max_bytes=None), first_images() and find_zarr().
• retrieve_many() raised BilError (HTTP 405, "no entries found") when every id in a 100-id batch was unknown, contrary to its contract; such a batch is now empty.
• list_files() recovers inventory paths that the server holds with underscores where the inventory has spaces: 36 of the 39 such rows 404'd and all 138 directory names under that project matched after the swap. Everything built on listings (thumbnail(), first_images(), terafly_levels(), extensions_under()) benefits; the entries carry the server's real URLs. A random sample of 300 other inventory paths found one stale, so this was the one systematic case.
• read_region() returned an empty array for a rectangle outside the page; it raises IndexError naming the page size. A page number past the end (read_tiff(key=), read_region(page=)) and a thumbnail() index past the last slice raise IndexError with the count instead of a bare index error from deep inside tifffile.
• thumbnail(max_size=0) divided by zero after downloading the slice; max_size < 1 and read_stack(step=0) are rejected before any request.
• thumbnail(channel=...) with no matching file, and read_stack() with a start/stop/step selecting nothing, reported "no slices found"; each now says what was found and what the filter excluded.
• BilCatalog.load(date=...) used the date in a URL and a cache file name without checking it; anything but eight digits is now a ValueError.
• A cache entry missing its value key raised KeyError; it is dropped.
• The CLI printed a traceback for every error (unknown id, stale path, unsupported format, bad index); it prints one line and exits 1.
• filter(extension="tif") (no dot) silently matched nothing; the dot is optional now, in the API and the CLI.

Added:

• preview_plane()/thumbnail() take max_bytes (PREVIEW_MAX_BYTES, 512 MB) and refuse a JPEG 2000 file larger than that: the format cannot be read partially on BIL, and single-file RGB sections cost 320 to 666 MB and 25 to 53 s each for a 256-pixel preview. Pass max_bytes=None to fetch anyway.

Measured and left alone: the inventory and /retrieve disagree on file counts for 16 of 40 sampled datasets and on size for some (ace-oat-old is 0 bytes in the inventory and 3.4 TB in the API), so size filters on the inventory can miss datasets; one dataset root listing of 61,000 entries takes 77 s from BIL's nginx on every cold call.

0.3.0

• JPEG 2000 (.jp2, .j2k, .jpx) reads in place: read_jp2() decodes exactly through imagecodecs' bundled OpenJPEG (a 12 MB STPT section to (11377, 8557) uint16 in 1.5 s; Dong-lab tracing sections to (12000, 16000, 3) uint16), with reduce=k for a 1/2**k decode through Pillow on the 75 to 550 MB RGB sections. read_image() dispatches by extension; slices(), read_stack(), first_images(), find(), thumbnail() and preview_plane() accept JPEG 2000 alongside TIFF. This covers 5,787 datasets that previously raised UnsupportedFormatError.
• Checked and documented: BIL's codestreams do not decode truncated, so the whole file is fetched; Pillow's reduced decode rejects 16-bit single-channel files, which fall back to a full decode.

0.2.0

Everything here came out of running the package against the whole archive (random samples across techniques, the largest trees and files, odd paths, sixteen threads on a cold cache) on 2026-09-08.

Fixed:

• Paths with #, spaces or non-ASCII characters are percent-encoded (encode_path, encode_url, idempotent). An unencoded # was a URL fragment, so 39 inventory datasets could not be listed or read.
• Directory-listing names are decoded (Virus_tracing-B1-#6), URLs keep the server's encoding.
• Two-word and one-word dataset ids (act-nod, 861 of them) were rejected by the id pattern.
• Six concurrent cold catalog loads crashed on a shared temp file name; each writer now uses its own.
• walk()/thumbnail() sat for over twenty minutes on a 1.6 million file fMOST tree. find_zarr() now has a directory budget, thumbnail() uses a bounded first-folder search (first_images), and TeraFly trees are recognised and previewed from their coarsest resolution folder.
• manifest() refuses manifests over 64 MB gzipped (a 4.7 million file dataset's is 710 MB) with ManifestTooLargeError; walk() and find_zarr() fall back to crawling. manifest_size() is the cheap check.
• A strip-sampled preview came out one pixel wide (decoded strip shape was misread).
• A dataset renamed on the server since the inventory was built, and a dataset still under the landing zone (/bil/lz/), each get a clear BilNotFoundError instead of a bare 404.
• tiff_info() no longer fails on files where tifffile cannot build a series ("incompatible keyframe"); pages are reported anyway.

Added:

• preview_plane(): bounded preview of a TIFF of any size (coarsest pyramid level, or every k-th row by offset for uncompressed planes, or sampled strips, or a centre region of a tiled page). A 10 GB single-strip plane previews from 63 MB of reads in 11 s.
• read_region(): a rectangle of a tiled page through tifffile's zarr store, fetching only the covering tiles. tiff_info(): shape, pages, tiling and pyramid levels from the IFDs alone.
• terafly_levels(), terafly_thumbnail(), first_images(), extensions_under(), manifest_size().
• HttpFile uses 256 KB blocks for metadata walks and one direct range request for any read of a block or more. Opening a 21.7 GB, 92-page BigTIFF went from 81 MB in 78 requests to 25 MB; reading a 1 GB page went from 108 s at 10 MB/s to 27 s at 38 MB/s.
• Latency-bound sampling reads (rows 20 MB apart, TeraFly blocks) run through a small thread pool; bulk transfers stay sequential.
• imagecodecs and pillow are now dependencies: fMOST blocks are LZW, and ImageJ Deflate stacks that libdeflate rejects with INSUFFICIENT_SPACE decode with Pillow.
• Unsupported-format errors name the tool for the extension found (glymur for .jp2, navis for .swc, nibabel, h5py, numpy for .dax).

0.1.0

First release.

• BilCatalog: the whole Brain Image Library in memory from BIL's daily inventory TSV (14,224 datasets on 2026-07-31), with case-insensitive structured filters, fulltext search joined to inventory rows, a light-sheet finder that unions the technique field with BIL's fulltext index (348 vs 808 datasets), a summary, and an optional pandas view.
• retrieve(), retrieve_many(), query(), fulltext(): typed wrappers over the metadata API, including the batched POST form of /retrieve.
• list_files(), manifest(), walk(), find(), find_zarr(), download(): lazy listing over the download server's nginx autoindex and BIL's per-dataset gzipped manifests. walk() never enters a zarr or N5 store (crawling one chunk directory at a time took 218 s on a real store before that guard existed).
• read_tiff(), read_stack(), slices(), thumbnail(), downsample(): single-slice TIFF reads over HTTP, sequential on purpose (eight parallel streams measured slower than one against BIL's server).
• HttpFile: a seekable read-only file over HTTP range requests with a block cache, so tifffile fetches only the pages requested.
• open_zarr(), zarr_levels(), zarr_thumbnail(): OME-Zarr stores opened lazily; pyramid levels are verified against the server listing because at least one BIL store declares a level it does not serve.
• Response cache on by default, 7-day expiry, image bytes never cached.
• CLI: summary, search, light-sheet, filter, info, files, thumbnail.