0.2.0: fixes and bounded previews from a full-archive stress test

Fixed: percent-encoding of paths with #, spaces, non-ASCII (39 datasets
were unreachable); decoded listing names; one- and two-word dataset ids;
a temp-file race on concurrent cold catalog loads; unbounded crawls in
find_zarr/thumbnail on million-file trees (20+ minutes); manifest size
guard (710 MB gzipped on a 4.7 M-file dataset); strip-sampled preview
shape; clear errors for renamed and landing-zone paths; tiff_info on
files without a buildable series.

Added: preview_plane (bounded reads for any file size: pyramid level,
row sampling by offset, strip sampling, tiled centre region), read_region,
tiff_info, terafly_levels/terafly_thumbnail for fMOST trees,
first_images, extensions_under, manifest_size. HttpFile: 256 KB blocks
for metadata, direct range requests for large reads (21.7 GB OME-TIFF
open 81 MB to 25 MB; 1 GB page 108 s to 27 s). Parallel pool for
latency-bound sampling only. imagecodecs and pillow become dependencies.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GDpV8346GxohHCft1jrrGT

Author: 2 people

CHANGELOG.md

@@ -1,5 +1,61 @@
 # Changelog
 
+## 0.2.0
+
+Everything here came out of running the package against the whole archive
+(random samples across techniques, the largest trees and files, odd paths,
+sixteen threads on a cold cache) on 2026-09-08.
+
+Fixed:
+
+- Paths with `#`, spaces or non-ASCII characters are percent-encoded
+  (`encode_path`, `encode_url`, idempotent). An unencoded `#` was a URL
+  fragment, so 39 inventory datasets could not be listed or read.
+- Directory-listing names are decoded (`Virus_tracing-B1-#6`), URLs keep
+  the server's encoding.
+- Two-word and one-word dataset ids (`act-nod`, 861 of them) were rejected
+  by the id pattern.
+- Six concurrent cold catalog loads crashed on a shared temp file name;
+  each writer now uses its own.
+- `walk()`/`thumbnail()` sat for over twenty minutes on a 1.6 million file
+  fMOST tree. `find_zarr()` now has a directory budget, `thumbnail()`
+  uses a bounded first-folder search (`first_images`), and TeraFly trees
+  are recognised and previewed from their coarsest resolution folder.
+- `manifest()` refuses manifests over 64 MB gzipped (a 4.7 million file
+  dataset's is 710 MB) with `ManifestTooLargeError`; `walk()` and
+  `find_zarr()` fall back to crawling. `manifest_size()` is the cheap
+  check.
+- A strip-sampled preview came out one pixel wide (decoded strip shape was
+  misread).
+- A dataset renamed on the server since the inventory was built, and a
+  dataset still under the landing zone (`/bil/lz/`), each get a clear
+  `BilNotFoundError` instead of a bare 404.
+- `tiff_info()` no longer fails on files where tifffile cannot build a
+  series ("incompatible keyframe"); pages are reported anyway.
+
+Added:
+
+- `preview_plane()`: bounded preview of a TIFF of any size (coarsest
+  pyramid level, or every k-th row by offset for uncompressed planes, or
+  sampled strips, or a centre region of a tiled page). A 10 GB single-strip
+  plane previews from 63 MB of reads in 11 s.
+- `read_region()`: a rectangle of a tiled page through tifffile's zarr
+  store, fetching only the covering tiles. `tiff_info()`: shape, pages,
+  tiling and pyramid levels from the IFDs alone.
+- `terafly_levels()`, `terafly_thumbnail()`, `first_images()`,
+  `extensions_under()`, `manifest_size()`.
+- `HttpFile` uses 256 KB blocks for metadata walks and one direct range
+  request for any read of a block or more. Opening a 21.7 GB, 92-page
+  BigTIFF went from 81 MB in 78 requests to 25 MB; reading a 1 GB page
+  went from 108 s at 10 MB/s to 27 s at 38 MB/s.
+- Latency-bound sampling reads (rows 20 MB apart, TeraFly blocks) run
+  through a small thread pool; bulk transfers stay sequential.
+- `imagecodecs` and `pillow` are now dependencies: fMOST blocks are LZW,
+  and ImageJ Deflate stacks that libdeflate rejects with
+  INSUFFICIENT_SPACE decode with Pillow.
+- Unsupported-format errors name the tool for the extension found (glymur
+  for `.jp2`, navis for `.swc`, nibabel, h5py, numpy for `.dax`).
+
 ## 0.1.0
 
 First release.

README.md

@@ -36,14 +36,18 @@ The Brain Image Library is the BRAIN Initiative's repository for whole-brain mic
 
 BIL's own [`brainimagelibrary`](https://pypi.org/project/brainimagelibrary/) package (py-brain-sdk, 0.0.23, GPL-3.0) wraps the metadata API and downloads datasets, with resumable transfers and citation lookups. Its source (read, not assumed, on 2026-09-08) contains no image reading at all: the only route from a BIL id to pixels is `DatasetInventory.download()`, which fetches the files to disk. For a 27 GB light-sheet stack that is the whole stack, to look at one slice.
 
+This package was then run against the archive rather than a handful of friendly datasets: random samples across every technique, the largest trees (4.7 million files), the largest single files (a 21.7 GB OME-TIFF, a 10 GB single-strip plane, 199 GB TIFFs), paths with `#` and spaces and non-ASCII characters, two-word and one-word dataset ids, stale inventory paths, landing-zone paths, and sixteen threads on a cold cache. What that found is fixed and covered by tests; what it cannot read yet is stated below.
+
 This package is the other half. It reads BIL in place:
 
 - **A typed, in-memory catalog of the whole archive** from BIL's daily inventory TSV, with structured filters and a light-sheet finder that unions the inventory's technique field with BIL's fulltext index. The technique field alone says light sheet for 348 datasets; the union finds 808. The rest are tagged `other` and only say light sheet in their abstract or instrument record.
 - **Single-slice TIFF reads** straight from the download server. The dominant BIL layout is one TIFF per z-plane, about 16 MB each; one slice is one request.
 - **OME-Zarr stores** opened lazily through zarr's HTTP store, with pyramid levels checked against what the server actually has (one BIL store declares eight levels and serves seven).
 - **Thumbnails that read as little as possible**: the middle slice of a TIFF stack, or the coarsest level of a zarr pyramid.
-- **A seekable HTTP file object** (`HttpFile`) so tifffile fetches only the pages you ask for from a large multi-page TIFF.
-- **BIL's per-dataset manifest** (path, size, MD5, URL for every file) as one gzipped GET, so a deep tree lists in one request.
+- **Bounded previews of files of any size.** `preview_plane()` reads the coarsest pyramid level, or every k-th row by offset from an uncompressed plane, or a sample of strips, or a centre region of a tiled page. A 10 GB single-strip MERFISH mosaic previews from 63 MB of reads; `asarray()` would fetch all 10 GB.
+- **fMOST TeraFly trees** (`RES_<x>x<y>x<z>_/` folders, 985 datasets) previewed from the coarsest resolution folder, stitched. The full-resolution folder of one such brain holds 1.6 million files and is never listed.
+- **A seekable HTTP file object** (`HttpFile`) so tifffile fetches only the IFDs, pages, tiles or regions you ask for from a large TIFF, with `read_region()` for a rectangle of a tiled page.
+- **BIL's per-dataset manifest** (path, size, MD5, URL for every file) as one gzipped GET, so a deep tree lists in one request. Guarded by size: a 4.7 million file dataset's manifest is 710 MB gzipped, and `walk()` crawls instead of downloading that.
 
 Measured on 2026-09-08 from a residential connection, against the live archive:
 
@@ -52,12 +56,19 @@ Measured on 2026-09-08 from a residential connection, against the live archive:
 | Load the full catalog (14,224 datasets) | 1.0 s cold, 0.3 s from the on-disk copy |
 | Light-sheet datasets found | 348 by technique field, 808 with fulltext union |
 | Read one 16 MB light-sheet slice (4501 x 3828 uint16) | 0.40 s, 45 MB/s single stream |
-| Thumbnail of a 1,923-slice, 27 GB stack | 0.25 s to 0.56 s, one file read |
+| Thumbnail of a 1,923-slice, 27 GB stack | 0.3 s, one file read |
 | Thumbnail of a 50 GB OME-Zarr store (848 x 6300 x 9600) | 0.65 s, coarsest level only |
-| Eight parallel streams | 19 MB/s aggregate, slower than one stream, so reads are sequential |
+| Thumbnail of a 4.5 TB fMOST TeraFly brain | 4 to 5 s, coarsest resolution folder, about 25 small block reads |
+| Preview of a 10 GB single-strip plane (84289 x 61974 uint16) | 11 s, 512 rows fetched by offset |
+| Open a 21.7 GB, 92-page BigTIFF OME-TIFF and read a 512 x 512 region | 7 s to walk the IFDs, 1.3 s for the region |
+| Read one full 1 GB page of that file | 27 s at 38 MB/s in 6 range requests |
+| Eight parallel streams on 16 MB slices | 32 MB/s aggregate against 39 MB/s for one stream |
 | Manifest for a 1,923-file dataset | 352 KB gzipped, one request |
+| 1,000 full metadata records | 1.4 s via batched POST |
+
+Bulk transfers are sequential because parallel streams measured slower. Sampling reads (rows 20 MB apart, TeraFly blocks) are latency-bound and go through a small pool: 512 sequential range requests took 68 s for 63 MB, eight workers take a few seconds.
 
-Dependencies are `requests`, `numpy` and `tifffile`. zarr and pandas are extras.
+Dependencies are `requests`, `numpy`, `tifffile`, `imagecodecs` (LZW and Deflate strips are common in BIL TIFFs) and `pillow` (a second decoder for ImageJ Deflate stacks that libdeflate rejects). zarr and pandas are extras.
 
 ## Data license
 
@@ -113,7 +124,7 @@ bil.find_zarr("ace-cup-eel")             # ['https://download.brainimagelibrary.
 bil.download(entry, "out/")              # the one function that writes image bytes to disk
 ```
 
-Any of these accept a BIL id, a `Dataset`, a `/bil/data/...` path from the metadata, or a download-server URL.
+Any of these accept a BIL id, a `Dataset`, a `/bil/data/...` path from the metadata, or a download-server URL. Paths are percent-encoded for you: 39 inventory paths carry spaces, `#` or non-ASCII characters, and an unencoded `#` is a URL fragment. Two things the inventory gets wrong are reported rather than hidden: a directory renamed on the server since the inventory was built (the error names the parent to list) and a dataset still under BIL's landing zone, which is not public.
 
 ## Images
 
@@ -125,7 +136,22 @@ bil.thumbnail("ace-bin-run", max_size=512)                          # middle sli
 bil.thumbnail("ace-bin-run", index=100, channel="ch02")
 ```
 
-Formats this package does not decode raise `UnsupportedFormatError` naming what does: JPEG 2000 (`.jp2`, 5,787 datasets, mostly STPT and fMOST sections; use `download()` then glymur), Imaris (`.ims`, h5py), NIfTI. Reading `.jp2` in place is the obvious next addition; the format has resolution levels built in, so a thumbnail should not need the whole file.
+### Previews of large files
+
+```python
+bil.tiff_info(entry)                     # pages, shape, dtype, tiled, pyramid levels, from the IFDs only
+bil.preview_plane(entry, max_size=512)   # bounded read whatever the file size, see the table above
+bil.read_region(entry, rows=(16000, 18048), cols=(7000, 9048), page=0)   # tiles covering a rectangle, [zarr] extra
+bil.terafly_levels("ace-cap-cop")        # [((793, 1268, 350), url), ...] coarsest first, for fMOST trees
+bil.terafly_thumbnail("ace-cap-cop")     # one stitched plane through the middle of the coarsest level
+bil.first_images("ace-owl-cot")          # the first folder of TIFFs by bounded descent, never a crawl
+```
+
+`thumbnail()` tries these in order: a TeraFly tree, an OME-Zarr store, then the first folder of TIFFs. It never lists a whole tree; the unbounded version sat on a 1.6 million file fMOST dataset for over twenty minutes before the bound existed.
+
+### What it does not read
+
+Formats this package does not decode raise `UnsupportedFormatError` naming what does. Across the archive's techniques the big one is JPEG 2000 (`.jp2`, 5,787 datasets: most STPT sections, viral tracing and enhancer-labelling brains): `download()` then glymur. Also Imaris `.ims` (HDF5, h5py), NIfTI (nibabel), MERFISH `.dax` frames (raw uint16, numpy), SWC morphologies (navis), and tables (pandas). Reading `.jp2` in place is the obvious next addition; the format has resolution levels built in, so a preview should not need the whole file.
 
 ### OME-Zarr
 
@@ -162,7 +188,7 @@ $ scigantic-bil thumbnail ace-bin-run slice.png --size 512
 
 ## Testing
 
-Every test runs live against BIL, no mocks, the same philosophy as the rest of the scigantic-* packages. The suite takes about 15 seconds. CI runs Python 3.10 through 3.14 plus `mypy --strict`.
+Every test runs live against BIL, no mocks, the same philosophy as the rest of the scigantic-* packages. The suite takes one to two minutes depending on the server and includes the stress cases above: the 21 GB OME-TIFF, the 10 GB plane, the `#` path, the 4.7 million file manifest guard, the TeraFly tree, the Pillow fallback, six threads loading the catalog cold. CI runs Python 3.10 through 3.14 plus `mypy --strict`.
 
 ## License
 

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "scigantic-bil"
-version = "0.1.0"
+version = "0.2.0"
 description = "Search the Brain Image Library (14,000+ BICCN/BICAN brain microscopy datasets, 6 PB) and read light-sheet, fMOST and STPT volumes over HTTP without downloading them: a typed catalog over BIL's daily inventory and metadata API, lazy file listing, single-slice TIFF reads, thumbnails, and OME-Zarr access."
 readme = "README.md"
 requires-python = ">=3.10"
@@ -31,6 +31,15 @@ dependencies = [
   "requests>=2.28,<3",
   "numpy>=1.24,<3",
   "tifffile>=2023.7,<2027",
+  # tifffile decodes LZW, Deflate and JPEG-in-TIFF only through imagecodecs.
+  # BIL's fMOST TeraFly blocks are LZW (ace-cap-cop, verified 2026-09-08);
+  # without this, read_tiff() on them raises "requires the 'imagecodecs'
+  # package". Binary wheels exist for every CI target.
+  "imagecodecs>=2023.9",
+  # Fallback decoder for TIFFs tifffile rejects (ImageJ Deflate stacks whose
+  # strips overrun their declared rows: libdeflate says INSUFFICIENT_SPACE,
+  # Pillow decodes them). Also renders the CLI's thumbnail PNGs.
+  "pillow>=10,<13",
 ]
 
 [project.optional-dependencies]
@@ -48,7 +57,7 @@ zarr = [
   "aiohttp>=3.9,<4; python_version >= '3.11'",
 ]
 pandas = ["pandas>=1.5,<4"]
-dev = ["pytest>=7", "mypy>=1.10", "types-requests", "pandas-stubs", "pillow>=10"]
+dev = ["pytest>=7", "mypy>=1.10", "types-requests", "pandas-stubs"]
 
 [project.urls]
 Homepage = "https://scigantic.com"

src/scigantic_bil/__init__.py

@@ -15,26 +15,56 @@
 from .api import fulltext, query, retrieve, retrieve_many
 from .cache import cache_dir, clear as clear_cache, disable_cache, enable_cache, is_cache_enabled
 from .catalog import BilCatalog, available_inventory_dates
-from .files import download, find, find_zarr, is_store_dir, list_files, manifest, resolve_url, walk
+from .files import (
+    MANIFEST_MAX_BYTES,
+    ManifestTooLargeError,
+    download,
+    extensions_under,
+    find,
+    find_zarr,
+    first_images,
+    is_store_dir,
+    list_files,
+    manifest,
+    manifest_size,
+    resolve_url,
+    walk,
+)
 from .images import (
     HttpFile,
     UnsupportedFormatError,
     downsample,
     open_zarr,
+    preview_plane,
+    read_region,
     read_stack,
     read_tiff,
     slices,
+    terafly_levels,
+    terafly_thumbnail,
     thumbnail,
+    tiff_info,
     zarr_levels,
     zarr_thumbnail,
 )
-from .models import Contributor, Dataset, DatasetDetail, FileEntry, Publication, dataset_url
+from .models import (
+    Contributor,
+    Dataset,
+    DatasetDetail,
+    FileEntry,
+    Publication,
+    dataset_url,
+    encode_path,
+    encode_url,
+)
 
 __all__ = [
     "__version__",
     "BilError",
     "BilNotFoundError",
     "UnsupportedFormatError",
+    "ManifestTooLargeError",
+    "MANIFEST_MAX_BYTES",
     "BilCatalog",
     "available_inventory_dates",
     "Dataset",
@@ -43,22 +73,32 @@
     "Publication",
     "FileEntry",
     "dataset_url",
+    "encode_path",
+    "encode_url",
     "query",
     "fulltext",
     "retrieve",
     "retrieve_many",
     "list_files",
     "manifest",
+    "manifest_size",
     "walk",
     "find",
     "find_zarr",
+    "first_images",
     "is_store_dir",
     "download",
+    "extensions_under",
     "resolve_url",
     "read_tiff",
+    "read_region",
+    "preview_plane",
+    "tiff_info",
     "read_stack",
     "slices",
     "thumbnail",
+    "terafly_levels",
+    "terafly_thumbnail",
     "downsample",
     "open_zarr",
     "zarr_levels",

src/scigantic_bil/_version.py

@@ -1 +1 @@
-__version__ = "0.1.0"
+__version__ = "0.2.0"

src/scigantic_bil/cache.py

@@ -120,7 +120,7 @@ def clear() -> int:
     Returns how many files were removed."""
     d = _resolve_dir()
     n = 0
-    for pattern in ("*.json", "*.part", "inventory-*.tsv"):
+    for pattern in ("*.json", "*.part", "inventory-*.tsv", "inventory-*.tsv.*.part"):
         for f in d.glob(pattern):
             f.unlink()
             n += 1

src/scigantic_bil/catalog.py

@@ -19,7 +19,9 @@
 
 import csv
 import io
+import os
 import re
+import uuid
 from collections import Counter
 from dataclasses import dataclass, field
 from typing import TYPE_CHECKING, Any, Iterable, Iterator
@@ -57,9 +59,15 @@ def _inventory_text(date: str, refresh: bool = False) -> str:
     resp = send("GET", f"{_INVENTORY_DIR}{date}.tsv", timeout=300.0)
     text = resp.content.decode("utf-8")
     if cache.is_cache_enabled():
-        tmp = path.with_suffix(".tsv.part")
+        # Unique per writer: two threads loading the catalog cold at the same
+        # time (a notebook with a thread pool, a test session) must not share
+        # a temp path, or the second os.replace() fails with FileNotFoundError
+        # once the first has moved it. Measured: 2 of 6 concurrent cold loads
+        # crashed before this. Both write an identical immutable snapshot, so
+        # last-writer-wins is fine.
+        tmp = path.with_suffix(f".tsv.{uuid.uuid4().hex}.part")
         tmp.write_text(text, encoding="utf-8")
-        tmp.replace(path)
+        os.replace(tmp, path)
     return text