Skip to content

Describe the resample as a VRT instead of writing it out - #13

Open
wrignj08 wants to merge 4 commits into
perf-memoryfrom
vsimem-resample
Open

Describe the resample as a VRT instead of writing it out#13
wrignj08 wants to merge 4 commits into
perf-memoryfrom
vsimem-resample

Conversation

@wrignj08

Copy link
Copy Markdown
Contributor

Stacks on #11 — based on perf-memory, so retarget to main once that merges.

The problem

resample_input read every band at the new resolution and wrote them out as a GeoTIFF next to the output, so the pipeline could reopen that path and read band_order back. That's a full second copy of the scene on disk — including bands the caller never asks for — plus a second decode of the ones it does.

resample left on disk
4-band S2 tile → 20 m 362 MB
4-band S2 tile → 5 m 5.8 GB

Nothing downstream wanted a raster. Of the four consumers of that path, three — both build_targets threads and export_to_disk — read only crs, bounds and transform, and never touch a pixel. The file existed because a path was the only way to hand the resampled scene to all of them.

The change

A VRT keeps the path and drops the copy. resample_input wraps the source in a WarpedVRT on the target grid and serialises that description to /vsimem, returning its path — 3 KB of XML saying "I am a raster of this size on this grid; fetch my pixels from that GeoTIFF".

It opens like any other raster, so build_targets, water_inf_helpers and export_to_disk are unchanged apart from widening Path to str | Path. The resampled scene only ever exists as the array the pipeline reads.

Alternatives considered: MemoryFile (same API, but holds an uncompressed duplicate of the resampled raster in RAM — ~240 MB on the 20 m case, wrong direction for this branch), and passing a lightweight georeferencing object instead of a path (no duplicate, but a new type and the same signature churn, since a WarpedVRT has no path either).

Verification

Old vs new, same source, across down- and up-sampling:

res=   20  5490x5490   pixels identical: True   crs/transform/size/bounds: all True
res=   30  3660x3660   pixels identical: True   crs/transform/size/bounds: all True
res=    5  21960x21960 pixels identical: True   crs/transform/size/bounds: all True
res=  7.5  14640x14640 pixels identical: True   crs/transform/size/bounds: all True

Resample-and-read on a 4-band 5490×5490, end to end:

old: write GeoTIFF + reopen   0.62s   artefact 361.7 MB
new: /vsimem VRT + open       0.22s   artefact 0.0 MB

The grid arithmetic is carried over unchanged, and the VRT resamples with nearest — which is what a decimated rasterio read already did, so this is not a behaviour change. (It does make Resampling.average a one-word change if the downsampling aliasing is worth fixing later — deliberately not part of this.)

197 unit tests, 36 e2e, mypy strict and ruff all pass.

Notes for review

  • The pipeline's diff looks larger than it is: 141 lines, but 31 ignoring the reindent from wrapping the loop body in try/finally.
  • The scene path, the output name and the bands read now come from three places rather than all from the written file's name. The _resample_<res>m suffix is rebuilt explicitly, and input_image keeps naming the real scene in the logs.
  • /vsimem is process-global and nothing reclaims it, so the per-scene VRT is released in a finally. Tests cover release on both the success and TargetBuildError paths.
  • The exists() cache is gone. It skipped a resample that is now free to redo, and never checked the cached file still matched its input.
  • The skip-if-exists branch no longer writes a whole raster before discovering the output was already there.
  • The VRT records an absolute path to its source, so it is strictly a within-run object — which is why it lives in /vsimem behind a finally rather than next to the outputs.
  • Carries a merge of main for the e2e conftest fix (Let e2e tests resolve the live Overture release #12); without it the e2e suite can't run.

🤖 Generated with Claude Code

wrignj08 and others added 4 commits August 19, 2026 15:37
The pin_overture_release fixture is autouse with no scope guard, so it
replaced the resolved release with "test-release" for every test in the
suite - including the e2e tests whose entire purpose is to fetch from the
live release. Every Overture read in those tests 404'd on a release that
does not exist, failing all six TestOvertureLiveFetch tests plus the four
pipeline configurations that pass use_osm_water=True.

CI deselects e2e by default (addopts = -m 'not e2e'), so this only shows
up when the marker is run explicitly.

Skip the pin for e2e-marked tests. Unit tests stay independent of release
discovery; e2e tests go back to exercising it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Let e2e tests resolve the live Overture release
resample_input read every band of the scene at the new resolution and
wrote them out as a GeoTIFF next to the output, so the pipeline could
reopen that path and read band_order back. That is a full second copy of
the scene on disk - including the bands the caller never asks for - and a
second decode of the ones it does. Downsampling a 4-band Sentinel-2 tile
to 20 m left 362 MB behind; upsampling it to 5 m left 5.8 GB.

Nothing downstream actually wanted a raster. Of the four consumers of
that path, three - both build_targets threads and export_to_disk - read
only crs, bounds and transform, and never touch a pixel. The file existed
because a path was the only way to hand the resampled scene to all of
them.

A VRT keeps the path and drops the copy. resample_input now wraps the
source in a WarpedVRT on the target grid and serialises that description
to /vsimem, returning its path: 3 KB of XML saying "I am a raster of this
size on this grid; fetch my pixels from that GeoTIFF". It opens like any
other raster, so build_targets, water_inf_helpers and export_to_disk are
unchanged apart from widening Path to str | Path. The resampled scene
only ever exists as the array the pipeline reads.

The grid arithmetic is carried over unchanged, and the VRT resamples with
nearest, which is what a decimated rasterio read already did. Measured on
a 4-band 5490x5490 resample: pixels bitwise identical, crs, transform,
size and bounds identical, 0.62s -> 0.22s, 362 MB -> nothing.

The scene path, the output name and the bands read now come from three
places rather than all from the written file's name, so the pipeline
rebuilds the _resample_<res>m suffix itself and keeps input_image naming
the real scene in the logs. /vsimem is process-global and nothing
reclaims it, so the per-scene VRT is released in a finally.

Two things go with the file. The exists() cache is gone - it skipped a
resample that is now free to redo, and never checked the cached file
still matched its input. And the skip-if-exists branch no longer writes a
whole raster before discovering the output was already there.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant