Describe the resample as a VRT instead of writing it out - #13
Open
wrignj08 wants to merge 4 commits into
Open
Conversation
The pin_overture_release fixture is autouse with no scope guard, so it replaced the resolved release with "test-release" for every test in the suite - including the e2e tests whose entire purpose is to fetch from the live release. Every Overture read in those tests 404'd on a release that does not exist, failing all six TestOvertureLiveFetch tests plus the four pipeline configurations that pass use_osm_water=True. CI deselects e2e by default (addopts = -m 'not e2e'), so this only shows up when the marker is run explicitly. Skip the pin for e2e-marked tests. Unit tests stay independent of release discovery; e2e tests go back to exercising it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Let e2e tests resolve the live Overture release
resample_input read every band of the scene at the new resolution and wrote them out as a GeoTIFF next to the output, so the pipeline could reopen that path and read band_order back. That is a full second copy of the scene on disk - including the bands the caller never asks for - and a second decode of the ones it does. Downsampling a 4-band Sentinel-2 tile to 20 m left 362 MB behind; upsampling it to 5 m left 5.8 GB. Nothing downstream actually wanted a raster. Of the four consumers of that path, three - both build_targets threads and export_to_disk - read only crs, bounds and transform, and never touch a pixel. The file existed because a path was the only way to hand the resampled scene to all of them. A VRT keeps the path and drops the copy. resample_input now wraps the source in a WarpedVRT on the target grid and serialises that description to /vsimem, returning its path: 3 KB of XML saying "I am a raster of this size on this grid; fetch my pixels from that GeoTIFF". It opens like any other raster, so build_targets, water_inf_helpers and export_to_disk are unchanged apart from widening Path to str | Path. The resampled scene only ever exists as the array the pipeline reads. The grid arithmetic is carried over unchanged, and the VRT resamples with nearest, which is what a decimated rasterio read already did. Measured on a 4-band 5490x5490 resample: pixels bitwise identical, crs, transform, size and bounds identical, 0.62s -> 0.22s, 362 MB -> nothing. The scene path, the output name and the bands read now come from three places rather than all from the written file's name, so the pipeline rebuilds the _resample_<res>m suffix itself and keeps input_image naming the real scene in the logs. /vsimem is process-global and nothing reclaims it, so the per-scene VRT is released in a finally. Two things go with the file. The exists() cache is gone - it skipped a resample that is now free to redo, and never checked the cached file still matched its input. And the skip-if-exists branch no longer writes a whole raster before discovering the output was already there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacks on #11 — based on
perf-memory, so retarget tomainonce that merges.The problem
resample_inputread every band at the new resolution and wrote them out as a GeoTIFF next to the output, so the pipeline could reopen that path and readband_orderback. That's a full second copy of the scene on disk — including bands the caller never asks for — plus a second decode of the ones it does.Nothing downstream wanted a raster. Of the four consumers of that path, three — both
build_targetsthreads andexport_to_disk— read onlycrs,boundsandtransform, and never touch a pixel. The file existed because a path was the only way to hand the resampled scene to all of them.The change
A VRT keeps the path and drops the copy.
resample_inputwraps the source in aWarpedVRTon the target grid and serialises that description to/vsimem, returning its path — 3 KB of XML saying "I am a raster of this size on this grid; fetch my pixels from that GeoTIFF".It opens like any other raster, so
build_targets,water_inf_helpersandexport_to_diskare unchanged apart from wideningPathtostr | Path. The resampled scene only ever exists as the array the pipeline reads.Alternatives considered:
MemoryFile(same API, but holds an uncompressed duplicate of the resampled raster in RAM — ~240 MB on the 20 m case, wrong direction for this branch), and passing a lightweight georeferencing object instead of a path (no duplicate, but a new type and the same signature churn, since aWarpedVRThas no path either).Verification
Old vs new, same source, across down- and up-sampling:
Resample-and-read on a 4-band 5490×5490, end to end:
The grid arithmetic is carried over unchanged, and the VRT resamples with
nearest— which is what a decimated rasterio read already did, so this is not a behaviour change. (It does makeResampling.averagea one-word change if the downsampling aliasing is worth fixing later — deliberately not part of this.)197 unit tests, 36 e2e, mypy strict and ruff all pass.
Notes for review
try/finally._resample_<res>msuffix is rebuilt explicitly, andinput_imagekeeps naming the real scene in the logs./vsimemis process-global and nothing reclaims it, so the per-scene VRT is released in afinally. Tests cover release on both the success andTargetBuildErrorpaths.exists()cache is gone. It skipped a resample that is now free to redo, and never checked the cached file still matched its input./vsimembehind afinallyrather than next to the outputs.mainfor the e2e conftest fix (Let e2e tests resolve the live Overture release #12); without it the e2e suite can't run.🤖 Generated with Claude Code