Skip to content

Switch vector data to Overture Maps by default - #3

Merged
wrignj08 merged 2 commits into
mainfrom
perf-easy-wins
Aug 11, 2026
Merged

Switch vector data to Overture Maps by default#3
wrignj08 merged 2 commits into
mainfrom
perf-easy-wins

Conversation

@wrignj08

Copy link
Copy Markdown
Contributor

Summary

Vector data now comes from Overture Maps instead of the Overpass API. Overpass is a live query service that routinely rate-limits or times out on dense urban bounding boxes; Overture serves static monthly GeoParquet releases from cloud storage. On the Sydney example scene, Overpass did not return within 10 minutes, while Overture returned water, road and building vectors in 7–13 s each.

Pass vector_source="osm" to restore the previous Overpass behaviour.

Changes

  • New parametersvector_source ("overture" default, "osm") and include_ocean (default True) on make_water_mask / make_water_mask_debug.
  • Vector cache bumped to v2 — adds source and ocean columns so entries from different sources cannot be served for one another. An existing v1 cache is ignored rather than migrated, since its rows carry no record of which source produced them.
  • A scene whose vector targets fail to build is now skipped, not exported without them. Such a mask is not obviously wrong on inspection, and its presence on disk made a later overwrite=False run skip the scene — one transient outage silently became a permanent result.
  • Landform filtering — Overture files cape, blowhole and shoal under its water theme. Shoals appear as polygons and would have marked land as water; these are now dropped, keeping the two sources aligned with OSM_water_tags.
  • Reliability — Overture fetches retry transient failures (3 attempts, 2s/4s backoff); empty-but-valid regions are told apart from fetch errors via STAC file coverage; Overpass rate-limiting is no longer invisible; a 200 response with unparseable JSON is no longer cached as "no features here".
  • Resource fixes — rasterio dataset handles leaked once or more per scene (target threads and export_to_disk) and are now closed in a finally after threads are joined.
  • Error surfacing — the osmnx version check ran inside the per-scene thread where its raise was lost; it now runs up front so the actionable ImportError is what the caller sees.
  • Dependencies — adds overturemaps>=1.0.0; raises the declared pyarrow floor to >=15.0.2 to match what overturemaps actually requires. osmnx is retained for the vector_source="osm" path.

Testing

  • 163 unit tests pass (up from 158); ruff, ruff format and mypy strict clean
  • 36 e2e tests pass against live Overture releases (2m41s), asserting fetched schema, subtype vocabulary and land-class filtering
  • Verified the staged tree in isolation imports and passes the full suite, i.e. a fresh checkout is complete

Follow-up

The conda-forge recipe needs five edits when the autotick bot opens its version PR — it bumps version/sha only and will not pick up the new dependency.

🤖 Generated with Claude Code

wrignj08 and others added 2 commits August 11, 2026 12:39
Overpass rate-limits and times out on dense urban bounding boxes;
Overture serves static monthly GeoParquet from cloud storage. Adds
vector_source ("overture" default, "osm" to restore Overpass) and
include_ocean, bumps the vector cache to v2 so entries from different
sources cannot be served for one another.

A scene whose vector targets fail to build is now skipped rather than
exported without them, since such a mask looks plausible and its
presence on disk makes a later overwrite=False run skip the scene.

Also closes rasterio dataset handles that leaked once per scene, and
raises the osmnx version check up front so its ImportError is not lost
inside a target thread.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3.8.15 has the same defect as 3.8.14: neither a cp314 wheel nor an
sdist, so there is nothing installable on 3.14 (3.8.13 ships both).
CI resolved to it and failed at install.

Scoped to 3.14 with a marker so other interpreters still track the
latest spaCy, and left as exclusions so a release that restores cp314
artifacts is picked up without another change here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@wrignj08
wrignj08 merged commit c65bdc7 into main Aug 11, 2026
10 checks passed
@wrignj08
wrignj08 deleted the perf-easy-wins branch August 19, 2026 06:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant