Merge pull request #3 from DPIRD-DMA/perf-easy-wins

Switch vector data to Overture Maps by default

Author: wrignj08

.gitignore

@@ -27,3 +27,4 @@ OWM_cache/*
 data/S2A_56HLH_20231011_0_L2A_RGBNIR.tif
 examples/water_vectors_cache/gdfs/*.parquet
 examples/water_vectors_cache/*.db
+*.tif.aux.xml

CHANGELOG.md

@@ -1,5 +1,37 @@
 # Changelog
 
+## [0.6.0] - Aug 11, 2026
+
+### Changed
+- **Vector data now comes from [Overture Maps](https://overturemaps.org) by default** instead of the Overpass API. Overpass is a live query service that routinely rate-limits or times out on dense urban bounding boxes; Overture serves static monthly GeoParquet releases from cloud storage. On the Sydney example scene, Overpass did not return within 10 minutes, while Overture returned water, road and building vectors in 7–13 s each. Pass `vector_source="osm"` to restore the previous Overpass behaviour.
+- Vector cache database bumped to `geodataframes_v2.db`, adding `source` and `ocean` columns so entries from different vector sources cannot be served for one another. An existing v1 cache is ignored rather than migrated, since its rows carry no record of which source produced them.
+
+### Added
+- `vector_source` parameter on `make_water_mask` and `make_water_mask_debug`, accepting `"overture"` (default) or `"osm"`.
+- `include_ocean` parameter (default True). Overture's `ocean` water features cover everything seaward of the OSM coastline — signal the previous OSM tag set had no equivalent for, since it did not query `natural=coastline`. Set to False where coastline/tide offsets cause false positives.
+- `overturemaps>=1.0.0` dependency. `osmnx` is retained for the `vector_source="osm"` path.
+- The spaCy exclusion now covers 3.8.15 as well as 3.8.14, and is scoped to Python 3.14 with an environment marker. Both releases publish neither a cp314 wheel nor an sdist, so there is nothing installable on 3.14; 3.8.13 ships both. Other interpreters are no longer held back by the exclusion, and a later release that restores cp314 artifacts will be picked up without another change here.
+- The declared `pyarrow` floor is raised from 10.0.0 to 15.0.2. `overturemaps` requires that version, so 10.0.0 was never actually installable alongside it; the old floor only misdescribed what the package supports.
+
+### Fixed
+- Overture fetches now retry transient failures (3 attempts, 2s then 4s backoff). Overture is served from S3, where a throttled range read or dropped connection is common and the client does not retry on its own; previously a single blip cost the scene all of its vector targets. A persistent failure now raises rather than returning an empty frame, so a network problem fails the build instead of being read as "no water here".
+- A bounding box that genuinely intersects no Overture files (open ocean, Antarctica) is no longer treated as a fetch error. `record_batch_reader` returns `None` for both the empty and the failed case, so the STAC file coverage is checked to tell them apart; if that check is unavailable the `None` is treated as an error, which is the conservative reading.
+- Argument validation in `build_targets` moved outside its catch-all `except`, so an invalid `vector_source` raises instead of degrading to a full run with no vector targets and one line in the log.
+- A target-building thread that dies without reporting no longer hangs the run. `build_targets` returns its result through a queue, and the reader blocked forever if the thread raised before it could put anything there; the wait now surfaces a `RuntimeError` instead.
+- A failed vector build is logged with its traceback (`logging.exception`) rather than just the exception message.
+- **A scene whose vector targets fail to build is now skipped instead of exported without them.** Previously the run continued and wrote a mask derived from NDWI and the model alone — not obviously wrong on inspection, and its presence on disk made a later run with `overwrite=False` skip the scene, so one transient outage silently became a permanent result. The scene is now logged at ERROR and left unwritten, and is omitted from the returned output paths; the rest of the batch continues, and a re-run reprocesses it. `build_targets` signals this with a new `TargetBuildError` (put on its queue when threaded, since a raise from a thread would be lost), which is distinct from the `None` it returns when there was nothing to build.
+- A skipped scene no longer leaves its other target thread running. Positive and negative targets are built in two threads; propagating the first failure without joining the second left it fetching on into the next scene, so a widespread outage piled orphaned threads up across a batch. Both are now joined before either failure propagates.
+- Overpass rate-limiting is no longer invisible. osmnx pauses for its advertised slot time and retries 429/504 after 55s, but reports that through its own logger, which writes nowhere by default — an overloaded Overpass looked like a multi-minute hang. Those messages now route to the standard `logging` module (without turning on osmnx's log files).
+- An Overpass response that returns 200 with a body that will not parse as JSON is no longer treated as "no features in this area". osmnx raises `InsufficientResponseError` for both that and a genuinely empty query; the parse failure is now told apart by its chained `JSONDecodeError` and propagates, so a fetch failure cannot be written to the vector cache as an empty result.
+- Landforms filed under Overture's water theme are no longer rasterized as positive water targets. Overture's `WaterClass` enum includes `cape` (a headland), `blowhole` (a coastal rock formation) and `shoal` (a routinely exposed sandbank), all carried under `subtype="physical"`. Capes and blowholes are usually points, which `combine_vector_targets` already drops, but shoals do appear as polygons — a Cape Cod bounding box returns two. These would have marked land as water. `natural=cape`/`natural=shoal` were never in `OSM_water_tags`, so dropping them also keeps the two sources aligned. Genuine water in the same subtype (`bay`, `strait`, `sound`) is retained.
+- Rasterio dataset handles no longer leak, one or more per scene. The two target threads each read from a dataset opened by the integration layer, and neither that layer nor `export_to_disk` ever closed what it opened, so a long batch could exhaust the process's file descriptors. The handles are now closed in a `finally`, after both threads are joined — closing a dataset a thread is still reading from would take that thread down with it — so they are released whether the scene succeeds, is skipped, or fails during inference.
+- An osmnx too old for the Overpass path is now reported as the `ImportError` that says so. The version check ran inside the per-scene target thread, where the raise was lost and the caller saw only "the vector target thread exited without a result", with the real message going to stderr through the threading excepthook rather than the logging module. `make_water_mask`/`make_water_mask_debug` now check up front, before any scene is opened, alongside the existing `vector_source` validation.
+
+### Notes
+- Road targets use Overture `segment` features with `subtype="road"`, matching what `highway=*` returned from OSM; rail and waterway segments are excluded.
+- Overture building footprints include machine-learning-derived data beyond OSM, so negative building targets have broader coverage than before.
+- The Overture path is covered by live network tests marked `e2e`, which assert the fetched schema, subtype vocabulary and land-class filtering against real releases. Run them with `pytest -m e2e`.
+
 ## [0.5.0] - Jun 3, 2026
 
 ### Changed

README.md

@@ -86,7 +86,9 @@ water_mask_path = make_water_mask(
 
 ## Usage tips
 
--   OWM requires an active internet connection to function properly, as it needs to download OpenStreetMap (OSM) data.
+-   OWM requires an active internet connection to function properly, as it needs to download vector data.
+-   Vector data comes from [Overture Maps](https://overturemaps.org) by default. If you would rather query OpenStreetMap live through the Overpass API, set `vector_source="osm"`. Overture serves static monthly GeoParquet releases from cloud storage, so it avoids the rate limits and timeouts Overpass returns on large or dense bounding boxes. The underlying data is largely the same — Overture's water and road layers are derived from OSM — though its building footprints add machine-learning-derived data beyond OSM. Note that Overture files a few landforms (`cape`, `blowhole`, `shoal`) under its water theme; OWM filters these out so they are not treated as water.
+-   If a scene's vector data cannot be fetched, that scene is skipped rather than processed without it — a mask built without its vector targets looks plausible but is quietly worse. Overture fetches retry transient failures first (3 attempts, 2s then 4s apart). A skipped scene is logged at ERROR, is left out of the returned list of output paths, and has no file written, so re-running the same call reprocesses it while the rest of the batch is untouched.
 -   Hardware acceleration is strongly recommended:
     -   NVIDIA GPU
     -   Apple Silicon Mac
@@ -142,9 +144,13 @@ This matters because OWM optimises its detection thresholds both **locally** (pe
 
 -    `use_cache`: Whether to cache vector data processing results. Defaults to True
 
--    `use_osm_building`: Whether to use OpenStreetMap building data to reduce false positives. Defaults to True
+-    `use_osm_building`: Whether to use building data to reduce false positives. Defaults to True
 
--    `use_osm_roads`: Whether to use OpenStreetMap road data to reduce false positives. Defaults to True
+-    `use_osm_roads`: Whether to use road data to reduce false positives. Defaults to True
+
+-    `vector_source`: Where water, road and building vectors come from — `"overture"` (Overture Maps GeoParquet) or `"osm"` (OpenStreetMap via the Overpass API). Defaults to "overture"
+
+-    `include_ocean`: Whether Overture ocean polygons count as positive water targets. These cover everything seaward of the OSM coastline, which the OSM tag set does not provide. Set to False if coastline/tide offsets cause false positives on your scenes. Only applies when `vector_source="overture"`. Defaults to True
 
 -    `cache_dir`: Directory for storing cached vector data. Defaults to "OWM_cache" in current directory
 

omniwatermask/overture_source.py

@@ -0,0 +1,212 @@
+"""Fetch vector features from Overture Maps.
+
+Overture is an alternative to the Overpass API that ``osmnx`` queries. Overpass
+is a live query service that routinely rate-limits or times out on dense urban
+bounding boxes; Overture serves static, monthly-released GeoParquet from cloud
+storage, so a fetch is a bounded range-read rather than a server-side query.
+
+The underlying data is largely the same: Overture's ``base`` and
+``transportation`` themes are derived from OpenStreetMap. The ``buildings``
+theme adds machine-learning-derived footprints on top of OSM.
+"""
+
+import logging
+import time
+from typing import Any, Optional
+
+import geopandas as gpd
+from overturemaps import core
+
+# Overture type names for each kind of feature OWM targets, chosen to match the
+# OSM tag sets in target_builders: OSM_water_tags -> water, OSM_roads_tags
+# (highway=*) -> road segments, OSM_buildings_tags -> buildings.
+OVERTURE_TYPES: dict[str, str] = {
+    "water": "water",
+    "roads": "segment",
+    "buildings": "building",
+}
+
+# The transportation theme also carries "rail" and "water" segments. Only
+# "road" is kept so negative targets match what highway=* returned from OSM.
+OVERTURE_SUBTYPES: dict[str, frozenset[str]] = {
+    "roads": frozenset({"road"}),
+}
+
+# Everything seaward of the OSM coastline is a single "ocean" water feature.
+# OSM_water_tags has no coastline equivalent, so this is signal OWM did not
+# previously receive - but it is pinned to the coastline vector regardless of
+# tide, hence the opt-out in build_targets.
+OCEAN_SUBTYPE = "ocean"
+
+# Overture files a few landforms under the water theme's "physical" subtype:
+# a cape is a headland, a blowhole is a coastal rock formation, and a shoal is
+# a sandbank that is routinely exposed. Rasterizing them as positive water
+# targets marks land as water. Capes and blowholes are usually points (which
+# combine_vector_targets drops anyway), but shoals do appear as polygons.
+# OSM_water_tags never selected these - it queried natural=water/strait, not
+# natural=cape/shoal - so dropping them also keeps the two sources aligned.
+OVERTURE_LAND_CLASSES = frozenset({"cape", "blowhole", "shoal"})
+
+# Only these are needed downstream: combine_vector_targets uses geometry alone,
+# and the subtype/class pair is kept for filtering, cache inspection and
+# debugging.
+# Dropping the rest keeps the concat in combine_vector_targets and the parquet
+# cache free of Overture's deeply nested attribute columns.
+KEEP_COLUMNS = ["subtype", "class", "geometry"]
+
+# Overture is served from S3, so failures are transient far more often than not:
+# a throttled range read, a dropped connection, a timeout on a dense bbox. The
+# client itself does not retry, so a single blip would otherwise cost the scene
+# all of its vector targets. Delays are 2s, 4s - short enough not to stall a
+# batch run, long enough to clear a throttle.
+MAX_ATTEMPTS = 3
+BACKOFF_SECONDS = 2.0
+
+
+def get_overture_features(
+    gdf_bounds_4326: gpd.GeoDataFrame,
+    kind: str,
+    include_ocean: bool = True,
+) -> gpd.GeoDataFrame:
+    """Download Overture features of ``kind`` within a bounding box.
+
+    ``kind`` is one of ``water``, ``roads`` or ``buildings``. ``include_ocean``
+    only applies to ``water``.
+    """
+    if kind not in OVERTURE_TYPES:
+        raise ValueError(
+            f"Unknown Overture feature kind: {kind!r}. "
+            f"Expected one of {sorted(OVERTURE_TYPES)}."
+        )
+    overture_type = OVERTURE_TYPES[kind]
+    bounds = gdf_bounds_4326.total_bounds
+    bbox = (
+        float(bounds[0]),
+        float(bounds[1]),
+        float(bounds[2]),
+        float(bounds[3]),
+    )
+
+    features = _fetch_with_retry(overture_type=overture_type, bbox=bbox, kind=kind)
+
+    if features.empty:
+        logging.info(f"No {kind} features found within bbox: {bbox}")
+        return gpd.GeoDataFrame()
+
+    features = _filter_features(features, kind=kind, include_ocean=include_ocean)
+    if features.empty:
+        logging.info(f"No {kind} features left after filtering")
+        return gpd.GeoDataFrame()
+
+    features = features[[c for c in KEEP_COLUMNS if c in features.columns]]
+    features = features.set_crs("EPSG:4326", allow_override=True)
+    features = gpd.clip(features, gdf_bounds_4326)
+    return gpd.GeoDataFrame(features)
+
+
+def _fetch_once(
+    overture_type: str,
+    bbox: tuple[float, float, float, float],
+) -> Optional[gpd.GeoDataFrame]:
+    """Run a single fetch attempt, or return None if Overture gave no reader.
+
+    ``record_batch_reader`` streams lazily, so a failed range read surfaces out
+    of ``from_arrow`` rather than out of the reader call. Both live here so the
+    retry above covers the whole request.
+    """
+    # stac=True lets the client resolve the release and target only the
+    # relevant files via the STAC catalogue. Without it the whole dataset
+    # listing is opened, which measured ~5x slower per request.
+    reader = core.record_batch_reader(overture_type, bbox=bbox, stac=True)
+    if reader is None:
+        return None
+    return gpd.GeoDataFrame.from_arrow(reader)
+
+
+def _bbox_has_no_files(
+    overture_type: str,
+    bbox: tuple[float, float, float, float],
+) -> bool:
+    """Report whether STAC finds no files at all intersecting ``bbox``.
+
+    ``record_batch_reader`` returns None both when the bbox is genuinely empty
+    (STAC matched zero files, e.g. open ocean or Antarctica) and when opening
+    the dataset failed. Only the second is an error, and the two are
+    indistinguishable from the None alone. ``_prepare_query`` is what returns
+    None for the empty case, so ask it directly. It is private, so an
+    overturemaps release that drops or renames it makes this report False and
+    the caller treats the None as an error - the conservative reading.
+    """
+    prepare_query = getattr(core, "_prepare_query", None)
+    if prepare_query is None:
+        return False
+    try:
+        return prepare_query(overture_type, bbox=bbox, stac=True) is None
+    except Exception as e:
+        logging.debug(f"Could not check STAC file coverage for {overture_type}: {e}")
+        return False
+
+
+def _fetch_with_retry(
+    overture_type: str,
+    bbox: tuple[float, float, float, float],
+    kind: str,
+) -> gpd.GeoDataFrame:
+    """Fetch ``overture_type`` over ``bbox``, retrying transient failures.
+
+    Returns an empty frame when the bbox genuinely holds no features. Raises
+    once the attempts are exhausted, so a persistent failure fails the build
+    rather than passing an empty frame off as "no water here".
+    """
+    last_error: Optional[Exception] = None
+
+    for attempt in range(1, MAX_ATTEMPTS + 1):
+        try:
+            features = _fetch_once(overture_type, bbox)
+        except Exception as e:
+            last_error = e
+        else:
+            if features is not None:
+                return features
+            if _bbox_has_no_files(overture_type, bbox):
+                logging.info(f"Overture has no {kind} files intersecting bbox: {bbox}")
+                return gpd.GeoDataFrame()
+            last_error = RuntimeError(
+                f"Overture returned no reader for type {overture_type!r} "
+                f"within bbox: {bbox}"
+            )
+
+        if attempt < MAX_ATTEMPTS:
+            delay = BACKOFF_SECONDS * 2 ** (attempt - 1)
+            logging.warning(
+                f"Overture {kind} fetch failed (attempt {attempt}/{MAX_ATTEMPTS}): "
+                f"{last_error}. Retrying in {delay:.0f}s"
+            )
+            time.sleep(delay)
+
+    raise RuntimeError(
+        f"Overture {kind} fetch failed after {MAX_ATTEMPTS} attempts for bbox: "
+        f"{bbox}. This usually indicates a network or cloud storage error."
+    ) from last_error
+
+
+def _filter_features(
+    features: gpd.GeoDataFrame,
+    kind: str,
+    include_ocean: bool,
+) -> gpd.GeoDataFrame:
+    """Restrict features to those matching the equivalent OSM tag sets."""
+    if kind == "water" and "class" in features.columns:
+        features = features[~features["class"].isin(OVERTURE_LAND_CLASSES)]
+
+    if "subtype" not in features.columns:
+        return gpd.GeoDataFrame(features)
+
+    keep: Any = OVERTURE_SUBTYPES.get(kind)
+    if keep is not None:
+        features = features[features["subtype"].isin(keep)]
+
+    if kind == "water" and not include_ocean:
+        features = features[features["subtype"] != OCEAN_SUBTYPE]
+
+    return gpd.GeoDataFrame(features)

omniwatermask/raster_helpers.py

@@ -54,18 +54,18 @@ def export_to_disk(
     embedded inside the GeoTIFF (``GDAL_TIFF_INTERNAL_MASK``) rather than written
     as a separate ``.tif.msk`` sidecar, so it travels with the file.
     """
-    src = rio.open(source_path)
-    profile = {
-        "dtype": array.dtype,
-        "count": array.shape[0],
-        "compress": "lzw",
-        "nodata": None,
-        "driver": "GTiff",
-        "height": array.shape[1],
-        "width": array.shape[2],
-        "transform": src.transform,
-        "crs": src.crs,
-    }
+    with rio.open(source_path) as src:
+        profile = {
+            "dtype": array.dtype,
+            "count": array.shape[0],
+            "compress": "lzw",
+            "nodata": None,
+            "driver": "GTiff",
+            "height": array.shape[1],
+            "width": array.shape[2],
+            "transform": src.transform,
+            "crs": src.crs,
+        }
 
     with rio.Env(GDAL_TIFF_INTERNAL_MASK=True):
         with rio.open(export_path, "w", **profile) as dst: