Skip to content

Imagery-age poll: size the nightly batch per city so a full pass takes ~2 weeks (GSV metadata is free) #5402

Description

@jonfroehlich

The imagery-age poll (ImageryFreshnessService.pollImageryAges, #4384) processes street-imagery-poll.batch-size = 500 streets a night, three GSV Street View metadata calls per street. Prod on 2026-09-17: every night succeeds with 500 of 500 polled, 0 skipped, in every city checked — so the rotation works, it is just slow for big cities:

City Open streets Nights per full pass First pass started
Newberg 1,544 ~3 done
DC 14,037 ~28 2026-09-04 (8 nights in, 3,711 medians)
Seattle 23,995 ~48 2026-08-25

Chicago and Houston are larger still. Until a street has been polled once it has no median_newest_capture and can never be flagged, so for the first month or two after deploy a big city under-reports its re-audit backlog by design, and after that a newly re-driven street waits up to a pass length to be noticed.

The GSV metadata endpoint is free of charge (the header comment on pollImageryAges says so), Mapillary is free within rate limits, and Panoramax is public. 1,500 requests/night is far below anything that matters.

Proposal

  • Size the batch per city rather than a flat 500: e.g. ceil(open_streets / 14) so every city completes a pass in about two weeks, with a floor of 500 and a configurable ceiling (say 5,000 streets ≈ 15k requests) to keep Mapillary rate limits and run time bounded. Keep STREET_IMAGERY_POLL_BATCH_SIZE as an override.
  • Log the pass length (open streets ÷ batch) in PollResult.runDetails so the admin Imagery page can say "full pass every N nights" instead of the operator computing it.
  • Consider provider-specific ceilings (GSV high, Mapillary lower) since the cost profiles differ.

Interim: the env var already exists per stage, so a large city can be raised by hand today.

Related: #4704 (Mapillary dense-bbox miss), #4899 (bulk seed from streetscape-tracker), the rotation-key ticket filed alongside this one.

🤖 Generated with Claude Code — Fable 5.1, claude-fable-5-1

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions