Skip to content

Latest commit

 

History

History
191 lines (160 loc) · 15.4 KB

File metadata and controls

191 lines (160 loc) · 15.4 KB

AI subsystems in Project Sidewalk

Project Sidewalk's AI/CV work spans ~13 repositories accumulated since 2018, plus a hosted inference service and a HuggingFace org. This page maps what is in production, what each repo does, and which repos are historical. (Written July 2026; repo-by-repo audit in the tables below.)

The one-paragraph version

There are two fundamentally different AI tasks, reflecting our research finding that validating an existing human label is much easier than finding and labeling panoramas:

  1. AI as a validator/tagger (mature, ~55 cities): given an existing human label, hosted models judge whether it's correct and suggest tags. Validator models are trained in sidewalk-validator-ai, tagger models in sidewalk-tagger-ai; both are served by sidewalk-ai-api and orchestrated by a daily actor in this repo.
  2. AI as a labeler (new, piloting): a detection model (RampNet) scans every GSV panorama in a city and submits curb-ramp labels from scratch, via sidewalk-auto-labeler and this repo's /ai/submitLabelsOnPano.

Plus a third, smaller feature: AI Guidance — real-time auditing tips generated by Google Gemini at audit-task start (added Aug 2025).

Production subsystems

Subsystem A — AI validator / tagger

What it does: validates existing human labels (Agree/Disagree/Unsure) and suggests tags, for CurbRamp, NoCurbRamp, Obstacle, SurfaceProblem, Crosswalk.

Flow:

  1. GetAiValidationsActor (app/actor/GetAiValidationsActor.scala) runs nightly at the time app/actor/ScheduledJobs.scala gives it (staggered per city), selecting up to 800 labels/day via LabelTable.getLabelsToValidateWithAi (not assessed as its current type, GSV-only, prioritized). A label whose type was edited is assessed again right after the edit (AiService.reassessAfterTypeChange, same eligibility rules, #3671), with this sweep as the fallback.
  2. AiService.callAiApi (app/service/AiService.scala) POSTs {label_type, panorama_id, x, y, city} to https://sidewalk-ai-api.cs.washington.edu/process (code: sidewalk-ai-api — Dockerized GPU service, ≥ 9–10 GB VRAM, serving both model families from HuggingFace).
  3. The response (validation result + estimated accuracy + per-tag scores + model provenance) is stored in label_ai_assessment, along with the type it was about, which is what makes an assessment stale once the label is retyped (#3671) — the AI is asked about one type, so its answer only speaks to that type. If AI validations are enabled for the city and estimated accuracy ≥ ai-validation-min-accuracy (0.92 everywhere today), a real label_validation is submitted as the SidewalkAI user; below the threshold it downgrades to Unsure. HTTP 502 → label_ai_failure (permanently excluded).
  4. Surfaced in the Humans vs AI admin dashboard (/admin/humans-vs-ai) and the AI icon (public/js/common/aiLabelIndicator.js) across Gallery/Validate/LabelMap. The icon carries the "AI can make mistakes" tooltip everywhere except Validate's marker, where the label card the same hover opens shows the sentence instead (LabelCardView, #5359).

Models:

  • Validator (from sidewalk-validator-ai): DINOv2 fine-tuned per label type as a binary correct/incorrect classifier, trained on user-agreement data, with Depth Anything V2-derived depth-aware crops. Published to the HF collection project-sidewalk-validator-models. SLURM-based training (Hyak).
  • Tagger (from sidewalk-tagger-ai): DINOv2 / CLIP multi-label classifiers over 640×640 label-centered crops, 33 tag classes (ASSETS'24 paper). Per-tag ADD/REMOVE thresholds chosen by analyze_thresholds.py to maximize recall subject to precision ≥ 0.92 (Wilson lower bound ≥ 0.90) — deliberately the same 0.92 as ai-validation-min-accuracy.

Config: ai-enabled, sidewalk-ai-api-hostname (application.conf); ai-tag-suggestions-enabled, ai-validation-enabled, ai-validation-min-accuracy (cityparams.conf, per city).

DB: label_ai_assessment, label_ai_failure, AI user 51b0b927-3c8a-45b2-93de-bd878d1e5cf4 (role AI), mission type aiValidation (evolutions 281/282, 321/322). The AI user's per-schema rows — its user_stat row and one aiValidation mission per label type — are re-created at boot by AiSeedRowsRepair wherever a schema lacks them, since a city cloned from a donor or restored from an onboarding dump never ran 281 (#5349).

Subsystem B — AI-generated labels (RampNet + auto-labeler)

What it does: detects curb ramps in every GSV panorama inside a city polygon and submits them as new labels from the SidewalkAI user — no human in the loop until validation.

Flow:

  1. sidewalk-auto-labeler (main.py) scans coverage tiles for all pano IDs in a GeoJSON area, downloads each equirectangular pano (via streetlevel), runs projectsidewalk/rampnet-model (heatmap → peak_local_max, threshold 0.55 → normalized points), and writes one JSONL record per pano; send_to_ps.py converts to pixel coords and POSTs to POST /ai/submitLabelsOnPano (AiController.submitAiLabel).
  2. ExploreService.submitAiLabelData computes lat/lng + POV, creates an AI mission/audit task, inserts real label rows under the AI user, and records provenance in label_ai_info. The pano's copyright is reduced to the contributor's bare name on the way in (ImageryAttribution.normalizeCopyright): the labeler sends a whole attribution, © name / Mapillary (CC BY-SA 4.0), and the server composes the sign, provider and licence around the stored name itself, so storing the whole thing credited everything twice (#5360).
  3. Labels then enter the normal human-validation pipeline — which is also the feedback signal for improving the model (continual-improvement roadmap: sidewalk-auto-labeler/docs/design-review-2026-07.md). A lone AI vote leaves a label one vote short of consensus, so the crowd queue keeps serving it until a human concurs, and a label the humans lean against goes to Expert Validate's triage queue (docs/validation-queue.md).

City gate: submitAiLabel is gated by the per-city ai-label-submission-enabled flag in cityparams.conf (default false; unlisted cities reject submissions). Onboarding another city to AI labeling (e.g. Bend) means setting the flag to true for that city. Note that this only opens the gate — submission still requires a valid INTERNAL_API_KEY on that instance, which fails closed when the key is unset.

Model (from RampNet, ICCV'25 workshop paper): ConvNeXtV2-Base + heatmap head, input 2048×4096, trained on ~214k panoramas auto-labeled from open-government curb-ramp data (NYC, Portland, Bend); P=0.938 / R=0.935 on a 1k-pano manual gold set at the deployed threshold.

Subsystem C — AI Guidance (Gemini)

What it does: at audit-task start, sends the street's GSV images to Google Gemini (gemini-2.5-flash-lite) and shows the user two lines of context-specific auditing tips (e.g. what to look for on this street type in this city). Added by Mikey in Aug 2025 alongside the AI-label endpoint; model id updated June 2026.

Flow: AiGuidance.js (Explore) → POST /ai/analyzeSceneAiController.analyzeScene → Gemini API. Enabled only where GEMINI_API_KEY is configured. Independent of subsystems A/B — no PS-trained models, no DB tables, no validation loop. (Note: this is our only subsystem that ships GSV imagery to a third-party API; A and B keep imagery within UW-controlled infrastructure.)

AI project timeline — every AI/CV repo in the org

Chronological by repo creation. Dates are repo-created → last substantive commit. Status: ✅ Active · 🗄 Archived (read-only on GitHub; still public and citable — historical repos were archived in July 2026 to make the active set obvious).

Years Repo What it does / why it exists Paper Status
2017– sidewalk-panorama-tools GSV panorama download + label-crop scripts (DownloadRunner/CropRunner, adapted from @kotarohara's code) feeding the training repos. Downloader still in production use; "new version in the works". ✅ Active
2018 Sidewalk_CV First CV experiments: classical features (SIFT/HOG) + random forests/SVMs on label crops. Pre-deep-learning exploration. 🗄 Archived (2026-07)
2018–20 sidewalk-cv-assets19 First deep-learning work: ResNet on GSV crops, multimodal (image + geo features). Tackled both tasks — automatic validation of crowd labels and automatic labeling of panoramas — establishing that validation is the far easier problem. ASSETS'19 (Weld et al.) 🗄 Archived (2026-07)
2019 label-intersection-proximity Supporting tool: distance from a label to the nearest OSM intersection — used as a geo feature in the CV work. 🗄 Archived (2026-07)
2019 sidewalk-cv-tools First attempt to package CV labeling + validation as reusable functions for production use. Python 2.7; superseded by sidewalk-ai-api. 🗄 Archived (2026-07)
2019 cv-siamese-network Experiment: Siamese networks (+ SVM baseline) for sidewalk-feature detection. Exploratory dead end. 🗄 Archived (2026-07)
2021– label-latlng-estimation Fits and validates the estimator that places a label's real-world lat/lng from its position in the pano. Its 2026-08 refit is what production runs: a saturating-cotangent distance blend on the label's depression angle over a single camera height, calibrated against depth truth measured on current imagery (0.42 m median on held-out modern truth). Reported under reports/ and transcribed into PanoDataService.LatLngEstimation. ✅ Active
2021–23 sidewalk-cv-2021 Second-generation automatic label validation (better crop quality, multi-size crops, cross-city training), with labeling/segmentation experiments alongside (e.g. citysurfaces/ surface classification). Direct ancestor of sidewalk-validator-ai. ASSETS'22 (Duan et al.) 🗄 Archived (2026-07)
2023 BusStopCV Sibling crowd+AI project: real-time in-browser CV assistant for labeling bus-stop features (seating, shelter, signage, trash cans). ASSETS'23 (Kulkarni et al.) 🗄 Archived (2026-07)
2024– sidewalk-tagger-ai Trains the tagger models: DINOv2/CLIP multi-label classifiers, 33 tag classes over label crops; per-tag deployment thresholds at precision ≥ 0.92. ASSETS'24 (Liu, Wu, et al.) ✅ Active
2024– sidewalk-ai-api The serving layer: Dockerized GPU API (/process) hosting the validator + tagger models from HuggingFace; called daily by SidewalkWebpage for ~55 cities. ✅ Active
2025 gsv-location-extraction-analysis Completed one-off study: GSV's fromContainerPixelToLatLng vs. the linear regression PS deployed at the time for label lat/lng (verdict: the regression was slightly better; both are beaten by the cotangent blend label-latlng-estimation fit in 2026). Conclusions and a decision-record banner in its README. 🗄 Archived (2026-08)
2025– RampNet Trains the curb-ramp detector + auto-generates its 214k-pano dataset from open-gov data. ⚠ Tag an ICCV-paper-state release before changing (issue #2). ICCV'25 wksp (O'Meara et al.) ✅ Active
2025– sidewalk-auto-labeler Deploys RampNet at city scale: finds every pano in a polygon, runs detection, submits AI labels to PS. ✅ Active
2025– sidewalk-validator-ai Trains the validator models: DINOv2 binary correct/incorrect per label type, Depth-Anything-V2 depth-aware crops, agreement-based training labels. ✅ Active
SidewalkWebpage Production server: AI orchestration (daily validation actor), ingestion endpoints, DB tables, Humans-vs-AI dashboard, Gemini guidance. ✅ Active

(SidewalkWebpageMTurk, the 2017 MTurk experiment fork, is 🗄 already archived.)

Lineage: both of today's threads trace back to sidewalk-cv-assets19 (2019), which attempted validation and auto-labeling with one ResNet approach (after Sidewalk_CV's 2018 classical-CV exploration):

  • Validation/tagging thread (the easier task — production first): cv-assets19 → sidewalk-cv-2021 (ASSETS'22) → validator-ai + tagger-ai + ai-api (in production, ~55 cities).
  • Labeling/detection thread (the harder task — human-level only in 2025): cv-assets19's auto-labeling experiments → cv-2021's segmentation side experiments → RampNet (ICCV'25) + auto-labeler (piloting).

sidewalk-panorama-tools has supplied imagery to both threads throughout; sidewalk-cv-tools (2019) was an early attempt to package the ResNet-era work for production, superseded by sidewalk-ai-api.

Artifact flow

sidewalk-validator-ai ─┐ (HF: project-sidewalk-validator-models)
sidewalk-tagger-ai ────┤ (HF: sidewalk-tagger-ai-models)          RampNet
                       ▼                                             │ (HF: rampnet-model)
                sidewalk-ai-api  /process                            ▼
                       ▲                                   sidewalk-auto-labeler
                       │ daily actor (≤800 labels/city)              │ /ai/submitLabelsOnPano
                       ▼                                             ▼
   SidewalkWebpage: label_ai_assessment + label_validation   SidewalkWebpage: label + label_ai_info
                       └────────── human validations close the loop ─────────┘

Known gaps (July 2026)

  • Each pipeline fetches GSV imagery its own way (streetlevel, sidewalk-panorama-tools, RampNet's bespoke scraping, the ai-api's fetch/cache). Google changed these undocumented endpoints in June 2026 and broke consumers independently (see sidewalk-auto-labeler/docs/design-review-2026-07.md §3) — worth consolidating on one maintained fetch path.
  • Model provenance (model_id, training date) is hardcoded in the auto-labeler rather than read from the HF model config.
  • RampNet must get a tagged ICCV'25-state release before any code changes land (RampNet issue #2).