Project Sidewalk's AI/CV work spans ~13 repositories accumulated since 2018, plus a hosted inference service and a HuggingFace org. This page maps what is in production, what each repo does, and which repos are historical. (Written July 2026; repo-by-repo audit in the tables below.)
There are two fundamentally different AI tasks, reflecting our research finding that validating an existing human label is much easier than finding and labeling panoramas:
- AI as a validator/tagger (mature, ~55 cities): given an existing human label, hosted
models judge whether it's correct and suggest tags. Validator models are trained in
sidewalk-validator-ai, tagger models insidewalk-tagger-ai; both are served bysidewalk-ai-apiand orchestrated by a daily actor in this repo. - AI as a labeler (new, piloting): a detection model (RampNet) scans every GSV panorama
in a city and submits curb-ramp labels from scratch, via
sidewalk-auto-labelerand this repo's/ai/submitLabelsOnPano.
Plus a third, smaller feature: AI Guidance — real-time auditing tips generated by Google Gemini at audit-task start (added Aug 2025).
What it does: validates existing human labels (Agree/Disagree/Unsure) and suggests tags, for CurbRamp, NoCurbRamp, Obstacle, SurfaceProblem, Crosswalk.
Flow:
GetAiValidationsActor(app/actor/GetAiValidationsActor.scala) runs nightly at the timeapp/actor/ScheduledJobs.scalagives it (staggered per city), selecting up to 800 labels/day viaLabelTable.getLabelsToValidateWithAi(not assessed as its current type, GSV-only, prioritized). A label whose type was edited is assessed again right after the edit (AiService.reassessAfterTypeChange, same eligibility rules, #3671), with this sweep as the fallback.AiService.callAiApi(app/service/AiService.scala) POSTs{label_type, panorama_id, x, y, city}tohttps://sidewalk-ai-api.cs.washington.edu/process(code:sidewalk-ai-api— Dockerized GPU service, ≥ 9–10 GB VRAM, serving both model families from HuggingFace).- The response (validation result + estimated accuracy + per-tag scores + model provenance)
is stored in
label_ai_assessment, along with the type it was about, which is what makes an assessment stale once the label is retyped (#3671) — the AI is asked about one type, so its answer only speaks to that type. If AI validations are enabled for the city and estimated accuracy ≥ai-validation-min-accuracy(0.92 everywhere today), a reallabel_validationis submitted as theSidewalkAIuser; below the threshold it downgrades to Unsure. HTTP 502 →label_ai_failure(permanently excluded). - Surfaced in the Humans vs AI admin dashboard (
/admin/humans-vs-ai) and the AI icon (public/js/common/aiLabelIndicator.js) across Gallery/Validate/LabelMap. The icon carries the "AI can make mistakes" tooltip everywhere except Validate's marker, where the label card the same hover opens shows the sentence instead (LabelCardView, #5359).
Models:
- Validator (from
sidewalk-validator-ai): DINOv2 fine-tuned per label type as a binary correct/incorrect classifier, trained on user-agreement data, with Depth Anything V2-derived depth-aware crops. Published to the HF collectionproject-sidewalk-validator-models. SLURM-based training (Hyak). - Tagger (from
sidewalk-tagger-ai): DINOv2 / CLIP multi-label classifiers over 640×640 label-centered crops, 33 tag classes (ASSETS'24 paper). Per-tag ADD/REMOVE thresholds chosen byanalyze_thresholds.pyto maximize recall subject to precision ≥ 0.92 (Wilson lower bound ≥ 0.90) — deliberately the same 0.92 asai-validation-min-accuracy.
Config: ai-enabled, sidewalk-ai-api-hostname (application.conf);
ai-tag-suggestions-enabled, ai-validation-enabled, ai-validation-min-accuracy
(cityparams.conf, per city).
DB: label_ai_assessment, label_ai_failure, AI user
51b0b927-3c8a-45b2-93de-bd878d1e5cf4 (role AI), mission type aiValidation
(evolutions 281/282, 321/322). The AI user's per-schema rows — its user_stat row and
one aiValidation mission per label type — are re-created at boot by
AiSeedRowsRepair wherever a schema lacks them, since a city cloned from a donor or
restored from an onboarding dump never ran 281 (#5349).
What it does: detects curb ramps in every GSV panorama inside a city polygon and
submits them as new labels from the SidewalkAI user — no human in the loop until
validation.
Flow:
sidewalk-auto-labeler(main.py) scans coverage tiles for all pano IDs in a GeoJSON area, downloads each equirectangular pano (viastreetlevel), runsprojectsidewalk/rampnet-model(heatmap →peak_local_max, threshold 0.55 → normalized points), and writes one JSONL record per pano;send_to_ps.pyconverts to pixel coords and POSTs toPOST /ai/submitLabelsOnPano(AiController.submitAiLabel).ExploreService.submitAiLabelDatacomputes lat/lng + POV, creates an AI mission/audit task, inserts reallabelrows under the AI user, and records provenance inlabel_ai_info. The pano'scopyrightis reduced to the contributor's bare name on the way in (ImageryAttribution.normalizeCopyright): the labeler sends a whole attribution,© name / Mapillary (CC BY-SA 4.0), and the server composes the sign, provider and licence around the stored name itself, so storing the whole thing credited everything twice (#5360).- Labels then enter the normal human-validation pipeline — which is also the feedback signal
for improving the model (continual-improvement roadmap:
sidewalk-auto-labeler/docs/design-review-2026-07.md). A lone AI vote leaves a label one vote short of consensus, so the crowd queue keeps serving it until a human concurs, and a label the humans lean against goes to Expert Validate's triage queue (docs/validation-queue.md).
City gate: submitAiLabel is gated by the per-city ai-label-submission-enabled flag in
cityparams.conf (default false; unlisted cities reject submissions). Onboarding another
city to AI labeling (e.g. Bend) means setting the flag to true for that city. Note that this
only opens the gate — submission still requires a valid INTERNAL_API_KEY on that instance,
which fails closed when the key is unset.
Model (from RampNet, ICCV'25 workshop
paper): ConvNeXtV2-Base + heatmap head, input 2048×4096, trained on ~214k panoramas
auto-labeled from open-government curb-ramp data (NYC, Portland, Bend); P=0.938 / R=0.935 on
a 1k-pano manual gold set at the deployed threshold.
What it does: at audit-task start, sends the street's GSV images to Google Gemini
(gemini-2.5-flash-lite) and shows the user two lines of context-specific auditing tips
(e.g. what to look for on this street type in this city). Added by Mikey in Aug 2025
alongside the AI-label endpoint; model id updated June 2026.
Flow: AiGuidance.js (Explore) → POST /ai/analyzeScene → AiController.analyzeScene
→ Gemini API. Enabled only where GEMINI_API_KEY is configured. Independent of subsystems
A/B — no PS-trained models, no DB tables, no validation loop. (Note: this is our only
subsystem that ships GSV imagery to a third-party API; A and B keep imagery within
UW-controlled infrastructure.)
Chronological by repo creation. Dates are repo-created → last substantive commit. Status: ✅ Active · 🗄 Archived (read-only on GitHub; still public and citable — historical repos were archived in July 2026 to make the active set obvious).
| Years | Repo | What it does / why it exists | Paper | Status |
|---|---|---|---|---|
| 2017– | sidewalk-panorama-tools |
GSV panorama download + label-crop scripts (DownloadRunner/CropRunner, adapted from @kotarohara's code) feeding the training repos. Downloader still in production use; "new version in the works". |
— | ✅ Active |
| 2018 | Sidewalk_CV |
First CV experiments: classical features (SIFT/HOG) + random forests/SVMs on label crops. Pre-deep-learning exploration. | — | 🗄 Archived (2026-07) |
| 2018–20 | sidewalk-cv-assets19 |
First deep-learning work: ResNet on GSV crops, multimodal (image + geo features). Tackled both tasks — automatic validation of crowd labels and automatic labeling of panoramas — establishing that validation is the far easier problem. | ASSETS'19 (Weld et al.) | 🗄 Archived (2026-07) |
| 2019 | label-intersection-proximity |
Supporting tool: distance from a label to the nearest OSM intersection — used as a geo feature in the CV work. | — | 🗄 Archived (2026-07) |
| 2019 | sidewalk-cv-tools |
First attempt to package CV labeling + validation as reusable functions for production use. Python 2.7; superseded by sidewalk-ai-api. |
— | 🗄 Archived (2026-07) |
| 2019 | cv-siamese-network |
Experiment: Siamese networks (+ SVM baseline) for sidewalk-feature detection. Exploratory dead end. | — | 🗄 Archived (2026-07) |
| 2021– | label-latlng-estimation |
Fits and validates the estimator that places a label's real-world lat/lng from its position in the pano. Its 2026-08 refit is what production runs: a saturating-cotangent distance blend on the label's depression angle over a single camera height, calibrated against depth truth measured on current imagery (0.42 m median on held-out modern truth). Reported under reports/ and transcribed into PanoDataService.LatLngEstimation. |
— | ✅ Active |
| 2021–23 | sidewalk-cv-2021 |
Second-generation automatic label validation (better crop quality, multi-size crops, cross-city training), with labeling/segmentation experiments alongside (e.g. citysurfaces/ surface classification). Direct ancestor of sidewalk-validator-ai. |
ASSETS'22 (Duan et al.) | 🗄 Archived (2026-07) |
| 2023 | BusStopCV |
Sibling crowd+AI project: real-time in-browser CV assistant for labeling bus-stop features (seating, shelter, signage, trash cans). | ASSETS'23 (Kulkarni et al.) | 🗄 Archived (2026-07) |
| 2024– | sidewalk-tagger-ai |
Trains the tagger models: DINOv2/CLIP multi-label classifiers, 33 tag classes over label crops; per-tag deployment thresholds at precision ≥ 0.92. | ASSETS'24 (Liu, Wu, et al.) | ✅ Active |
| 2024– | sidewalk-ai-api |
The serving layer: Dockerized GPU API (/process) hosting the validator + tagger models from HuggingFace; called daily by SidewalkWebpage for ~55 cities. |
— | ✅ Active |
| 2025 | gsv-location-extraction-analysis |
Completed one-off study: GSV's fromContainerPixelToLatLng vs. the linear regression PS deployed at the time for label lat/lng (verdict: the regression was slightly better; both are beaten by the cotangent blend label-latlng-estimation fit in 2026). Conclusions and a decision-record banner in its README. |
— | 🗄 Archived (2026-08) |
| 2025– | RampNet |
Trains the curb-ramp detector + auto-generates its 214k-pano dataset from open-gov data. ⚠ Tag an ICCV-paper-state release before changing (issue #2). | ICCV'25 wksp (O'Meara et al.) | ✅ Active |
| 2025– | sidewalk-auto-labeler |
Deploys RampNet at city scale: finds every pano in a polygon, runs detection, submits AI labels to PS. | — | ✅ Active |
| 2025– | sidewalk-validator-ai |
Trains the validator models: DINOv2 binary correct/incorrect per label type, Depth-Anything-V2 depth-aware crops, agreement-based training labels. | — | ✅ Active |
| — | SidewalkWebpage |
Production server: AI orchestration (daily validation actor), ingestion endpoints, DB tables, Humans-vs-AI dashboard, Gemini guidance. | — | ✅ Active |
(SidewalkWebpageMTurk, the 2017 MTurk experiment fork, is 🗄 already archived.)
Lineage: both of today's threads trace back to sidewalk-cv-assets19 (2019), which attempted validation and auto-labeling with one ResNet approach (after Sidewalk_CV's 2018 classical-CV exploration):
- Validation/tagging thread (the easier task — production first): cv-assets19 → sidewalk-cv-2021 (ASSETS'22) → validator-ai + tagger-ai + ai-api (in production, ~55 cities).
- Labeling/detection thread (the harder task — human-level only in 2025): cv-assets19's auto-labeling experiments → cv-2021's segmentation side experiments → RampNet (ICCV'25) + auto-labeler (piloting).
sidewalk-panorama-tools has supplied imagery to both threads throughout;
sidewalk-cv-tools (2019) was an early attempt to package the ResNet-era work for
production, superseded by sidewalk-ai-api.
sidewalk-validator-ai ─┐ (HF: project-sidewalk-validator-models)
sidewalk-tagger-ai ────┤ (HF: sidewalk-tagger-ai-models) RampNet
▼ │ (HF: rampnet-model)
sidewalk-ai-api /process ▼
▲ sidewalk-auto-labeler
│ daily actor (≤800 labels/city) │ /ai/submitLabelsOnPano
▼ ▼
SidewalkWebpage: label_ai_assessment + label_validation SidewalkWebpage: label + label_ai_info
└────────── human validations close the loop ─────────┘
- Each pipeline fetches GSV imagery its own way (
streetlevel,sidewalk-panorama-tools, RampNet's bespoke scraping, the ai-api's fetch/cache). Google changed these undocumented endpoints in June 2026 and broke consumers independently (seesidewalk-auto-labeler/docs/design-review-2026-07.md§3) — worth consolidating on one maintained fetch path. - Model provenance (
model_id, training date) is hardcoded in the auto-labeler rather than read from the HF model config. - RampNet must get a tagged ICCV'25-state release before any code changes land (RampNet issue #2).