Skip to content

dev-environment.md

Latest commit

 

History

History
588 lines (454 loc) · 34.9 KB

dev-environment.md

File metadata and controls

588 lines (454 loc) · 34.9 KB
Raw

Development Environment Setup

This guide gets a local copy of Project Sidewalk running on your machine. Everything runs in Docker, so you do not install Scala, Node, or Postgres directly — you only need Docker, Git, and a terminal.

New contributor? Read CONTRIBUTING.md first for the branch/PR workflow and coding standards. For a deeper tour of the architecture, see docs/architecture.md.

Stuck? Jump to Troubleshooting or Getting help.

What you'll need

  • Docker (Docker Desktop on macOS/Windows, Docker Engine + Compose on Linux) — installed and running.
  • Git and terminal/shell access.
  • From a maintainer (the app will not start without these): a docker-compose.override.yml with local secrets and one or more database dumps to seed Postgres. Email the lead engineer, Mikey (saugstad@cs.washington.edu), to request them.
  • API keys — the override file from a maintainer includes our keys. If you're outside the team, you'll need to create your own Google Maps and Mapbox keys and add them to the override file.

The dev setup is geared toward team members. If you're outside the team and want to stand up a server for a city we don't support, start with Onboarding a city — the tooling builds the streets and regions from open data and fills a schema; the server side is still on you.

Platform setup

Install Docker and Git for your platform, then clone the repo. The workflow after that is identical everywhere.

git clone https://github.com/ProjectSidewalk/SidewalkWebpage.git
cd SidewalkWebpage

Linux (Ubuntu)

  1. Install Docker. Consider rootless Docker — a bit more setup now, but smoother development later.
  2. Install Docker Compose (bundled on Mac/Windows, separate on Linux).
  3. Clone the repo (above).

macOS

  1. Install Docker Desktop — pick Apple Chip for M-series Macs or Intel Chip for older models.
  2. Clone the repo (above).

Windows (WSL2)

We recommend and only support the WSL2 path — it runs a real Linux kernel in a lightweight VM, giving faster compiles and better Docker support than legacy WSL.

  1. Install Docker Desktop. When prompted, select Use WSL 2 (not Hyper-V).
  2. Install WSL2 with the default Ubuntu distro: open PowerShell as administrator and run wsl --install. (Pin Ubuntu to your taskbar for easy access.)
  3. Update WSL: wsl --update.
  4. In Docker Desktop → Settings → General, check Use the WSL 2 based engine. Then under Resources → WSL Integration, enable integration with your default distro and check Ubuntu.
  5. From inside the Ubuntu (Linux) shell — not /mnt/c — clone the repo into your Linux home (e.g. ~/projects/). Running from the Linux filesystem is required for acceptable performance.

You'll need make inside Ubuntu: sudo apt install make.

Moving files (e.g. database dumps) into the Linux VM
  • File Explorer: in the left sidebar open Linux → Ubuntu → home → <username> → SidewalkWebpage and drag files in. (Right-click the folder → "Pin to Quick access" to find it easily later.) Copied files often get a companion :Zone.Identifier file — it's safe to delete those.
  • Command line: your Windows drives are mounted under /mnt (e.g. /mnt/c), so you can cp /mnt/c/path/to/dump ~/SidewalkWebpage/db/.
Starting / shutting down WSL2 + Docker

WSL and Docker use significant memory in the background. When you're not working on Project Sidewalk:

  • Shut down: close anything using Docker/WSL, quit Docker Desktop from the tray, then run wsl --shutdown.
  • Start back up: run wsl -d Ubuntu (or just open your IDE in WSL), then launch Docker Desktop.

First-time setup

Make sure Docker is running (you'll see the whale icon in your tray; you can set Docker to start on login).

  1. Add the secrets file. Place the docker-compose.override.yml from your maintainer in the repo root, then edit it for the city you want to run:

    • SIDEWALK_CITY_ID — the city to run (e.g. seattle-wa); see the City IDs table.
    • DATABASE_USER — that city's database user, replacing the default sidewalk (e.g. sidewalk_seattle).
    • platform — leave it commented. Every host builds the web image for its own architecture as of #5069, so Apple Silicon is native without it. Uncomment it only if a native build fails, to force an emulated linux/amd64 build.

  2. Stage the database dumps. Put the dump files from your maintainer in the db/ directory and rename them to the exact names the import scripts expect:

    • City data dump → <database_user>-dump (e.g. sidewalk_seattle-dump).
    • Users dump → sidewalk_users-dump (same name regardless of city).

  3. Build and start the containers, dropping into a shell in the web container:

    make dev

    make dev starts the db container and then opens an interactive shell in the web container. The first build downloads images, spins up containers, and initializes (but does not yet populate) the database — expect ~5–30+ minutes depending on your connection. Success ends with a root@<container-id>:/home# prompt.

  4. Import users and data from a second terminal on your host (outside the web-container shell):

    make import-users replace=1         # load sidewalk_users-dump (the login schema)
make import-dump db=<database_user> # load <database_user>-dump; db= defaults to "sidewalk"

    Both restore from a binary dump and show a live elapsed-time clock — the users dump is ~1 GB, so it runs for a couple of minutes (the restore is parallelized to keep that short); a city dump varies with its size. replace=1 is for this first import only: a fresh DB has no accounts of its own to keep, and merging all ~6M accounts one by one takes about 20 minutes. Later imports merge (see Switching / adding another city). Read the output carefully — if it errors, don't continue; check Troubleshooting and ask. (A schema "public" already exists notice is the one error you can safely ignore.) For what each script does and the full set of DB lifecycle/maintenance targets, see db/scripts/README.md.

  5. Start the app from inside the web-container shell opened by make dev:

    npm start

    npm start runs Grunt (JS/CSS concatenation + watch) in the background, then sbt ~ run for continuous recompile. The first compile takes 5+ minutes; later ones are seconds. Use npm run debug if you want a JVM debug port attached. It's ready when you see Listening for HTTP on .../9000.

  6. Open the app: http://localhost:9000 (or 127.0.0.1:9000). The first load is slow while Play applies evolutions and compiles on demand.

Daily workflow

After a restart you don't repeat the import — just:

cd SidewalkWebpage
make dev          # start db + open the web-container shell
npm start         # (inside that shell) build assets + run the app

Then visit http://localhost:9000. To stop everything: make docker-stop.

make dev also keeps node_modules in step with package-lock.json, reinstalling only when the two have diverged. See npm dependencies.

Other handy targets:

Command What it does
make docker-up Start all services detached (no shell). Useful for db only: the web container exits at once, its image command being jshell, which reads EOF without a TTY.
make npm-sync Reinstall node_modules from package-lock.json if they've diverged.
make ssh target=web Open a shell in a running container (target=web or target=db).

Switching / adding another city

Each city is a separate database. To switch:

  1. Put the new dump in db/, renamed to <database_user>-dump (see the City IDs table).
  2. If that dump is newer than your users dump, get a users dump at least as new, rename it sidewalk_users-dump, and run make import-users (ask a maintainer if unsure — the creation date is in the original filename). It merges: accounts you're missing are added, and the ones you have, including local test accounts your other cities point at, are kept, so the cities you already imported keep working. make import-users replace=1 wipes the login schema and restores the dump from scratch instead. That also drops the parts of every city that depend on it (foreign keys, a survey column, a view), so after it, re-import every other city you have.
  3. make import-dump db=<database_user> (from the host, outside the Docker shell).
  4. Update DATABASE_USER and SIDEWALK_CITY_ID in docker-compose.override.yml to match.
  5. make dev again.

To switch back and forth later: exit the Docker shell, change the two override values, and re-run make dev.

City IDs

The SIDEWALK_CITY_ID and DATABASE_USER must correspond. In the repo, conf/cityparams.conf is the source of truth; the snapshot below is a convenience copy (it may lag as new cities are added).

City ID Database User City ID Database User
seattle-wa sidewalk_seattle taichung-tw sidewalk_taichung
columbus-oh sidewalk_columbus cliffside-park-nj sidewalk_cliffside_park
cdmx sidewalk_cdmx blackhawk-hills-il sidewalk_blackhawk_hills
spgg sidewalk_spgg columbia-sc sidewalk_columbia
pittsburgh-pa sidewalk_pittsburgh west-chester-pa sidewalk_west_chester
newberg-or sidewalk_newberg danville-il sidewalk_danville
washington-dc sidewalk detroit-mi sidewalk_detroit
chicago-il sidewalk_chicago hackensack-nj sidewalk_hackensack
amsterdam sidewalk_amsterdam clifton-nj sidewalk_clifton
la-piedad sidewalk_la_piedad maywood-nj sidewalk_maywood
la-piedad-old sidewalk_la_piedad_old madison-wi sidewalk_madison
oradell-nj sidewalk_oradell tainan-tw sidewalk_tainan
validation-study sidewalk_validation niagara-falls-ny sidewalk_niagara_falls
zurich sidewalk_zurich chandigarh-india sidewalk_chandigarh
zurich-infra3d sidewalk_zurich_infra3d rancagua-chile sidewalk_rancagua
taipei sidewalk_taipei vancouver-wa sidewalk_vancouver
new-taipei-tw sidewalk_new_taipei santiago-chile sidewalk_santiago
keelung-tw sidewalk_keelung tucson-az sidewalk_tucson
auckland sidewalk_auckland paterson-nj sidewalk_paterson
cuenca sidewalk_cuenca staging sidewalk_zurich
crowdstudy sidewalk_crowdstudy richmond-va sidewalk_richmond
burnaby sidewalk_burnaby fort-wayne-in sidewalk_fort_wayne
teaneck-nj sidewalk_teaneck virden-il sidewalk_virden
walla-walla-wa sidewalk_walla_walla gainesville-fl sidewalk_gainesville
st-louis-mo sidewalk_st_louis sao-paulo-brazil sidewalk_sao_paulo
la-ca sidewalk_la winterthur-infra3d sidewalk_winterthur_infra3d
mendota-il sidewalk_mendota waltham-ma sidewalk_waltham
knox-oh sidewalk_knox houston-tx sidewalk_houston
kaohsiung-tw sidewalk_kaohsiung newport-ky sidewalk_newport_ky
bayonne-fr sidewalk_bayonne_fr laurens-ia sidewalk_laurens_ia

Editor and database tools

Editor: use whatever you're productive in — the team uses both, and the codebase has two halves with different sweet spots:

See docs/editor-setup.md for full setup of either (JDK, plugins/extensions, format-on-save). Whichever you use, configure it to run scalafmt on Scala files (see CONTRIBUTING.md).

Database client: Valentina Studio (cross-platform), Postico (Mac), or pgAdmin (Windows/Mac). Connect with:

Host:     localhost
Port:     5432
Database: sidewalk
User:     postgres
Password: sidewalk

Making changes

The dev server hot-reloads, so you rarely restart it.

  • Scala / Twirl views — sbt ~ run recompiles on save; reload the browser once compilation finishes.
  • JavaScript / CSS — Grunt's watch re-concatenates your src/ edits into public/js/*/build/ automatically. Edit src/ files only; never edit build/ output, and don't run grunt by hand. If a new src/ file isn't picked up, check that its path matches a glob in Gruntfile.js.
  • build.sbt or config changes — these aren't hot-reloaded. In the Docker shell press Ctrl+D, then run sbt clean, then npm start again.
  • Python (the standalone utilities in scripts/) — the container has two interpreters. python3 is the base image's 3.8, kept because the app shells out to it for in-band clustering; python3.13 is where the offline tooling and its libraries live. Run offline scripts as python3.13 scripts/.... Details in scripts/README.md; pins in docs/upgrading-libraries.md.

npm dependencies

package-lock.json is committed, and it — not package.json — decides which versions actually get installed. CI runs npm ci, which installs it exactly and refuses to run if it disagrees with package.json, so every developer and every required check share one toolchain.

To add or change a dependency, edit package.json, run npm install inside the container, and commit the resulting package-lock.json alongside it. Don't hand-edit the lockfile. On a rootful Docker daemon, run that install as docker exec -u $(id -u) projectsidewalk-web … — npm rewrites the lockfile by rename, so the new file lands owned by container root, and a tracked root-owned file makes the next host-side git checkout of it fail.

package.json's engines records the Node and npm the container and CI use. There's no .npmrc, so it's advisory: npm prints an EBADENGINE warning and installs anyway. It's there to tell you what the supported pair is, which matters for make test-e2e-host — the one path that installs on your host rather than in the container.

The container never uses your checkout's node_modules. The bind mount would otherwise lay a host-built copy — the one make test-e2e-host needs — over the container's, so docker-compose.yml mounts a named volume over that path. The volume outlives the container and so isn't refreshed by rebuilding the image; make dev and make npm-sync are what refresh it, against a stamp covering package.json, package-lock.json and the container's node/npm versions. To start completely clean:

make docker-stop
docker volume rm "$(basename "$PWD" | tr '[:upper:]' '[:lower:]')_node_modules"   # your checkout dir, lowercased
make dev

Checking that backend changes compile

The quickest pass/fail on a Scala change is a compile. The sbt thin client hands the command to a background sbt server rather than starting its own:

make compile

The first call after a container boot starts the compile server (~30s); later calls are near-instant. build.sbt sets -Xfatal-warnings, so a [success] is also warning-clean.

Use --jvm-client, not --client: the native client (sbtn) needs a newer glibc than the container's focal base, so it dies on startup, though sbt --client --version still prints happily (#5268). A server belongs to one project directory, so each worktree gets its own; sbt shutdownall stops every one of them, the running ~ run included. An idle one stops itself after an hour (serverIdleTimeout) instead of holding ~1GB until you reboot.

That server serves one command at a time, so it will not run anything while ~ run is up in the same checkout — run holds the task for as long as the app lives, and your command queues behind it forever with nothing on screen. (A ~ compile is fine; a watch loop yields between runs.) So run the app and compile it from different checkouts. tools/sbt-run.sh refuses with an explanation rather than hanging.

Running the backend tests

ScalaTest specs live under test/ — mostly functional specs for the public API, plus service and submission specs. They boot the real app against Postgres+PostGIS, so the db container has to be up:

make test-scala
make test-scala only=controllers.api.PublicApiSpec

Only one checkout tests at a time. They share one db container and one city schema, and most specs commit rather than roll back, so simultaneous runs overwrite each other's rows and stack two multi-GB JVMs — which is how earlyoom comes to kill one mid-run. A second make test-scala says it's waiting, then starts when the first finishes.

The backend-tests CI job is a required check and runs all of test/ (sbt coverage test, since #5042), so a new spec file is picked up with nothing to enroll it in. Still run the suite locally before you trust it — and read the CANCELED lines, not just the green ones. Specs assume their preconditions instead of failing on them, and they cancel in both directions: most want rows your schema may not have, while NightlyJobStatusSpec and ImageryPollOutcomeSpec cancel on a database holding more than CI's — every nightly job already recorded, say. CI's seeded schema (#5115) is expected to cancel nothing, so a CANCELED line there means the seed stopped covering something.

There are also Python unit tests for the scripts/ utilities (make test-python) and a jsdom Jest suite for frontend modules (make test-js — Jest's node_modules are in the container, not on your host). docs/testing-and-ci.md covers what each layer is for.

Checking that pages still load in a browser

Compiling and linting can't see a runtime JavaScript error — a stale bundle, a missing global, a method that throws only when clicked. The browser smoke suite covers that: it loads every core page in headless Chromium and fails on any uncaught page error or console error. With your dev app running, in a second terminal:

make test-e2e                               # the whole suite
make test-e2e args="-g labelMap --no-deps"  # one page
make test-e2e wt=<worktree-name>            # a worktree's specs, from anywhere

Nothing to install: the runner is a container, so it behaves the same on macOS, Linux, and WSL — including Apple Silicon, where it runs a native browser. CI runs the same suite on every PR as the e2e-smoke job. Full details, including how to watch a test run headed, are in test/e2e/README.md.

Running a branch from a git worktree

If you keep in-progress branches in git worktrees (.claude/worktrees/<name>) — for example to review a colleague's branch, or to run a second branch alongside your main checkout — you can bring that branch's app up on http://localhost:9000 with one command:

make qa-worktree wt=<worktree-name>

A worktree needs more setup than the main repo (its node_modules and built asset bundles aren't checked in, and sbt's caches and config have to be pointed at the right places), so this target handles all of it: it links the main repo's node_modules, builds that branch's JS/CSS bundles, starts a backgrounded grunt watch so later edits rebuild automatically, frees :9000, kills any stray sbt server or hung sbt task sharing the worktree's target/ (either deadlocks ~ run on compile locks), and launches sbt ~ run against the worktree's own config while reusing the main repo's warm sbt caches. The first request triggers the dev compile; Ctrl+C stops it and reaps the grunt watch. To tear a session down out-of-band, run make qa-worktree-stop wt=<name> (add clean=1 to also drop the node_modules symlink). It behaves the same on macOS, Linux, and WSL because the work runs inside the web container.

Both targets run the worktree's own copy of tools/qa-worktree.sh when it has one (falling back to the main checkout's), so the branch being QA'd supplies its own tooling. make itself still reads the main checkout's Makefile, so when that checkout sits on a branch without the target, make reports No rule to make target; either check out a branch that has it or run the script directly: docker exec -it projectsidewalk-web bash /home/.claude/worktrees/<name>/tools/qa-worktree.sh <name>.

Every other container target checks the checkout you run it from. The container mounts the main checkout at /home and so sees the worktrees inside it: make lint, make test-js, make compile, make test-scala, make scalafmt, make test-python and the rest, run from a worktree, check that worktree, and wt=<name> points them at one from anywhere. make lint opens by naming the tree it checks. Make stops with an error for a checkout the container can't see (one outside the main checkout). This takes the worktree's own Makefile, so a branch older than #5291 needs develop merged in first. The exceptions:

  • make test-e2e runs the worktree's specs against whatever app is on :9000, and warns when that's another checkout's. Start the worktree's app with make qa-worktree wt=<name> first.
  • make build-city-data and make check-imagery always run in the main checkout, whose db/ the db container reads.
  • A hand-typed docker exec … "cd /home && …" always runs in the main checkout.

The sbt server that make compile, make test-scala, or make scalafmt starts for a worktree stays up until it idles out after an hour, or until make qa-worktree-stop wt=<name> or make worktree-remove wt=<name> stops it.

Each checkout has its own target/. What grows without bound there is packaged build output: every sbt dist/stage writes ~1GB of jars named after the current version and removes none of the older ones. Clear just those, keeping compiled classes so the next make compile is still incremental:

make clean-dist              # this checkout
make clean-dist wt=<name>    # a worktree's

When you're done with a worktree for good, remove it with:

make worktree-remove wt=<worktree-name>

That stops its QA session, deletes the directory and the registration git keeps for it, and deletes its branch once that branch is fully merged into develop — it tells you what it kept otherwise. Deleting the directory yourself leaves the registration behind, so the worktree lingers in git worktree list; run the target and it cleans that up instead. It stops and tells you if the worktree still has uncommitted or untracked files (add force=1 to delete those along with it) or if the worktree is locked: an active Claude Code worktree session holds a lock, and git refuses a locked worktree even with --force. Unlike the QA targets it runs host-side, because a worktree's .git file points at the main repo by absolute host path; the direct invocation is bash .claude/worktrees/<name>/tools/worktree-remove.sh <name>.

To QA admin-only pages you need an account with a role. The dev database is seeded from a dump that includes real accounts, so if your own account is in it you can sign in normally — password checks work the same locally as in production. Otherwise — or if you'd rather use a throwaway account — create a fresh one through the sign-up form and grant it a role directly in the dev database (roles are checked per request, so you don't need to sign in again). Open a psql shell (docker exec -it projectsidewalk-db psql -U sidewalk -d sidewalk) and run:

UPDATE sidewalk_login.user_role
SET role = 'Owner'
WHERE user_id = (SELECT user_id FROM sidewalk_login.sidewalk_user WHERE username = '<your-username>');

Exercising authenticated routes

Most routes need a session. Grab an anonymous cookie once, then reuse the jar:

curl -s -c /tmp/sidewalk_cookies.txt "http://localhost:9000/anonSignUp?url=%2F"
curl -s -b /tmp/sidewalk_cookies.txt "http://localhost:9000/v3/api/labelTypes"

Inspecting the database

Each city lives in its own schema (sidewalk_<city>); authentication lives in sidewalk_login. For ad-hoc queries, prefer the read-only role so you can't accidentally write:

docker exec projectsidewalk-db psql -U readonly_user -d sidewalk -c "\dt sidewalk_seattle.*"

The schemas are essentially identical, so sidewalk_seattle is a safe default for schema questions. For anything about data or migration state, first find out which city is actually running — don't assume:

docker exec projectsidewalk-web bash -lc 'echo $DATABASE_USER'   # this value IS the active schema name

DATABASE_USER selects the schema and is authoritative; SIDEWALK_CITY_ID only selects cityparams.conf entries (map center, bounds, display name). The two are supposed to correspond (see City IDs), but a container can be left with them mismatched, in which case the app renders one city's params over another city's data: an empty map with no error in any log. Confirm what the app believes it is with curl -s -b <cookie-jar> localhost:9000/labelmap | grep -oE 'cityId: "[^"]*"'.

Two more things to know before drawing conclusions from a query:

  • readonly_user cannot see every schema, and the failure is silent. It is granted per-schema, so it may have no rights on the active city's schema, and information_schema / \dt simply omit what you can't see rather than erroring, which reads as "that schema doesn't exist" or "that evolution never applied". pg_namespace is world-readable, so enumerate with it, then query the city as its own role:

    docker exec projectsidewalk-db psql -U readonly_user -d sidewalk -tAc \
  "SELECT nspname FROM pg_namespace WHERE nspname LIKE 'sidewalk%' ORDER BY 1"
docker exec projectsidewalk-db psql -U sidewalk_teaneck -d sidewalk -c \
  "SELECT max(id) FROM sidewalk_teaneck.play_evolutions"

  • The dev DB is not representative of production size, and some tables may be absent. The two largest production tables by a wide margin are audit_task_interaction and validation_task_interaction (raw per-action interaction logs). The dev dumps omit them to stay manageable, so locally they are typically empty or missing. Never infer a table's production size or existence from the local DB; when reasoning about query cost or indexes, treat those two, not webpage_activity, as the heavyweight logs.

Troubleshooting

Roughly ordered by when you'd hit them during setup.

Symptom Fix
make: docker-compose: No such file or directory Newer Docker uses docker compose (no hyphen). Edit the Makefile to replace docker-compose with docker compose.
Docker-Compose command fails on Mac Recreate the symlink per the Compose install docs.
gpg: keyserver receive failed during build (Windows) Add ENV APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1 near the top of the Dockerfile.
pg_restore: ... schema "public" already exists Safe to ignore — no effect.
import-dump otherwise errors Don't skip ahead. Re-check the dump filename and db= value, then see the Troubleshooting wiki and ask.
Execution exception [NoSuchElementException: None.get] at runtime The data wasn't imported — run make import-dump (the init only creates the schema, not the data).
Database suddenly looks empty (role "sidewalk_<city>" does not exist, no city schemas) Your data is most likely parked on an orphaned Docker volume, not gone — docker volume ls -qf dangling=true lists the candidates, and you can copy one back onto this project's data volume (<project>_pgdata, where <project> is your checkout directory lowercased). Don't run docker volume prune while you're looking; that is what actually destroys them.
Cannot create container for service web: Conflict ... name "/projectsidewalk-web" already in use A prior web container wasn't shut down cleanly: docker container rm /projectsidewalk-web.
Errors after the computer was shut off mid-run (WSL) Run wsl --shutdown; when Docker offers to restart WSL, accept. Otherwise restart Docker manually.
Can't connect to the database The db container may not be listening on all addresses. make ssh target=db, edit /var/lib/postgresql/data/postgresql.conf, set listen_addresses = '*'.
make commands "just don't work" Reinstall make. As a fallback, run the underlying command from the Makefile directly (e.g. make ssh target=web ≈ docker exec -it projectsidewalk-web /bin/bash).
relation "role" does not exist while a schema is applying evolutions That schema is behind evolution 372, which dropped the shared sidewalk_login.role lookup table that evolutions 270, 295, 337 and 355 all read. Recoverable with the data intact: Recovering a schema stranded below evolution 372.
A new src/ JS file isn't bundled Make sure its path matches a glob in Gruntfile.js.
First compile seems stuck It isn't — initial dependency resolution is genuinely slow. Watch the container logs.
Compiles are slow on Apple Silicon Your projectsidewalk/web image may predate #5069 and still be x86_64 — Compose reuses a locally tagged image instead of rebuilding it, so pulling that change alone doesn't help. Check with docker image inspect projectsidewalk/web --format '{{.Architecture}}' (expect arm64); if it says amd64, rebuild with make docker-stop && docker compose build web. Also make sure platform is commented out in your docker-compose.override.yml.

Recovering a schema stranded below evolution 372

Evolution 372 turned the shared sidewalk_login.role lookup table into an enum of the same name. Evolutions 270, 295, 337 and 355 read that table, so once 372 has run for any one city, a schema below 356 can't replay them.

Re-importing does not help: every committed city dump is at evolution 335 or below, so it lands in the same hole. Only sidewalk_init-dump, the empty template, is past 372. Editing the old evolution files isn't the fix either — Play would revert every schema that already applied them.

What does work is giving the old evolutions the table they expect for the length of the replay. The conflict is only over the name: Postgres won't hold a table and an enum called role in one schema at once.

  1. Move 372.sql and 373.sql out of conf/evolutions/default/, so no city drops the table mid-replay.
  2. Park the enum and rebuild the lookup table. Evolution 295 also writes user_role.role_id, which 372 renamed and retyped, and 355 joins on it, so it needs a populated shim — not just a placeholder: 
    ALTER TYPE sidewalk_login.role RENAME TO role_parked;
CREATE TABLE sidewalk_login.role (role_id SERIAL PRIMARY KEY, role TEXT NOT NULL);
ALTER TABLE sidewalk_login.role OWNER TO sidewalk;
INSERT INTO sidewalk_login.role (role_id, role) VALUES
  (1, 'Registered'), (2, 'Turker'), (3, 'Researcher'), (4, 'Administrator'), (5, 'Owner'),
  (6, 'Anonymous'), (7, 'AI');
SELECT setval('sidewalk_login.role_role_id_seq', 7);
ALTER TABLE sidewalk_login.user_role ADD COLUMN role_id INT;
UPDATE sidewalk_login.user_role SET role_id = CASE role::TEXT
  WHEN 'Registered' THEN 1 WHEN 'Turker' THEN 2 WHEN 'Researcher' THEN 3 WHEN 'Administrator' THEN 4
  WHEN 'Owner' THEN 5 WHEN 'Anonymous' THEN 6 WHEN 'AI' THEN 7 END;
  3. Boot the app once per stranded schema, which takes each to 371. Evolutions apply on the first request, so docker exec -e DATABASE_USER=<schema> -e SIDEWALK_CITY_ID=<city-id> projectsidewalk-web a sbt "run 9000" and curl it, one schema at a time.
  4. Undo step 2, in reverse. CASCADE clears the survey_question FKs that 337 re-added; 372 drops them itself on the way past: 
    DROP TABLE sidewalk_login.role CASCADE;
ALTER TABLE sidewalk_login.user_role DROP COLUMN role_id;
ALTER TYPE sidewalk_login.role_parked RENAME TO role;
  5. Put 372.sql and 373.sql back and boot each schema once more. 372's shared half self-skips on its enum guard, so only its per-city half runs, then 373.

Schemas already past 372 are never touched, and need no down-migration.

Two other evolutions can strand a schema the same way, because they read shared state a later evolution removed:

  • column "infra3d_access" does not exist — 313 reads it and 316 drops it. A schema below 303 self-heals, since 303 re-adds the column with IF NOT EXISTS; one in the 303–312 window needs it restored (exactly as 316's own Downs does) before each run, because 316 drops it again on the way past.
  • function replace(jsonb, unknown, unknown) does not exist — 276 converts config.excluded_tags from text to jsonb. A dump whose play_evolutions predates 276 but whose config was already converted can't replay it; restore the column to the pre-276 text value from that schema's own dump.

Both are symptoms of the same underlying problem: a committed dump whose contents disagree with the play_evolutions it ships with. If an evolution fails on a table or column that should exist at that level, check the dump (pg_restore -s -t <table> -f - db/<dump>) before assuming your local DB drifted.

Slick query errors while developing:

  • value transactionally is not a member of slick.dbio.DBIOAction... → add import models.utils.MyPostgresProfile.api._.
  • type mismatch ... NoStream,Nothing ... (often misleading) → try wrapping the queries in .transactionally, or use DBIO.seq().andThen().

Getting help

If you solve something not covered here, please add it.