Skip to content

fix things missed in the ingest migration - #2622

Draft
damonmcc wants to merge 7 commits into
mainfrom
dm-library-facdb
Draft

fix things missed in the ingest migration#2622
damonmcc wants to merge 7 commits into
mainfrom
dm-library-facdb

Conversation

@damonmcc

@damonmcc damonmcc commented Sep 12, 2026

Copy link
Copy Markdown
Member

related to #1255 and a follow-up to recent PRs/ingest runs

Fixes the four product builds that nightly QA has been failing on since the mappluto and pops ingest migrations landed. Consumers were still reading wkb_geometry from datasets whose ingest archives now carry geom, so FacDB, KPDB and CPDB died in SQL and green_fast_track died on a source test.

  • facdb _create_facdb_spatial.sql, kpdb combine.sql and dcp_housing.sql, cpdb stg__dcp_mappluto_wi.sql now read geom for dcp_mappluto_wi
  • dcp_pops.yml no longer renames geom to wkb_geometry. That rename was mine and it was pointless: the only postgres consumer builds its own geometry from location/xcoordinate, while green_fast_track reads the parquet through duckdb and expects geom like every other parquet source. Re-archived at 20260713, same 38 columns and 392 rows, only the column name differs
  • stg__pluto.sql casts spdist3 to varchar. Arrow nulls land as INTEGER in duckdb, so the LIKE against it failed there even though the same SQL is fine on postgres
  • status csv regenerated: 246 rows, 183 archived via ingest, 0 flagged stale

Two unrelated fixes that were in the way:

  • build commands now stream their output instead of buffering it. capture_output=True plus a single logger.info turned a whole dbt build into one log record, so CI showed nothing for five minutes and then a wall of text that the log viewer wrapped mid-word, with the formatter's color codes nested inside dbt's
  • cpdb's projects_in_geographies builds one engine instead of 210. It was calling create_engine twice per geography and never disposing

Built all four on this branch: cpdb, facilities, knownprojects, green_fast_track. green_fast_track is PASS=313 WARN=6 ERROR=0 SKIP=0 across 319 nodes, the 6 warns being the stale equality fixtures already tracked in that product's issues.md.

@damonmcc damonmcc changed the title fix mappluto geometry column references missed in the ingest migration fix things missed in the ingest migration Sep 13, 2026
@codecov

codecov Bot commented Sep 13, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 68.70%. Comparing base (9432a53) to head (7346e4a).

Files with missing lines Patch % Lines
dcpy/lifecycle/builds/build.py 0.00% 1 Missing ⚠️
Additional details and impacted files
Files with missing lines Coverage Δ
dcpy/lifecycle/builds/build.py 0.00% <0.00%> (ø)

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant