Repository navigation
refactor(dagster): make asset metadata keys consistent across the pipelines - #53
Conversation
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Didn't find any major issues. Swish! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
|
@codex review |
|
Codex Review: Didn't find any major issues. Delightful! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
What & why
Asset metadata used different keys for the same fact across pipelines, and some facts appeared on only a few of them. This aligns every source, raw asset, export and ingestion on Dagster's standard keys and one name per fact.
Closes #17
Changes
dagster/uri: DAC1, DAC2A, DAC2B and CRS renameurl, anddfo_skills_sourceadds the skills repository's URL.versionon every single-input source.imf_weo_sourcerenameslatest_releaseand keeps theOctober 2026form, anddfo_skills_sourcerenamescommit.one_imputed_multilateral_sectors_sourcereads two files, so it reports each one's address and token under prefixed keys:crs_url,crs_version,multisystem_urlandmultisystem_version.world_bank_ids_sourceputs its six per-file entries under onefileskey.imf_weo_raw_fileswitches toOctober 2026,dfo_skills_raw_directoryrenamescommit, and Imputed Multilateral Sectors reports the tokens it checks before computing.world_bank_ids_raw_filereportsfilesfrom the same HEAD lookup the source observation makes, just before it downloads.dagster/row_count; Imputed Multilateral Sectors renamesrow_count.import_name. Every ingestion reports it, as the import's identity in Data Commons.DataVersionis unchanged.Notes for reviewers
resolve_*_bulk_file(), thenbulk_download_*inoda_readerresolves again, and IDS makes the HEAD requests the source observation makes, then the GET requests. A republish between lookup and download would label the new file with the older version, the same gap the OECD release date already has.imf_weo_sourcehas nodagster/uribecause it is observed through the IMF's SDMX API. The skills and ontology raw assets are directories, so they report nodagster/row_count.Checklist
Testing:
pre-commit run --all-files(with CI'sUV_NO_ENV_FILE=1),dg check defs,dbt buildon the CI fixtures andpytestall pass, 444 tests. New assertions were checked by mutation: dropping the IDS rawfiles, the DAC1 rawversionor the skills sourcedagster/uri, or reverting the WEO version format, fails the matching tests.🤖 Generated with Claude Code