Skip to content

refactor(dagster): make asset metadata keys consistent across the pipelines - #53

Merged
tillywoodfield merged 8 commits into
devfrom
refactor/17-consistent-asset-metadata
Oct 27, 2026
Merged

tillywoodfield merged 8 commits into
devfrom
refactor/17-consistent-asset-metadata

Conversation

@tillywoodfield

@tillywoodfield tillywoodfield commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

What & why

Asset metadata used different keys for the same fact across pipelines, and some facts appeared on only a few of them. This aligns every source, raw asset, export and ingestion on Dagster's standard keys and one name per fact.

Closes #17

Changes

  • Sources observed from one file or repository report their address as dagster/uri: DAC1, DAC2A, DAC2B and CRS rename url, and dfo_skills_source adds the skills repository's URL.
  • The publisher's token is version on every single-input source. imf_weo_source renames latest_release and keeps the October 2026 form, and dfo_skills_source renames commit. one_imputed_multilateral_sectors_source reads two files, so it reports each one's address and token under prefixed keys: crs_url, crs_version, multisystem_url and multisystem_version. world_bank_ids_source puts its six per-file entries under one files key.
  • Raw assets report the version they downloaded under their source's key and in its format. The OECD four report the token they resolved, imf_weo_raw_file switches to October 2026, dfo_skills_raw_directory renames commit, and Imputed Multilateral Sectors reports the tokens it checks before computing. world_bank_ids_raw_file reports files from the same HEAD lookup the source observation makes, just before it downloads.
  • Every raw file reports dagster/row_count; Imputed Multilateral Sectors renames row_count.
  • No export reports import_name. Every ingestion reports it, as the import's identity in Data Commons.
  • Every source's DataVersion is unchanged.

Notes for reviewers

  • The OECD and IDS raw assets look up the version just before they download. The OECD four call their own resolve_*_bulk_file(), then bulk_download_* in oda_reader resolves again, and IDS makes the HEAD requests the source observation makes, then the GET requests. A republish between lookup and download would label the new file with the older version, the same gap the OECD release date already has.
  • imf_weo_source has no dagster/uri because it is observed through the IMF's SDMX API. The skills and ontology raw assets are directories, so they report no dagster/row_count.

Checklist

  • Addresses one issue, linked above
  • Tests updated where relevant
  • Docs updated where relevant
  • Self-reviewed the diff

Testing: pre-commit run --all-files (with CI's UV_NO_ENV_FILE=1), dg check defs, dbt build on the CI fixtures and pytest all pass, 444 tests. New assertions were checked by mutation: dropping the IDS raw files, the DAC1 raw version or the skills source dagster/uri, or reverting the WEO version format, fails the matching tests.

🤖 Generated with Claude Code

@tillywoodfield

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-27T15:23:16.154687Z 542b5b4 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: b745f02941

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@tillywoodfield

Copy link
Copy Markdown
Collaborator Author

@codex review

@tillywoodfield

Copy link
Copy Markdown
Collaborator Author

@codex review

@tillywoodfield
tillywoodfield marked this pull request as ready for review October 27, 2026 15:20
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: 542b5b4f8b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@tillywoodfield
tillywoodfield merged commit dc1b589 into dev Oct 27, 2026
4 checks passed
@tillywoodfield
tillywoodfield deleted the refactor/17-consistent-asset-metadata branch October 27, 2026 15:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make asset metadata consistent across the pipelines

1 participant