Skip to content

feat(dagster): report each source's release date on its raw file - #58

Merged
tillywoodfield merged 1 commit into
devfrom
feat/50-report-release-dates
Oct 29, 2026
Merged

tillywoodfield merged 1 commit into
devfrom
feat/50-report-release-dates

Conversation

@tillywoodfield

@tillywoodfield tillywoodfield commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

What & why

Each raw file asset already fetches or parses its source's release date to write the metadata.json sidecar, but never reported it to Dagster, so the UI couldn't show which release a download holds. The raw file assets of DAC1, DAC2A, DAC2B, CRS, WEO, IDS and Imputed Multilateral Sectors now report it as YYYY-MM-DD metadata, under the sidecar's keys (source_release_date, or crs_release_date and multisystem_release_date).

Closes #50

Changes

  • Each raw file asset adds the date it already holds to its materialization metadata. CRS and WEO keep the parsed or fetched date in a variable, since it is now used twice.
  • Each raw file test asserts the new key.

Notes for reviewers

  • The observations and exports report no date. An observation already identifies a release by its version metadata, and a date there would cost a second request a day that must not fail release detection. What the graph serves is on its provenance's importedSourceReleaseDate.

Checklist

  • Addresses one issue, linked above
  • Tests updated where relevant
  • Docs updated where relevant
  • Self-reviewed the diff

Testing: pre-commit checks, dg check defs and pytest (451 passed) pass locally, rebased on dev.

🤖 Generated with Claude Code

@tillywoodfield

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-29T11:11:52.783963Z 60212b1 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Co-authored-by: Claude <noreply@anthropic.com>
@tillywoodfield
tillywoodfield force-pushed the feat/50-report-release-dates branch from 71b1af4 to 60212b1 Compare October 29, 2026 11:07
@tillywoodfield tillywoodfield changed the title feat(dagster): report each source's release date in asset metadata feat(dagster): report each source's release date on its raw file Oct 29, 2026
@tillywoodfield
tillywoodfield marked this pull request as ready for review October 29, 2026 11:08
@tillywoodfield

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: 60212b1c8d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@tillywoodfield
tillywoodfield merged commit 111fb97 into dev Oct 29, 2026
4 checks passed
@tillywoodfield
tillywoodfield deleted the feat/50-report-release-dates branch October 29, 2026 11:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Show each source's release date on its raw file asset

1 participant