Skip to content

feat(dagster): re-export a pipeline when its documentation changes - #57

Merged
tillywoodfield merged 4 commits into
devfrom
feat/21-re-export-on-docs-change
Oct 29, 2026
Merged

tillywoodfield merged 4 commits into
devfrom
feat/21-re-export-on-docs-change

Conversation

@tillywoodfield

@tillywoodfield tillywoodfield commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

What & why

Closes #21

A pipeline's transformations.md and changelog.md go onto its provenance, but a change to either left Dagster with nothing to track, so the graph kept serving the old text. Each export now takes a hash of those files as its code version, and its automation condition counts a new code version as an update. A docs-only change shows the export as unsynced, automation re-exports it, and the hand-run ingestion then shows stale, ready to run. Dagster does not pass staleness down a chain, so the ingestion turns stale once the export has re-run, which is one step later than the issue's wording.

Changes

  • ExportSpec.documentation_version in data_commons/export.py hashes the DOCUMENTATION_FILES with a length prefix per file, so text moved from one file to the other gives a new hash.
  • The nine exports built from an ExportSpec pass it as code_version. The dfo_shared, dfo_ontology and dfo_skills exports have no documentation files and are unchanged.
  • EAGER_ON_DEPS_OR_CODE_CHANGE_WHEN_CHECKS_PASS in defs/automation.py is EAGER_WHEN_CHECKS_PASS with code_version_changed() added beside any_deps_updated() inside eager(), and the nine exports use it. A docs change that lands while a parent's blocking check fails waits until the check passes. EAGER_WHEN_CHECKS_PASS and the dbt models are unchanged.

Notes for reviewers

  • After the first deploy all nine exports show "has a new code version", because earlier materializations recorded their run ID as the code version. code_version_changed() reads a first evaluation as unchanged, so automation leaves them alone and each flag clears when that export next runs.
  • The code version covers the documentation files alone. An export's data version now derives from that code version and its inputs, so a hand re-export after a fix to export.py with unchanged inputs keeps the same data version and leaves the ingestion looking fresh.
  • A missing documentation file fails the code location load, since the hash is computed when definitions load.

Checklist

  • Addresses one issue, linked above
  • Tests updated where relevant
  • Docs updated where relevant
  • Self-reviewed the diff

Testing: pre-commit, dg check defs, dbt build (1138 passed) and pytest (457 passed), run locally as in ci.yml. The new tests fail when the hash ignores a file or the length prefix, when an export drops its code version or condition, when the condition lacks the trigger, and when the trigger sits outside eager()'s since_last_handled().

🤖 Generated with Claude Code

@tillywoodfield

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-27T16:48:54.786906Z 8a48a1f Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: 8a48a1fd27

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

tillywoodfield and others added 4 commits October 29, 2026 10:44
A docs-only edit to a pipeline's transformations.md or changelog.md
changed nothing Dagster tracks, so its export never showed unsynced.
ExportSpec.documentation_version hashes those files, and each export
asset takes it as its code_version.
eager() ignores code versions, so a new documentation version left the
export unsynced without requesting it. The exports switch to
EAGER_OR_NEW_CODE_VERSION_WHEN_CHECKS_PASS, which counts a new code
version as an update inside eager()'s since_last_handled(), so a change
that lands while a parent's blocking check fails waits until it passes.
Co-authored-by: Claude <noreply@anthropic.com>
…zed test

Co-authored-by: Claude <noreply@anthropic.com>
@tillywoodfield
tillywoodfield force-pushed the feat/21-re-export-on-docs-change branch from 60b9ce1 to 8987d5c Compare October 29, 2026 10:44
@tillywoodfield
tillywoodfield marked this pull request as ready for review October 29, 2026 10:44
@tillywoodfield
tillywoodfield merged commit 95ac22c into dev Oct 29, 2026
4 checks passed
@tillywoodfield
tillywoodfield deleted the feat/21-re-export-on-docs-change branch October 29, 2026 10:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Re-export a pipeline's bundle when only its docs change

1 participant