Repository navigation
feat(dagster): re-export a pipeline when its documentation changes - #57
Merged
Merged
Conversation
Collaborator
Author
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Didn't find any major issues. Bravo. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
2 tasks
A docs-only edit to a pipeline's transformations.md or changelog.md changed nothing Dagster tracks, so its export never showed unsynced. ExportSpec.documentation_version hashes those files, and each export asset takes it as its code_version.
eager() ignores code versions, so a new documentation version left the export unsynced without requesting it. The exports switch to EAGER_OR_NEW_CODE_VERSION_WHEN_CHECKS_PASS, which counts a new code version as an update inside eager()'s since_last_handled(), so a change that lands while a parent's blocking check fails waits until it passes.
Co-authored-by: Claude <noreply@anthropic.com>
…zed test Co-authored-by: Claude <noreply@anthropic.com>
tillywoodfield
force-pushed
the
feat/21-re-export-on-docs-change
branch
from
October 29, 2026 10:44
60b9ce1 to
8987d5c
Compare
tillywoodfield
marked this pull request as ready for review
October 29, 2026 10:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Closes #21
A pipeline's
transformations.mdandchangelog.mdgo onto its provenance, but a change to either left Dagster with nothing to track, so the graph kept serving the old text. Each export now takes a hash of those files as its code version, and its automation condition counts a new code version as an update. A docs-only change shows the export as unsynced, automation re-exports it, and the hand-run ingestion then shows stale, ready to run. Dagster does not pass staleness down a chain, so the ingestion turns stale once the export has re-run, which is one step later than the issue's wording.Changes
ExportSpec.documentation_versionindata_commons/export.pyhashes theDOCUMENTATION_FILESwith a length prefix per file, so text moved from one file to the other gives a new hash.ExportSpecpass it ascode_version. Thedfo_shared,dfo_ontologyanddfo_skillsexports have no documentation files and are unchanged.EAGER_ON_DEPS_OR_CODE_CHANGE_WHEN_CHECKS_PASSindefs/automation.pyisEAGER_WHEN_CHECKS_PASSwithcode_version_changed()added besideany_deps_updated()insideeager(), and the nine exports use it. A docs change that lands while a parent's blocking check fails waits until the check passes.EAGER_WHEN_CHECKS_PASSand the dbt models are unchanged.Notes for reviewers
code_version_changed()reads a first evaluation as unchanged, so automation leaves them alone and each flag clears when that export next runs.export.pywith unchanged inputs keeps the same data version and leaves the ingestion looking fresh.Checklist
Testing: pre-commit,
dg check defs,dbt build(1138 passed) andpytest(457 passed), run locally as inci.yml. The new tests fail when the hash ignores a file or the length prefix, when an export drops its code version or condition, when the condition lacks the trigger, and when the trigger sits outsideeager()'ssince_last_handled().🤖 Generated with Claude Code