Repository navigation
feat(dbt): report row counts on table-materialized models - #56
tillywoodfield wants to merge 2 commits into
Conversation
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a6dc8131be
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 032f98475c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
What & why
No dbt model reported a row count in Dagster, so a model that lost rows showed nothing in the UI. This sets
include_metadata: [row_count]on the dbt component, so dagster-dbt runs onecount(*)per table-materialized model after each build and reports it asdagster/row_count. Views report nothing.Closes #51
Changes
defs/dbt_ingest/defs.yamlsetsinclude_metadatatorow_counttests/dbt_ingest/test_defs.pybuilds every dbt asset through the project's definitions against the CI fixtures and checks each table's count against the databaseNotes for reviewers
dbt builduntil the dbt process exits, since it can't open the database file while dbt holds the lock. During a run, the asset graph shows the dbt models and their checks all at once when the build ends. The dbt logs still stream, models that succeed before a failure are still recorded, and automation reads the events once they exist. On BigQuery, events stream as each model finishes.relation_namefrom the component's prepared manifest, which carries the catalog from theDUCKDB_PATHfile name at preparation.dg devre-prepares on every load. A run from a manifest prepared against a differently named DuckDB file, such as after switchingDUCKDB_PATHoutsidedg dev, logs an error per count and materializes without one. On BigQuery the relation names come from the profile's project and dataset, so they match while the deployment prepares and runs against the same target.DAGSTER_IS_DEV_CLIso the manifest is prepared against its temporary database, and counts rows in a subprocess because dbt's in-process DuckDB connection outlives the run. It takes about 30 seconds.Checklist
Testing:
pre-commit run --all-files(with CI'sUV_NO_ENV_FILE=1),dg check defs,dbt buildon the CI fixtures andpytestall pass, 445 tests. The new test fails with all 40 tables missing a count wheninclude_metadatais removed, and fails with count errors when the manifest is left prepared against another database.🤖 Generated with Claude Code