Fix the Airflow version condition check to parse inlets/outlets syntax according to the dataset by pankajastro · Pull Request #485 · astronomer/dag-factory

pankajastro · 2025-07-21T06:15:04Z

We should be able to parse inlets/outlets syntax according to the dataset for Airflow < 3.0.0, because in Airflow 3, the dataset has been replaced with an asset. This PR fixes the wrong condition check for Airflow version for parsing in inlets/outlets for datasets

This PR also add an example DAG for Asset.

Before this PR, DAG below will not work.

default:
  default_args:
    start_date: 2025-01-01
  catchup: false

producer_dag:
  schedule: "@daily"
  tasks:
    produce_data:
      operator: "airflow.providers.standard.operators.python.PythonOperator"
      python_callable: utils.generate_data
      outlets:
        - __type__: airflow.sdk.Asset
          uri: "file:///$AIRFLOW_HONE/data.csv"
          name: "data_asset"

consumer_dag:
  schedule:
    - __type__: airflow.sdk.Asset
      uri: "file:///$AIRFLOW_HONE/data.csv"
      name: "data_asset"
  tasks:
    consume_data:
      operator: "airflow.providers.standard.operators.bash.BashOperator"
      bash_command: "echo 'Asset was updated, running DAG!'"

codecov-commenter · 2025-07-25T10:51:37Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.06%. Comparing base (9f7ccdb) to head (f7c1fa8).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #485   +/-   ##
=======================================
  Coverage   93.06%   93.06%           
=======================================
  Files          13       13           
  Lines        1212     1212           
=======================================
  Hits         1128     1128           
  Misses         84       84

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

pankajkoti

Sorry, I’m not sure what we’re trying to accomplish here. Could you link the relevant issue for context?

The PR description says, “We should parse inlets/outlets syntax as per the dataset for Airflow < 2.4,” which I find a bit confusing.

pankajastro · 2025-07-31T19:12:44Z

Sorry, I’m not sure what we’re trying to accomplish here. Could you link the relevant issue for context?

The PR description says, “We should parse inlets/outlets syntax as per the dataset for Airflow < 2.4,” which I find a bit confusing.

My bad. I have improved the PR description. Please let me know if there is any further confusion.

pankajkoti

I am good on this one if we have tested the DAG being added in the PR (at least manually) with Airflow.

pankajastro · 2025-07-31T19:32:44Z

I am good on this one if we have tested the DAG being added in the PR (at least manually) with Airflow.

Tested with Astro-CLI

@pankajastro

### Breaking Changes - Airflow providers are now optional dependencies by @pankajastro in [#486](#486) - Previously, `dag-factory` enforced the installation of `apache-airflow-providers-http` and `apache-airflow-providers-cncf-kubernetes`. These Airflow providers dependencies are now optional. If your DAGs depend on these providers, you must install them manually. Alternatively, you can install `dag-factory` with extras like `dag-factory[all]`, `dag-factory[kubernetes]`, etc. - Removed `clean_dags` function by @pankajastro in [#498](#498) - You no longer need to call `example_dag_factory.clean_dags(globals())` in your DAG files. DAG cleanup is now controlled via the Airflow config setting `AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL`. - Remove `schedule_interval` parameter from DAG configuration YAML by @viiccwen in [#503](#503) - Use `schedule` parameter instead of `schedule_interval`. - Change `DagFactory` class access to private by @pankajastro in [#509](#509) - The import path `from dagfactory import DagFactory` has been removed. - The class `DagFactory` has been renamed to `_DagFactory`. - The `generate_dags` method of `DagFactory` has been renamed to `_generate_dags`. - Remove Inconsistent Parameters for Airflow Consistent by @pankajastro in [#512](#512) - Removed `dagrun_timeout_sec` from dag param. - Removed `retry_delay_sec`, `sla_secs` from default_args. - Removed accepting `execution_timeout` as integer. - Removed `execution_timeout_secs`, `sla_secs` and `execution_delta_secs` from task param. - Remove custom parsing for Kubernetes object and refactor KPO to use `__type__` syntax by @pankajastro in [#523](#523) - The custom parsing for Kubernetes objects has been removed. You can no longer pass a custom YAML dictionary to DAG-Factory configuration unless accepted by the KubernetesPodOperator. We suggest you to use `__type__` syntax to supply Kubernetes object in your YAML DAG. For an example KPO configuration, visit: [KubernetesPodOperator Documentation](https://astronomer.github.io/dag-factory/dev/features/kpo/). - Consolidate `!and`, `!or`, `!join`, `and` and `or` key in YAML DAG configuration by @pankajastro in [#525](#525) - Use `__and__`, `__or__` and `join__` instead - Remove custom parsing for DAG parameter `timetable` by @pankajastro in [#533](#533) - Use the `__type__` annotation for the `timetable` parameter. - Rename parameter of `load_yaml_dags` and `_DagFactory` to reflect behaviour by @pankajastro in [#546](#546) - Rename `config` to `config_dict` - Rename `default_args_config_path` to `defaults_config_path` - Rename `default_args_config_dict` to `defaults_config_dict` ### Added - Support dag-level arguments in global defaults by @gyli in [#480](#480) - Support `*args` in custom Python object by @pankajastro in [#484](#484) - Support tasks and task_groups as lists by @pankajkoti in [#487](#487) - Support overriding `defaults.yml` based on the directory hierarchy by @tatiana in [#500](#500) - Introduced DAG Factory CLI by @tatiana in [#510](#510) - Added `lint` command to CLI by @tatiana in [#513](#513) - Add convert CLI command to migrate from af2 to af3 by @tatiana in [539](#539) - Allow yaml/yml suffix for shared defaults by @pankajastro in [#567](#567) ### Fixed - Fix the Airflow version condition check to parse inlets/outlets syntax according to the dataset by @pankajastro in [#485](#485) - Ensure `dag_params` contain `schedule` before operating on it by @pankajkoti in [#488](#488) - Fix `start_date`, `end_date` at the DAG level by @pankajastro in [#495](#495) - Allow `execution_timeout` in `default_args` by @pankajastro in [#501](#501) - Capture Telemetry DNS gaierror and handle it gracefully by @tatiana in [#544](#544) - Fix require response_check param in http sensor by @pankajastri in [#576](#576) - Add none check before evaluating lambda func in HttpSensor @pankajastro in [#577](#577) ### Docs - Restore basic DAG example by @pankajastro in [#483](#483) - Replace the usages in example dags, tests and docs for tasks and taskgroups to be list by @pankajkoti in [#492](#492) - Update default documentation based on #500 by @tatiana in [#504](#504) - Add more examples for Custom Python object by @pankajastro in [#506](#506) - Add documentation for DAG Factory CLI by @tatiana in [#511](#511) - Add documentation and example YAMLs for task and task_group configuration formats by @pankajkoti in [#530](#530) - Add migration guide docs by @pankajastro in [#532](#532) - Docs: Fix rendering of note block by @pankajastro in [#537](#537) - Document the Asset example DAG by @pankajastro in [#538](#538) - Add docs for the CLI convert command by @tatiana in [#541](#541) - Add remaining breaking changes in migration guide by @pankajastro in [#549](#549) - Fix typos in scheduling and datasets docs by @viiccwen in [#565](#565) - docs: Make markdownlint happy (fix MD007 ul-indent in dev/README.md) by @viiccwen in [#566](#566) - Update Index.md by @pankajastro in [#570](#570) - Document dag-factory reserved keys by @pankajastro in [#571](#571) - Add an introduction to the migration guide by @pankajastro in [#572](#572) ### Other Changes - Improve unit tests to disregard `$AIRFLOW_HOME` by @tatiana in [#490](#490) - Resolve unpinned action reference error alerts raised by Zizmor by @pankajkoti in [#493](#493) - Resolve 'credential persistence through GitHub Actions artifacts' warnings from Zizmor by @pankajkoti in [#494](#494) - Resolve 'overly broad permissions' warnings from Zizmor by @pankajkoti in [#496](#496) - CI: Add GitHub CodeQL analysis workflow (`codeql.yml`) by @pankajkoti in [#497](#497) - Fix deploy pages job missing credentials by @pankajkoti in [#499](#499) - Add the breaking changes to changelog by @pankajastro in [#502](#502) - Add pre-commit to update `uv.lock` by @pankajastro in [#514](#514) - Remove `clean_dags` usage from object storage DAG by @pankajastro in [#515](#515) - Remove broad exceptions and catch more specific exceptions by @pankajastro in [#519](#519) - Add missing env in contributing doc by @pankajastro in [#522](#522) - Enhance PyPI Stats API error handling by @viiccwen in [#535](#535) - Update example to be Airflow 3 compatible by @tatiana in [540](#540) - Remove AUTO_CONVERT_TO_AF3 from tests by @tatiana in [#543](#543) - Bump actions/download-artifact from 4 to 5 by @dependabot in [#548](#548), [#558](#558), [#559](#559), [#560](#560), [#562](#562), [#653](#563) and [#575](#575) - Update pyproject.toml to Sync Test Versions with CI/CD Pipeline by @viiccwen in [#553](#553) - Fix file URI format in ObjectStoragePath example to prevent duplicate slashes by @viiccwen in [#556](#556) - CI: Only build docs on PR push event and deploy for merge and release by @pankajastro in [#568](#568)

pankajastro temporarily deployed to internal July 21, 2025 06:15 — with GitHub Actions Inactive

pankajastro commented Jul 21, 2025

View reviewed changes

Comment thread dagfactory/dagbuilder.py Outdated

pankajastro temporarily deployed to internal July 21, 2025 06:15 — with GitHub Actions Inactive

pankajastro commented Jul 21, 2025

View reviewed changes

Comment thread dagfactory/dagbuilder.py

pankajastro temporarily deployed to internal July 21, 2025 06:16 — with GitHub Actions Inactive

pankajastro temporarily deployed to internal July 21, 2025 06:19 — with GitHub Actions Inactive

pankajastro temporarily deployed to internal July 21, 2025 06:21 — with GitHub Actions Inactive

pankajastro temporarily deployed to internal July 21, 2025 06:34 — with GitHub Actions Inactive

pankajastro temporarily deployed to internal July 21, 2025 07:28 — with GitHub Actions Inactive

pankajastro temporarily deployed to internal July 24, 2025 19:41 — with GitHub Actions Inactive

pankajastro force-pushed the fix_outlet branch from dd4c235 to 417a6ae Compare July 24, 2025 19:42

pankajastro temporarily deployed to internal July 24, 2025 19:43 — with GitHub Actions Inactive

pankajastro temporarily deployed to internal July 25, 2025 10:38 — with GitHub Actions Inactive

pankajastro and others added 10 commits July 25, 2025 16:09

Fix the outlets parsing for Airflow 3

828b6e8

Update dagfactory/dagbuilder.py

e7048c9

Update dagfactory/dagbuilder.py

3fca3c1

Add example dag

87d300c

Add example dag

2392461

Add example dag

721ae08

Install airflow-3.0.3

f376753

Fix the AF version in pre-install script

53e4114

Fix the AF version in pre-install script

9510ad8

Add fn to create csv

da9fea0

pankajastro force-pushed the fix_outlet branch from 3535403 to da9fea0 Compare July 25, 2025 10:39

pankajastro temporarily deployed to internal July 25, 2025 10:39 — with GitHub Actions Inactive

Disable test

fc1d30e

pankajastro temporarily deployed to internal July 25, 2025 10:47 — with GitHub Actions Inactive

pankajastro marked this pull request as ready for review July 25, 2025 10:51

pankajastro requested a review from a team as a code owner July 25, 2025 10:51

tatiana reviewed Jul 28, 2025

View reviewed changes

Comment thread scripts/test/pre-install-airflow.sh Outdated

pankajastro commented Jul 29, 2025

View reviewed changes

Comment thread dev/dags/asset_triggered_dags.py Outdated

Update dev/dags/asset_triggered_dags.py

e2a46ba

pankajastro commented Jul 29, 2025

View reviewed changes

Comment thread scripts/test/pre-install-airflow.sh Outdated

pankajastro added 2 commits July 29, 2025 13:07

Update scripts/test/pre-install-airflow.sh

9a02238

Merge branch 'main' into fix_outlet

27612a3

pankajastro temporarily deployed to internal July 29, 2025 07:39 — with GitHub Actions Inactive

Merge branch 'main' into fix_outlet

9b514db

pankajastro temporarily deployed to internal July 29, 2025 08:57 — with GitHub Actions Inactive

Merge branch 'main' into fix_outlet

22e25b3

pankajastro temporarily deployed to internal July 29, 2025 13:18 — with GitHub Actions Inactive

pankajastro requested a review from tatiana July 30, 2025 11:36

Merge branch 'main' into fix_outlet

889a9de

pankajastro temporarily deployed to internal July 31, 2025 16:24 — with GitHub Actions Inactive

pankajastro commented Jul 31, 2025

View reviewed changes

Comment thread scripts/test/pre-install-airflow.sh Outdated

Update scripts/test/pre-install-airflow.sh

f7c1fa8

pankajastro temporarily deployed to internal July 31, 2025 17:01 — with GitHub Actions Inactive

pankajkoti reviewed Jul 31, 2025

View reviewed changes

pankajastro changed the title ~~Fix the outlets parsing for Airflow 3~~ We should parse inlets/outlets syntax as per the dataset for Airflow < 3.0.0 Jul 31, 2025

pankajastro changed the title ~~We should parse inlets/outlets syntax as per the dataset for Airflow < 3.0.0~~ We should be able to parse inlets/outlets syntax according to the dataset for Airflow < 3.0.0 Jul 31, 2025

pankajastro changed the title ~~We should be able to parse inlets/outlets syntax according to the dataset for Airflow < 3.0.0~~ Fix the Airflow version condition check to parse inlets/outlets syntax according to the dataset Jul 31, 2025

pankajkoti approved these changes Jul 31, 2025

View reviewed changes

Comment thread dev/dags/asset_triggered_dags.yml

pankajastro merged commit 86ab50d into main Jul 31, 2025
66 checks passed

pankajastro deleted the fix_outlet branch July 31, 2025 19:33

pankajastro added this to the DAG Factory 1.0.0 milestone Jul 31, 2025

pankajastro mentioned this pull request Aug 1, 2025

Release 1.0.0 🎉 #521

Merged

Navigation Menu

Uh oh!

Fix the Airflow version condition check to parse inlets/outlets syntax according to the dataset - #485

Fix the Airflow version condition check to parse inlets/outlets syntax according to the dataset#485
pankajastro merged 18 commits into
mainfrom
fix_outlet

pankajastro commented Jul 21, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Jul 25, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pankajkoti left a comment

Uh oh!

pankajastro commented Jul 31, 2025

Uh oh!

pankajkoti left a comment

Uh oh!

Uh oh!

pankajastro commented Jul 31, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

pankajastro commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pankajkoti left a comment

Choose a reason for hiding this comment

Uh oh!

pankajastro commented Jul 31, 2025

Uh oh!

pankajkoti left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pankajastro commented Jul 31, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pankajastro commented Jul 21, 2025 •

edited

Loading

codecov-commenter commented Jul 25, 2025 •

edited

Loading