Skip to content

Make Airflow providers an optional dependency - #486

Merged
pankajastro merged 13 commits into
mainfrom
optional__depndency
Jul 24, 2025
Merged

Make Airflow providers an optional dependency#486
pankajastro merged 13 commits into
mainfrom
optional__depndency

Conversation

@pankajastro

@pankajastro pankajastro commented Jul 21, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

closes: #409

  • Guard the import for SqlSensor, HttpOperator, HttpSensor KPO and other Kubernetes import
  • If Package is not installed, log message about it
  • Add comments that will help in future cleanup if we upgrade packages
  • Introduce additional installation target, dag-factory[all, kubernetes, http, common-sql]

@codecov-commenter

codecov-commenter commented Jul 21, 2025 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

Attention: Patch coverage is 73.33333% with 16 lines in your changes missing coverage. Please review.

Project coverage is 92.29%. Comparing base (b4f7d44) to head (fb1c6cd).

Files with missing lines Patch % Lines
dagfactory/dagbuilder.py 73.33% 16 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #486      +/-   ##
==========================================
- Coverage   93.76%   92.29%   -1.47%     
==========================================
  Files          11       11              
  Lines        1042     1064      +22     
==========================================
+ Hits          977      982       +5     
- Misses         65       82      +17

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@pankajastro pankajastro changed the title Make providers as optional dependency Make providers an optional dependency Jul 22, 2025
@pankajastro
pankajastro marked this pull request as ready for review July 22, 2025 06:34
@pankajastro
pankajastro requested a review from a team as a code owner July 22, 2025 06:34
tatiana
tatiana reviewed Jul 24, 2025
View reviewed changes
Comment thread dev/dags/example_http_operator_task.py Outdated
tatiana
tatiana reviewed Jul 24, 2025
View reviewed changes
Comment thread dev/dags/example_http_operator_task.py Outdated
tatiana
tatiana reviewed Jul 24, 2025
View reviewed changes
Comment thread dagfactory/dagbuilder.py Outdated
tatiana
tatiana approved these changes Jul 24, 2025
View reviewed changes

@tatiana tatiana left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, @pankajastro , thank you very much! It is great that we're making the K8s provider become an optional dependency. I left minor feedback inline.

@pankajastro
pankajastro merged commit 3ffb657 into main Jul 24, 2025
63 checks passed
@pankajastro
pankajastro deleted the optional__depndency branch July 24, 2025 16:48
@pankajastro pankajastro changed the title Make providers an optional dependency Make aproviders an optional dependency Jul 25, 2025
@pankajastro pankajastro changed the title Make aproviders an optional dependency Make Airflow providers an optional dependency Jul 25, 2025
@pankajastro pankajastro mentioned this pull request Jul 25, 2025
Merged
pankajastro added a commit that referenced this pull request Jul 28, 2025
@pankajastro
Add the breaking changes in changelog (#502)
57119eb 
We introduced breaking changes in PR:
#486 and
#498. This PR documents
them to make it easy to create release note
@pankajastro pankajastro mentioned this pull request Jul 29, 2025
Merged
@pankajastro pankajastro mentioned this pull request Aug 8, 2025
Merged
tatiana pushed a commit that referenced this pull request Aug 8, 2025
@pankajastro
Add remaining breaking changes in migration guide (#549)
29b72f5 
Document the breaking changes of PR's:
#486 #498 #503 #509 #512 #523 #525 #533 #546
tatiana pushed a commit that referenced this pull request Sep 3, 2025
@pankajastro
Release 1.0.0 🎉 (#521)
0e34e4c 
### Breaking Changes

- Airflow providers are now optional dependencies by @pankajastro in
[#486](#486)
- Previously, `dag-factory` enforced the installation of
`apache-airflow-providers-http` and
`apache-airflow-providers-cncf-kubernetes`. These Airflow providers
dependencies are now optional. If your DAGs depend on these providers,
you must install them manually. Alternatively, you can install
`dag-factory` with extras like `dag-factory[all]`,
`dag-factory[kubernetes]`, etc.
- Removed `clean_dags` function by @pankajastro in
[#498](#498)
- You no longer need to call `example_dag_factory.clean_dags(globals())`
in your DAG files. DAG cleanup is now controlled via the Airflow config
setting `AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL`.
- Remove `schedule_interval` parameter from DAG configuration YAML by
@viiccwen in [#503](#503)
  - Use `schedule` parameter instead of `schedule_interval`.
- Change `DagFactory` class access to private by @pankajastro in
[#509](#509)
- The import path `from dagfactory import DagFactory` has been removed.
  - The class `DagFactory` has been renamed to `_DagFactory`.
- The `generate_dags` method of `DagFactory` has been renamed to
`_generate_dags`.
- Remove Inconsistent Parameters for Airflow Consistent by @pankajastro
in [#512](#512)
  - Removed `dagrun_timeout_sec` from dag param.
  - Removed `retry_delay_sec`, `sla_secs` from default_args.
  - Removed accepting `execution_timeout` as integer.
- Removed `execution_timeout_secs`, `sla_secs` and
`execution_delta_secs` from task param.
- Remove custom parsing for Kubernetes object and refactor KPO to use
`__type__` syntax by @pankajastro in
[#523](#523)
- The custom parsing for Kubernetes objects has been removed. You can no
longer pass a custom YAML dictionary to DAG-Factory configuration unless
accepted by the KubernetesPodOperator. We suggest you to use `__type__`
syntax to supply Kubernetes object in your YAML DAG. For an example KPO
configuration, visit: [KubernetesPodOperator
Documentation](https://astronomer.github.io/dag-factory/dev/features/kpo/).
- Consolidate `!and`, `!or`, `!join`, `and` and `or` key in YAML DAG
configuration by @pankajastro in
[#525](#525)
  - Use `__and__`, `__or__` and `join__` instead
- Remove custom parsing for DAG parameter `timetable` by @pankajastro in
[#533](#533)
  - Use the `__type__` annotation for the `timetable` parameter.
- Rename parameter of `load_yaml_dags` and `_DagFactory` to reflect
behaviour by @pankajastro in
[#546](#546)
  - Rename `config` to `config_dict`
  - Rename `default_args_config_path` to `defaults_config_path`
  - Rename `default_args_config_dict`  to `defaults_config_dict`

### Added

- Support dag-level arguments in global defaults by @gyli in
[#480](#480)
- Support `*args` in custom Python object by @pankajastro in
[#484](#484)
- Support tasks and task_groups as lists by @pankajkoti in
[#487](#487)
- Support overriding `defaults.yml` based on the directory hierarchy by
@tatiana in [#500](#500)
- Introduced DAG Factory CLI by @tatiana in
[#510](#510)
- Added `lint` command to CLI by @tatiana in
[#513](#513)
- Add convert CLI command to migrate from af2 to af3 by @tatiana in
[539](#539)
- Allow yaml/yml suffix for shared defaults by @pankajastro in
[#567](#567)

### Fixed

- Fix the Airflow version condition check to parse inlets/outlets syntax
according to the dataset by @pankajastro in
[#485](#485)
- Ensure `dag_params` contain `schedule` before operating on it by
@pankajkoti in
[#488](#488)
- Fix `start_date`, `end_date` at the DAG level by @pankajastro in
[#495](#495)
- Allow `execution_timeout` in `default_args` by @pankajastro in
[#501](#501)
- Capture Telemetry DNS gaierror and handle it gracefully by @tatiana in
[#544](#544)
- Fix require response_check param in http sensor by @pankajastri in
[#576](#576)
- Add none check before evaluating lambda func in HttpSensor
@pankajastro in
[#577](#577)

### Docs

- Restore basic DAG example by @pankajastro in
[#483](#483)
- Replace the usages in example dags, tests and docs for tasks and
taskgroups to be list by @pankajkoti in
[#492](#492)
- Update default documentation based on #500 by @tatiana in
[#504](#504)
- Add more examples for Custom Python object by @pankajastro in
[#506](#506)
- Add documentation for DAG Factory CLI by @tatiana in
[#511](#511)
- Add documentation and example YAMLs for task and task_group
configuration formats by @pankajkoti in
[#530](#530)
- Add migration guide docs by @pankajastro in
[#532](#532)
- Docs: Fix rendering of note block by @pankajastro in
[#537](#537)
- Document the Asset example DAG by @pankajastro in
[#538](#538)
- Add docs for the CLI convert command by @tatiana in
[#541](#541)
- Add remaining breaking changes in migration guide by @pankajastro in
[#549](#549)
- Fix typos in scheduling and datasets docs by @viiccwen in
[#565](#565)
- docs: Make markdownlint happy (fix MD007 ul-indent in dev/README.md)
by @viiccwen in
[#566](#566)
- Update Index.md by @pankajastro in
[#570](#570)
- Document dag-factory reserved keys by @pankajastro in
[#571](#571)
- Add an introduction to the migration guide by @pankajastro in
[#572](#572)

### Other Changes

- Improve unit tests to disregard `$AIRFLOW_HOME` by @tatiana in
[#490](#490)
- Resolve unpinned action reference error alerts raised by Zizmor by
@pankajkoti in
[#493](#493)
- Resolve 'credential persistence through GitHub Actions artifacts'
warnings from Zizmor by @pankajkoti in
[#494](#494)
- Resolve 'overly broad permissions' warnings from Zizmor by @pankajkoti
in [#496](#496)
- CI: Add GitHub CodeQL analysis workflow (`codeql.yml`) by @pankajkoti
in [#497](#497)
- Fix deploy pages job missing credentials by @pankajkoti in
[#499](#499)
- Add the breaking changes to changelog by @pankajastro in
[#502](#502)
- Add pre-commit to update `uv.lock` by @pankajastro in
[#514](#514)
- Remove `clean_dags` usage from object storage DAG by @pankajastro in
[#515](#515)
- Remove broad exceptions and catch more specific exceptions by
@pankajastro in
[#519](#519)
- Add missing env in contributing doc by @pankajastro in
[#522](#522)
- Enhance PyPI Stats API error handling by @viiccwen in
[#535](#535)
- Update example to be Airflow 3 compatible by @tatiana in
[540](#540)
- Remove AUTO_CONVERT_TO_AF3 from tests by @tatiana in
[#543](#543)
- Bump actions/download-artifact from 4 to 5 by @dependabot in
[#548](#548),
[#558](#558),
[#559](#559),
[#560](#560),
[#562](#562),
[#653](#563) and
[#575](#575)
- Update pyproject.toml to Sync Test Versions with CI/CD Pipeline by
@viiccwen in [#553](#553)
- Fix file URI format in ObjectStoragePath example to prevent duplicate
slashes by @viiccwen in
[#556](#556)
- CI: Only build docs on PR push event and deploy for merge and release
by @pankajastro in
[#568](#568)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tatiana tatiana tatiana approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Make apache-airflow-providers-cncf-kubernetes an optional dependency

3 participants

@pankajastro @codecov-commenter @tatiana