Support tasks and task_groups as lists - #487
Merged
Merged
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #487 +/- ##
==========================================
- Coverage 93.76% 93.70% -0.06%
==========================================
Files 11 11
Lines 1042 1080 +38
==========================================
+ Hits 977 1012 +35
- Misses 65 68 +3 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
pankajkoti
commented
Jul 22, 2025
Contributor
Author
|
I have logged a follow-up ticket: #491 to update example DAGs and docs to reflect authoring tasks as list. Will work on a quick follow-up PR to do that once this change is avaialble and merged. Keeping the scope of this PR limited to ease reviews. |
tatiana
approved these changes
Jul 24, 2025
tatiana
left a comment
Collaborator
There was a problem hiding this comment.
This is a very exciting feature, and it's great it is backwards compatible.
I left an implementation feedback inline, but happy with the feature!
pankajkoti
force-pushed
the
task-taskgroups-as-list
branch
from
July 24, 2025 11:26
e0457d2 to
bdbf8df
Compare
This was referenced Jul 24, 2025
Merged
This was referenced Jul 31, 2025
pankajkoti
added a commit
that referenced
this pull request
Aug 1, 2025
…tion formats (#530) This PR enhances documentation to clearly show both supported formats (list & dictionary) for defining tasks and task groups in DAG Factory configurations, helping users understand their options and migrate toward the recommended list-based approach. We already have tests covering both list and dictionary-based tasks & taskgroups being added previously in #487 and #491, hence the PR solely focuses on adding example YAMLs and reflecting those in docs. Docs preview: <img width="767" height="1141" alt="Screenshot 2025-08-01 at 1 43 57 AM" src="https://github.com/user-attachments/assets/8e970dee-d056-46bd-b572-ca54fc9b3f27" /> closes: #529
tatiana
pushed a commit
that referenced
this pull request
Sep 3, 2025
### Breaking Changes - Airflow providers are now optional dependencies by @pankajastro in [#486](#486) - Previously, `dag-factory` enforced the installation of `apache-airflow-providers-http` and `apache-airflow-providers-cncf-kubernetes`. These Airflow providers dependencies are now optional. If your DAGs depend on these providers, you must install them manually. Alternatively, you can install `dag-factory` with extras like `dag-factory[all]`, `dag-factory[kubernetes]`, etc. - Removed `clean_dags` function by @pankajastro in [#498](#498) - You no longer need to call `example_dag_factory.clean_dags(globals())` in your DAG files. DAG cleanup is now controlled via the Airflow config setting `AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL`. - Remove `schedule_interval` parameter from DAG configuration YAML by @viiccwen in [#503](#503) - Use `schedule` parameter instead of `schedule_interval`. - Change `DagFactory` class access to private by @pankajastro in [#509](#509) - The import path `from dagfactory import DagFactory` has been removed. - The class `DagFactory` has been renamed to `_DagFactory`. - The `generate_dags` method of `DagFactory` has been renamed to `_generate_dags`. - Remove Inconsistent Parameters for Airflow Consistent by @pankajastro in [#512](#512) - Removed `dagrun_timeout_sec` from dag param. - Removed `retry_delay_sec`, `sla_secs` from default_args. - Removed accepting `execution_timeout` as integer. - Removed `execution_timeout_secs`, `sla_secs` and `execution_delta_secs` from task param. - Remove custom parsing for Kubernetes object and refactor KPO to use `__type__` syntax by @pankajastro in [#523](#523) - The custom parsing for Kubernetes objects has been removed. You can no longer pass a custom YAML dictionary to DAG-Factory configuration unless accepted by the KubernetesPodOperator. We suggest you to use `__type__` syntax to supply Kubernetes object in your YAML DAG. For an example KPO configuration, visit: [KubernetesPodOperator Documentation](https://astronomer.github.io/dag-factory/dev/features/kpo/). - Consolidate `!and`, `!or`, `!join`, `and` and `or` key in YAML DAG configuration by @pankajastro in [#525](#525) - Use `__and__`, `__or__` and `join__` instead - Remove custom parsing for DAG parameter `timetable` by @pankajastro in [#533](#533) - Use the `__type__` annotation for the `timetable` parameter. - Rename parameter of `load_yaml_dags` and `_DagFactory` to reflect behaviour by @pankajastro in [#546](#546) - Rename `config` to `config_dict` - Rename `default_args_config_path` to `defaults_config_path` - Rename `default_args_config_dict` to `defaults_config_dict` ### Added - Support dag-level arguments in global defaults by @gyli in [#480](#480) - Support `*args` in custom Python object by @pankajastro in [#484](#484) - Support tasks and task_groups as lists by @pankajkoti in [#487](#487) - Support overriding `defaults.yml` based on the directory hierarchy by @tatiana in [#500](#500) - Introduced DAG Factory CLI by @tatiana in [#510](#510) - Added `lint` command to CLI by @tatiana in [#513](#513) - Add convert CLI command to migrate from af2 to af3 by @tatiana in [539](#539) - Allow yaml/yml suffix for shared defaults by @pankajastro in [#567](#567) ### Fixed - Fix the Airflow version condition check to parse inlets/outlets syntax according to the dataset by @pankajastro in [#485](#485) - Ensure `dag_params` contain `schedule` before operating on it by @pankajkoti in [#488](#488) - Fix `start_date`, `end_date` at the DAG level by @pankajastro in [#495](#495) - Allow `execution_timeout` in `default_args` by @pankajastro in [#501](#501) - Capture Telemetry DNS gaierror and handle it gracefully by @tatiana in [#544](#544) - Fix require response_check param in http sensor by @pankajastri in [#576](#576) - Add none check before evaluating lambda func in HttpSensor @pankajastro in [#577](#577) ### Docs - Restore basic DAG example by @pankajastro in [#483](#483) - Replace the usages in example dags, tests and docs for tasks and taskgroups to be list by @pankajkoti in [#492](#492) - Update default documentation based on #500 by @tatiana in [#504](#504) - Add more examples for Custom Python object by @pankajastro in [#506](#506) - Add documentation for DAG Factory CLI by @tatiana in [#511](#511) - Add documentation and example YAMLs for task and task_group configuration formats by @pankajkoti in [#530](#530) - Add migration guide docs by @pankajastro in [#532](#532) - Docs: Fix rendering of note block by @pankajastro in [#537](#537) - Document the Asset example DAG by @pankajastro in [#538](#538) - Add docs for the CLI convert command by @tatiana in [#541](#541) - Add remaining breaking changes in migration guide by @pankajastro in [#549](#549) - Fix typos in scheduling and datasets docs by @viiccwen in [#565](#565) - docs: Make markdownlint happy (fix MD007 ul-indent in dev/README.md) by @viiccwen in [#566](#566) - Update Index.md by @pankajastro in [#570](#570) - Document dag-factory reserved keys by @pankajastro in [#571](#571) - Add an introduction to the migration guide by @pankajastro in [#572](#572) ### Other Changes - Improve unit tests to disregard `$AIRFLOW_HOME` by @tatiana in [#490](#490) - Resolve unpinned action reference error alerts raised by Zizmor by @pankajkoti in [#493](#493) - Resolve 'credential persistence through GitHub Actions artifacts' warnings from Zizmor by @pankajkoti in [#494](#494) - Resolve 'overly broad permissions' warnings from Zizmor by @pankajkoti in [#496](#496) - CI: Add GitHub CodeQL analysis workflow (`codeql.yml`) by @pankajkoti in [#497](#497) - Fix deploy pages job missing credentials by @pankajkoti in [#499](#499) - Add the breaking changes to changelog by @pankajastro in [#502](#502) - Add pre-commit to update `uv.lock` by @pankajastro in [#514](#514) - Remove `clean_dags` usage from object storage DAG by @pankajastro in [#515](#515) - Remove broad exceptions and catch more specific exceptions by @pankajastro in [#519](#519) - Add missing env in contributing doc by @pankajastro in [#522](#522) - Enhance PyPI Stats API error handling by @viiccwen in [#535](#535) - Update example to be Airflow 3 compatible by @tatiana in [540](#540) - Remove AUTO_CONVERT_TO_AF3 from tests by @tatiana in [#543](#543) - Bump actions/download-artifact from 4 to 5 by @dependabot in [#548](#548), [#558](#558), [#559](#559), [#560](#560), [#562](#562), [#653](#563) and [#575](#575) - Update pyproject.toml to Sync Test Versions with CI/CD Pipeline by @viiccwen in [#553](#553) - Fix file URI format in ObjectStoragePath example to prevent duplicate slashes by @viiccwen in [#556](#556) - CI: Only build docs on PR push event and deploy for merge and release by @pankajastro in [#568](#568)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What?
This change adds first-class support for list-based syntax in DAG-Factory YAML:
Until now, both sections had to be dictionaries keyed by task_id / group_id. Authors can now opt for a cleaner, order-preserving list format while everything still works in dictionary form.
Why?
The list style is more natural, readable, and aligns with common YAML patterns.
Example before ⬇️
Example after ⬇️
Same goes for
task_groups:Implementation details
DagBuilder.build()now:Detects if
tasksortask_groupsis a list.Converts each list to the existing dict structure (task_id / group_name key becomes dict key).
Performs validation:
No downstream code had to change – all later logic still sees a dictionary.
Backwards compatibility
Existing dictionary syntax continues to work unchanged.
Tests covering legacy pattern remain green.
closes: #468