Skip to content

Support tasks and task_groups as lists - #487

Merged
pankajkoti merged 4 commits into
mainfrom
task-taskgroups-as-list
Jul 24, 2025
Merged

Support tasks and task_groups as lists#487
pankajkoti merged 4 commits into
mainfrom
task-taskgroups-as-list

Conversation

@pankajkoti

@pankajkoti pankajkoti commented Jul 22, 2025

Copy link
Copy Markdown
Contributor

What?

This change adds first-class support for list-based syntax in DAG-Factory YAML:

  1. Tasks as a list
  2. Task Groups as a list

Until now, both sections had to be dictionaries keyed by task_id / group_id. Authors can now opt for a cleaner, order-preserving list format while everything still works in dictionary form.

Why?

The list style is more natural, readable, and aligns with common YAML patterns.

Example before ⬇️

tasks:
  task_1:
    operator: airflow.operators.bash.BashOperator
    bash_command: "echo 1"

Example after ⬇️

tasks:
  - task_id: task_1
    operator: airflow.operators.bash.BashOperator
    bash_command: "echo 1"

Same goes for task_groups:

task_groups:
  - group_name: extract
    tooltip: "extraction steps"

Implementation details

DagBuilder.build() now:
Detects if tasks or task_groups is a list.
Converts each list to the existing dict structure (task_id / group_name key becomes dict key).
Performs validation:

  1. Each entry must be a mapping with the required identifier field.
  2. Duplicate identifiers raise DagFactoryConfigException.
    No downstream code had to change – all later logic still sees a dictionary.

Backwards compatibility

Existing dictionary syntax continues to work unchanged.
Tests covering legacy pattern remain green.

closes: #468

@codecov-commenter

codecov-commenter commented Jul 22, 2025

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.10526% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.70%. Comparing base (ea0ab42) to head (bdbf8df).
⚠️ Report is 127 commits behind head on main.

Files with missing lines Patch % Lines
dagfactory/dagbuilder.py 92.10% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #487      +/-   ##
==========================================
- Coverage   93.76%   93.70%   -0.06%     
==========================================
  Files          11       11              
  Lines        1042     1080      +38     
==========================================
+ Hits          977     1012      +35     
- Misses         65       68       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@pankajkoti
pankajkoti marked this pull request as ready for review July 22, 2025 10:04
@pankajkoti
pankajkoti requested a review from a team as a code owner July 22, 2025 10:04
Comment thread dagfactory/dagbuilder.py Outdated
Comment thread tests/test_tasks_and_task_groups_list.py Outdated
@pankajkoti

Copy link
Copy Markdown
Contributor Author

I have logged a follow-up ticket: #491 to update example DAGs and docs to reflect authoring tasks as list. Will work on a quick follow-up PR to do that once this change is avaialble and merged. Keeping the scope of this PR limited to ease reviews.

@tatiana tatiana left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a very exciting feature, and it's great it is backwards compatible.
I left an implementation feedback inline, but happy with the feature!

Comment thread dagfactory/dagbuilder.py Outdated
@pankajkoti
pankajkoti force-pushed the task-taskgroups-as-list branch from e0457d2 to bdbf8df Compare July 24, 2025 11:26
@pankajkoti
pankajkoti merged commit f35955f into main Jul 24, 2025
63 checks passed
@pankajkoti
pankajkoti deleted the task-taskgroups-as-list branch July 24, 2025 11:37
@pankajastro pankajastro mentioned this pull request Jul 29, 2025
tatiana pushed a commit that referenced this pull request Jul 31, 2025
…groups to be list (#492)

Following up on PR #487, this PR demonstrates and replaces the usages of
tasks and task groups across all example dags, tests and static yamls in
docs to be a list.

closes: #491
pankajkoti added a commit that referenced this pull request Aug 1, 2025
…tion formats (#530)

This PR enhances documentation to clearly show both supported formats
(list & dictionary) for defining tasks and task groups in DAG Factory
configurations, helping users understand their options and migrate
toward the recommended list-based approach.

We already have tests covering both list and dictionary-based tasks &
taskgroups being added previously in #487 and #491, hence the PR solely
focuses on adding example YAMLs and reflecting those in docs.

Docs preview:
<img width="767" height="1141" alt="Screenshot 2025-08-01 at 1 43 57 AM"
src="https://github.com/user-attachments/assets/8e970dee-d056-46bd-b572-ca54fc9b3f27"
/>



closes: #529
pankajastro added a commit that referenced this pull request Aug 1, 2025
Following up on PR #487, this PR demonstrates and replaces the usages of
tasks and task groups across all example dags, tests and static yamls in
docs to be a list.

closes: #491

Fix yml
pankajastro added a commit that referenced this pull request Aug 1, 2025
Following up on PR #487, this PR demonstrates and replaces the usages of
tasks and task groups across all example dags, tests and static yamls in
docs to be a list.

closes: #491

Fix yml
pankajastro added a commit that referenced this pull request Aug 4, 2025
Following up on PR #487, this PR demonstrates and replaces the usages of
tasks and task groups across all example dags, tests and static yamls in
docs to be a list.

closes: #491

Fix yml
tatiana pushed a commit that referenced this pull request Sep 3, 2025
### Breaking Changes

- Airflow providers are now optional dependencies by @pankajastro in
[#486](#486)
- Previously, `dag-factory` enforced the installation of
`apache-airflow-providers-http` and
`apache-airflow-providers-cncf-kubernetes`. These Airflow providers
dependencies are now optional. If your DAGs depend on these providers,
you must install them manually. Alternatively, you can install
`dag-factory` with extras like `dag-factory[all]`,
`dag-factory[kubernetes]`, etc.
- Removed `clean_dags` function by @pankajastro in
[#498](#498)
- You no longer need to call `example_dag_factory.clean_dags(globals())`
in your DAG files. DAG cleanup is now controlled via the Airflow config
setting `AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL`.
- Remove `schedule_interval` parameter from DAG configuration YAML by
@viiccwen in [#503](#503)
  - Use `schedule` parameter instead of `schedule_interval`.
- Change `DagFactory` class access to private by @pankajastro in
[#509](#509)
- The import path `from dagfactory import DagFactory` has been removed.
  - The class `DagFactory` has been renamed to `_DagFactory`.
- The `generate_dags` method of `DagFactory` has been renamed to
`_generate_dags`.
- Remove Inconsistent Parameters for Airflow Consistent by @pankajastro
in [#512](#512)
  - Removed `dagrun_timeout_sec` from dag param.
  - Removed `retry_delay_sec`, `sla_secs` from default_args.
  - Removed accepting `execution_timeout` as integer.
- Removed `execution_timeout_secs`, `sla_secs` and
`execution_delta_secs` from task param.
- Remove custom parsing for Kubernetes object and refactor KPO to use
`__type__` syntax by @pankajastro in
[#523](#523)
- The custom parsing for Kubernetes objects has been removed. You can no
longer pass a custom YAML dictionary to DAG-Factory configuration unless
accepted by the KubernetesPodOperator. We suggest you to use `__type__`
syntax to supply Kubernetes object in your YAML DAG. For an example KPO
configuration, visit: [KubernetesPodOperator
Documentation](https://astronomer.github.io/dag-factory/dev/features/kpo/).
- Consolidate `!and`, `!or`, `!join`, `and` and `or` key in YAML DAG
configuration by @pankajastro in
[#525](#525)
  - Use `__and__`, `__or__` and `join__` instead
- Remove custom parsing for DAG parameter `timetable` by @pankajastro in
[#533](#533)
  - Use the `__type__` annotation for the `timetable` parameter.
- Rename parameter of `load_yaml_dags` and `_DagFactory` to reflect
behaviour by @pankajastro in
[#546](#546)
  - Rename `config` to `config_dict`
  - Rename `default_args_config_path` to `defaults_config_path`
  - Rename `default_args_config_dict`  to `defaults_config_dict`

### Added

- Support dag-level arguments in global defaults by @gyli in
[#480](#480)
- Support `*args` in custom Python object by @pankajastro in
[#484](#484)
- Support tasks and task_groups as lists by @pankajkoti in
[#487](#487)
- Support overriding `defaults.yml` based on the directory hierarchy by
@tatiana in [#500](#500)
- Introduced DAG Factory CLI by @tatiana in
[#510](#510)
- Added `lint` command to CLI by @tatiana in
[#513](#513)
- Add convert CLI command to migrate from af2 to af3 by @tatiana in
[539](#539)
- Allow yaml/yml suffix for shared defaults by @pankajastro in
[#567](#567)

### Fixed

- Fix the Airflow version condition check to parse inlets/outlets syntax
according to the dataset by @pankajastro in
[#485](#485)
- Ensure `dag_params` contain `schedule` before operating on it by
@pankajkoti in
[#488](#488)
- Fix `start_date`, `end_date` at the DAG level by @pankajastro in
[#495](#495)
- Allow `execution_timeout` in `default_args` by @pankajastro in
[#501](#501)
- Capture Telemetry DNS gaierror and handle it gracefully by @tatiana in
[#544](#544)
- Fix require response_check param in http sensor by @pankajastri in
[#576](#576)
- Add none check before evaluating lambda func in HttpSensor
@pankajastro in
[#577](#577)

### Docs

- Restore basic DAG example by @pankajastro in
[#483](#483)
- Replace the usages in example dags, tests and docs for tasks and
taskgroups to be list by @pankajkoti in
[#492](#492)
- Update default documentation based on #500 by @tatiana in
[#504](#504)
- Add more examples for Custom Python object by @pankajastro in
[#506](#506)
- Add documentation for DAG Factory CLI by @tatiana in
[#511](#511)
- Add documentation and example YAMLs for task and task_group
configuration formats by @pankajkoti in
[#530](#530)
- Add migration guide docs by @pankajastro in
[#532](#532)
- Docs: Fix rendering of note block by @pankajastro in
[#537](#537)
- Document the Asset example DAG by @pankajastro in
[#538](#538)
- Add docs for the CLI convert command by @tatiana in
[#541](#541)
- Add remaining breaking changes in migration guide by @pankajastro in
[#549](#549)
- Fix typos in scheduling and datasets docs by @viiccwen in
[#565](#565)
- docs: Make markdownlint happy (fix MD007 ul-indent in dev/README.md)
by @viiccwen in
[#566](#566)
- Update Index.md by @pankajastro in
[#570](#570)
- Document dag-factory reserved keys by @pankajastro in
[#571](#571)
- Add an introduction to the migration guide by @pankajastro in
[#572](#572)

### Other Changes

- Improve unit tests to disregard `$AIRFLOW_HOME` by @tatiana in
[#490](#490)
- Resolve unpinned action reference error alerts raised by Zizmor by
@pankajkoti in
[#493](#493)
- Resolve 'credential persistence through GitHub Actions artifacts'
warnings from Zizmor by @pankajkoti in
[#494](#494)
- Resolve 'overly broad permissions' warnings from Zizmor by @pankajkoti
in [#496](#496)
- CI: Add GitHub CodeQL analysis workflow (`codeql.yml`) by @pankajkoti
in [#497](#497)
- Fix deploy pages job missing credentials by @pankajkoti in
[#499](#499)
- Add the breaking changes to changelog by @pankajastro in
[#502](#502)
- Add pre-commit to update `uv.lock` by @pankajastro in
[#514](#514)
- Remove `clean_dags` usage from object storage DAG by @pankajastro in
[#515](#515)
- Remove broad exceptions and catch more specific exceptions by
@pankajastro in
[#519](#519)
- Add missing env in contributing doc by @pankajastro in
[#522](#522)
- Enhance PyPI Stats API error handling by @viiccwen in
[#535](#535)
- Update example to be Airflow 3 compatible by @tatiana in
[540](#540)
- Remove AUTO_CONVERT_TO_AF3 from tests by @tatiana in
[#543](#543)
- Bump actions/download-artifact from 4 to 5 by @dependabot in
[#548](#548),
[#558](#558),
[#559](#559),
[#560](#560),
[#562](#562),
[#653](#563) and
[#575](#575)
- Update pyproject.toml to Sync Test Versions with CI/CD Pipeline by
@viiccwen in [#553](#553)
- Fix file URI format in ObjectStoragePath example to prevent duplicate
slashes by @viiccwen in
[#556](#556)
- CI: Only build docs on PR push event and deploy for merge and release
by @pankajastro in
[#568](#568)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make tasks a list

3 participants