Skip to content

Commit 46a6605

Browse files
authored
Add documentation and example YAMLs for task and task_group configuration formats (#530)
This PR enhances documentation to clearly show both supported formats (list & dictionary) for defining tasks and task groups in DAG Factory configurations, helping users understand their options and migrate toward the recommended list-based approach. We already have tests covering both list and dictionary-based tasks & taskgroups being added previously in #487 and #491, hence the PR solely focuses on adding example YAMLs and reflecting those in docs. Docs preview: <img width="767" height="1141" alt="Screenshot 2025-08-01 at 1 43 57 AM" src="https://github.com/user-attachments/assets/8e970dee-d056-46bd-b572-ca54fc9b3f27" /> closes: #529
1 parent ae69f1e commit 46a6605

3 files changed

Lines changed: 60 additions & 2 deletions

File tree

dev/dags/example_dag_factory.yml

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,10 @@ basic_example_dag:
1010
schedule_interval: "0 3 * * *"
1111
render_template_as_native_obj: True
1212
catchup: false
13+
task_groups:
14+
- group_name: "example_task_group"
15+
tooltip: "this is an example task group"
16+
dependencies: [task_1]
1317
tasks:
1418
- task_id: "task_1"
1519
operator: airflow.operators.bash.BashOperator
@@ -20,6 +24,7 @@ basic_example_dag:
2024
dependencies: [task_1]
2125
- task_id: "task_3"
2226
operator: airflow.operators.bash.BashOperator
23-
bash_command: "echo 2"
27+
bash_command: "echo 3"
2428
dependencies: [task_1]
29+
task_group_name: "example_task_group"
2530
# ----8<--- [ end: example_dag_yaml_configuration ]
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
default:
2+
default_args:
3+
start_date: 2024-11-11
4+
5+
# ----8<--- [ start: example_dag_dict_configuration ]
6+
basic_example_dag_dict_format:
7+
default_args:
8+
owner: "custom_owner"
9+
description: "this is an example dag using dictionary format for tasks and task_groups"
10+
schedule_interval: "0 3 * * *"
11+
render_template_as_native_obj: True
12+
catchup: false
13+
task_groups:
14+
example_task_group:
15+
tooltip: "this is an example task group"
16+
dependencies: [task_1]
17+
tasks:
18+
task_1:
19+
operator: airflow.operators.bash.BashOperator
20+
bash_command: "echo 1"
21+
task_2:
22+
operator: airflow.operators.bash.BashOperator
23+
bash_command: "echo 2"
24+
dependencies: [task_1]
25+
task_3:
26+
operator: airflow.operators.bash.BashOperator
27+
bash_command: "echo 3"
28+
dependencies: [task_1]
29+
task_group_name: example_task_group
30+
# ----8<--- [ end: example_dag_dict_configuration ]

docs/configuration/configuring_workflows.md

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,13 +9,36 @@ You can define multiple workflows within a single YAML file based on your requir
99
- **default_args**: Common arguments for all tasks.
1010
- **schedule**/**schedule_interval**: Specifies the execution schedule.
1111
- **tasks**: Defines the [Airflow tasks](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html) in your workflow.
12+
- **task_groups**: Defines [Airflow task groups](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html#taskgroups) to organize and group related tasks.
1213

13-
### Example DAG Configuration
14+
## Example DAG Configuration
15+
16+
### Task and Task Group Configuration Formats
17+
18+
DAG Factory supports two formats for defining `tasks` and `task_groups`:
19+
20+
#### List Format (Recommended)
21+
22+
The **list format** is the recommended and more readable approach. In this format, tasks are defined as a list where each task includes a `task_id` field, and task groups are also defined as a list where each group includes a `group_name` field:
23+
24+
!!! info "Version Support"
25+
List format support was introduced in version 1.0.0.
1426

1527
```title="example_dag_factory.yml"
1628
--8<-- "dev/dags/example_dag_factory.yml:example_dag_yaml_configuration"
1729
```
1830

31+
#### Dictionary Format (Legacy)
32+
33+
The **dictionary format** is also supported for backward compatibility. In this format, tasks are defined as a dictionary where the key is the task ID, and task groups are also defined as a dictionary where the key is the group name:
34+
35+
```title="example_dag_factory_tasks_taskgroups_as_dict_format.yml"
36+
--8<-- "dev/dags/example_dag_factory_tasks_taskgroups_as_dict_format.yml:example_dag_dict_configuration"
37+
```
38+
39+
!!! note "Format Recommendation"
40+
While both formats are supported, **we recommend using the list format** as it is more readable and easier to maintain.
41+
1942
### Check out more configuration params
2043

2144
- [Environment variables](environment_variables.md)

0 commit comments

Comments
 (0)