Skip to content

Commit f8a290e

Browse files
Add a new tutorial which shows how to debug a failing pipeline (#16225)
Co-authored-by: Jeff Hale <discdiver@users.noreply.github.com>
1 parent 340e296 commit f8a290e

3 files changed

Lines changed: 120 additions & 1 deletion

File tree

docs/mint.json

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,8 @@
7373
{
7474
"group": "For platform engineers",
7575
"pages": [
76-
"v3/tutorials/platform"
76+
"v3/tutorials/platform",
77+
"v3/tutorials/debug"
7778
]
7879
}
7980
],

docs/v3/tutorials/debug.mdx

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
---
2+
title: Debug a data pipeline
3+
description: Learn how to troubleshoot flow runs that fail.
4+
---
5+
6+
In the [Set up a platform for data pipelines](/v3/tutorials/platform) tutorial, you used Prefect Cloud to set up a platform for data pipelines.
7+
In this tutorial, you'll learn what to do when those data pipelines fail.
8+
9+
<Info>
10+
This tutorial starts where the [previous tutorial](/v3/tutorials/platform) leaves off, so complete that one first.
11+
</Info>
12+
13+
## Find failures
14+
15+
You can use the Prefect Cloud dashboard to find failures.
16+
17+
1. Sign in to Prefect Cloud
18+
1. Use the workspace switcher to open the `staging` workspace that you created in the last tutorial.
19+
1. Go to **Home**, and look for red bars in the **Flow Runs** section, these indicate failed flow runs.
20+
1. Hover over a red bar to see more details about the flow run: name, deployment, duration, timestamp, and tags.
21+
22+
<Note>
23+
You can filter by a specific tag (e.g. `team-a`) if you're only interested in a specific set of flows.
24+
</Note>
25+
26+
## Debug a failure
27+
28+
A single flow might experience failures on several runs.
29+
When this happens, it can be helpful to inspect the first failure in the series.
30+
31+
1. In the **Flow Runs** section on the **Home** page, expand the `data-pipeline` flow.
32+
1. You will see a list of failing `data-pipeline` flow runs, in reverse chronological order.
33+
1. Use the pagination controls to navigate to the last failure in the list, this is the first failure that occurred.
34+
1. Click the name of the flow run to go to its detail page.
35+
1. From the flow run detail page, scroll down to the **Logs** section in the right panel.
36+
1. Look for an error message similar to the following:
37+
+
38+
```
39+
File "/opt/prefect/demos/simulate_failures.py", line 12, in process_data
40+
raise Exception(f"Run failed")
41+
```
42+
43+
It looks like there's an error in the `simulate_failures.py` file.
44+
Now that you've found the failure, the next step is to fix the underlying code.
45+
46+
## Update the code
47+
48+
Open the `simulate_failures.py` file and look at line 12.
49+
50+
```python simulate_failures.py {12}
51+
from prefect import flow, task
52+
import argparse
53+
import asyncio
54+
from prefect.client.orchestration import get_client
55+
56+
57+
@task
58+
def process_data(run: int, fail_at_run: int | None = None) -> bool:
59+
"""Simulate data processing with failures"""
60+
61+
# Simulate persistent failures
62+
if fail_at_run and run > fail_at_run:
63+
raise Exception(f"Run failed")
64+
65+
return True
66+
67+
# ...
68+
```
69+
70+
The `if` statement is the problem.
71+
If you specify the `--fail_at_run` flag, once the flow runs more than `fail_at_run` times, the flow fails with an exception.
72+
Remove the `if` statement to fix this failure.
73+
We added this statement to give you something to fix. :)
74+
75+
```python simulate_failures.py
76+
from prefect import flow, task
77+
import argparse
78+
import asyncio
79+
from prefect.client.orchestration import get_client
80+
81+
@task
82+
def process_data(run: int, fail_at_run: int | None = None) -> bool:
83+
"""Simulate data processing with failures"""
84+
85+
return True
86+
87+
# ...
88+
```
89+
90+
Now, all flow runs succeed in spite of the `--fail-at-run` flag.
91+
Deploy the fix to the staging workspace to confirm this new behavior.
92+
93+
```bash
94+
prefect cloud workspace --set "<account>/staging"
95+
python simulate_failures.py --fail-at-run 3
96+
```
97+
98+
After the script finishes, open the **Home** page in Prefect Cloud to verify that the flow run is no longer failing.
99+
100+
You can now switch workspaces to update the code used in the production workspace as well.
101+
102+
```bash
103+
prefect cloud workspace --set "<account>/production"
104+
python simulate_failures.py
105+
```
106+
107+
108+
## Next steps
109+
110+
In this tutorial, you successfully used Prefect Cloud to fix a failing data pipeline.
111+
112+
To take this to the next level, learn how to [set up an alert](/v3/automate/events/automations-triggers) so that you get notified about failures automatically.
113+
114+
<Tip>
115+
Need help? [Book a meeting](https://calendly.com/prefect-experts/prefect-product-advocates?utm_campaign=prefect_docs_cloud&utm_content=prefect_docs&utm_medium=docs&utm_source=docs) with a Prefect Product Advocate to get your questions answered.
116+
</Tip>

docs/v3/tutorials/platform.mdx

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -188,6 +188,8 @@ If this doesn't perfectly match your use case, here are some variations you can
188188
- You can [write flows from scratch](/v3/develop/write-flows).
189189
- You can [automate deployments with GitHub Actions](/v3/deploy/infrastructure-concepts/deploy-ci-cd).
190190

191+
Next, learn how to [debug a flow run](/v3/tutorials/debug) when things go wrong.
192+
191193
<Tip>
192194
Need help? [Book a meeting](https://calendly.com/prefect-experts/prefect-product-advocates?utm_campaign=prefect_docs_cloud&utm_content=prefect_docs&utm_medium=docs&utm_source=docs) with a Prefect Product Advocate to get your questions answered.
193195
</Tip>

0 commit comments

Comments
 (0)