Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/mint.json
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,8 @@
{
"group": "For platform engineers",
"pages": [
"v3/tutorials/platform"
"v3/tutorials/platform",
"v3/tutorials/debug"
]
}
],
Expand Down
116 changes: 116 additions & 0 deletions docs/v3/tutorials/debug.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
---
title: Debug a data pipeline
description: Learn how to troubleshoot flow runs that fail.
---

In the [Set up a platform for data pipelines](/v3/tutorials/platform) tutorial, you used Prefect Cloud to set up a platform for data pipelines.
In this tutorial, you'll learn what to do when those data pipelines fail.

<Info>
This tutorial starts where the [previous tutorial](/v3/tutorials/platform) leaves off, so complete that one first.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can they skip to this state easily?

@daniel-prefect daniel-prefect Dec 9, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would love to have a Terraform script they can run to provision Prefect Cloud automatically with the requisite workspaces and work pools. I might need some help with this though.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ooo. That would be cool. Could add as a linear issue.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

</Info>

## Find failures

You can use the Prefect Cloud dashboard to find failures.

1. Sign in to Prefect Cloud
1. Use the workspace switcher to open the `staging` workspace that you created in the last tutorial.
1. Go to **Home**, and look for red bars in the **Flow Runs** section, these indicate failed flow runs.
1. Hover over a red bar to see more details about the flow run: name, deployment, duration, timestamp, and tags.

<Note>
You can filter by a specific tag (e.g. `team-a`) if you're only interested in a specific set of flows.
</Note>

## Debug a failure

A single flow might experience failures on several runs.
When this happens, it can be helpful to inspect the first failure in the series.

1. In the **Flow Runs** section on the **Home** page, expand the `data-pipeline` flow.
1. You will see a list of failing `data-pipeline` flow runs, in reverse chronological order.
1. Use the pagination controls to navigate to the last failure in the list, this is the first failure that occurred.
1. Click the name of the flow run to go to its detail page.
1. From the flow run detail page, scroll down to the **Logs** section in the right panel.
1. Look for an error message similar to the following:
+
```
File "/opt/prefect/demos/simulate_failures.py", line 12, in process_data
raise Exception(f"Run failed")
```

It looks like there's an error in the `simulate_failures.py` file.
Now that you've found the failure, the next step is to fix the underlying code.

## Update the code

Open the `simulate_failures.py` file and look at line 12.

```python simulate_failures.py {12}
from prefect import flow, task
import argparse
import asyncio
from prefect.client.orchestration import get_client

Comment thread
daniel-prefect marked this conversation as resolved.

@task
def process_data(run: int, fail_at_run: int | None = None) -> bool:
"""Simulate data processing with failures"""

# Simulate persistent failures
if fail_at_run and run > fail_at_run:
raise Exception(f"Run failed")

return True

# ...
```

The `if` statement is the problem.
If you specify the `--fail_at_run` flag, once the flow runs more than `fail_at_run` times, the flow fails with an exception.
Remove the `if` statement to fix this failure.
Comment thread
daniel-prefect marked this conversation as resolved.
We added this statement to give you something to fix. :)

```python simulate_failures.py
from prefect import flow, task
import argparse
import asyncio
from prefect.client.orchestration import get_client

@task
def process_data(run: int, fail_at_run: int | None = None) -> bool:
"""Simulate data processing with failures"""

return True

# ...
```

Now, all flow runs succeed in spite of the `--fail-at-run` flag.
Deploy the fix to the staging workspace to confirm this new behavior.

```bash
prefect cloud workspace --set "<account>/staging"
python simulate_failures.py --fail-at-run 3
```

After the script finishes, open the **Home** page in Prefect Cloud to verify that the flow run is no longer failing.

You can now switch workspaces to update the code used in the production workspace as well.

```bash
prefect cloud workspace --set "<account>/production"
python simulate_failures.py
```


## Next steps

In this tutorial, you successfully used Prefect Cloud to fix a failing data pipeline.

To take this to the next level, learn how to [set up an alert](/v3/automate/events/automations-triggers) so that you get notified about failures automatically.

<Tip>
Need help? [Book a meeting](https://calendly.com/prefect-experts/prefect-product-advocates?utm_campaign=prefect_docs_cloud&utm_content=prefect_docs&utm_medium=docs&utm_source=docs) with a Prefect Product Advocate to get your questions answered.
</Tip>
2 changes: 2 additions & 0 deletions docs/v3/tutorials/platform.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,8 @@ If this doesn't perfectly match your use case, here are some variations you can
- You can [write flows from scratch](/v3/develop/write-flows).
- You can [automate deployments with GitHub Actions](/v3/deploy/infrastructure-concepts/deploy-ci-cd).

Next, learn how to [debug a flow run](/v3/tutorials/debug) when things go wrong.

<Tip>
Need help? [Book a meeting](https://calendly.com/prefect-experts/prefect-product-advocates?utm_campaign=prefect_docs_cloud&utm_content=prefect_docs&utm_medium=docs&utm_source=docs) with a Prefect Product Advocate to get your questions answered.
</Tip>