-
Notifications
You must be signed in to change notification settings - Fork 2.5k
Add a new tutorial which shows how to debug a failing pipeline #16225
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
6810a25
Add a stub for the new debug tutorial
djsauble 86d9241
Update the title to be more similar to the first tutorial
djsauble 3e5c51c
Complete the first draft of the 'Debug a data pipeline' tutorial
djsauble 932d483
Make some small copy improvements to the tutorial
djsauble 10a54de
Merge branch 'main' into debug_flow_runs_tutorial
daniel-prefect d403483
Apply suggestions from code review
daniel-prefect b1c711b
Merge branch 'main' into debug_flow_runs_tutorial
daniel-prefect d7063b5
Update docs/v3/tutorials/debug.mdx
discdiver File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,116 @@ | ||
| --- | ||
| title: Debug a data pipeline | ||
| description: Learn how to troubleshoot flow runs that fail. | ||
| --- | ||
|
|
||
| In the [Set up a platform for data pipelines](/v3/tutorials/platform) tutorial, you used Prefect Cloud to set up a platform for data pipelines. | ||
| In this tutorial, you'll learn what to do when those data pipelines fail. | ||
|
|
||
| <Info> | ||
| This tutorial starts where the [previous tutorial](/v3/tutorials/platform) leaves off, so complete that one first. | ||
| </Info> | ||
|
|
||
| ## Find failures | ||
|
|
||
| You can use the Prefect Cloud dashboard to find failures. | ||
|
|
||
| 1. Sign in to Prefect Cloud | ||
| 1. Use the workspace switcher to open the `staging` workspace that you created in the last tutorial. | ||
| 1. Go to **Home**, and look for red bars in the **Flow Runs** section, these indicate failed flow runs. | ||
| 1. Hover over a red bar to see more details about the flow run: name, deployment, duration, timestamp, and tags. | ||
|
|
||
| <Note> | ||
| You can filter by a specific tag (e.g. `team-a`) if you're only interested in a specific set of flows. | ||
| </Note> | ||
|
|
||
| ## Debug a failure | ||
|
|
||
| A single flow might experience failures on several runs. | ||
| When this happens, it can be helpful to inspect the first failure in the series. | ||
|
|
||
| 1. In the **Flow Runs** section on the **Home** page, expand the `data-pipeline` flow. | ||
| 1. You will see a list of failing `data-pipeline` flow runs, in reverse chronological order. | ||
| 1. Use the pagination controls to navigate to the last failure in the list, this is the first failure that occurred. | ||
| 1. Click the name of the flow run to go to its detail page. | ||
| 1. From the flow run detail page, scroll down to the **Logs** section in the right panel. | ||
| 1. Look for an error message similar to the following: | ||
| + | ||
| ``` | ||
| File "/opt/prefect/demos/simulate_failures.py", line 12, in process_data | ||
| raise Exception(f"Run failed") | ||
| ``` | ||
|
|
||
| It looks like there's an error in the `simulate_failures.py` file. | ||
| Now that you've found the failure, the next step is to fix the underlying code. | ||
|
|
||
| ## Update the code | ||
|
|
||
| Open the `simulate_failures.py` file and look at line 12. | ||
|
|
||
| ```python simulate_failures.py {12} | ||
| from prefect import flow, task | ||
| import argparse | ||
| import asyncio | ||
| from prefect.client.orchestration import get_client | ||
|
|
||
|
daniel-prefect marked this conversation as resolved.
|
||
|
|
||
| @task | ||
| def process_data(run: int, fail_at_run: int | None = None) -> bool: | ||
| """Simulate data processing with failures""" | ||
|
|
||
| # Simulate persistent failures | ||
| if fail_at_run and run > fail_at_run: | ||
| raise Exception(f"Run failed") | ||
|
|
||
| return True | ||
|
|
||
| # ... | ||
| ``` | ||
|
|
||
| The `if` statement is the problem. | ||
| If you specify the `--fail_at_run` flag, once the flow runs more than `fail_at_run` times, the flow fails with an exception. | ||
| Remove the `if` statement to fix this failure. | ||
|
daniel-prefect marked this conversation as resolved.
|
||
| We added this statement to give you something to fix. :) | ||
|
|
||
| ```python simulate_failures.py | ||
| from prefect import flow, task | ||
| import argparse | ||
| import asyncio | ||
| from prefect.client.orchestration import get_client | ||
|
|
||
| @task | ||
| def process_data(run: int, fail_at_run: int | None = None) -> bool: | ||
| """Simulate data processing with failures""" | ||
|
|
||
| return True | ||
|
|
||
| # ... | ||
| ``` | ||
|
|
||
| Now, all flow runs succeed in spite of the `--fail-at-run` flag. | ||
| Deploy the fix to the staging workspace to confirm this new behavior. | ||
|
|
||
| ```bash | ||
| prefect cloud workspace --set "<account>/staging" | ||
| python simulate_failures.py --fail-at-run 3 | ||
| ``` | ||
|
|
||
| After the script finishes, open the **Home** page in Prefect Cloud to verify that the flow run is no longer failing. | ||
|
|
||
| You can now switch workspaces to update the code used in the production workspace as well. | ||
|
|
||
| ```bash | ||
| prefect cloud workspace --set "<account>/production" | ||
| python simulate_failures.py | ||
| ``` | ||
|
|
||
|
|
||
| ## Next steps | ||
|
|
||
| In this tutorial, you successfully used Prefect Cloud to fix a failing data pipeline. | ||
|
|
||
| To take this to the next level, learn how to [set up an alert](/v3/automate/events/automations-triggers) so that you get notified about failures automatically. | ||
|
|
||
| <Tip> | ||
| Need help? [Book a meeting](https://calendly.com/prefect-experts/prefect-product-advocates?utm_campaign=prefect_docs_cloud&utm_content=prefect_docs&utm_medium=docs&utm_source=docs) with a Prefect Product Advocate to get your questions answered. | ||
| </Tip> | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can they skip to this state easily?
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would love to have a Terraform script they can run to provision Prefect Cloud automatically with the requisite workspaces and work pools. I might need some help with this though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ooo. That would be cool. Could add as a linear issue.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
prefect-archive/demos#2