|
| 1 | +--- |
| 2 | +title: Debug a data pipeline |
| 3 | +description: Learn how to troubleshoot flow runs that fail. |
| 4 | +--- |
| 5 | + |
| 6 | +In the [Set up a platform for data pipelines](/v3/tutorials/platform) tutorial, you used Prefect Cloud to set up a platform for data pipelines. |
| 7 | +In this tutorial, you'll learn what to do when those data pipelines fail. |
| 8 | + |
| 9 | +<Info> |
| 10 | +This tutorial starts where the [previous tutorial](/v3/tutorials/platform) leaves off, so complete that one first. |
| 11 | +</Info> |
| 12 | + |
| 13 | +## Find failures |
| 14 | + |
| 15 | +You can use the Prefect Cloud dashboard to find failures. |
| 16 | + |
| 17 | +1. Sign in to Prefect Cloud |
| 18 | +1. Use the workspace switcher to open the `staging` workspace that you created in the last tutorial. |
| 19 | +1. Go to **Home**, and look for red bars in the **Flow Runs** section, these indicate failed flow runs. |
| 20 | +1. Hover over a red bar to see more details about the flow run: name, deployment, duration, timestamp, and tags. |
| 21 | + |
| 22 | +<Note> |
| 23 | +You can filter by a specific tag (e.g. `team-a`) if you're only interested in a specific set of flows. |
| 24 | +</Note> |
| 25 | + |
| 26 | +## Debug a failure |
| 27 | + |
| 28 | +A single flow might experience failures on several runs. |
| 29 | +When this happens, it can be helpful to inspect the first failure in the series. |
| 30 | + |
| 31 | +1. In the **Flow Runs** section on the **Home** page, expand the `data-pipeline` flow. |
| 32 | +1. You will see a list of failing `data-pipeline` flow runs, in reverse chronological order. |
| 33 | +1. Use the pagination controls to navigate to the last failure in the list, this is the first failure that occurred. |
| 34 | +1. Click the name of the flow run to go to its detail page. |
| 35 | +1. From the flow run detail page, scroll down to the **Logs** section in the right panel. |
| 36 | +1. Look for an error message similar to the following: |
| 37 | ++ |
| 38 | +``` |
| 39 | +File "/opt/prefect/demos/simulate_failures.py", line 12, in process_data |
| 40 | + raise Exception(f"Run failed") |
| 41 | +``` |
| 42 | + |
| 43 | +It looks like there's an error in the `simulate_failures.py` file. |
| 44 | +Now that you've found the failure, the next step is to fix the underlying code. |
| 45 | + |
| 46 | +## Update the code |
| 47 | + |
| 48 | +Open the `simulate_failures.py` file and look at line 12. |
| 49 | + |
| 50 | +```python simulate_failures.py {12} |
| 51 | +from prefect import flow, task |
| 52 | +import argparse |
| 53 | +import asyncio |
| 54 | +from prefect.client.orchestration import get_client |
| 55 | + |
| 56 | + |
| 57 | +@task |
| 58 | +def process_data(run: int, fail_at_run: int | None = None) -> bool: |
| 59 | + """Simulate data processing with failures""" |
| 60 | + |
| 61 | + # Simulate persistent failures |
| 62 | + if fail_at_run and run > fail_at_run: |
| 63 | + raise Exception(f"Run failed") |
| 64 | + |
| 65 | + return True |
| 66 | + |
| 67 | +# ... |
| 68 | +``` |
| 69 | + |
| 70 | +The `if` statement is the problem. |
| 71 | +If you specify the `--fail_at_run` flag, once the flow runs more than `fail_at_run` times, the flow fails with an exception. |
| 72 | +Remove the `if` statement to fix this failure. |
| 73 | +We added this statement to give you something to fix. :) |
| 74 | + |
| 75 | +```python simulate_failures.py |
| 76 | +from prefect import flow, task |
| 77 | +import argparse |
| 78 | +import asyncio |
| 79 | +from prefect.client.orchestration import get_client |
| 80 | + |
| 81 | +@task |
| 82 | +def process_data(run: int, fail_at_run: int | None = None) -> bool: |
| 83 | + """Simulate data processing with failures""" |
| 84 | + |
| 85 | + return True |
| 86 | + |
| 87 | +# ... |
| 88 | +``` |
| 89 | + |
| 90 | +Now, all flow runs succeed in spite of the `--fail-at-run` flag. |
| 91 | +Deploy the fix to the staging workspace to confirm this new behavior. |
| 92 | + |
| 93 | +```bash |
| 94 | +prefect cloud workspace --set "<account>/staging" |
| 95 | +python simulate_failures.py --fail-at-run 3 |
| 96 | +``` |
| 97 | + |
| 98 | +After the script finishes, open the **Home** page in Prefect Cloud to verify that the flow run is no longer failing. |
| 99 | + |
| 100 | +You can now switch workspaces to update the code used in the production workspace as well. |
| 101 | + |
| 102 | +```bash |
| 103 | +prefect cloud workspace --set "<account>/production" |
| 104 | +python simulate_failures.py |
| 105 | +``` |
| 106 | + |
| 107 | + |
| 108 | +## Next steps |
| 109 | + |
| 110 | +In this tutorial, you successfully used Prefect Cloud to fix a failing data pipeline. |
| 111 | + |
| 112 | +To take this to the next level, learn how to [set up an alert](/v3/automate/events/automations-triggers) so that you get notified about failures automatically. |
| 113 | + |
| 114 | +<Tip> |
| 115 | +Need help? [Book a meeting](https://calendly.com/prefect-experts/prefect-product-advocates?utm_campaign=prefect_docs_cloud&utm_content=prefect_docs&utm_medium=docs&utm_source=docs) with a Prefect Product Advocate to get your questions answered. |
| 116 | +</Tip> |
0 commit comments