Automatically provision a Prefect Cloud account with resources for the debug tutorial - #2
Automatically provision a Prefect Cloud account with resources for the debug tutorial#2daniel-prefect wants to merge 21 commits into
Conversation
…or the debugging tutorial
|
We'll need to update the debug tutorial to include instructions for starting workers (since this script automatically kills those once the initial runs are complete). |
…ince it's not needed
EmilRex
left a comment
EmilRex
left a comment
There was a problem hiding this comment.
@daniel-prefect here are a few high level comments/questions.
| @@ -0,0 +1,32 @@ | |||
| #!/bin/bash | |||
There was a problem hiding this comment.
What do you think about converting this script to Python? We can safely assume our users are familiar with Python whereas bash syntax might be new. You can then also use Prefect's built in conveniences for things like getting the active profile.
There was a problem hiding this comment.
I'm down with converting this to Python if it simplifies the script. I started with Terraform, and then wrapped it in a bash script when I realized it didn't do everything I needed.
| @@ -0,0 +1,38 @@ | |||
| terraform { | |||
| # Check if Python is installed and determine the Python command | ||
| echo "🐍 Checking if Python is installed..." | ||
| if command -v python3 &> /dev/null; then | ||
| PYTHON_CMD="python3" | ||
| elif command -v python &> /dev/null; then | ||
| PYTHON_CMD="python" | ||
| else | ||
| echo "❌ Error: Python is not installed. Please install Python 3.9 or higher and try again." | ||
| exit 1 | ||
| fi | ||
|
|
||
| # Verify Python version is 3.9 or higher | ||
| if ! $PYTHON_CMD -c "import sys; assert sys.version_info >= (3, 9), 'Python 3.9 or higher is required'" &> /dev/null; then | ||
| echo "❌ Error: Python 3.9 or higher is required. Found $($PYTHON_CMD --version)" | ||
| exit 1 | ||
| fi | ||
|
|
||
| echo "✅ Python $(${PYTHON_CMD} --version) is installed" | ||
|
|
||
| ############################################################################### | ||
| # Set up virtual environment | ||
| ############################################################################### | ||
|
|
||
| # Create and activate virtual environment | ||
| echo "🌟 Setting up Python virtual environment..." | ||
| $PYTHON_CMD -m venv temp_venv | ||
| source temp_venv/bin/activate | ||
|
|
||
| # Install requirements | ||
| echo "📦 Installing Python packages..." | ||
| pip install -r requirements.txt |
| # Create default work pool in staging workspace | ||
| resource "prefect_work_pool" "staging_default" { | ||
| name = "default-work-pool" | ||
| workspace_id = prefect_workspace.staging.id | ||
| type = "docker" | ||
| } | ||
|
|
||
| # Create default work pool in production workspace | ||
| resource "prefect_work_pool" "production_default" { | ||
| name = "default-work-pool" | ||
| workspace_id = prefect_workspace.production.id | ||
| type = "docker" | ||
| } |
There was a problem hiding this comment.
What do you think about picking a Cloud based work pool, for example GCP Cloud Run? There would be more infrastructure but it would also be a more real world example.
There was a problem hiding this comment.
I like that idea. The current script assumes users have Docker running on their local machines, which is a big ask.
| echo "🔑 Reading Prefect API key and account ID..." | ||
|
|
||
| ############################################################################### | ||
| # Provision Prefect Cloud resources |
There was a problem hiding this comment.
In practice a big part of this kind of setup work is creating all the related objects: blocks, variables, automations, etc.
It would be great to show an example of creating a few blocks as well as a simple "notify on failure" automation. I believe that can all be expressed in both Terraform and Python.
There was a problem hiding this comment.
Since this script is in service of the debug tutorial, it depends what kind of scenario we want them to debug. If that scenario includes blocks and automations (the current scenario doesn't), it could make sense to provision more resources.
This reverts commit 2be0fd2.
3 participants
Closes https://linear.app/prefect/issue/DOC-122/automatically-provision-a-prefect-cloud-account-with-resources
(Please suggest better ways to accomplish this, I'm just being a script kiddie :-) )
Run
./setup_envto provision a paid Prefect Cloud account with the following resources:productionandstaging)stagingworkspace)The user can then complete the Debug a data pipeline tutorial without having to first manually perform all tasks in the Set up a platform for data pipelines tutorial.
NOTE: This script doesn't work on personal Prefect accounts because it requires multiple workspaces. There's a check which fails the script if you try to use it with a personal account.