Skip to content

Setup_Redshift_Connection_Airflow.md

Latest commit

 

History

History
118 lines (83 loc) · 2.13 KB

Setup_Redshift_Connection_Airflow.md

File metadata and controls

118 lines (83 loc) · 2.13 KB
Raw

Airflow – Redshift Connection Setup

This guide explains how to configure Apache Airflow to connect to Amazon Redshift and AWS S3 for the ETL pipeline.

1. Configure AWS Credentials

In Airflow, create an AWS connection:

Connection ID

aws_credentials

Connection Type

Amazon Web Services

Add your AWS Access Key and Secret Access Key.

Never commit AWS credentials to GitHub. Store them securely in Airflow Connections or environment variables.

2. Configure Redshift Connection

In Airflow, create a new connection:

Connection ID

redshift

Connection Type

Postgres

Enter the Redshift cluster details:

Host: <redshift-endpoint>
Schema: <database-name>
Login: <username>
Password: <password>
Port: 5439

The connection ID must match:

redshift_conn_id = "redshift"

used in the Airflow DAG.

3. S3 Configuration

The pipeline reads the source data from Amazon S3.

Example configuration:

s3_bucket = "your-bucket-name"
song_s3_key = "song_data"
log_s3_key = "log-data"

Make sure the configured AWS user/role has permission to read the required S3 objects.

4. Airflow Connections

The pipeline requires the following Airflow connections:

Connection ID Purpose
aws_credentials Access AWS S3
redshift Connect to Amazon Redshift

5. Run the Pipeline

After configuring the connections:

  1. Start the Airflow scheduler and webserver.
  2. Place the DAG files inside the Airflow dags directory.
  3. Open the Airflow UI.
  4. Enable the pipeline DAG.
  5. Trigger the DAG manually or wait for its scheduled execution.

The pipeline performs the following steps:

Amazon S3
    ↓
Stage data in Redshift
    ↓
Load Fact Table
    ↓
Load Dimension Tables
    ↓
Run Data Quality Checks

6. Important Notes

  • Do not hardcode AWS credentials in Python files.
  • Do not commit passwords, access keys, or secret keys to GitHub.
  • Verify that the Redshift cluster is accessible from the Airflow environment.
  • Make sure the required Airflow connections are created before running the DAG.