This repo contains details about how youtube trends are implemented with sample mock data generator and leveraging Pyspark using Dataproc cluster and scheduling the Pyspark scripts leveraging Airflow and creating tables in Apache Iceberg format in GCS bucket and also creating Big Query aggregated tables, Thanks
End to End documentation: https://deepwiki.com/ViinayKumaarMamidi/GCP_Youtube_Trends_GCS_Iceberg_BigQuery_Pipeline
