🚀 PySpark Optimization Project – Banking Transactions

This project showcases practical PySpark optimization techniques using a synthetic banking transactions dataset (~5,000 records). The goal is to build efficient, scalable data pipelines using Spark best practices.

✅ Optimization Topics Covered

• Broadcast Join – Join small lookup tables without shuffle
• Shuffle Partition Tuning – Control partitioning for groupBy/joins
• Caching / Persisting – Improve performance on reused data
• Repartition vs Coalesce – Optimize parallelism and output size
• Delta Lake + Time Travel – Store and query historical data versions
• Adaptive Query Execution (AQE) – Smart query plan optimization
• Partitioning – Save Parquet by column folders for faster filtering
• Bucketing – Reduce shuffle during joins using hash-based buckets

---

Tools Used

• PySpark
• Databricks Notebook
• Delta Lake
• Parquet