Data Engineer | Azure · Databricks · Snowflake · AWS
I am a Data Engineer with 6+ years of experience building scalable cloud-native data platforms on Azure and AWS. My expertise spans real-time data ingestion, ELT/ETL pipelines, and data modeling using Databricks, Snowflake, Delta Lake, and PySpark — delivering analytics-ready solutions that power executive dashboards and business decision-making.
-
Netflix Azure Pipeline
ETL pipeline using ADF + Databricks + Delta Live Tables. Implements Bronze → Silver → Gold with Unity Catalog and Power BI dashboards. -
Real-Time Weather Streaming
Compared Azure Functions vs Databricks for streaming ingestion → Event Hubs → Fabric Eventhouse → KQL & dashboards. -
AI Animal Gateway
Computer vision + IoT project for monitoring animal activity using Python, FastAPI, and ML models.
Languages: Python, SQL, Scala, Java
Big Data: Spark, Hive, Kafka, Flink, Hadoop
ETL/Orchestration: ADF, Databricks, Airflow, AWS Glue, Matillion
Clouds: Azure (ADLS, Functions, Event Hubs, Synapse), AWS (S3, EMR, Lambda, Redshift)
Databases: Snowflake, SQL Server, MongoDB, DynamoDB, Cassandra
DevOps: Docker, Kubernetes, Terraform, Azure DevOps, GitHub Actions
Modeling: Kimball, Data Vault 2.0, SCD (Type 1, 2, 3)
Senior Data Engineer – Rainbird (Boston Data Pro) | Jan 2025 – Present
- Migrated Hive Metastore → Unity Catalog in Databricks for centralized governance.
- Built near real-time ingestion with Auto Loader + Delta Live Tables (50GB+ daily).
- Automated monitoring using Azure Functions + Logic Apps → reduced incident resolution time by 30%.
- Delivered dimensional models (Kimball) powering Power BI executive dashboards.
Data Engineer – Polaris | Feb 2023 – Dec 2024
- Developed PySpark/Spark SQL pipelines handling JSON, Avro, Parquet, CSV, XML.
- Implemented CI/CD for Synapse + Databricks using Azure DevOps & Docker.
- Migrated legacy SQL Server, Oracle, and MySQL → Synapse & ADLS (30% latency reduction).
- Designed a Data Vault 2.0 model for marketing data → improved traceability & audit readiness.
Data Engineer – Sigma Infosolutions | Jun 2018 – Dec 2021
- Built Kafka → Spark Streaming → Cassandra pipelines (20% faster).
- Designed AWS Glue + EMR pipelines feeding S3, Redshift, and RDS.
- Created REST APIs with AWS Lambda + API Gateway for business logic orchestration.
Associate Database Support Engineer – Magna Infotech | May 2017 – Apr 2018
- Developed interactive Power BI dashboards & paginated reports for finance and audit compliance.
- Designed and implemented SSIS packages to automate ETL pipelines.
- Ensured data governance and security compliance for reporting solutions.
- Microsoft Certified: Azure Data Engineer Associate (DP-203)
- Microsoft Certified: Azure Data Fundamentals (DP-900)
- Snowflake Certified: SnowPro Core (COF-C02)