OctopuFS library helps managing cloud storage, ADLSgen2 specifically. It allows you to operate on files (moving, copying, setting ACLs) in very efficient manner. Designed to work on databricks, but should work on any other platform as well.
Fluentd output plugin for Azure Datalake Storage Gen2 (append support)
Data Engineering Project on Supply Chain ETL. Creating a dynamic ADF pipeline to ingest both Full Load and Incremental Load data from SQL Server and then transform these datasets based on medallion architecture using Databricks.
Explore the Tokyo Olympics data journey! We ingested a GitHub CSV into Azure via Data Factory, stored it in Data Lake Storage Gen2, performed transformations in Databricks, conducted advanced analytics in Azure Synapse, and visualized insights in Synapse or Power BI.
Designed and implemented an end-to-end Azure Data Engineering platform using Azure Data Factory, ADLS Gen2, Databricks, Synapse Analytics, and Power BI. Built metadata-driven pipelines and Medallion Architecture (Bronze, Silver, Gold) to ingest, transform, and serve analytics-ready data.
This sample demonstrates how to create a Linux Virtual Machine in a virtual network that privately accesses a blob storage account using an Azure Private Endpoint.
End-to-end Azure Data Engineering project for Healthcare Revenue Cycle Management
A Python package for Azure Datalake Storage adlsgen2 [abfss://] and Microsoft Fabric Lakehouse [abfss://], Google Cloud Storage [gs://bucket], AWS S3 bucket [s3://bucket] enables format detection and schema retrieval for Iceberg, Delta, and Parquet formats. It helps identify partitioned columns for parquet datasets. It also supports querying Delta,
Using SAS to authenticate and access to ADLS Gen 2 from Azure Databricks
COVID19-ADF is a project that leverages Azure services to collect, analyze, and visualize COVID-19 data. With seamless data integration and advanced analytics, it provides valuable insights into the pandemic's impact, enabling informed decision-making in the fight against COVID-19.
Deploy apache spark in client mode on Kubernetes cluster, integrate with Jupyter notebook through Jupyterhub server.
This project demonstrates an end-to-end Data Engineering pipeline built on Databricks. It showcases modern ETL/ELT practices using Delta Lake, Delta Live Tables (DLT), Auto Loader, and Unity Catalog. The solution follows the Medallion Architecture (Bronze → Silver → Gold) to ensure scalable, reliable, and high-quality data processing.
POC projects working on Cloud Platforms
ETL project with Spark and Airflow
Production-style Azure analytics engineering project for adult social care using ADF, ADLS, Synapse and Power BI.
End-to-end Azure Data Engineering Pipeline using Azure Data Factory, Azure Databricks, ADLS Gen2, Medallion Architecture, Metadata-Driven ETL, PySpark, SQL and Power BI.
Implemented Azure Databricks for real-time data processing and governance using Unity Catalog, Spark Structured Streaming, Delta Lake features, Medallion Architecture, and end-to-end CI/CD pipelines. Focused on incremental loading, compute cluster management, maintaining data quality, and creating workflows.
To associate your repository with the adlsgen2 topic, visit your repo's landing page and select "manage topics."