data-pipeline-automation

End-to-end data pipeline transforming Olist e-commerce data through Azure cloud services. Implements medallion architecture (Bronze-Silver-Gold) with multi-source ingestion, Spark-based processing, and OLTP-to-OLAP optimization for analytics-ready datasets.

Pulls Meta ad data automatically, syncs to Google Sheets, delivers a live analytics dashboard with rule-based insights, decision engine, and Telegram alerts.

Standalone music intelligence layer that transforms your Spotify listening history into structured, filterable playlists with the goal of giving you explicit control over your taste

A complete reference implementation of a local-first ecosystem for AI-powered analytics. This repository contains the source code for the SBDK.dev website, the central hub for the SBDK suite of open-source tools.

A full-stack financial application featuring real-time forex rates and machine learning-powered price predictions across multiple currency pairs. Built with automated data pipelines and Prophet time series forecasting.

A toml based data pipeline creation tool/framework

A unified ETL product for turning repetitive file-based integration work into repeatable, reliable business flows

An end-to-end, multi-container solution for collecting and visualizing stock market data

Data Engineering project implementing a fully automated Medallion Architecture (Bronze, Silver, Gold) using Snowpipe, Streams, Tasks, and SQL to build scalable cloud-native data pipelines for analytics.

This repository contains scripts to build and process a Real-Time GDP (RTD) dataset for Peru, focusing on extracting, cleaning, and analyzing GDP revisions from the BCRP's Weekly Reports. Future updates will include econometric models and visualizations.

VizFlow is a VS Code extension that brings a visual workflow builder, RBQL querying, database connectors (MongoDB, MySQL, PostgreSQL), HTTP/REST integration, file transformations, and built-in scheduling — all running locally on your machine. Build repeatable data pipelines with 35 activity types, no cloud account or vendor lock-in required.

End-to-end data pipeline integrating manufacturing and financial data. Demonstrates ETL processes, a PostgreSQL data warehouse with a star schema, and business intelligence capabilities using PowerBI. Built with Python, Apache Airflow, and Docker.

The monorepo for microservices and microfrontends used in data augmentation.

End-to-End PV Monitoring & Streaming Pipeline with Delta Lake

Production-grade cold email outreach infrastructure with strict data normalization, deterministic lead scoring, and fault-tolerant multi-stage campaigns.

Pipeline automatisé de suivi mensuel des demandeurs d'emploi, téléchargement, nettoyage et rapport, données DARES/France Travail

Automated data pipeline scraping fuel prices daily, apply transformations and load the data to a PostgreSQL database.

Data automation involves automating the extraction, transformation, and loading (ETL) processes to streamline data workflows. GitHub Actions enables automated execution of tasks, such as building, testing, and deploying code, in response to events. This integration simplifies continuous deployment and ensures repeatable data pipeline operations

Python data pipeline project that tracks Amazon product prices, stores historical scrape results, and sends automated price-drop alerts.

Data-Engineering learning pg