Databricks:
DashBoards:
Featured Projects:
A selection of data engineering and analytics work
All pipelines are production-ready, implemented with Spark Declarative Bundles) for deployment automation and managed under version control via GitHub.
A portfolio of ETL and Analytics projects built on the Databricks ecosystem using Medallion Architecture. Each project ingests data from a different source platform using a purpose-fit ingestion method, applies transformations through the Silver layer, and delivers a Gold layer ready for AI and analytics workloads. Every pipeline showcases a distinct Databricks capability — reflecting real-world design choices tailored to different data sources, volumes, and business needs.
Stocks – Spark Declarative Pipelines
A production-ready ETL pipeline for processing stock market data using Databricks LakeHouse architecture. This project implements a medallion architecture (Bronze → Silver → Gold) with automated data quality checks and orchestrated execution.
| Bronze Layer | Silver Layer | Gold – Layer |
|---|---|---|
| Raw, unprocessed stock data ingestion with full historical refresh capability | Cleaned and validated data with quality checks applied | Business-ready aggregations and analytics-optimized datasets |
WikiMedia- Structure Streaming | CDC
A real-time data pipeline that consumes the Wikimedia EventStreams feed , lands raw events in a Bronze layer, and applies CDC (merge/upsert) logic to maintain deduplicated, current-state Silver tables of page edits.
| Bronze Layer | Silver Layer |
|---|---|
| Generates raw Wikimedia event data via defined and persists it to the bronze Delta table. | Reads raw Wikimedia edit events from a Bronze Delta table, deduplicates them, and upserts them into a Silver Delta table. |
GA4 Pipeline – Lakeflow Ingestion
This project builds a production-grade data pipeline that ingests raw Google Analytics 4 (GA4) event data, cleans and flattens it, and makes it ready for analysis following the Medallion Architecture pattern.
| INGESTION | SILVER LAYER | GOLD LAYER |
|---|---|---|
| Raw web traffic data is ingested from Google BigQuery into Databricks using Lakeflow Ingestion. | Transform bronze GA4 events into clean, analytics-ready silver tables with flattened nested structures. | Transforms silver table into pre-aggregated analytical views optimized for reporting and dashboards. |
Airflights
An end-to-end ETL pipeline built with Python, Serpapi, and Delta Lake that ingests raw Google Flights data, transforms it through a medallion architecture, and produces aggregated flight intelligence for trip planning.
| Bronze Layer | Silver Layer | Gold Layer |
|---|---|---|
| Extracting and storing raw flight data to a Delta table in Databricks. | Processing raw flight data from Bronze tables into cleaned, curated silver tables. | Processing cleaned flight data to create business-ready analytics for round-trip flight combinations |
IBM Stocks – Structure Stream | Merge
This project is an automated, scheduled ETL pipeline built in Databricks that ingests IBM daily stock data from an external API and processes it through a three-layers Delta Lake architecture (Bronze → Silver → Gold).
| Bronze Layer | Silver Layer | Dashboard – Tableau |
|---|---|---|
| Implements Auto loader using StructureStreaming for incremental files ingestion. with Schema enforcemement . | Merge Raw data into Curated Truth — Silver Layer Delta . | Interactive Tableau dashboards surfacing IBM stock performance metrics and market trends for business consumption. |
IT professional Survey
A multi-layer ETL pipeline for analyzing IT professional survey data
using Medallion Architecture with Python and PostgreSQL
| ELT | Analysis – EDA |
|---|---|
| Processing large IT professional survey dataset through a structured three-layers Medallion Architecture. | In-depth Exploratory Data Analysis (EDA) |
Crypto Medallion ETL – Dashboard – EDA
| Crypto – Medallion ETL | Uber Drive | Cat Breed |
|---|---|---|
| Processed cryptocurrency market data through a Medallion ETL pipeline. | Analyzed Uber ride data to improve operational efficiency metrics. DASHBOARD |
Comming Soon
Databricks Pipeline Monitoring Framework
Built a lightweight, native monitoring framework for Databricks pipelines that tracks both operational health and data quality in a single queryable system. The framework captures per-run metrics — duration, row counts, SLO compliance — alongside granular quality check results, all linked by a shared run_id key stored in two Delta tables in Unity Catalog.