# Rana Nasri Ghazzi

> Turning Data into Decisions


## Posts

- [AIR Flights](https://ranaghazzi.com/?p=5604)
- [Sales Dashboard 2](https://ranaghazzi.com/?p=5599)
- [📊 Sales Dashboard](https://ranaghazzi.com/?p=5594): 📊 Sales Dashboard 2022
- [💻 IT Survey Dashboard](https://ranaghazzi.com/?p=5589): 💻 IT Survey Dashboard
- [Uber Analytics Dashboard](https://ranaghazzi.com/?p=5585): 🚗 Uber Analytics Dashboard
- [Handling Duplicates in Spark Streaming Pipelines: 3 Practical Approaches](https://ranaghazzi.com/?p=5475): Handling Duplicates in Spark Streaming Pipelines: 3 Practical Approaches Duplicate records are one of the most common and frustrating problems in data engineering. Whether you’re pulling from an API, a message queue, or a CDC feed, duplicates will eventually appear. Here’s a breakdown of three practical approaches to deal with them — and when to use each. The Problem In streaming or periodic ingestion pipelines, duplicates can sneak in through: Left unchecked, duplicates silently corrupt aggregations, inflate metrics, and break downstream consumers. Option 1: dropDuplicates() Before Writing The simplest fix — deduplicate the DataFrame before it hits the sink. How it […]
- [Comprehensive Guide: Data Ingestion in Databricks — Lakeflow Connect & External Ingestion](https://ranaghazzi.com/?p=5381): The Six Lanes of Data Ingestion in Databricks: A Practical Guide Every lakehouse project starts with the same unglamorous question: how does the data actually get in? Databricks offers a surprisingly rich set of answers, and that’s both a blessing and a source of confusion. Auto Loader, COPY INTO, Lakeflow Connect, Partner Connect, Kafka connectors, custom Python — they all “ingest data,” but they solve very different problems. This post lays out a mental model I find useful: six lanes of ingestion, each defined by what your source looks like and how much engineering you want to own. By the end, […]
- [Guide to Job Clusters, All-Purpose Clusters, and Serverless Compute](https://ranaghazzi.com/?p=5352): Databricks Clusters Demystified: A Practical Guide to Job Clusters, All-Purpose Clusters, and Serverless Compute Introduction One of the most important — and often misunderstood — concepts in Databricks is compute management. Choosing the wrong cluster type can cost your organization money, slow down pipelines, or introduce security and governance risks. This guide breaks down everything you need to know about cluster types in Databricks, with a deep focus on Job Clusters and how they behave in production environments. The Three Types of Compute in Databricks 1. All-Purpose Clusters All-purpose clusters are interactive clusters designed for collaborative development. They are manually started […]
- [PostgreSQL (SQL) vs pandas vs PySpark.](https://ranaghazzi.com/?p=5291): PostgreSQL (SQL) vs pandas vs PySpark. Read / Inspect Task PostgreSQL pandas PySpark Preview rows SELECT * FROM t LIMIT 5; df.head(5) df.show(5) Row count SELECT count(*) FROM t; len(df) df.count() Columns/schema \d t df.dtypes df.printSchema() Distinct SELECT DISTINCT a FROM t; df.a.drop_duplicates() df.select(“a”).distinct() Select / Filter Task PostgreSQL pandas PySpark Select cols SELECT a, b FROM t; df[[“a”,”b”]] df.select(“a”,”b”) Filter WHERE age > 30 df[df.age > 30] df.filter(F.col(“age”) > 30) Multi-condition WHERE a > 1 AND b < 5 df[(df.a>1) & (df.b<5)] df.filter((F.col(“a”)>1) & (F.col(“b”)<5)) In list WHERE c IN (1,2) df[df.c.isin([1,2])] df.filter(F.col(“c”).isin(1,2)) Columns / Expressions Task PostgreSQL pandas PySpark […]
- [Structured Streaming vs.  DLT Live Table(@dp Declarative Pipeline](https://ranaghazzi.com/?p=5268): Structured Streaming vs.  DLT Live Table – Declarative Pipeline Conceptually, DLT and Structured Streaming differ in who is in control, not in formats or parsing. The one core idea: Structured Streaming is imperative — you write how to run the stream. DLT is declarative — you write what the table should contain, and the framework runs it. From that single distinction, everything conceptual follows: Auto Loader / design choices that work the same way in both. Auto Loader (cloudFiles) is a source and is independent of whether DLT or Streaming is driving. One-line summary: Streaming = you orchestrate the pipeline; DLT = you declare the result and the framework orchestrates. That’s the only conceptual axis — […]
- [Databricks Pipeline Monitoring Framework](https://ranaghazzi.com/?p=4556): How We Built a Custom Monitoring Framework for Databricks Pipelines Data pipelines fail silently. That’s the uncomfortable truth most data engineering teams learn the hard way — usually when a business stakeholder asks why last week’s report looks off, or why a dashboard hasn’t refreshed since Tuesday. By then, the damage is done. Rather than reaching for an expensive third-party observability tool, we built something lean, purposeful, and entirely native to our Databricks environment. This post walks through what I built, why I built it that way, and the key design decisions that made it work. The Problem With “Just Check […]
- [Windowing and Watermarking in Databricks Streaming](https://ranaghazzi.com/?p=5203): Windowing & Watermarking in Databricks Streaming What Is Windowing? In a continuous stream, data never stops arriving. Windowing is the mechanism that groups streaming events into finite, time-bounded buckets so you can compute meaningful aggregations — counts, averages, sums — over a defined period rather than over all data since the beginning of time. Spark Structured Streaming supports three window types. Tumbling Windows are fixed-size, non-overlapping buckets. Every event belongs to exactly one window — the simplest and most memory-efficient option. Sliding Windows are fixed-size but advance at a shorter interval, creating overlap. An event can belong to multiple windows simultaneously, […]
- [Databricks Auto Loader: Predefined Schemas vs. Schema Inference](https://ranaghazzi.com/?p=5184): Databricks Auto Loader: Predefined Schemas vs. Schema Inference Published: May 30, 2026 Data engineers using Databricks Auto Loader often face a critical decision early in their pipeline development: should they define table schemas upfront, or let Auto Loader automatically infer them? It seems like a simple choice, but it has profound implications for data quality, pipeline reliability, and operational complexity. In this post, I’ll break down both approaches, reveal why predefined schemas are the production standard, and show you how to get the best of both worlds using schema evolution modes. The Auto Loader Dilemma Auto Loader is Databricks’ incremental data […]
- [Auto Loader: The Smartest Way to Ingest Streaming Data in Databricks](https://ranaghazzi.com/?p=5165): Auto Loader: The Smartest Way to Ingest Streaming Data in Databricks If you’ve ever built a data pipeline that ingests files from cloud storage, you know the pain: polling for new files, tracking what’s already been processed, handling duplicates, and scaling when data volumes spike. Databricks Auto Loader was built to solve exactly these problems — elegantly and at scale. What Is Auto Loader? Auto Loader is a Databricks-native structured streaming source that incrementally and efficiently ingests new data files as they arrive in cloud storage (S3, ADLS, GCS). It’s built on top of Apache Spark’s Structured Streaming engine and handles […]
- [Streaming Tables vs. Materialized Views in Databricks](https://ranaghazzi.com/?p=5144): Streaming Tables vs. Materialized Views If you’re building pipelines in Databricks, you’ll eventually hit a fork in the road: should this dataset be a Streaming Table or a Materialized View? They look similar on the surface—both are managed by Unity Catalog, both are backed by Lakeflow/Delta Live Tables (DLT), and both handle incremental processing for you under the hood. But they solve fundamentally different problems. Get the choice right and your pipeline is fast, cheap, and correct. Get it wrong and you end up with stale metrics, runaway compute bills, or joins that quietly return the wrong numbers. Here’s how to […]
- [Streaming Tables vs. Regular Delta Tables](https://ranaghazzi.com/?p=5124): Streaming Tables vs. Regular Delta Tables Not all data pipelines are built the same. Some need to react the instant new records land; others run on a tidy schedule and care more about flexible transformations than speed. In Databricks, that choice often comes down to two options: Streaming Tables and Regular Delta Tables. Picking the wrong one can mean paying to reprocess the same data over and over, or fighting your pipeline every time you need to update a row. Here’s how to tell them apart and choose with confidence. The Short Version Streaming Tables are built for continuous, incremental ingestion […]
- [Data Governance in Databricks](https://ranaghazzi.com/?p=5021): Databricks governance Unity Catalog Databricks governance is centered around Unity Catalog, a powerful unified governance layer that manages data and AI assets across your entire organization. It operates beneath every data interaction, automatically enforcing access control when you query a table, tracking lineage as data moves, and logging activity for auditing. Unity Catalog operates consistently across AWS, Azure, and GCP, providing unified governance regardless of cloud provider. Managed Assets Unity Catalog governs: Core Governance Pillars Databricks governance focuses on four essential components: data quality checks, access control, lineage tracking, and auditing with monitoring. Centralized Metadata Management Unity Catalog serves as the […]
- [CI/CD on Databricks](https://ranaghazzi.com/?p=4999): CI/CD on Databricks Databricks Asset Bundles (DABs) CI/CD on Databricks centers on automating the testing and deployment of data pipelines, notebooks, and jobs across environments using Databricks Asset Bundles (DABs) as the primary infrastructure-as-code tool, with everything version-controlled in Git. The CI side covers linting, unit testing with pytest or Nutter, and bundle validation on every code push, while the CD side automates promotion from dev to staging to prod using platform tools like GitHub Actions or Azure DevOps. Credentials are managed through service principals and Databricks Secrets — never hardcoded — and Unity Catalog provides clean environment separation at the data layer. Key best […]
- [Data Quality Checks](https://ranaghazzi.com/?p=4927): Quality Checks: 1. Lakeflow Declarative Pipelines expectations: What are expectations? Expectations are optional clauses in pipeline materialized view, streaming table, or view creation statements that apply data quality checks on each record passing through a query. Expectations use standard SQL Boolean statements to specify constraints. You can combine multiple expectations for a single dataset and set expectations across all dataset declarations in a pipeline. Behavior on Violation: Specify an action to determine what happens when a record fails the validation check. The following table describes the available actions: dp.expect_or_fail – This expectation causes a failure of a single flow dp.expect_or_drop – Invalid records […]
- [AirFlights Gold Layer](https://ranaghazzi.com/?p=4412): AirFlights Gold Layer – Round-Trip Analysis   Tools: Databricks | Pandas | PyArrow | PySpark | Delta Lake | Tableau Description: This notebook processes cleaned flight data from Silver tables to create business-ready analytics for round-trip flight combinations. It transforms cleaned Silver-layer flight data into actionable Gold-layer insights by identifying the best-priced round-trip flight combinations across airlines and routes. The Notebook begins by loading the necessary libraries and defining strict PyArrow schemas to ensure data consistency throughout processing. It then reads both direct and connecting flight records from Unity Catalog’s Silver Delta tables, consolidates them into a single dataset, and categorizes […]
- [AirFlights Silver Layer](https://ranaghazzi.com/?p=4116): Air Flights Silver Layer – Data Cleaning & Transformation   Tools: Databricks | Pandas | PyArrow | PySpark | Delta Lake Description: This notebook implements a Silver layer that processes raw flight data from Bronze tables into cleaned, curated Silver tables. What it does: Schema includes: flight number, airline, origin/destination, times, duration, stops, price, currency, trip type, and fetch timestamp.
- [AirFlights Bronze Layer](https://ranaghazzi.com/?p=3902): AirFlights Bronze Layer Tools: Databricks | SerpAPI | Pandas | PyArrow | PySpark | Delta Lake | datetime Moving Airplane Description: This layer is responsible for extracting and storing raw flight data, no transformations or business logic applied. It consists of two identical notebooks: – The first notebook extracts outbound flights between the defined origin airports , in this Example :(DC metro — IAD, DCA, BWI) to destination airports (Florida — 5 airports) – The second notebook runs the same logic to capture return flights. Each notebook calls SerpAPI’s Google Flights engine across all airport combinations (15 API calls per run), […]
- [Interactive Real-Time Stock Charts & Financial Insights](https://ranaghazzi.com/?p=3716): Interactive Real-Time Stock Charts & Financial Insights Top stories by TradingView
- [Top News](https://ranaghazzi.com/?p=3568): Top News
- [IBM Stocks](https://ranaghazzi.com/?p=3398): 📈 IBM Stock Dashboard
- [AI-Powered Real-Time Crypto ETL](https://ranaghazzi.com/?p=2238): AI-Powered Real-Time Crypto ETL Project: Crypto Medallion ETL (Bronze → Silver → Gold) Overview A multi-layer (Bronze → Silver → Gold) ETL pipeline that ingests live cryptocurrency market data from the CoinGecko API, stores raw and cleaned records in PostgreSQL, and produces daily aggregated insights. The pipeline is orchestrated by pipeline.py, version-controlled on GitHub, and implemented in Python (Anaconda). The Claude AI agent assisted during development. Logs are written to etl.log. Goals Ingest top N coins from CoinGecko on an hourly cadence. Preserve raw extracted records (Bronze) for auditing and reprocessing. Enrich and normalize data (Silver) for analysis-ready consumption. Produce daily aggregated summaries […]
- [Survey Analysis Medallion Pipeline](https://ranaghazzi.com/?p=1968): IT professional survey data – ELT Tools: Jupiter Notebook | PostgreSQL | Python: Pandas | NumPy | Matplotlib | Seaborn | SciPy Project Overview This project implements a multi-layer ETL pipeline for analyzing survey data using the Medallion Architecture pattern. The pipeline extracts raw survey data, performs data quality checks, applies transformations, and prepares, clean and normalized data using Python and PostgreSQL for analysis. Architecture The project follows the **Medallion Architecture** pattern: Pipeline Components Bronze Layer The Bronze layer handles: – Data extraction from external APIs – Connection to PostgreSQL database – Loading raw survey data – Initial column filtering and selection […]
- [Exploratory Data Analysis (EDA) of IT Professionals](https://ranaghazzi.com/?p=1728): Exploratory Data Analysis (EDA) of IT Professionals Tools: Jupiter Notebook | Python: Pandas | NumPy | Matplotlib | Seaborn | SciPy Introduction This Exploratory Data Analysis (EDA) project examines compensation, skills, demographics, and work patterns among IT professionals. Using a cleaned, annualized-salary dataset filtered for full-time, USD-denominated responses and with multi-valued fields (e.g., programming languages) normalized, the analysis aims to reveal actionable insights for hiring managers, data practitioners, and professionals planning career moves. Objectives Methodology Expected Insights Dataset: Our data set is a survey works among IT professional , collected and published in the link below. “https://api.example.com/data” It has 11551 records […]

## Pages

- [Blogs](https://ranaghazzi.com/?page_id=3303)
- [Observability in Databricks](https://ranaghazzi.com/?page_id=4666): Why Observability Matters in Data Engineering Observability in data engineering is the ability to understand the internal state of your entire data platform — not just knowing something broke, but knowing why, where, when, and what was affected — before your users or business stakeholders notice. The Core Problem It Solves Data pipelines are inherently complex. Data flows through dozens of systems, transformations, and dependencies. Without observability you are essentially flying blind: Observability flips this — from reactive firefighting to proactive intelligence. The 3 Pillars of Data Observability 1. 📊 Metrics Quantitative measurements over time — query durations, job success rates, […]
- [Contact](https://ranaghazzi.com/?page_id=2645): Have a project in mind? Reach out directly. Email Me rana@ghazzi.com
- [Uber Drive](https://ranaghazzi.com/?page_id=1661): Tools: Jupiter Notebook / Python /Pandas/ Tableau About The Data Set: This dataset represents a collection of 1,150 ride entries (likely from a personal Uber or business travel log). It captures the physical distance and the time investment for various trips. Insight and Conclusion Observations: n = 1,150 Miles Mean: 10.09 miles Median (50th percentile): 6.00 Miles Range: min 0.0, max 310.0 miles Spread: std dev ≈ 21.56 miles Q1–Q3: 2.0 miles (25th percentile) to 10.0 miles (75th percentile) Observation There is a wide range with a long tail toward higher mile values, and the mean is higher than the median, […]
- [IBM Stocks ETL – Bronze Layer](https://ranaghazzi.com/?page_id=1196): Tools: Databricks | Pyspark | Pandas| Numpy | delta.tables Description: This project is an automated, scheduled ETL pipeline built in Databricks that ingests IBM daily stock data from an external API and processes it through a two-layer Delta Lake architecture (Bronze → Silver). The pipeline connects to a financial API that returns the latest 100 trading days of IBM stock data in JSON format — including open, high, low, close prices, and volume — updated every two to three days. Rather than reloading the full dataset on each run, it implements a Change Data Capture (CDC) approach that processes only new or […]
- [IBM Stocks - ETL Pipeline](https://ranaghazzi.com/?page_id=1129): IBM Stocks ETL – Silver layer Tools: Databricks | Pyspark | Pandas| Numpy | delta.tables Description : The Silver layer takes the Bronze data and produces a clean, curated, always-current version of each record. It applies type casting, null validation, and deduplication, then uses a Delta MERGE (upsert) operation keyed on the date field — updating existing records if they’ve changed and inserting new ones if they haven’t been seen before. Task 3: Silver_Merge Notebook: IBM _silverDependencies: Waits for Auto_Loader_bronze to complete What it does: Output: The end result is a reliable, incrementally-updated data foundation ready to power business intelligence tools and real-time dashboards with accurate, deduplicated […]
- [IBM Stock Dashboard](https://ranaghazzi.com/?page_id=1111): Project Overview This IBM Stock Live Dashboard is a comprehensive visualization tool to track and analyze IBM’s stock market performance. It focuses on identifying trends, volatility, and trading activity over a multi-year period (ranging from late 2023 into 2026). Below is a breakdown of the key components and what they represent: 1. Top-Level Key Performance Indicators (KPIs) These metrics provide an instant snapshot of the stock’s status within the selected timeframe: 2. Performance & Volatility Charts The middle section of the dashboard focuses on the “heartbeat” of the stock: 3. Technical Analysis Trends
- [Home](https://ranaghazzi.com/)
- [ Portfolio](https://ranaghazzi.com/?page_id=207): Harvard Business School Certified Databricks Engineer Certified Tableau Specialist Certified Python Programmer PCEP B.A. / B.S. in Law Greetings, I am a results-driven data and technology professional who transforms complex datasets into strategic insights that move organizations forward. Professionally accredited in modern data visualization and engineering frameworks, with deep fluency in Python and SQL, I help businesses unlock measurable value from their data. My unique background bridging law and information technology sharpens my analytical precision and structured thinking — allowing me to architect robust data solutions while translating technical complexity into clear, compelling narratives for any audience. Key strengths: I thrive […]
- [Projects](https://ranaghazzi.com/?page_id=30): Databricks: DashBoards: Featured Projects: Inventory A selection of data engineering and analytics work A portfolio of ETL and Analytics projects built on the Databricks ecosystem using Medallion Architecture. Each project ingests data from a different source platform using a purpose-fit ingestion method, applies transformations through the Silver layer, and delivers a Gold layer ready for AI and analytics workloads. Every pipeline showcases a distinct Databricks capability — reflecting real-world design choices tailored to different data sources, volumes, and business needs. All pipelines are production-ready, implemented with Spark Declarative Bundles) for deployment automation and managed under version control via GitHub. Airflights Moving […]

## Optional

- [Agent (MCP protocol)](websites-agents.hostinger.com/ranaghazzi.com/mcp)

[comment]: # (Generated by Hostinger Tools Plugin)