{"id":4666,"date":"2026-05-15T00:05:36","date_gmt":"2026-05-15T00:05:36","guid":{"rendered":"https:\/\/ranaghazzi.com\/?page_id=4666"},"modified":"2026-05-26T02:15:34","modified_gmt":"2026-05-26T02:15:34","slug":"observability-in-databricks","status":"publish","type":"page","link":"https:\/\/ranaghazzi.com\/?page_id=4666","title":{"rendered":"Observability in Databricks"},"content":{"rendered":"<p><style>\n    .light-font-container, .light-font-container p, .light-font-container h2, .light-font-container li {<br \/>\n        font-weight: #FFFFFF !important;<br \/>\n    }<br \/>\n<\/style>\n<\/p>\n<div class=\"light-font-container\" style=\"background-color: #FFFFFF ; padding: 40px; border-radius: 15px;\">\n\n\n\n<div class=\"wp-block-group has-contrast-color has-text-color has-link-color wp-elements-649d348a09b2d446839758fb30966f94 has-global-padding is-layout-constrained wp-block-group-is-layout-constrained\">\n<div class=\"wp-block-group has-contrast-color has-text-color has-link-color wp-elements-ed1c5c0beab47ee3ba8f35656d049294 has-global-padding is-layout-constrained wp-block-group-is-layout-constrained\">\n<div class=\"wp-block-group has-contrast-color has-text-color has-link-color wp-elements-7d36ad4b6ffc3732b888a5d9a35130ad has-global-padding is-layout-constrained wp-block-group-is-layout-constrained\">\n<div class=\"wp-block-group has-contrast-color has-text-color has-link-color wp-elements-75327d5408e1cafa52d9776352c84b85 has-global-padding is-layout-constrained wp-block-group-is-layout-constrained\">\n<div class=\"wp-block-group has-contrast-color has-text-color has-link-color wp-elements-ad79176cca8b89866e91e4086fcb1ec2 has-global-padding is-layout-constrained wp-block-group-is-layout-constrained\">\n<div style=\"height:28px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h3 class=\"wp-block-heading has-x-large-font-size\"><strong>Why Observability Matters in Data Engineering<\/strong><\/h3>\n\n\n\n<div style=\"height:20px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">Observability in data engineering is the ability to understand the internal state of your entire data platform \u2014 not just knowing something broke, but knowing <em>why<\/em>, <em>where<\/em>, <em>when<\/em>, and <em>what was affected<\/em> \u2014 before your users or business stakeholders notice.<\/p>\n\n\n\n<div style=\"height:27px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h3 class=\"wp-block-heading has-x-large-font-size\">The Core Problem It Solves<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Data pipelines are inherently complex. Data flows through dozens of systems, transformations, and dependencies. Without observability you are essentially flying blind:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>A job silently produces wrong data instead of failing<\/li>\n\n\n\n<li>A cost spike goes unnoticed until the monthly bill arrives<\/li>\n\n\n\n<li>A slow query kills performance but <strong>n<\/strong>o one knows which team caused it<\/li>\n\n\n\n<li>A table gets dropped and lineage is <strong>l<\/strong>ost forever<\/li>\n\n\n\n<li>Compliance requires an audit trail but no logs exis<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Observability flips this \u2014 from reactive firefighting to proactive intelligence.<\/p>\n\n\n\n<div style=\"height:24px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h3 class=\"wp-block-heading has-x-large-font-size\">The 3 Pillars of Data Observability<\/h3>\n\n\n\n<h4 class=\"wp-block-heading has-large-font-size\">1. \ud83d\udcca Metrics<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">Quantitative measurements over time \u2014 query durations, job success rates, DBU consumption, row counts, cost per pipeline.<\/p>\n\n\n\n<h4 class=\"wp-block-heading has-large-font-size\">2. \ud83d\udccb Logs<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">Detailed event records \u2014 who ran what query, when a job started\/failed, what permissions changed. <\/p>\n\n\n\n<h4 class=\"wp-block-heading has-large-font-size\">3. \ud83d\udd0d Traces<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">End-to-end tracking of a data record as it moves through the system \u2014 from raw ingestion through bronze \u2192 silver \u2192 gold. <\/p>\n\n\n\n<div style=\"height:21px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading has-text-align-center has-x-large-font-size\">Observability Strategies in Databricks:<\/h2>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<div style=\"height:27px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h4 class=\"wp-block-heading has-x-large-font-size\">Strategy 1 \u2014 Unity Catalog System Tables (Native)<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">System tables are essentially a built-in observability layer \u2014 useful for monitoring costs, auditing access, tracking data lineage, and managing compute resources without any custom instrumentation.<\/p>\n\n\n\n<div style=\"height:26px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<figure class=\"wp-block-table has-medium-font-size\"><table class=\"has-contrast-color has-text-color has-link-color has-fixed-layout\"><thead><tr><th>Table Name<\/th><th>Purpose<\/th><\/tr><\/thead><tbody><tr><td><code>system.access.assistant_events<\/code><\/td><td>Tracks user messages sent to the Databricks Assistant<\/td><\/tr><tr><td><code>system.access.audit<\/code><\/td><td>Audit events from all workspaces in your region<\/td><\/tr><tr><td><code>system.access.table_lineage<\/code><\/td><td>Records every read\/write event on a Unity Catalog table or path<\/td><\/tr><tr><td><code>system.billing.list_prices<\/code><\/td><td>Historical log of SKU pricing changes<\/td><\/tr><tr><td><code>system.billing.usage<\/code><\/td><td>All billable usage across your account<\/td><\/tr><tr><td><code>system.compute.warehouse_events<\/code><\/td><td>SQL warehouse lifecycle events (start, stop, scale)<\/td><\/tr><tr><td><code>system.compute.warehouses<\/code><\/td><td>Full configuration history for SQL warehouses<\/td><\/tr><tr><td><code>system.data_classification.results<\/code><\/td><td>Column-level sensitive data detections across catalogs<\/td><\/tr><tr><td><code>system.lakeflow.job_run_timeline<\/code><\/td><td>Start and end times of job runs<\/td><\/tr><tr><td><code>system.lakeflow.job_tasks<\/code><\/td><td>All job tasks running in the account<\/td><\/tr><tr><td><code>system.lakeflow.jobs<\/code><\/td><td>All jobs created in the account<\/td><\/tr><tr><td><code>system.query.history<\/code><\/td><td>All queries run on SQL warehouses and serverless compute<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Best for:<\/strong> Cost monitoring, governance, compliance, job health, query analysis.<\/p>\n\n\n\n<div style=\"height:23px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h4 class=\"wp-block-heading has-x-large-font-size\">Strategy 2 \u2014 Delta Table Audit Logging (Medallion Layer)<\/h4>\n\n\n\n<div style=\"height:15px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">Build a <strong>m<\/strong>onitoring data lake on top of system tables using the medallion architecture.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Bronze \u2192 Raw system table snapshots (incremental, append-only)<br>Silver \u2192 Cleaned, joined, enriched metrics<br>Gold \u2192 Aggregated KPIs ready for dashboards and alerts<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This decouples monitoring layer from live system tables, improves query performance, and creates a historical record you own and control.<\/p>\n\n\n\n<div style=\"height:19px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h4 class=\"wp-block-heading has-x-large-font-size\">Strategy 3 \u2014 Databricks Lakehouse Monitoring<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">To draw useful insights from your data, you must have confidence in the quality of your data. Monitoring your data provides quantitative measures that help you track and confirm the quality and consistency of your data over time.<\/p>\n\n\n\n<div style=\"height:21px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h1 class=\"wp-block-heading has-x-large-font-size\"><strong>Attach a monitor to any Delta table<\/strong><\/h1>\n\n\n\n<p class=\"wp-block-paragraph\">To monitor a table in Databricks, you create a monitor attached to the table. To monitor the performance of a machine learning model, you attach the monitor to an inference table that holds the model&#8217;s inputs and corresponding predictions.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This automatically tracks:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Schema drift<\/strong> \u2014 did a column change type or disappear?<\/li>\n\n\n\n<li><strong>Data distribution shifts<\/strong> \u2014 are values suddenly out of range?<\/li>\n\n\n\n<li><strong>Null rates<\/strong> \u2014 are critical columns going empty?<\/li>\n\n\n\n<li><strong>Row count anomalies<\/strong> \u2014 did a pipeline produce zero rows?<\/li>\n<\/ul>\n\n\n\n<div style=\"height:24px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h4 class=\"wp-block-heading has-x-large-font-size\">Strategy 4 \u2014 Structured Streaming Metrics<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Structured Streaming Metrics<\/strong> are used for <em>observability<\/em>\u2014the telemetry data generated by Apache Spark engines to monitor the health and performance of real-time data pipelines<\/p>\n\n\n\n<p class=\"has-large-font-size wp-block-paragraph\"><strong>Where you access this \u2014 two options:<\/strong><\/p>\n\n\n\n<div style=\"height:8px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Option 1 \u2014 Spark UI (visual, built-in)<\/strong> : You see graphs of throughput, batch duration, and input rate updating live \u2014 no code needed. <\/p>\n\n\n\n<div style=\"height:9px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Option 2 \u2014 Code (programmatic): Query lastProgress <\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Spark&#8217;s&nbsp;StreamingQuery.lastProgress&nbsp;gives a live snapshot of your streaming job&#8217;s throughput and latency after each micro-batch.<\/p>\n\n\n\n<div style=\"height:19px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<div class=\"wp-block-group has-global-padding is-layout-constrained wp-block-group-is-layout-constrained\">\n<div class=\"wp-block-group is-style-section-5 has-global-padding is-layout-constrained wp-block-group-is-layout-constrained is-style-section-5--1\">\n<p class=\"wp-block-paragraph\"><strong>python<\/strong><\/p>\n\n\n\n<div class=\"wp-block-group has-global-padding is-layout-constrained wp-block-group-is-layout-constrained\">\n<p class=\"has-medium-font-size wp-block-paragraph\">query = df.writeStream.format(&#8220;console&#8221;).start()<\/p>\n\n\n\n<p class=\"has-medium-font-size wp-block-paragraph\"># After each micro-batch, inspect the metrics:<\/p>\n\n\n\n<p class=\"has-medium-font-size wp-block-paragraph\">import time<br>time.sleep(10)<br>print(query.lastProgress)<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n<div style=\"height:30px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h2 class=\"wp-block-heading\"><strong> <\/strong>Limitation: <\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">StreamingQuery.lastProgress exists purely in memory inside the Spark driver process, storing only the most recent micro-batch snapshot. It does not survive query restarts or cluster shutdowns. On Databricks, however, streaming metrics are automatically persisted to system tables, so the data is not lost when a query stops.<\/p>\n\n\n\n<div style=\"height:12px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h2 class=\"wp-block-heading has-large-font-size\"><strong>What gets saved automatically on Databricks:<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Databricks automatically logs structured streaming metrics into Unity Catalog system tables. Two tables are relevant: system.compute.query_history stores query-level metadata, while system.lakeflow.job_run_timeline captures job and pipeline run information. For Delta Live Tables pipelines specifically, Spark writes richer streaming metrics into a dedicated event_log table that you can query directly like any Delta table.<\/p>\n\n\n\n<div style=\"height:30px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<div style=\"height:9px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h2 class=\"wp-block-heading has-large-font-size\"><strong>StreamingQueryListener:<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">For raw Structured Streaming jobs, Spark does not automatically save metrics anywhere permanent. StreamingQueryListener solves this \u2014 it is a built-in Spark hook that fires automatically after every micro-batch completes, handing you the full progress payload so you can forward it wherever you need: a Delta table, a logging system, or an alerting tool \u2014 without any manual polling.<\/p>\n\n\n\n<div class=\"wp-block-group is-style-section-5 has-global-padding is-layout-constrained wp-block-group-is-layout-constrained is-style-section-5--2\">\n<h1 class=\"wp-block-heading has-large-font-size\"># Option 1 \u2014 write to Delta manually in foreachBatch<\/h1>\n\n\n\n<p class=\"has-medium-font-size wp-block-paragraph\"><em>def log_metrics(batch_df, batch_id):<br>progress = query.lastProgress<br>spark.createDataFrame([progress]).write \\<br>.mode(&#8220;append&#8221;).saveAsTable(&#8220;monitoring.stream_metrics&#8221;)<\/em><\/p>\n\n\n\n<div style=\"height:45px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h1 class=\"wp-block-heading has-large-font-size\">#Option 2 \u2014 use a StreamingQueryListener<\/h1>\n\n\n\n<p class=\"has-medium-font-size wp-block-paragraph\">clas<em>s MetricsListener(StreamingQueryListener):<br>def onQueryProgress(self, event):<br>progress = event.progress<br># save to Delta, send to Slack, push to Prometheus, etc.<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">spark.streams.addListener(MetricsListener())<\/p>\n<\/div>\n\n\n\n<div style=\"height:38px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<h4 class=\"wp-block-heading has-x-large-font-size\">Strategy 5 \u2014 Alerting &amp; Notification Pipelines<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">Observability is only complete when it triggers action automatically. Using the Databricks SQL Alert API, you can run a query against system tables on a schedule and fire an alert the moment a threshold is crossed <\/p>\n\n\n\n<div style=\"height:12px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">For example, <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Counting failed jobs in the last hour from system.lakeflow.job_run_timeline and notifying your team if that count exceeds three. These alerts can chain directly into webhooks that post to Slack, PagerDuty, or Microsoft Teams, or they can auto-trigger a remediation notebook to respond without any human intervention.<\/p>\n\n\n\n<div style=\"height:16px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<div class=\"wp-block-group has-contrast-color has-text-color has-link-color wp-elements-92bf471bae7ad0282a65a49a1e6a61b6 has-global-padding is-layout-constrained wp-block-group-is-layout-constrained\">\n<div class=\"wp-block-group has-global-padding is-layout-constrained wp-block-group-is-layout-constrained\"><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Why Observability Matters in Data Engineering Observability in data engineering is the ability to understand the internal state of your entire data platform \u2014 not just knowing something broke, but knowing why, where, when, and what was affected \u2014 before your users or business stakeholders notice. The Core Problem It Solves Data pipelines are inherently [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-4666","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.2 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Observability in Databricks - Rana Nasri Ghazzi<\/title>\n<meta name=\"description\" content=\"Browse real-world data projects by Rana Ghazzi, covering data cleaning, analysis, and storytelling with Python, SQL, and Tableau Explore Rana Ghazzi&#039;s data analytics portfolio \u2014 dashboards, visualizations, and insights built with Tableau, Power BI &amp; Python.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ranaghazzi.com\/?page_id=4666\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Observability in Databricks - Rana Nasri Ghazzi\" \/>\n<meta property=\"og:description\" content=\"Browse real-world data projects by Rana Ghazzi, covering data cleaning, analysis, and storytelling with Python, SQL, and Tableau Explore Rana Ghazzi&#039;s data analytics portfolio \u2014 dashboards, visualizations, and insights built with Tableau, Power BI &amp; Python.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ranaghazzi.com\/?page_id=4666\" \/>\n<meta property=\"og:site_name\" content=\"Rana Nasri Ghazzi\" \/>\n<meta property=\"article:modified_time\" content=\"2026-05-26T02:15:34+00:00\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/ranaghazzi.com\/?page_id=4666\",\"url\":\"https:\/\/ranaghazzi.com\/?page_id=4666\",\"name\":\"Observability in Databricks - Rana Nasri Ghazzi\",\"isPartOf\":{\"@id\":\"https:\/\/ranaghazzi.com\/#website\"},\"datePublished\":\"2026-05-15T00:05:36+00:00\",\"dateModified\":\"2026-05-26T02:15:34+00:00\",\"description\":\"Browse real-world data projects by Rana Ghazzi, covering data cleaning, analysis, and storytelling with Python, SQL, and Tableau Explore Rana Ghazzi's data analytics portfolio \u2014 dashboards, visualizations, and insights built with Tableau, Power BI & Python.\",\"breadcrumb\":{\"@id\":\"https:\/\/ranaghazzi.com\/?page_id=4666#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/ranaghazzi.com\/?page_id=4666\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/ranaghazzi.com\/?page_id=4666#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/ranaghazzi.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Observability in Databricks\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/ranaghazzi.com\/#website\",\"url\":\"https:\/\/ranaghazzi.com\/\",\"name\":\"Rana Nasri Ghazzi\",\"description\":\"Turning Data into Decisions\",\"publisher\":{\"@id\":\"https:\/\/ranaghazzi.com\/#\/schema\/person\/d8ee34f53cb0df9faaf816fb5363a4cc\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/ranaghazzi.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/ranaghazzi.com\/#\/schema\/person\/d8ee34f53cb0df9faaf816fb5363a4cc\",\"name\":\"Rana Ghazzi\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/ranaghazzi.com\/wp-content\/uploads\/2025\/11\/logo.png\",\"url\":\"https:\/\/ranaghazzi.com\/wp-content\/uploads\/2025\/11\/logo.png\",\"contentUrl\":\"https:\/\/ranaghazzi.com\/wp-content\/uploads\/2025\/11\/logo.png\",\"width\":1024,\"height\":1024,\"caption\":\"Rana Ghazzi\"},\"logo\":{\"@id\":\"https:\/\/ranaghazzi.com\/wp-content\/uploads\/2025\/11\/logo.png\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Observability in Databricks - Rana Nasri Ghazzi","description":"Browse real-world data projects by Rana Ghazzi, covering data cleaning, analysis, and storytelling with Python, SQL, and Tableau Explore Rana Ghazzi's data analytics portfolio \u2014 dashboards, visualizations, and insights built with Tableau, Power BI & Python.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ranaghazzi.com\/?page_id=4666","og_locale":"en_US","og_type":"article","og_title":"Observability in Databricks - Rana Nasri Ghazzi","og_description":"Browse real-world data projects by Rana Ghazzi, covering data cleaning, analysis, and storytelling with Python, SQL, and Tableau Explore Rana Ghazzi's data analytics portfolio \u2014 dashboards, visualizations, and insights built with Tableau, Power BI & Python.","og_url":"https:\/\/ranaghazzi.com\/?page_id=4666","og_site_name":"Rana Nasri Ghazzi","article_modified_time":"2026-05-26T02:15:34+00:00","twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/ranaghazzi.com\/?page_id=4666","url":"https:\/\/ranaghazzi.com\/?page_id=4666","name":"Observability in Databricks - Rana Nasri Ghazzi","isPartOf":{"@id":"https:\/\/ranaghazzi.com\/#website"},"datePublished":"2026-05-15T00:05:36+00:00","dateModified":"2026-05-26T02:15:34+00:00","description":"Browse real-world data projects by Rana Ghazzi, covering data cleaning, analysis, and storytelling with Python, SQL, and Tableau Explore Rana Ghazzi's data analytics portfolio \u2014 dashboards, visualizations, and insights built with Tableau, Power BI & Python.","breadcrumb":{"@id":"https:\/\/ranaghazzi.com\/?page_id=4666#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ranaghazzi.com\/?page_id=4666"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/ranaghazzi.com\/?page_id=4666#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/ranaghazzi.com\/"},{"@type":"ListItem","position":2,"name":"Observability in Databricks"}]},{"@type":"WebSite","@id":"https:\/\/ranaghazzi.com\/#website","url":"https:\/\/ranaghazzi.com\/","name":"Rana Nasri Ghazzi","description":"Turning Data into Decisions","publisher":{"@id":"https:\/\/ranaghazzi.com\/#\/schema\/person\/d8ee34f53cb0df9faaf816fb5363a4cc"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ranaghazzi.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/ranaghazzi.com\/#\/schema\/person\/d8ee34f53cb0df9faaf816fb5363a4cc","name":"Rana Ghazzi","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ranaghazzi.com\/wp-content\/uploads\/2025\/11\/logo.png","url":"https:\/\/ranaghazzi.com\/wp-content\/uploads\/2025\/11\/logo.png","contentUrl":"https:\/\/ranaghazzi.com\/wp-content\/uploads\/2025\/11\/logo.png","width":1024,"height":1024,"caption":"Rana Ghazzi"},"logo":{"@id":"https:\/\/ranaghazzi.com\/wp-content\/uploads\/2025\/11\/logo.png"}}]}},"_hostinger_reach_plugin_has_subscription_block":false,"_hostinger_reach_plugin_is_elementor":false,"_links":{"self":[{"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=\/wp\/v2\/pages\/4666","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4666"}],"version-history":[{"count":187,"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=\/wp\/v2\/pages\/4666\/revisions"}],"predecessor-version":[{"id":5082,"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=\/wp\/v2\/pages\/4666\/revisions\/5082"}],"wp:attachment":[{"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4666"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}