{"id":4556,"date":"2026-06-01T22:48:41","date_gmt":"2026-06-01T22:48:41","guid":{"rendered":"https:\/\/ranaghazzi.com\/?p=4556"},"modified":"2026-06-13T19:13:54","modified_gmt":"2026-06-13T19:13:54","slug":"databricks-pipeline-monitoring-framework","status":"publish","type":"post","link":"https:\/\/ranaghazzi.com\/?p=4556","title":{"rendered":"Databricks Pipeline Monitoring Framework"},"content":{"rendered":"<p><style>\n    .light-font-container, .light-font-container p, .light-font-container h2, .light-font-container li {<br \/>\n        font-weight: #FFFFFF !important;<br \/>\n    }<br \/>\n<\/style>\n<\/p>\n<div class=\"light-font-container\" style=\"background-color: #FFFFFF; padding: 40px; border-radius: 15px;\">\n\n<h2 class=\"has-text-align-center wp-block-post-title\">Databricks Pipeline Monitoring Framework<\/h2>\n\n\n<h3 class=\"wp-block-heading\"> How We Built a Custom Monitoring Framework for Databricks Pipelines<\/h3>\n\n\n<div class=\"wp-block-post-date\"><time datetime=\"2026-06-13T19:13:47.993Z\">June 13, 2026<\/time><\/div>\n\n\n<p class=\"wp-block-paragraph\">Data pipelines fail silently. That&#8217;s the uncomfortable truth most data engineering teams learn the hard way \u2014 usually when a business stakeholder asks why last week&#8217;s report looks off, or why a dashboard hasn&#8217;t refreshed since Tuesday. By then, the damage is done.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Rather than reaching for an expensive third-party observability tool, we built something lean, purposeful, and entirely native to our Databricks environment. This post walks through what I built, why I built it that way, and the key design decisions that made it work.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">The Problem With &#8220;Just Check the Logs&#8221;<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">When a pipeline fails, Databricks marks the job as failed. But operational failure is only half the story. A pipeline can succeed \u2014 finishing on time, writing rows, returning exit code 0 \u2014 while quietly producing garbage data. Null primary keys. Negative claim amounts. Row counts that dropped 40% with no explanation.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Traditional job monitoring doesn&#8217;t catch that. You need a second layer: data quality monitoring. And you need both layers to speak the same language so you can ask: <em>&#8220;For every run that breached our SLO in the last 30 days, which quality checks also failed?&#8221;<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That was our north star.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Two Tables, One Foreign Key<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The entire framework rests on two Delta tables in Unity Catalog:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>pipeline_metrics<\/strong> captures operational health \u2014 one row per pipeline run.<br><strong>pipeline_quality_checks<\/strong> captures data quality \u2014 one row per check per run.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Both share a run_id column \u2014 a UUID generated at the start of every run. That single foreign key is what makes the whole system queryable.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Section 1 \u2014 Configuration<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Set these values once per environment. All secrets live in Databricks Secret Scopes \u2014 never hardcoded.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Catalog and schema for monitoring tables\nMONITORING_CATALOG = \"monitoring\"\nMONITORING_SCHEMA = \"pipeline_ops\"\n\n# Full table names\nMETRICS_TABLE = f\"{MONITORING_CATALOG}.{MONITORING_SCHEMA}.pipeline_metrics\"\nQUALITY_TABLE = f\"{MONITORING_CATALOG}.{MONITORING_SCHEMA}.pipeline_quality_checks\"\n\n# Slack webhook \u2014 stored in Databricks Secrets\nSLACK_WEBHOOK_URL = dbutils.secrets.get(scope=\"monitoring\", key=\"slack_webhook_url\")\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Section 2 \u2014 Table Setup<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Call setup_monitoring_tables() once when deploying to a new environment. It is idempotent \u2014 safe to re-run because IF NOT EXISTS prevents overwrites.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>def setup_monitoring_tables():\n    # pipeline_metrics \u2014 one row per run\n    spark.sql(f\"\"\"\n        CREATE TABLE IF NOT EXISTS {METRICS_TABLE} (\n            run_id              STRING,\n            pipeline_name       STRING,\n            pipeline_version    STRING,\n            started_at          TIMESTAMP,\n            completed_at        TIMESTAMP,\n            duration_minutes    DOUBLE,\n            rows_processed      LONG,\n            rows_quarantined    LONG,\n            slo_target_minutes  INT,\n            slo_met             BOOLEAN,\n            status              STRING,\n            error_message       STRING\n        ) USING DELTA\n    \"\"\")\n\n    # pipeline_quality_checks \u2014 one row per check per run\n    spark.sql(f\"\"\"\n        CREATE TABLE IF NOT EXISTS {QUALITY_TABLE} (\n            run_id          STRING,\n            pipeline_name   STRING,\n            checked_at      TIMESTAMP,\n            check_name      STRING,\n            column_name     STRING,\n            check_type      STRING,\n            rows_checked    LONG,\n            rows_failed     LONG,\n            failure_pct     DOUBLE,\n            threshold_pct   DOUBLE,\n            passed          BOOLEAN\n        ) USING DELTA\n    \"\"\")\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Section 3 \u2014 Pipeline Metrics Writer<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Call write_pipeline_metrics() at the end of every run \u2014 in both the success path and the except block. The completed_at timestamp is captured inside the function so it always reflects when the work actually finished.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>def write_pipeline_metrics(\n    pipeline_name, started_at, rows_processed,\n    rows_quarantined, run_id, pipeline_version='1.0',\n    slo_target_minutes=45, status='success', error_message=None\n):\n    completed_at = datetime.utcnow()\n    duration_minutes = (completed_at - started_at).total_seconds() \/ 60\n    slo_met = duration_minutes &lt;= slo_target_minutes\n\n    row = Row(\n        run_id              = run_id,\n        pipeline_name       = pipeline_name,\n        pipeline_version    = pipeline_version,\n        started_at          = started_at,\n        completed_at        = completed_at,\n        duration_minutes    = round(duration_minutes, 2),\n        rows_processed      = rows_processed,\n        rows_quarantined    = rows_quarantined,\n        slo_target_minutes  = slo_target_minutes,\n        slo_met             = slo_met,\n        status              = status,\n        error_message       = error_message\n    )\n\n    # Append only \u2014 this table is an audit log, never overwrite\n    spark.createDataFrame(&#91;row]) \\\n        .write.format('delta').mode('append').saveAsTable(METRICS_TABLE)\n\n    return duration_minutes, slo_met\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Section 4 \u2014 Quality Check Runner<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Call run_quality_check() once per check. Each call appends one row to pipeline_quality_checks. The condition argument is a PySpark Column expression that evaluates to True for rows that <strong>fail<\/strong> the check.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>def run_quality_check(\n    df, run_id, pipeline_name, check_name,\n    column_name, check_type, condition, threshold_pct=1.0\n):\n    rows_checked = df.count()\n    rows_failed  = df.filter(condition).count()\n    failure_pct  = (rows_failed \/ rows_checked * 100) if rows_checked &gt; 0 else 0.0\n    passed       = failure_pct &lt;= threshold_pct\n\n    spark.createDataFrame(&#91;Row(\n        run_id        = run_id,\n        pipeline_name = pipeline_name,\n        checked_at    = datetime.utcnow(),\n        check_name    = check_name,\n        column_name   = column_name,\n        check_type    = check_type,\n        rows_checked  = rows_checked,\n        rows_failed   = rows_failed,\n        failure_pct   = round(failure_pct, 4),\n        threshold_pct = threshold_pct,\n        passed        = passed\n    )]).write.format('delta').mode('append').saveAsTable(QUALITY_TABLE)\n\n    return passed\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Example Checks<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code># Null check \u2014 zero tolerance on primary key\nrun_quality_check(\n    df=silver_df, run_id=run_id,\n    pipeline_name=PIPELINE_NAME, check_name='patient_id_not_null',\n    column_name='patient_id', check_type='null_check',\n    condition=col('patient_id').isNull(), threshold_pct=0.0\n)\n\n# Range check \u2014 claim amounts must be positive\nrun_quality_check(\n    df=silver_df, run_id=run_id,\n    pipeline_name=PIPELINE_NAME, check_name='claim_amount_positive',\n    column_name='claim_amount', check_type='range_check',\n    condition=col('claim_amount') &lt;= 0, threshold_pct=0.5\n)\n\n# Volume check \u2014 row count must not drop vs last run\nrun_quality_check(\n    df=silver_df, run_id=run_id,\n    pipeline_name=PIPELINE_NAME, check_name='volume_vs_last_run',\n    column_name='*', check_type='volume_check',\n    condition=lit(rows_processed &lt; expected_min), threshold_pct=0.0\n)\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Section 5 \u2014 The Pipeline Wrapper<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">This is the pattern used in every pipeline notebook. It ties together start time, run_id, transformation logic, quality checks, metrics writing, and alerting in a single try\/except block.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>start_time        = datetime.utcnow()   # capture before any work\nrun_id            = str(uuid.uuid4())   # shared key between both tables\nrows_processed    = 0\nrows_quarantined  = 0\nquality_failures  = &#91;]\n\ntry:\n    # STEP 1 \u2014 READ\n    bronze_df      = spark.table('bronze.claims')\n    rows_processed = bronze_df.count()\n\n    # STEP 2 \u2014 TRANSFORM\n    silver_df = transform_bronze_to_silver(bronze_df)\n\n    # STEP 3 \u2014 QUALITY CHECKS (before writing)\n    if not run_quality_check(...):   # null check\n        quality_failures.append('patient_id_not_null')\n\n    if not run_quality_check(...):   # range check\n        quality_failures.append('claim_amount_positive')\n\n    # STEP 4 \u2014 WRITE (split passing and failing rows)\n    silver_df.filter(col('quality_flag') == 'pass') \\\n        .write.format('delta').mode('append').saveAsTable('silver.claims')\n\n    silver_df.filter(col('quality_flag') == 'fail') \\\n        .write.format('delta').mode('append').saveAsTable('silver.claims_quarantine')\n\n    # STEP 5 \u2014 WRITE METRICS\n    duration, slo_met = write_pipeline_metrics(\n        pipeline_name      = PIPELINE_NAME,\n        started_at         = start_time,\n        rows_processed     = rows_processed,\n        rows_quarantined   = rows_quarantined,\n        run_id             = run_id,\n        slo_target_minutes = 45,\n        status             = 'success'\n    )\n\n    # STEP 6 \u2014 ALERT if needed\n    if not slo_met:\n        send_alert(PIPELINE_NAME, run_id, f'SLO breached: {duration:.1f} min')\n    if quality_failures:\n        send_alert(PIPELINE_NAME, run_id, f'Checks failed: {quality_failures}')\n\nexcept Exception as e:\n    # Still write metrics so the failure is recorded\n    write_pipeline_metrics(..., status='failed', error_message=str(e))\n    send_alert(PIPELINE_NAME, run_id, f'Pipeline FAILED: {str(e)}')\n    raise   # re-raise so Databricks marks the job as failed\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Section 6 \u2014 Monitoring Queries<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">SLO Summary \u2014 Last 7 Days<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>SELECT\n    pipeline_name,\n    COUNT(*)                                            AS total_runs,\n    SUM(CASE WHEN slo_met THEN 1 ELSE 0 END)           AS slo_met_count,\n    SUM(CASE WHEN NOT slo_met THEN 1 ELSE 0 END)       AS slo_breach_count,\n    ROUND(AVG(duration_minutes), 1)                    AS avg_duration_min\nFROM monitoring.pipeline_ops.pipeline_metrics\nWHERE started_at &gt;= CURRENT_DATE - INTERVAL 7 DAYS\nGROUP BY pipeline_name\nORDER BY slo_breach_count DESC;\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Full Picture for One Run \u2014 Operational + Quality<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>SELECT\n    m.pipeline_name,\n    m.duration_minutes,\n    m.slo_met,\n    q.check_name,\n    q.failure_pct,\n    q.passed\nFROM monitoring.pipeline_ops.pipeline_metrics m\nJOIN monitoring.pipeline_ops.pipeline_quality_checks q\n    ON m.run_id = q.run_id\nWHERE m.pipeline_name = 'silver_transform_claims'\n  AND DATE(m.started_at) = CURRENT_DATE\nORDER BY q.passed ASC;   -- failed checks first\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Quarantine Rate Trend \u2014 Last 30 Days<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>SELECT\n    pipeline_name,\n    DATE(started_at)                                            AS run_date,\n    SUM(rows_quarantined)                                       AS quarantined,\n    ROUND(SUM(rows_quarantined) * 100.0\n          \/ NULLIF(SUM(rows_processed), 0), 2)                 AS quarantine_pct\nFROM monitoring.pipeline_ops.pipeline_metrics\nWHERE started_at &gt;= CURRENT_DATE - INTERVAL 30 DAYS\nGROUP BY pipeline_name, DATE(started_at)\nORDER BY run_date DESC;\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Key Rules to Live By<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>One shared metrics table for all pipelines<\/strong> \u2014 pipeline_name separates them<\/li>\n\n\n\n<li><strong>Use the same run_id in both tables<\/strong> \u2014 it is the foreign key linking operational and quality data<\/li>\n\n\n\n<li><strong>Always write metrics in the except block too<\/strong> \u2014 a failed run still needs a record<\/li>\n\n\n\n<li><strong>Always re-raise after writing failure metrics<\/strong> \u2014 so Databricks marks the job as failed<\/li>\n\n\n\n<li><strong>Store all secrets in Databricks Secret Scopes<\/strong> \u2014 never hardcode credentials<\/li>\n\n\n\n<li><strong>Bump pipeline_version on significant logic changes<\/strong> \u2014 enables before\/after comparisons<\/li>\n\n\n\n<li><strong>Set threshold_pct = 0.0 for zero-tolerance checks<\/strong> like primary keys<\/li>\n\n\n\n<li><strong>Alert on SLO breach and quality failures separately<\/strong> \u2014 they have different on-call responses<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Closing Thoughts<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Data pipelines are only as trustworthy as your ability to verify them. This framework doesn&#8217;t try to replace full observability platforms \u2014 it gives Databricks teams a structured, queryable, audit-grade foundation that takes an afternoon to deploy and pays dividends every time something goes wrong.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you&#8217;re running pipelines in Databricks today without structured monitoring, this is a very good place to start.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Write metrics on failure, not just success.<\/strong> An unrecorded failure is a blind spot. Always write to the metrics table in the except block.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Re-raise after logging.<\/strong> Your monitoring framework should record what happened. Your orchestrator should know the job failed. These are separate responsibilities.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Make thresholds explicit.<\/strong> A quality check with no documented tolerance is a quality check waiting to cause confusion. Store the threshold alongside the result.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Name pipelines consistently.<\/strong> A naming convention like silver_transform_claims makes filtering by layer trivial and keeps dashboards readable as the number of pipelines grows.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p class=\"wp-block-paragraph\">Data pipelines are only as trustworthy as your ability to verify them. A monitoring framework doesn&#8217;t have to be complex to be effective \u2014 it has to be consistent, complete, and queryable. This one is all three.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you&#8217;re running pipelines in Databricks today without structured monitoring, this is a good place to start.<\/p>\n\n\n\n<div style=\"height:44px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n","protected":false},"excerpt":{"rendered":"<p>How We Built a Custom Monitoring Framework for Databricks Pipelines Data pipelines fail silently. That&#8217;s the uncomfortable truth most data engineering teams learn the hard way \u2014 usually when a business stakeholder asks why last week&#8217;s report looks off, or why a dashboard hasn&#8217;t refreshed since Tuesday. By then, the damage is done. Rather than [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-4556","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.0 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Databricks Pipeline Monitoring Framework - Rana Nasri Ghazzi<\/title>\n<meta name=\"description\" content=\"Explore Rana Ghazzi&#039;s data analytics portfolio \u2014 dashboards, visualizations, and insights built with Tableau, Power BI &amp; Python.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ranaghazzi.com\/?p=4556\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Databricks Pipeline Monitoring Framework - Rana Nasri Ghazzi\" \/>\n<meta property=\"og:description\" content=\"Explore Rana Ghazzi&#039;s data analytics portfolio \u2014 dashboards, visualizations, and insights built with Tableau, Power BI &amp; Python.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ranaghazzi.com\/?p=4556\" \/>\n<meta property=\"og:site_name\" content=\"Rana Nasri Ghazzi\" \/>\n<meta property=\"article:published_time\" content=\"2026-06-01T22:48:41+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-06-13T19:13:54+00:00\" \/>\n<meta name=\"author\" content=\"Rana Ghazzi\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Rana Ghazzi\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"8 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/ranaghazzi.com\\\/?p=4556#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ranaghazzi.com\\\/?p=4556\"},\"author\":{\"name\":\"Rana Ghazzi\",\"@id\":\"https:\\\/\\\/ranaghazzi.com\\\/#\\\/schema\\\/person\\\/d8ee34f53cb0df9faaf816fb5363a4cc\"},\"headline\":\"Databricks Pipeline Monitoring Framework\",\"datePublished\":\"2026-06-01T22:48:41+00:00\",\"dateModified\":\"2026-06-13T19:13:54+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/ranaghazzi.com\\\/?p=4556\"},\"wordCount\":768,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/ranaghazzi.com\\\/#\\\/schema\\\/person\\\/d8ee34f53cb0df9faaf816fb5363a4cc\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/ranaghazzi.com\\\/?p=4556#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/ranaghazzi.com\\\/?p=4556\",\"url\":\"https:\\\/\\\/ranaghazzi.com\\\/?p=4556\",\"name\":\"Databricks Pipeline Monitoring Framework - Rana Nasri Ghazzi\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ranaghazzi.com\\\/#website\"},\"datePublished\":\"2026-06-01T22:48:41+00:00\",\"dateModified\":\"2026-06-13T19:13:54+00:00\",\"description\":\"Explore Rana Ghazzi's data analytics portfolio \u2014 dashboards, visualizations, and insights built with Tableau, Power BI & Python.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/ranaghazzi.com\\\/?p=4556#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/ranaghazzi.com\\\/?p=4556\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/ranaghazzi.com\\\/?p=4556#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/ranaghazzi.com\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Databricks Pipeline Monitoring Framework\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/ranaghazzi.com\\\/#website\",\"url\":\"https:\\\/\\\/ranaghazzi.com\\\/\",\"name\":\"Rana Nasri Ghazzi\",\"description\":\"Turning Data into Decisions\",\"publisher\":{\"@id\":\"https:\\\/\\\/ranaghazzi.com\\\/#\\\/schema\\\/person\\\/d8ee34f53cb0df9faaf816fb5363a4cc\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/ranaghazzi.com\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\\\/\\\/ranaghazzi.com\\\/#\\\/schema\\\/person\\\/d8ee34f53cb0df9faaf816fb5363a4cc\",\"name\":\"Rana Ghazzi\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ranaghazzi.com\\\/wp-content\\\/uploads\\\/2025\\\/11\\\/logo.png\",\"url\":\"https:\\\/\\\/ranaghazzi.com\\\/wp-content\\\/uploads\\\/2025\\\/11\\\/logo.png\",\"contentUrl\":\"https:\\\/\\\/ranaghazzi.com\\\/wp-content\\\/uploads\\\/2025\\\/11\\\/logo.png\",\"width\":1024,\"height\":1024,\"caption\":\"Rana Ghazzi\"},\"logo\":{\"@id\":\"https:\\\/\\\/ranaghazzi.com\\\/wp-content\\\/uploads\\\/2025\\\/11\\\/logo.png\"},\"url\":\"https:\\\/\\\/ranaghazzi.com\\\/?author=2\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Databricks Pipeline Monitoring Framework - Rana Nasri Ghazzi","description":"Explore Rana Ghazzi's data analytics portfolio \u2014 dashboards, visualizations, and insights built with Tableau, Power BI & Python.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ranaghazzi.com\/?p=4556","og_locale":"en_US","og_type":"article","og_title":"Databricks Pipeline Monitoring Framework - Rana Nasri Ghazzi","og_description":"Explore Rana Ghazzi's data analytics portfolio \u2014 dashboards, visualizations, and insights built with Tableau, Power BI & Python.","og_url":"https:\/\/ranaghazzi.com\/?p=4556","og_site_name":"Rana Nasri Ghazzi","article_published_time":"2026-06-01T22:48:41+00:00","article_modified_time":"2026-06-13T19:13:54+00:00","author":"Rana Ghazzi","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Rana Ghazzi","Est. reading time":"8 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ranaghazzi.com\/?p=4556#article","isPartOf":{"@id":"https:\/\/ranaghazzi.com\/?p=4556"},"author":{"name":"Rana Ghazzi","@id":"https:\/\/ranaghazzi.com\/#\/schema\/person\/d8ee34f53cb0df9faaf816fb5363a4cc"},"headline":"Databricks Pipeline Monitoring Framework","datePublished":"2026-06-01T22:48:41+00:00","dateModified":"2026-06-13T19:13:54+00:00","mainEntityOfPage":{"@id":"https:\/\/ranaghazzi.com\/?p=4556"},"wordCount":768,"commentCount":0,"publisher":{"@id":"https:\/\/ranaghazzi.com\/#\/schema\/person\/d8ee34f53cb0df9faaf816fb5363a4cc"},"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/ranaghazzi.com\/?p=4556#respond"]}]},{"@type":"WebPage","@id":"https:\/\/ranaghazzi.com\/?p=4556","url":"https:\/\/ranaghazzi.com\/?p=4556","name":"Databricks Pipeline Monitoring Framework - Rana Nasri Ghazzi","isPartOf":{"@id":"https:\/\/ranaghazzi.com\/#website"},"datePublished":"2026-06-01T22:48:41+00:00","dateModified":"2026-06-13T19:13:54+00:00","description":"Explore Rana Ghazzi's data analytics portfolio \u2014 dashboards, visualizations, and insights built with Tableau, Power BI & Python.","breadcrumb":{"@id":"https:\/\/ranaghazzi.com\/?p=4556#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ranaghazzi.com\/?p=4556"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/ranaghazzi.com\/?p=4556#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/ranaghazzi.com\/"},{"@type":"ListItem","position":2,"name":"Databricks Pipeline Monitoring Framework"}]},{"@type":"WebSite","@id":"https:\/\/ranaghazzi.com\/#website","url":"https:\/\/ranaghazzi.com\/","name":"Rana Nasri Ghazzi","description":"Turning Data into Decisions","publisher":{"@id":"https:\/\/ranaghazzi.com\/#\/schema\/person\/d8ee34f53cb0df9faaf816fb5363a4cc"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ranaghazzi.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/ranaghazzi.com\/#\/schema\/person\/d8ee34f53cb0df9faaf816fb5363a4cc","name":"Rana Ghazzi","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ranaghazzi.com\/wp-content\/uploads\/2025\/11\/logo.png","url":"https:\/\/ranaghazzi.com\/wp-content\/uploads\/2025\/11\/logo.png","contentUrl":"https:\/\/ranaghazzi.com\/wp-content\/uploads\/2025\/11\/logo.png","width":1024,"height":1024,"caption":"Rana Ghazzi"},"logo":{"@id":"https:\/\/ranaghazzi.com\/wp-content\/uploads\/2025\/11\/logo.png"},"url":"https:\/\/ranaghazzi.com\/?author=2"}]}},"_links":{"self":[{"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=\/wp\/v2\/posts\/4556","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4556"}],"version-history":[{"count":30,"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=\/wp\/v2\/posts\/4556\/revisions"}],"predecessor-version":[{"id":5501,"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=\/wp\/v2\/posts\/4556\/revisions\/5501"}],"wp:attachment":[{"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4556"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4556"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4556"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}