{"id":5184,"date":"2026-05-31T00:13:00","date_gmt":"2026-05-31T00:13:00","guid":{"rendered":"https:\/\/ranaghazzi.com\/?p=5184"},"modified":"2026-05-31T00:17:59","modified_gmt":"2026-05-31T00:17:59","slug":"5184","status":"publish","type":"post","link":"https:\/\/ranaghazzi.com\/?p=5184","title":{"rendered":""},"content":{"rendered":"<p><style>\n    .light-font-container, .light-font-container p, .light-font-container h2, .light-font-container li {<br \/>\n        font-weight: #FFFFFF !important;<br \/>\n    }<br \/>\n<\/style>\n<\/p>\n<div class=\"light-font-container\" style=\"background-color: #FFFFFF; padding: 40px; border-radius: 15px;\">\n\n\n<h2 class=\"wp-block-heading\">Databricks Auto Loader: Predefined Schemas vs. Schema Inference <\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Published:<\/strong> May 30, 2026<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Data engineers using Databricks Auto Loader often face a critical decision early in their pipeline development: should they define table schemas upfront, or let Auto Loader automatically infer them? It seems like a simple choice, but it has profound implications for data quality, pipeline reliability, and operational complexity.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In this post, I&#8217;ll break down both approaches, reveal why predefined schemas are the production standard, and show you how to get the best of both worlds using schema evolution modes.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">The Auto Loader Dilemma<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Auto Loader is Databricks&#8217; incremental data ingestion engine. It handles streaming, schema evolution, and data quality concerns\u2014making it the go-to for ingesting cloud data.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But here&#8217;s the question: when your data lands in cloud storage, should you:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Define the schema upfront and load data against that contract?<\/li>\n\n\n\n<li>Let Auto Loader discover the schema and evolve it as needed?<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">The answer isn&#8217;t black and white, but the evidence strongly favors predefined schemas in production.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Approach 1: Auto Loader Schema Inference (Ad-Hoc &amp; Quick)<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">How It Works<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">When you don&#8217;t provide a schema, Auto Loader automatically infers it by sampling the first 50 GB or 1,000 files\u2014whichever limit is crossed first. It then stores this schema in a _schemas directory and evolves it as new fields arrive.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">python<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># No schema provided\u2014Auto Loader infers it\ndf = spark.readStream \\\n    .format(\"cloudFiles\") \\\n    .option(\"cloudFiles.format\", \"csv\") \\\n    .option(\"cloudFiles.schemaLocation\", \"\/path\/to\/schema\") \\\n    .load(\"\/path\/to\/data\")<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Advantages<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">\u2705 <strong>Speed to First Load<\/strong>: Perfect for exploration and prototyping. You can start ingesting data without upfront schema design.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u2705 <strong>Automatic Schema Evolution<\/strong>: New fields are automatically detected and added to the schema.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u2705 <strong>Flexibility<\/strong>: Ideal for semi-structured data where the schema is genuinely unknown or highly volatile.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Disadvantages<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">\u274c <strong>Type Inference Issues<\/strong>: For untyped formats (CSV, JSON), Auto Loader defaults to inferring all columns as strings. This creates downstream headaches when you need numeric or timestamp operations.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u274c <strong>Non-Deterministic Results<\/strong>: Sampling variations can lead to inconsistent schema inference across different data batches, especially with heterogeneous files.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u274c <strong>Case Sensitivity Problems<\/strong>: Auto Loader arbitrarily chooses column name casing based on sampled data, potentially causing mismatches with downstream systems.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u274c <strong>Data Quality Blind Spot<\/strong>: Without an explicit schema contract, subtle data quality issues go undetected (a ZIP code stored as a number instead of a string, for example).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u274c <strong>Schema Creep<\/strong>: Over time, your schema becomes a dumping ground for random new fields, making it hard to distinguish intentional schema evolution from data corruption.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Approach 2: Predefined Schemas (Production-Ready)<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">How It Works<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">You define the expected schema upfront as a StructType, matching your understanding of the data source.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">python<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>from pyspark.sql.types import StructType, StructField, StringType, LongType, TimestampType\n\nschema = StructType(&#91;\n    StructField(\"customer_id\", LongType()),\n    StructField(\"name\", StringType()),\n    StructField(\"email\", StringType()),\n    StructField(\"created_at\", TimestampType()),\n    StructField(\"subscription_tier\", StringType())\n])\n\ndf = spark.readStream \\\n    .schema(schema) \\\n    .format(\"cloudFiles\") \\\n    .option(\"cloudFiles.format\", \"csv\") \\\n    .option(\"cloudFiles.schemaLocation\", \"\/path\/to\/schema\") \\\n    .load(\"\/path\/to\/data\")<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Advantages<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">\u2705 <strong>Type Safety<\/strong>: You enforce correct data types upfront. No more &#8220;customer_id&#8221; as a string by accident.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u2705 <strong>Data Quality Enforcement<\/strong>: Any mismatch between actual data and your schema is immediately visible\u2014great for catching upstream issues.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u2705 <strong>Predictability<\/strong>: Same schema every time. Your downstream consumers know exactly what they&#8217;re getting.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u2705 <strong>Performance<\/strong>: No sampling overhead. Auto Loader skips the inference phase and goes straight to parsing.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u2705 <strong>Documentation<\/strong>: Your schema serves as a living contract with data producers. &#8220;This is what we expect to receive.&#8221;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u2705 <strong>Regulatory Compliance<\/strong>: Many organizations require explicit data contracts. Predefined schemas make auditing and lineage tracking straightforward.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Disadvantages<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">\u274c <strong>Upfront Effort<\/strong>: You need to understand your data structure before building the pipeline.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u274c <strong>Maintenance Burden<\/strong>: Schema changes require code updates (though schema evolution modes help).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u274c <strong>Rigid<\/strong>: If your data source legitimately evolves, you need explicit handling.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">The Plot Twist: Schema Evolution Modes<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Here&#8217;s where it gets interesting. Databricks gives you a middle path: use a predefined schema AND enable schema evolution modes to handle changes gracefully.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Important Default Behavior<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">When you provide a predefined schema, Auto Loader&#8217;s default behavior changes:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Without a schema<\/strong>: addNewColumns is the default (automatically add new fields)<\/li>\n\n\n\n<li><strong>With a schema<\/strong>: <code>none<\/code> is the default (fail on schema changes)<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">This is intentional. With a strict schema, you want to know immediately when something unexpected arrives.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Schema Evolution Modes: Your Safety Net<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Mode 1: <code>rescue<\/code> (Recommended for Bronze\/Raw Layer)<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Behavior<\/strong>: Data keeps flowing. Unexpected columns and type mismatches are placed in a _rescued_data column as JSON.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">python<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>df = spark.readStream \\\n    .schema(schema) \\\n    .format(\"cloudFiles\") \\\n    .option(\"cloudFiles.format\", \"csv\") \\\n    .option(\"cloudFiles.schemaLocation\", \"\/path\/to\/schema\") \\\n    .option(\"cloudFiles.schemaEvolutionMode\", \"rescue\") \\\n    .option(\"cloudFiles.rescuedDataColumn\", \"_rescued_data\") \\\n    .load(\"\/path\/to\/data\")<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>When to use<\/strong>: Raw\/bronze layer ingestion where uptime is critical. You want to capture all data, even anomalies, for later investigation.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Example scenario<\/strong>: Your vendor adds a new field customer_segment to their CSV export. With rescue mode:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Your predefined schema stays unchanged<\/li>\n\n\n\n<li>The new field appears in _rescued_data<\/li>\n\n\n\n<li>Your pipeline keeps running<\/li>\n\n\n\n<li>You can inspect and handle it later<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Mode 2: <code>addNewColumns<\/code> (Flexible Evolution)<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Behavior<\/strong>: Pipeline fails on new columns, but Auto Loader automatically updates the schema. Restart the job, and it picks up the new column.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">python<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>.option(\"cloudFiles.schemaEvolutionMode\", \"addNewColumns\") \\<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>When to use<\/strong>: Semi-structured data that evolves gradually. Works well with Databricks Jobs configured for automatic restarts.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Caveat<\/strong>: This defeats some of the purpose of a predefined schema. If you want automatic evolution, you&#8217;re partly back to schema inference mode.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Mode 3: <code>addNewColumnsWithTypeWidening<\/code> (Smart Type Evolution)<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Behavior<\/strong>: Similar to addNewColumns, but also handles type widening (e.g., int \u2192 long, float \u2192 double) automatically without data rewriting.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>When to use<\/strong>: Data sources where numeric precision increases over time (DBR 16.4+).<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Mode 4: <code>none<\/code> (Strict Mode)<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Behavior<\/strong>: Pipeline fails and requires manual schema update or data file removal. No automatic recovery.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>When to use<\/strong>: Highly regulated environments (finance, healthcare) where schema drift must trigger alerts and require explicit approval.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Best Practice: The Hybrid Approach<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Here&#8217;s the pattern I recommend for production pipelines:<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Bronze Layer (Raw Ingestion)<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">python<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Predefined schema + rescue mode\n# Captures everything, enforces types, survives schema changes\n\nschema = StructType(&#91;\n    StructField(\"id\", LongType()),\n    StructField(\"event_type\", StringType()),\n    StructField(\"timestamp\", TimestampType())\n])\n\ndf = spark.readStream \\\n    .schema(schema) \\\n    .format(\"cloudFiles\") \\\n    .option(\"cloudFiles.format\", \"json\") \\\n    .option(\"cloudFiles.schemaLocation\", \"\/path\/to\/schema\") \\\n    .option(\"cloudFiles.schemaEvolutionMode\", \"rescue\") \\\n    .option(\"cloudFiles.rescuedDataColumn\", \"_rescued_data\") \\\n    .option(\"rescuedDataColumn\", \"_rescued_data\") \\\n    .load(\"\/bronze\/incoming\/events\")\n\ndf.writeStream \\\n    .option(\"checkpointLocation\", \"\/path\/to\/checkpoint\") \\\n    .mode(\"append\") \\\n    .table(\"bronze_events\")<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Silver Layer (Cleaned &amp; Validated)<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">python<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Inspect _rescued_data, validate against business rules\n# Promote validated fields to standard columns\n# Enforce stricter schema\n\ndf = spark.read.table(\"bronze_events\") \\\n    .filter(col(\"_rescued_data\").isNull()) \\\n    .select(\"id\", \"event_type\", \"timestamp\") \\\n    # Add validation logic here\n\ndf.write.mode(\"overwrite\").table(\"silver_events\")<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Gold Layer (Analytics Ready)<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">python<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Fully typed, fully validated, ready for BI tools\n# Use schema evolution mode \"none\" for maximum strictness<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This approach:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>\u2705 Captures all data without data loss (bronze)<\/li>\n\n\n\n<li>\u2705 Enforces schema contracts (silver)<\/li>\n\n\n\n<li>\u2705 Maintains data quality (gold)<\/li>\n\n\n\n<li>\u2705 Provides debugging visibility (_rescued_data)<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Decision Tree: Which Approach Should You Use?<\/h2>\n\n\n\n<pre class=\"wp-block-code\"><code>Is this a production pipeline?\n\u251c\u2500 NO \u2192 Use schema inference (addNewColumns mode)\n\u2502       Fast to prototype, fine for exploratory work\n\u2514\u2500 YES \u2192 Use predefined schema\n         \u251c\u2500 Is data structure stable &amp; well-known?\n         \u2502  \u251c\u2500 YES \u2192 Use \"none\" mode (strict)\n         \u2502  \u2502        Best for regulated industries\n         \u2502  \u2514\u2500 NO \u2192 Use \"rescue\" mode\n         \u2502          Capture unexpected data safely\n         \u2514\u2500 Is this the raw\/bronze layer?\n            \u251c\u2500 YES \u2192 Use \"rescue\" mode\n            \u2502        Maximize data capture\n            \u2514\u2500 NO \u2192 Use appropriate mode based on downstream needs<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Common Pitfalls to Avoid<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">\u274c Pitfall 1: Defining Schemas Too Narrowly<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">python<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># BAD: Assumes no future fields\nschema = StructType(&#91;\n    StructField(\"id\", LongType()),\n    StructField(\"name\", StringType())\n])\n.option(\"cloudFiles.schemaEvolutionMode\", \"none\")<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Fix<\/strong>: Use <code>rescue<\/code> mode to capture unexpected fields gracefully.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">\u274c Pitfall 2: Trusting Schema Inference for Untyped Formats<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">python<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># BAD: CSV \u2192 all string columns by default\ndf = spark.readStream \\\n    .format(\"cloudFiles\") \\\n    .option(\"cloudFiles.format\", \"csv\") \\\n    # No schema, so customer_id becomes a string!\n    .load(\"\/data\")<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Fix<\/strong>: Always provide a schema for CSV\/JSON files.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">\u274c Pitfall 3: Forgetting to Enable Rescued Data Column<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">python<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># BAD: rescue mode without explicitly enabling the column\n.option(\"cloudFiles.schemaEvolutionMode\", \"rescue\")\n# Where does unexpected data go? Unclear!<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Fix<\/strong>: Always pair rescue mode with explicit column naming:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">python<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>.option(\"cloudFiles.rescuedDataColumn\", \"_rescued_data\")<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">\u274c Pitfall 4: Ignoring Case Sensitivity<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">python<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># BAD: Source has \"CustomerID\", schema expects \"customer_id\"\nschema = StructType(&#91;\n    StructField(\"customer_id\", LongType())\n])<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Fix<\/strong>: Match casing or use readerCaseSensitive: false:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">python<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>.option(\"readerCaseSensitive\", \"false\")<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Real-World Example: E-Commerce Order Ingestion<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Let&#8217;s say you&#8217;re ingesting daily order files from a vendor via S3. Orders sometimes have extra fields (a new payment method appears, a regional field is added).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">python<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>from pyspark.sql.types import StructType, StructField, LongType, StringType, DoubleType, TimestampType\n\n# Define what you KNOW about orders\norder_schema = StructType(&#91;\n    StructField(\"order_id\", LongType()),\n    StructField(\"customer_id\", LongType()),\n    StructField(\"order_date\", TimestampType()),\n    StructField(\"amount\", DoubleType()),\n    StructField(\"status\", StringType()),\n    StructField(\"currency\", StringType())\n])\n\n# Read with schema safety + graceful evolution\ndf = spark.readStream \\\n    .schema(order_schema) \\\n    .format(\"cloudFiles\") \\\n    .option(\"cloudFiles.format\", \"parquet\") \\\n    .option(\"cloudFiles.schemaLocation\", \"\/checkpoint\/order_schema\") \\\n    .option(\"cloudFiles.schemaEvolutionMode\", \"rescue\") \\\n    .option(\"cloudFiles.rescuedDataColumn\", \"_rescued_data\") \\\n    .load(\"s3:\/\/vendor-bucket\/orders\/daily\/\")\n\n# Bronze: Capture everything\ndf.writeStream \\\n    .option(\"checkpointLocation\", \"\/checkpoint\/orders_bronze\") \\\n    .mode(\"append\") \\\n    .table(\"bronze_orders\")\n\n# Silver: Clean and validate\nspark.sql(\"\"\"\n    SELECT \n        order_id, customer_id, order_date, amount, status, currency,\n        CASE \n            WHEN _rescued_data IS NOT NULL THEN TRUE \n            ELSE FALSE \n        END AS has_unexpected_fields,\n        _rescued_data\n    FROM bronze_orders\n    WHERE order_date &gt;= current_date() - INTERVAL 1 DAY\n\"\"\").write.mode(\"overwrite\").table(\"silver_orders\")<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">With this setup:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>\u2705 Orders with new fields (e.g., region) don&#8217;t break the pipeline<\/li>\n\n\n\n<li>\u2705 You see new fields in _rescued_data for inspection<\/li>\n\n\n\n<li>\u2705 Your schema is intentional and documented<\/li>\n\n\n\n<li>\u2705 Data lineage is clear<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Final Verdict<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">For production pipelines, predefined schemas are the clear winner. They enforce data quality, provide explicit contracts, and prevent subtle bugs.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">However, don&#8217;t go full &#8220;rigid mode.&#8221; Instead:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Define your schema based on what you know about the data<\/li>\n\n\n\n<li>Use rescue mode for bronze\/raw layers to capture unexpected fields<\/li>\n\n\n\n<li>Investigate _rescued_data regularly to catch upstream issues<\/li>\n\n\n\n<li>Use stricter modes (like none) as you move to silver\/gold layers<\/li>\n\n\n\n<li>Document your schema as a contract between producers and consumers<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">This approach gives you the safety of predefined schemas with the flexibility to handle schema evolution gracefully.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Resources<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/docs.databricks.com\/aws\/en\/ingestion\/cloud-object-storage\/auto-loader\/schema\">Databricks Auto Loader Schema Configuration<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/docs.databricks.com\/aws\/en\/data-engineering\/schema-evolution\">Databricks Schema Evolution Modes<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/docs.databricks.com\/aws\/en\/ingestion\/cloud-object-storage\/auto-loader\/index.html\">Auto Loader Best Practices<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Databricks Auto Loader: Predefined Schemas vs. Schema Inference Published: May 30, 2026 Data engineers using Databricks Auto Loader often face a critical decision early in their pipeline development: should they define table schemas upfront, or let Auto Loader automatically infer them? It seems like a simple choice, but it has profound implications for data quality, [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-5184","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.2 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>- Rana Nasri Ghazzi<\/title>\n<meta name=\"description\" content=\"Explore Rana Ghazzi&#039;s data analytics portfolio \u2014 dashboards, visualizations, and insights built with Tableau, Power BI &amp; Python.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ranaghazzi.com\/?p=5184\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"- Rana Nasri Ghazzi\" \/>\n<meta property=\"og:description\" content=\"Explore Rana Ghazzi&#039;s data analytics portfolio \u2014 dashboards, visualizations, and insights built with Tableau, Power BI &amp; Python.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ranaghazzi.com\/?p=5184\" \/>\n<meta property=\"og:site_name\" content=\"Rana Nasri Ghazzi\" \/>\n<meta property=\"article:published_time\" content=\"2026-05-31T00:13:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-05-31T00:17:59+00:00\" \/>\n<meta name=\"author\" content=\"Rana Ghazzi\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Rana Ghazzi\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"8 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/ranaghazzi.com\/?p=5184#article\",\"isPartOf\":{\"@id\":\"https:\/\/ranaghazzi.com\/?p=5184\"},\"author\":{\"name\":\"Rana Ghazzi\",\"@id\":\"https:\/\/ranaghazzi.com\/#\/schema\/person\/d8ee34f53cb0df9faaf816fb5363a4cc\"},\"headline\":\"No title\",\"datePublished\":\"2026-05-31T00:13:00+00:00\",\"dateModified\":\"2026-05-31T00:17:59+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/ranaghazzi.com\/?p=5184\"},\"wordCount\":1171,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/ranaghazzi.com\/#\/schema\/person\/d8ee34f53cb0df9faaf816fb5363a4cc\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/ranaghazzi.com\/?p=5184#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/ranaghazzi.com\/?p=5184\",\"url\":\"https:\/\/ranaghazzi.com\/?p=5184\",\"name\":\"- Rana Nasri Ghazzi\",\"isPartOf\":{\"@id\":\"https:\/\/ranaghazzi.com\/#website\"},\"datePublished\":\"2026-05-31T00:13:00+00:00\",\"dateModified\":\"2026-05-31T00:17:59+00:00\",\"description\":\"Explore Rana Ghazzi's data analytics portfolio \u2014 dashboards, visualizations, and insights built with Tableau, Power BI & Python.\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/ranaghazzi.com\/?p=5184\"]}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/ranaghazzi.com\/#website\",\"url\":\"https:\/\/ranaghazzi.com\/\",\"name\":\"Rana Nasri Ghazzi\",\"description\":\"Turning Data into Decisions\",\"publisher\":{\"@id\":\"https:\/\/ranaghazzi.com\/#\/schema\/person\/d8ee34f53cb0df9faaf816fb5363a4cc\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/ranaghazzi.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/ranaghazzi.com\/#\/schema\/person\/d8ee34f53cb0df9faaf816fb5363a4cc\",\"name\":\"Rana Ghazzi\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/ranaghazzi.com\/wp-content\/uploads\/2025\/11\/logo.png\",\"url\":\"https:\/\/ranaghazzi.com\/wp-content\/uploads\/2025\/11\/logo.png\",\"contentUrl\":\"https:\/\/ranaghazzi.com\/wp-content\/uploads\/2025\/11\/logo.png\",\"width\":1024,\"height\":1024,\"caption\":\"Rana Ghazzi\"},\"logo\":{\"@id\":\"https:\/\/ranaghazzi.com\/wp-content\/uploads\/2025\/11\/logo.png\"},\"url\":\"https:\/\/ranaghazzi.com\/?author=2\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"- Rana Nasri Ghazzi","description":"Explore Rana Ghazzi's data analytics portfolio \u2014 dashboards, visualizations, and insights built with Tableau, Power BI & Python.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ranaghazzi.com\/?p=5184","og_locale":"en_US","og_type":"article","og_title":"- Rana Nasri Ghazzi","og_description":"Explore Rana Ghazzi's data analytics portfolio \u2014 dashboards, visualizations, and insights built with Tableau, Power BI & Python.","og_url":"https:\/\/ranaghazzi.com\/?p=5184","og_site_name":"Rana Nasri Ghazzi","article_published_time":"2026-05-31T00:13:00+00:00","article_modified_time":"2026-05-31T00:17:59+00:00","author":"Rana Ghazzi","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Rana Ghazzi","Est. reading time":"8 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ranaghazzi.com\/?p=5184#article","isPartOf":{"@id":"https:\/\/ranaghazzi.com\/?p=5184"},"author":{"name":"Rana Ghazzi","@id":"https:\/\/ranaghazzi.com\/#\/schema\/person\/d8ee34f53cb0df9faaf816fb5363a4cc"},"headline":"No title","datePublished":"2026-05-31T00:13:00+00:00","dateModified":"2026-05-31T00:17:59+00:00","mainEntityOfPage":{"@id":"https:\/\/ranaghazzi.com\/?p=5184"},"wordCount":1171,"commentCount":0,"publisher":{"@id":"https:\/\/ranaghazzi.com\/#\/schema\/person\/d8ee34f53cb0df9faaf816fb5363a4cc"},"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/ranaghazzi.com\/?p=5184#respond"]}]},{"@type":"WebPage","@id":"https:\/\/ranaghazzi.com\/?p=5184","url":"https:\/\/ranaghazzi.com\/?p=5184","name":"- Rana Nasri Ghazzi","isPartOf":{"@id":"https:\/\/ranaghazzi.com\/#website"},"datePublished":"2026-05-31T00:13:00+00:00","dateModified":"2026-05-31T00:17:59+00:00","description":"Explore Rana Ghazzi's data analytics portfolio \u2014 dashboards, visualizations, and insights built with Tableau, Power BI & Python.","inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ranaghazzi.com\/?p=5184"]}]},{"@type":"WebSite","@id":"https:\/\/ranaghazzi.com\/#website","url":"https:\/\/ranaghazzi.com\/","name":"Rana Nasri Ghazzi","description":"Turning Data into Decisions","publisher":{"@id":"https:\/\/ranaghazzi.com\/#\/schema\/person\/d8ee34f53cb0df9faaf816fb5363a4cc"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ranaghazzi.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/ranaghazzi.com\/#\/schema\/person\/d8ee34f53cb0df9faaf816fb5363a4cc","name":"Rana Ghazzi","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ranaghazzi.com\/wp-content\/uploads\/2025\/11\/logo.png","url":"https:\/\/ranaghazzi.com\/wp-content\/uploads\/2025\/11\/logo.png","contentUrl":"https:\/\/ranaghazzi.com\/wp-content\/uploads\/2025\/11\/logo.png","width":1024,"height":1024,"caption":"Rana Ghazzi"},"logo":{"@id":"https:\/\/ranaghazzi.com\/wp-content\/uploads\/2025\/11\/logo.png"},"url":"https:\/\/ranaghazzi.com\/?author=2"}]}},"_links":{"self":[{"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=\/wp\/v2\/posts\/5184","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=5184"}],"version-history":[{"count":14,"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=\/wp\/v2\/posts\/5184\/revisions"}],"predecessor-version":[{"id":5202,"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=\/wp\/v2\/posts\/5184\/revisions\/5202"}],"wp:attachment":[{"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=5184"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=5184"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ranaghazzi.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=5184"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}