Rana Nasri Ghazzi

The Six Lanes of Data Ingestion in Databricks: A Practical Guide

Every lakehouse project starts with the same unglamorous question: how does the data actually get in? Databricks offers a surprisingly rich set of answers, and that’s both a blessing and a source of confusion. Auto Loader, COPY INTO, Lakeflow Connect, Partner Connect, Kafka connectors, custom Python — they all “ingest data,” but they solve very different problems.

This post lays out a mental model I find useful: six lanes of ingestion, each defined by what your source looks like and how much engineering you want to own. By the end, you should be able to look at any new data source and know which lane it belongs in within seconds.

Lane 1: Auto Loader — for files that keep coming

Auto Loader (the cloudFiles source) is the workhorse for incremental file ingestion from cloud object storage — S3, ADLS, or GCS. You point it at a directory, and it detects and processes new files as they arrive, without you ever tracking what’s already been loaded.

python

df = (spark.readStream
  .format("cloudFiles")
  .option("cloudFiles.format", "json")
  .option("cloudFiles.schemaLocation", "/checkpoints/schema")
  .load("s3://bucket/raw/"))

What makes it the default choice for ongoing file feeds: it’s built on Structured Streaming, so it runs continuously or in cost-efficient triggered batches (Trigger.AvailableNow). Checkpointing gives you exactly-once guarantees — reruns never duplicate data. It infers and evolves schemas, capturing unexpected fields in a rescued-data column instead of failing. And for extreme volumes, file-notification mode taps cloud events (SQS, Event Grid) to scale to millions of files.

Reach for it when: logs, exports, IoT dumps, or vendor drops land in cloud storage on an ongoing basis. It is the standard bronze-layer ingestion tool and slots naturally into Lakeflow Declarative Pipelines.

Lane 2: Direct ingestion — for batch and ad-hoc loads

Not everything is a pipeline. Sometimes you just need to load some files, and Databricks has built-in paths for exactly that.

COPY INTO is an idempotent SQL command that loads files into a Delta table and remembers what it has loaded, so reruns skip already-processed files:

sql

COPY INTO my_table
FROM 's3://bucket/raw/'
FILEFORMAT = CSV
FORMAT_OPTIONS ('header' = 'true');

Alongside it sit the Add Data UI (drag-and-drop a CSV or Excel file straight into a table) and CTAS / read_files() for one-off loads or querying files in place.

Reach for it when: ingestion is occasional, batch-oriented, SQL-first, or exploratory. The usual rule of thumb: COPY INTO for thousands of files, Auto Loader once you’re into high volume or continuous arrival.

Lane 3: Lakeflow Connect — managed connectors for apps and databases

Lakeflow Connect is Databricks’ fully managed connector service for enterprise applications and databases — Salesforce, Workday, ServiceNow, Google Analytics, SharePoint, SQL Server (with change-data-capture-based incremental sync), and a growing list of others.

There’s no pipeline code at all. You pick a source in the UI or API, authenticate through a Unity Catalog connection object, select the tables or objects you want, and set a schedule. Databricks handles the incremental extraction, CDC mechanics, and writes everything into governed Delta tables.

Reach for it when: you want SaaS or database data in the lakehouse with minimal engineering effort and a native connector exists. It’s Databricks’ answer to “why pay a third party for simple ELT?”

Lane 4: Partner Connect — when the long tail calls

No platform covers every source natively. Through Partner Connect, Databricks integrates with third-party ELT vendors — Fivetran, Qlik, Informatica, Rivery, Hevo, and more — and automates the plumbing: it provisions the SQL warehouse, service principal, and credentials so the partner tool can write into your lakehouse in a few clicks.

The value is connector breadth. A tool like Fivetran has hundreds of pre-built connectors covering long-tail SaaS apps, plus years of accumulated handling for schema drift, API quirks, and CDC edge cases.

Reach for it when: your source isn’t covered by Lakeflow Connect, or your organization already standardizes on one of these tools. The trade-off is extra licensing cost and one more vendor in the stack.

Lane 5: Streaming event buses — Kafka, Kinesis, and friends

When data arrives as individual events rather than files, you skip object storage entirely. Spark has native Structured Streaming connectors for Kafka, Kinesis, Azure Event Hubs, Pulsar, and Pub/Sub, and you read the stream directly into Delta:

python

df = (spark.readStream
  .format("kafka")
  .option("kafka.bootstrap.servers", "broker:9092")
  .option("subscribe", "orders")
  .load())

Records arrive with the payload as binary plus metadata (key, partition, offset, timestamp); the first transformation is almost always from_json on the value column. Checkpointing tracks offsets, giving effectively exactly-once delivery into Delta even across restarts.

Two architectures are common here.

The direct path (bus → Structured Streaming → bronze) gives the lowest latency but ties you to the bus’s retention window.

The staged path (bus → Firehose or sink connector → cloud storage → Auto Loader) accepts a few minutes of delay in exchange for a durable raw archive and resilience to pipeline downtime. Many teams happily take that trade.

Reach for it when: clickstreams, telemetry, fraud signals, or CDC event feeds (think Debezium publishing database changes to Kafka) need seconds-level freshness.

Lane 6: Custom Python — the escape hatch

When nothing pre-built fits, you write it yourself — and there’s a clear maturity ladder.

Tier one is plain Python in a scheduled job: requests or a vendor SDK pulls from an API, the results become a DataFrame, and you write to Delta. You own everything the managed tools normally handle — secrets in secret scopes, watermark tables for incrementality, MERGE INTO for idempotency, retries and rate limiting.

Tier two is Spark’s JDBC reader for databases without a managed connector, with partitionColumn options to parallelize the read and watermark subqueries for incremental pulls.

Tier three is the PySpark Python Data Source API (DBR 15.3+ / Spark 4), which lets you package ingestion logic as a first-class Spark format: implement a DataSource and reader class, register it, and then spark.read.format(“myapi”) works everywhere — batch, streaming with proper offset tracking, even inside Lakeflow Declarative Pipelines. You are, in effect, building your own mini-connector with the same semantics the Kafka connector has.

Reach for it when: the source is a proprietary internal API, an obscure SaaS, SFTP drops, mainframe extracts — anything without an existing connector. The common path is to prototype in a notebook, harden into a watermarked job, and eventually refactor into a custom data source (or retire it the day an official connector ships).

The decision flow

Strip away the product names and the choice comes down to what your source looks like:

One destination, six roads

Here’s the part that keeps the architecture sane: no matter which lane you take, everything converges on the same landing zone — Delta tables in Unity Catalog, typically a bronze layer. From there, Lakeflow Declarative Pipelines or jobs take over for the silver and gold transformations, and the downstream architecture is identical regardless of how the data arrived.

The ingestion lane only decides how data gets to bronze. Choose the lane that matches your source’s shape and your team’s appetite for owning pipeline code, and let the lakehouse handle the rest. Most real organizations end up using three or four lanes at once — Lakeflow Connect for Salesforce, Auto Loader for log files, Kafka for clickstream, a custom Python job for that one internal API — and that’s not a smell. It’s the design working as intended.