Data Integration in Databricks: Every Way to Bring Data into the Lakehouse (2026 Edition)
Sep 26, 2026 · @Rana
Why data integration is the first job of any lakehouse
Databricks gives you at least ten ways to get data into the lakehouse, and picking the right one is mostly a question of where the data lives and how fast you need it. Batch files, change feeds from operational databases, SaaS records, event streams and shared datasets all land in Delta tables governed by Unity Catalog.
This post walks through each integration method, when to use it, and what Databricks has shipped recently that changes the picture.
1. Native ingestion tools
Databricks’ own ingestion stack now lives under the Lakeflow umbrella, and it should be your default starting point. Databricks recommends starting at the most managed layer and dropping down only when a source or requirement isn’t covered (docs).
Auto Loader
Auto Loader incrementally and idempotently loads new files from cloud object storage (S3, ADLS, GCS) as they arrive. It tracks which files it has processed, infers the schema, and handles schema evolution, so it scales to millions of files without custom bookkeeping. Use it for any recurring file drop: logs, exports, partner feeds.
Lakeflow Connect managed connectors
Lakeflow Connect is the fully managed, low-code way to replicate operational databases and SaaS apps into Unity Catalog-governed Delta tables. Pipelines run on serverless compute and read incrementally.
- Databases: SQL Server, MySQL, PostgreSQL, Oracle and more, via change data capture or the newer query-based capture.
- SaaS apps: Salesforce, Workday, ServiceNow, HubSpot, Google Analytics 4, Dynamics 365 and others.
- Unstructured sources: SharePoint, Google Drive, Confluence and Outlook, for AI and document-processing use cases.
As of June 2026 the catalog passed 100 managed connectors (Databricks blog).
Structured Streaming and Zerobus Ingest
Spark Structured Streaming reads continuous event data from Kafka, Azure Event Hubs, Amazon Kinesis and Pub/Sub. It remains the most flexible option when you need custom stateful logic.
Zerobus Ingest is the newer, Kafka-free alternative: producers write events straight into Delta tables over gRPC, REST or SDKs (Python, Java, Rust, Go, TypeScript), with writes landing in under 5 seconds. If you run a message bus only to get events into Databricks, Zerobus can remove it.
2. File and object storage integration
When data already sits in a cloud data lake, the simplest integration is to read it where it is.
Direct cloud storage access
Databricks reads and writes CSV, JSON, Parquet, Avro, ORC, XML and Delta directly from S3, ADLS Gen2 and GCS. The modern pattern is Unity Catalog external locations and storage credentials rather than legacy DBFS mounts, because access is then governed, audited and traced in lineage. Use COPY INTO or read_files for one-off or idempotent batch loads, and Auto Loader once the load becomes recurring.
Volumes and UI file uploads
For small, ad hoc datasets, users can upload local files (CSV, JSON, Excel and others) through the workspace UI into a Unity Catalog volume and turn them into managed tables in a few clicks. Volumes also store non-tabular files, such as PDFs, images and model artifacts, under the same permissions model as tables.
3. Database and API connectors
Not every source fits a managed connector, so Databricks keeps the classic connectivity options too.
JDBC/ODBC drivers
Spark’s JDBC reader connects to PostgreSQL, SQL Server, Oracle, MySQL and most relational databases, and pulls structured data with standard queries. It gives full control over partitioning and pushdown, but you own the incremental logic, retries and scheduling.
Query-based connectors (GA, May 2026)
A middle ground now exists. Lakeflow Connect’s query-based connectors ingest from databases by querying the source with a cursor column, with no CDC setup or ingestion gateway required. Supported sources include Oracle, Teradata, SQL Server, MySQL, MariaDB, PostgreSQL and every Lakehouse Federation source (May 2026 release notes). Pick this when DBAs won’t enable CDC logs but you still want managed, incremental loads.
Lakehouse Federation
Federation lets you query external databases (and other warehouses) in place through Unity Catalog foreign catalogs, without copying data. It suits exploration and low-volume joins; switch to ingestion when query volume or latency matters.
REST APIs
For applications without a connector, notebooks or jobs can call REST APIs directly and land the JSON payloads in Delta. For sources you reuse, Lakeflow’s Community Connectors (Beta) offer an open-source framework to package that logic as a shareable connector.
4. Open data sharing: from Delta Sharing to OpenSharing
Delta Sharing is now OpenSharing, a Linux Foundation project announced on June 10, 2026 (press release). The core idea is unchanged: providers share live data with recipients on any platform, with zero copy and no pipelines to maintain.
What’s new:
- AI assets: the protocol now covers agent skills, AI models and unstructured data, not just tables.
- Iceberg support: Apache Iceberg REST Catalog clients can now receive shares, widening who a provider can reach.
- On-premises data: storage partners implement the OpenSharing server, so on-prem estates connect to Unity Catalog without migration. MinIO is GA, with Everpure, Qumulo and VAST Data in or near private preview (Databricks blog).
Existing Delta Sharing deployments keep working; OpenSharing adds capabilities without breaking changes. For integration teams, this makes sharing a real alternative to ingestion whenever a partner already publishes the data you need.
5. Partner Connect and third-party tools
Most enterprises still run tools outside Databricks, and the platform is built to plug into them.
- Partner Connect: a built-in UI that connects validated partners (ingestion, BI, transformation, governance) to your workspace in a few clicks, handling credentials and compute setup for you.
- Transformation tools: dbt and Prophecy are common choices for modelling data once it has landed.
- Orchestrators: Azure Data Factory and Apache Airflow frequently trigger Databricks jobs and pipelines.
The trend, though, is consolidation. Lakeflow Jobs now offers External Orchestration, which can trigger Snowflake jobs, call REST APIs and send Slack or PagerDuty alerts from inside Databricks. Databricks is pitching it explicitly as a replacement for legacy orchestrators like Airflow (Databricks blog).
6. What’s new: the latest integration updates (2026)
The biggest shifts this year are 100+ managed connectors, Kafka-free event ingestion, and Delta Sharing’s evolution into OpenSharing. Here are the integration-related releases worth knowing, newest first.
| Date (2026) | Update | Why it matters |
|---|---|---|
| Sep 22 | Shopify connector (Beta) | Ingest orders, products, customers and inventory without custom API code |
| Sep 22 | Invite OpenSharing recipients by email (Beta) | No more swapping sharing identifiers or credential files |
| Sep 15 | Sharing foreign Delta tables (GA) | Share tables federated from OneLake, Hive metastore or AWS Glue without copying them |
| Sep 11 | Sharing foreign Iceberg tables and foreign schemas (GA) | Federated sources can now be shared onward, including to Iceberg clients |
| Sep 10 | Managed tables in Lakeflow pipelines (Beta) | import_checkpoint migrates existing Structured Streaming jobs without reprocessing |
| Sep 10 | Databricks Excel Add-in (GA) | Business users import governed tables into Excel and write data back to Unity Catalog |
| Sep 9 | Celigo, Anaplan and Anysphere connectors (Beta) | More SaaS sources in Lakeflow Connect |
| Sep 7 | Zerobus Ingest into default storage (Public Preview) | Event ingestion without configuring your own storage |
| Sep 1 | Apache Arrow for Zerobus Ingest (GA) | High-throughput columnar writes via Arrow Flight |
| Sep 1 | Automatic change data feed (GA) | Row-level changes computed at query time, with no per-table setup |
| Jun 16 | Lakeflow Connect passes 100 connectors; Lakeflow Designer GA | No-code, drag-and-drop pipelines for analysts |
| Jun 16 | Real-Time Mode for Spark Declarative Pipelines (Public Preview) | End-to-end latency as low as 5 ms |
| Jun 10 | OpenSharing announced | Delta Sharing successor, now under the Linux Foundation |
| May | Query-based connectors (GA) | Incremental database ingestion without CDC |
| May | Google Drive connector (Beta) | Structured and unstructured file ingestion from Drive |
| Apr 2 | Lakeflow Connect Free Tier | 100 free DBUs per workspace per day, about 100 million records |
Note that releases roll out in stages, so a feature can take a week or more to reach your account.
7. How to choose the right method
Start from the source and the freshness you need, then pick the most managed option that covers it.
| Source | Freshness needed | Recommended method | Fallback |
|---|---|---|---|
| Files landing in S3 / ADLS / GCS | Minutes to hours | Auto Loader | COPY INTO / read_files |
| One-off local file | Once | UI upload to a volume | Notebook read from a volume |
| Operational database, CDC allowed | Near real time | Lakeflow Connect CDC connector | JDBC + custom merge logic |
| Operational database, no CDC | Hourly / daily | Query-based connector | Lakehouse Federation |
| SaaS app (Salesforce, Workday, Shopify) | Hourly / daily | Lakeflow Connect managed connector | Partner tool or REST API |
| Event streams you produce | Seconds | Zerobus Ingest | Kafka + Structured Streaming |
| Existing Kafka / Kinesis / Event Hubs | Seconds to milliseconds | Structured Streaming (Real-Time Mode) | Spark Declarative Pipelines |
| Partner or cross-org data | Live | OpenSharing | Databricks Marketplace |
| Niche or in-house API | Varies | Community Connector | REST API in a job |
A simple rule: move data only when you must. Share or federate first, ingest when volume or performance demands it, and hand-code only for sources nothing else covers.
Conclusion
Databricks’ integration story in 2026 is about fewer tools and less data movement. Lakeflow Connect, Zerobus Ingest and OpenSharing now cover most sources natively, all governed by Unity Catalog. Classic options like JDBC, REST APIs and Partner Connect still fill the gaps.
If you are planning a new pipeline, audit which third-party ingestion tools you could retire, and check whether a share or federated query could replace a copy entirely.
Sources
- Databricks September 2026 release notes
- Databricks May 2026 release notes
- Lakeflow: A new era of agentic data engineering (Databricks blog, June 16, 2026)
- Databricks announces OpenSharing (press release, June 10, 2026)
- Introducing OpenSharing (Databricks blog)
- Lakeflow Connect Free Tier (Databricks blog, April 2, 2026)
- Standard connectors in Lakeflow Connect (Databricks docs)