Structured Streaming vs Lakeflow Declarative Pipeline
Conceptually, Lakeflow Declarative Pipeline and Structured Streaming differ in who is in control,
The one core idea: Structured Streaming is imperative — you write how to run the stream.LDP is declarative — you write what the table should contain, and the framework runs it.
From that single distinction, everything conceptual follows:
- Execution control — In Structure Streaming, you start the query, set the trigger, and call awaitTermination(). In LDP the framework plans and runs the queries; you never start or stop them.
- State management (checkpoint & schema location) — In Structure Streaming, you must specify and manage checkpointLocation and schemaLocation. In LDP, It manages checkpoints, triggers, retries, and the actual start/stop of each underlying query itself, every time you run a pipeline update. If a table’s logic hasn’t changed and there’s no new data, LDP may skip re-running it entirely — again, your decision never comes into play.
- Orchestration & dependencies — In Structure Streaming, you wire tables together manually. In LDP , you reference one table from another (dp.read/readStream) and LDP builds the dependency graph and runs them in order.
- Data quality — A first-class concept in LDP (expect, expect_or_drop). Streaming has no built-in notion; you filter manually.
- Observability — LDP emits a structured event log as part of the framework. Structure Streaming exposes runtime hooks (StreamingQueryListener) you attach yourself.
Auto Loader / design choices that work the same way in both. Auto Loader (cloudFiles) is a source and is independent of whether LDP or Streaming is driving.
One-line summary: Structure Streaming = you orchestrate the pipeline; LDP = you declare the result and the framework orchestrates. That’s the only conceptual axis — format and parsing are orthogonal.
Which to use:
- Want a managed, declarative pipeline with built-in quality + event log → LDP.
- Want manual control, custom triggers, listener-based monitoring, or a simple scheduled batch-incremental job → this writeStream version.