Rana Nasri Ghazzi

Trigger Types in Databricks: Jobs vs. Pipelines

October 11, 2026

Two Different Things Called “Trigger”

Databricks has two separate layers that both use the word “trigger,” and conflating them is the most common source of confusion:

1. Job triggers — decide when a Databricks Job runs at all: on a schedule, when a file lands, when a table changes, or continuously.
2. Pipeline execution mode — a setting inside an LDP (Lakeflow Declarative Pipelines, formerly DLT) pipeline that decides whether, once started, it processes available data once and stops (Triggered) or keeps running and picking up new data forever (Continuous).

A Job can wrap a Pipeline task, a notebook, a Python script, or plain Structured Streaming code — the Job trigger controls when that task starts; what happens after it starts depends entirely on what’s inside the task.

The Four Job Trigger Types

Scheduled — runs on a cron expression you define.

Classic “every hour / every night at 2am” behavior. Good default when you don’t have (or need) an event signal — e.g. a nightly batch rollup.

"schedule": {
  "quartz_cron_expression": "0 0 * * * ?",
  "timezone_id": "UTC"
}

File arrival — runs automatically when new files land in a cloud storage path, using file-notification events rather than polling.

"trigger": {
  "file_arrival": {
    "url": "s3://my-bucket/raw-landing/",
    "min_time_between_triggers_seconds": 60,
    "wait_after_last_change_seconds": 30
  }
}

wait_after_last_change_seconds debounces a burst of file writes (e.g. a batch upload dropping 200 files) so you don’t fire the job 200 times — it waits for the writes to settle before triggering once. Natural fit for a bronze ingestion job sitting on top of a raw landing zone.

Table change — runs automatically when a specified Delta/Unity Catalog table receives new committed data, detected via the table’s own commit history rather than watching files.

"trigger": {
  "table_update": {
    "table_names": ["main.bronze.raw_events"],
    "min_time_between_triggers_seconds": 60,
    "wait_after_last_change_seconds": 30
  }
}

Use this instead of File arrival when your input is already a Delta table another pipeline maintains — you don’t need to watch raw files when the table’s commit log already tells you “something changed.”

Continuous — the job runs forever, restarting automatically on failure, instead of running-and-stopping per trigger.

"continuous": {
  "pause_status": "UNPAUSED"
}

This is the odd one out: the other three are about when to start a bounded run; Continuous means there’s no “bounded run” at all. Which is exactly where the restriction we ran into comes from —

Task dependencies are not allowed for continuous jobs, remove them to convert to a continuous job.

A continuous job must be a single task with no upstream/downstream dependencies, because “wait for task A, then run task B” doesn’t mean anything for something that never stops.

Where This Gets Confusing: Job Trigger vs. Pipeline Mode

Here’s the part that trips people up. Suppose you have a Job with one task, and that task runs an LDP pipeline. You now have two “is this continuous?” settings that look related but aren’t the same knob:

If your pipeline’s own mode is already Continuous, the pipeline runs forever on its own the moment it’s started — the Job task that launched it doesn’t need the Job’s trigger to also be Continuous. In fact, setting both tends to be redundant (and if the task has dependencies, outright blocked by the restriction above). A simple one-time “run now” or a basic Scheduled trigger is usually enough to kick off a Continuous-mode pipeline; the pipeline handles staying alive from there.

If your pipeline’s own mode is Triggered, it processes whatever’s currently available, then stops — here the Job trigger is doing real work, because it’s the only thing deciding how often a fresh run happens. File arrival or Table change are good matches: “new data showed up, go run a Triggered pipeline update.”

Where Raw Structured Streaming Differs

This two-layer split only exists because LDP pipelines have their own internal mode. A plain Structured Streaming script (a notebook calling .writeStream.start() + .awaitTermination()) has no such internal setting — it’s just code that blocks until it’s stopped or crashes. For that kind of task, the Job trigger is the only lifecycle control you get:

• Continuous job trigger = your only way to get “restart automatically if this stream dies.”
• Scheduled / File arrival / Table change = the job runs the script once per trigger event — typically written with .trigger(availableNow=True) so each run processes what’s currently available and then exits cleanly, turning an otherwise-continuous stream into a series of discrete batch-style runs.

So the same four trigger options exist for both kinds of tasks, but they mean something different underneath: for LDP, the Job trigger decides when to ask the pipeline to run; for raw Structured Streaming, the Job trigger decides the entire run/restart lifecycle, because there’s no pipeline layer underneath providing that for free.

Decision Guide

TL;DR

• Databricks has two separate “trigger” concepts: Job triggers (Scheduled, File arrival, Table change, Continuous) decide when a job runs; Pipeline execution mode (Triggered vs Continuous) decides how an LDP pipeline behaves once started.
• A Continuous job trigger requires a single task with no dependencies — you can’t mix “runs forever” with “task B waits for task A.”
• For an LDP pipeline, the pipeline’s own mode is the primary lifecycle control; the Job trigger mostly just decides when to kick off an update.
• For a raw Structured Streaming script, there’s no equivalent pipeline-level mode — the Job’s trigger type is the entire lifecycle control, including restart-on-failure behavior.
• Match the trigger to the actual data-arrival pattern: files → File arrival, upstream Delta table → Table change, no natural event signal → Scheduled, always-on single task → Continuous.