What is orchestration in data engineering?
Running the steps of data pipelines in the right order and at the right time, retrying failures and alerting someone when a step breaks.
intermediate · Also called Workflow orchestration
A data platform rarely runs one pipeline. It runs dozens of steps that depend on each other. The orders must be loaded before they are cleaned, cleaned before they are summed, and summed before the morning dashboard refreshes. Orchestration is the part that runs those steps in the right order, at the right time, and says something when one fails.
A graph of tasks
Most orchestrators describe work the same way. Apache Airflow, an open-source orchestrator, calls a workflow a Dag, short for directed acyclic graph. It holds:
- Tasks: the units of work, such as "load orders" or "rebuild the revenue table".
- Dependencies: which task must finish before which other one starts.
- A schedule: when the workflow runs.
- Settings: such as how many times to retry a task that fails.
Databricks has the same model under other names. A Lakeflow Job is made of tasks, which can be notebooks, pipelines or Python scripts, and tasks can depend on each other.
When to run: the clock or the data
The oldest trigger is the clock: every day at 2 am. It is simple, and it is often wrong, because the clock does not know whether the data has arrived. If the source export is an hour late, the 2 am job runs happily on yesterday's files.
So orchestrators can now start work from the data itself. A Lakeflow Job can start when new files land in a storage location, when the tables it reads have been updated, or continuously, starting a new run as soon as the last one ends. Each of these starts the work when there is something new to work on.
When it fails
Orchestration earns its keep on the bad days. When a task fails, the orchestrator stops the tasks that depend on it, so a broken load does not flow into a report. It retries what is worth retrying, such as a timeout. And it tells a person: Lakeflow Jobs can send an email or call a webhook when a job fails or runs too long.
Retries need care. A retry runs the step again, so a step that appends rows will append them twice unless it was written to be safe to repeat. Writing each step so that running it twice gives the same result as running it once is most of what makes a retry safe.
- load_ordersWaitingload_customersWaiting
- clean_ordersWaiting
- build_revenueWaiting
- refresh_dashboardWaiting
The run’s log
- Nothing yet
Press Run. The 2 loads start together; every other task waits for the ones above it.
The job, its tasks and timings are made up. Lakeflow Jobs and Airflow both run tasks in dependency order and retry by a setting on the task, and Lakeflow Jobs can email someone when a run fails.
Why it matters for AI
An AI feature is a few more tasks in the same graph. The index an assistant searches should be refreshed after the documents are cleaned, not on a separate timer that might run first. A nightly job that rebuilds an evaluation set should run after the tables it reads.
When the AI steps are outside the orchestrator, on their own schedule, they drift out of step with the data. The assistant ends up answering from an index built before last night's load, and nothing records that it happened. Put them in the same graph, with the same dependencies and alerts, and lineageData LineageThe record of where a piece of data came from and everything it passed through, from source tables to the dashboards, models and AI answers built on it. can show what each answer was built on.
Sources