# Airflow vs Dagster vs Prefect: how the 3 orchestrators differ and which to choose

Apache Airflow, Dagster and Prefect all run data pipelines on a schedule, retry what fails and show every run. Airflow schedules Dags of tasks and has managed services on AWS and Google Cloud. Dagster is built around assets, the tables and files a pipeline produces, so lineage, daily partitions and data checks come with the model. Prefect turns ordinary Python functions into flows and tasks with a decorator, and is the lightest to start. Choose Airflow for its reach and managed options, Dagster when the data itself should be the unit you schedule and check, and Prefect when your pipelines are Python code that should stay plain Python.

- Page: https://tecyfy.com/data-engineering/airflow-vs-dagster-vs-prefect
- Part of: Data engineering (https://tecyfy.com/data-engineering)

- **Apache Airflow** (Apache Software Foundation project, started at Airbnb, https://airflow.apache.org/docs/apache-airflow/stable/): A platform that schedules Dags, Python files that describe tasks and the order they run in, and runs each task on a worker. Install: `pip install apache-airflow`
- **Dagster** (Dagster Labs, https://docs.dagster.io/): An orchestrator built around assets, the tables, files and models a pipeline produces, each defined in code with what it depends on. Install: `pip install dagster dagster-webserver`
- **Prefect** (Prefect Technologies, https://docs.prefect.io/v3/): A Python library that makes functions into flows and tasks with a decorator, with a server or Prefect Cloud to schedule and watch them. Install: `pip install prefect`

All 3 are [orchestrators](https://tecyfy.com/concepts/orchestration): they start each [pipeline](https://tecyfy.com/concepts/data-pipeline) on time, run its steps in the right order, retry what fails, and keep a record of every run you can look at the next morning. They differ in what they think a pipeline is. Airflow sees tasks to run, Dagster sees data to build, and Prefect sees Python code to watch.

## What each one is

**Apache Airflow** started at Airbnb and is now an Apache project, run as a managed service by AWS and Google Cloud. A pipeline is a Dag, a Python file that lists tasks and the order between them. Each run covers a data interval, so a daily Dag's run for 15 October loads 15 October, whenever it actually runs, and a backfill is a range of past intervals. Airflow also has assets, so a Dag can start when another Dag has updated the data it needs, and its task SDK lets tasks report back through the API server without reaching into the database.

**Dagster** starts from the other end. You define assets, the tables, files and models the pipeline produces, and each asset names the assets it is built from. The order of work, the lineage graph and the record of which days are built all come from that. Daily partitions, checks that can stop a bad table from feeding the next one, and an integration that loads every dbt model as an asset are part of the same model.

**Prefect** asks the least of your code. Put `@flow` on a function and `@task` on the functions it calls, and they gain retries, caching, logs and a record in the UI. The code keeps its own if statements and loops, so what runs is decided as it runs. To run it on a schedule, a deployment adds the schedule, and either a long-running `serve()` process or workers on a work pool start the runs.

## Side by side

| | Apache Airflow | Dagster | Prefect |
| --- | --- | --- | --- |
| **What you schedule** | A Dag, a set of tasks with the order between them. Each run covers a data interval. | Assets, the data a pipeline makes. A job selects assets, and a schedule or an automation condition runs it. | A flow, a Python function that calls tasks. A deployment gives it a schedule. |
| **How the order is known** | You set it with >> or by passing one task's result to the next. | From each asset's inputs, which name the assets it depends on. That is also the lineage graph. | From the code as it runs. Ordinary if statements and loops decide what runs. |
| **Schedules and triggers** | Cron or timedelta schedules, timetables, and assets, which start a Dag when another Dag updates them. | Cron schedules, sensors, and automation conditions such as on_cron, which waits for upstream assets to update. | Cron, interval and RRule schedules on a deployment, and event triggers. |
| **Retries** | retries and retry_delay on each task, with retry_exponential_backoff and max_retry_delay. | A RetryPolicy with max_retries, a delay, exponential backoff and jitter. | retries and retry_delay_seconds on a task or a flow, a list of delays, or exponential_backoff. |
| **One run per day of data** | Built in. Each run has its data interval, and backfill runs a range of past intervals. | Daily partitions on an asset. Each day is materialized, rerun and backfilled on its own. | A parameter you pass, such as the day. Backfills are runs with past values. |
| **Stopping on bad data** | A check task that fails stops the tasks after it, because the default trigger rule waits for all upstream tasks to succeed. | Asset checks. A blocking check that fails stops the assets downstream from being built. | A task that raises an error fails the flow there, as in any Python program. |
| **What runs in production** | A scheduler, a Dag processor, an API server with the UI, a metadata database, workers, and a triggerer for deferred tasks. | A webserver, a daemon for schedules, sensors and run queues, a code location server per project and a database. | A Prefect server and its database, or Prefect Cloud, plus a process that serves flows or workers that pick up runs from a work pool. |
| **Run it on a laptop** | dag.test() runs a real Dag run in one Python process. | materialize() builds assets in one process; dagster dev starts the UI and the daemon. | Call the flow like a function. A temporary server starts if none is running. |
| **Managed service** | Amazon MWAA, Google Cloud's Managed Service for Apache Airflow (formerly Cloud Composer) and others. | Dagster+, Serverless or Hybrid, from Dagster Labs. | Prefect Cloud, from Prefect Technologies. |
| **Licence** | Apache 2.0. | Apache 2.0. | Apache 2.0. |

## Which one fits

- **A managed service on AWS or Google Cloud:** Apache Airflow. Amazon MWAA and Google Cloud's Managed Service for Apache Airflow run Airflow inside the cloud account you already have, with its sign-in and billing.
- **Lineage, and checks that stop bad data:** Dagster. Assets give the lineage graph from the code itself, and a blocking asset check keeps a bad table from feeding the ones after it.
- **Each day of data built and rebuilt on its own:** Apache Airflow or Dagster. Airflow gives every run a data interval and backfills ranges of them; Dagster's daily partitions keep each day of an asset apart, and a backfill fills in the days that do not exist yet or rebuilds a chosen set.
- **Python scripts that should become scheduled jobs with little change:** Prefect. A decorator on the functions you already have makes a flow and its tasks, and the code keeps its own if statements and loops.
- **A team that already runs Airflow:** Apache Airflow. Its Dags, its providers and the team's habits carry over, and Airflow itself schedules on assets too, so data-driven runs do not need a new tool.
- **Runs that start when something happens, not on a clock:** Apache Airflow or Dagster or Prefect. All 3 can. Airflow starts a Dag when an asset updates, Dagster has sensors and automation conditions, and Prefect has event triggers on deployments.

## The same pipeline in each

The same daily pipeline in each: read the day's orders from a CSV file, stop if any amount is negative, sum revenue by country and write the day's report, overwriting it if the day runs again. The read retries 3 times, a minute apart, and the pipeline runs every day at 06:00 UTC. Each file was run for 15 October on a laptop and wrote the same report, {"DE": 162.0, "FR": 80.5, "IN": 15.25}.

### Apache Airflow: `daily_revenue_airflow.py`

```python
import csv, json, os
import pendulum
from airflow.sdk import dag, task


@dag(schedule="0 6 * * *", start_date=pendulum.datetime(2026, 10, 1, tz="UTC"), catchup=False)
def daily_revenue():

    @task(retries=3, retry_delay=pendulum.duration(minutes=1))
    def extract(ds=None) -> list[dict]:
        # ds is the run's date, so a rerun or a backfill reads its own day.
        with open(f"orders/{ds}.csv") as f:
            return list(csv.DictReader(f))

    @task
    def revenue_by_country(orders: list[dict]) -> dict:
        totals: dict[str, float] = {}
        for o in orders:
            if float(o["amount"]) < 0:
                raise ValueError(f"negative amount in {o['order_id']}")
            totals[o["country"]] = totals.get(o["country"], 0) + float(o["amount"])
        return totals

    @task
    def load(totals: dict, ds=None) -> None:
        os.makedirs("reports", exist_ok=True)
        with open(f"reports/{ds}.json", "w") as f:  # overwrite: the same day gives the same file
            json.dump(totals, f, sort_keys=True)
        print(totals)

    load(revenue_by_country(extract()))


dag_object = daily_revenue()

if __name__ == "__main__":
    # One real run of the whole Dag in this process, for 15 October.
    dag_object.test(logical_date=pendulum.datetime(2026, 10, 15, tz="UTC"))
```

The TaskFlow API makes each function a task, and passing one task's result to the next sets the order (the results travel as XComs, so keep them small). ds is the run's date, filled in by Airflow, so a backfill of 15 October reads 15 October's file. dag.test() needs Airflow's database (airflow db migrate) and the file's folder as the Dags folder.

### Dagster: `daily_revenue_dagster.py`

```python
import csv, json, os
import dagster as dg

days = dg.DailyPartitionsDefinition(start_date="2026-10-01")


@dg.asset(
    partitions_def=days,
    retry_policy=dg.RetryPolicy(max_retries=3, delay=60),
    # A check that runs with the asset; blocking: if it fails, the assets downstream are not built.
    check_specs=[dg.AssetCheckSpec("no_negative_amounts", asset="orders", blocking=True)],
)
def orders(context: dg.AssetExecutionContext):
    # Each day is a partition: a rerun or a backfill reads its own day.
    with open(f"orders/{context.partition_key}.csv") as f:
        rows = list(csv.DictReader(f))
    yield dg.Output(rows)
    yield dg.AssetCheckResult(check_name="no_negative_amounts", passed=all(float(o["amount"]) >= 0 for o in rows))


@dg.asset(partitions_def=days)
def revenue_by_country(context: dg.AssetExecutionContext, orders: list[dict]) -> dict:
    totals: dict[str, float] = {}
    for o in orders:
        totals[o["country"]] = totals.get(o["country"], 0) + float(o["amount"])
    os.makedirs("reports", exist_ok=True)
    with open(f"reports/{context.partition_key}.json", "w") as f:  # overwrite the day's file
        json.dump(totals, f, sort_keys=True)
    context.log.info(str(totals))
    return totals


daily_job = dg.define_asset_job("daily_revenue", selection=[orders, revenue_by_country])

defs = dg.Definitions(
    assets=[orders, revenue_by_country],
    jobs=[daily_job],
    schedules=[dg.build_schedule_from_partitioned_job(daily_job, hour_of_day=6)],
)

if __name__ == "__main__":
    # Build both assets, with the check, for the 15 October partition, in this process.
    result = dg.materialize([orders, revenue_by_country], partition_key="2026-10-15")
    print(result.success, result.output_for_node("revenue_by_country"))
```

There are no tasks here, only the 2 tables the pipeline makes. revenue_by_country takes orders as an input, and that is the whole dependency. Each day is a partition, and the check runs with the orders asset: when it fails, as it did in a test with a negative amount, revenue_by_country is not built. The schedule runs only under dagster dev or the daemon.

### Prefect: `daily_revenue_prefect.py`

```python
import csv, json, os, sys
from datetime import date, timedelta
from prefect import flow, task


@task(retries=3, retry_delay_seconds=60)
def extract(day: str) -> list[dict]:
    with open(f"orders/{day}.csv") as f:
        return list(csv.DictReader(f))


@task
def revenue_by_country(orders: list[dict]) -> dict:
    totals: dict[str, float] = {}
    for o in orders:
        if float(o["amount"]) < 0:
            raise ValueError(f"negative amount in {o['order_id']}")
        totals[o["country"]] = totals.get(o["country"], 0) + float(o["amount"])
    return totals


@task
def load(totals: dict, day: str) -> None:
    os.makedirs("reports", exist_ok=True)
    with open(f"reports/{day}.json", "w") as f:  # overwrite: the same day gives the same file
        json.dump(totals, f, sort_keys=True)


@flow(log_prints=True)
def daily_revenue(day: str | None = None) -> dict:
    # The day is a parameter, so a rerun or a backfill passes the day it wants; by default, yesterday.
    day = day or (date.today() - timedelta(days=1)).isoformat()
    totals = revenue_by_country(extract(day))
    load(totals, day)
    print(totals)
    return totals


if __name__ == "__main__":
    if sys.argv[1:] == ["serve"]:
        # A long-running process that runs the flow at 06:00 every day.
        daily_revenue.serve(name="daily-revenue", cron="0 6 * * *")
    else:
        daily_revenue(day="2026-10-15")  # one run, now, in this process
```

It is a Python script with 4 decorators. The day is an ordinary parameter, so a backfill is a run with an older day. serve() keeps a process running that starts the flow at 06:00, but only against a real Prefect server (prefect server start) or Prefect Cloud: on the temporary server it warns that it cannot schedule.

The 3 files do the same job, and the differences are in what each one makes you name. The Airflow file names tasks and lets Airflow hand the day to each one, which is why the same file can be backfilled for any past day. The Dagster file names no tasks at all, only the 2 things that exist at the end, and the check sits on the data it guards. The Prefect file is the plainest Python of the 3, and the day is just a parameter.

What the files do not show is what runs around them. The Airflow file needs a scheduler, a Dag processor, an API server and a database before 06:00 means anything. The Dagster schedule runs only while the Dagster daemon is up. The Prefect `serve()` process refused to schedule on the temporary server and asked for a real one.

## Things that trip people up

- **Airflow runs after the interval, not at its start.** A daily run for 15 October starts once 15 October has ended. A new Dag with today's start date first runs tomorrow.
- **Airflow results are not for data.** What one task returns goes to the next through XCom, stored in Airflow's database. Pass a path or a count, never a dataframe.
- **Dagster's checks do not block by default.** A failed asset check is recorded and the assets after it still build, unless the check is marked `blocking`.
- **Dagster schedules need the daemon.** `materialize()` builds assets on demand, but schedules and sensors run only in the daemon, which `dagster dev` starts for you.
- **Prefect needs a server to schedule.** A flow runs anywhere you call it, but its schedules live in a Prefect server or Prefect Cloud, and something must be running to pick them up.
- **None of them makes a task safe to rerun.** Retries and backfills run your code again. Overwrite the day's output, as all 3 files do, so a second run gives the same result: see [idempotent pipelines](https://tecyfy.com/concepts/idempotent-pipeline).

## Before you choose

Most of the cost of an orchestrator is in running it, not writing for it. Count what you would host, or price the managed service, before comparing features. For the schedules themselves, the [cron expression builder](/data-engineering/cron-builder) writes the cron line all 3 accept and shows the next runs in your time zone.

## Questions

### What is the difference between Airflow, Dagster and Prefect?

All 3 schedule and run data pipelines. Airflow schedules Dags of tasks, each run covering a data interval. Dagster schedules assets, the tables and files a pipeline makes, so lineage, partitions and data checks are part of the model. Prefect makes Python functions into flows and tasks with decorators, and the code's own if statements and loops decide what runs.

### Is Dagster better than Airflow?

Neither is better for every team. Dagster fits when the data is what you want to track, with lineage, partitions per day and checks that stop bad data. Airflow fits when you want the widest set of providers, people who already know it, and managed services from AWS and Google Cloud. Airflow also schedules Dags on assets, which narrows the gap.

### Is Prefect easier than Airflow?

It is quicker to start. A Prefect flow is a decorated Python function that runs when you call it, with a temporary server starting by itself. Airflow needs its database and, in production, a scheduler, a Dag processor, an API server and workers. For scheduled runs Prefect also needs a server or Prefect Cloud, and a process or worker to run the flows.

### Are Airflow, Dagster and Prefect free?

All 3 are open source under the Apache 2.0 licence and free to run yourself. Each also has paid hosted options. Dagster+ and Prefect Cloud come from their makers, and Airflow is offered by Amazon MWAA, Google Cloud's Managed Service for Apache Airflow and others.

### Can Dagster and Prefect run dbt?

Yes, all 3 orchestrators can run dbt. Dagster can load each dbt model as an asset in its lineage graph, and Airflow and Prefect can run dbt commands as tasks. A dbt run is just another step that the orchestrator schedules, retries and watches.

### Which orchestrator handles backfills best?

Airflow and Dagster both build backfills in. Airflow's runs each have a data interval, and airflow backfill create runs a range of them with a limit on how many run at once. Dagster's backfills run a chosen set of an asset's partitions, such as the days that do not exist yet, submitting runs for several at a time. In Prefect, a backfill is a set of runs with older parameter values.

## Sources

- [Project](https://airflow.apache.org/docs/apache-airflow/stable/project.html)
- [Architecture Overview](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/overview.html)
- [Dag Runs](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dag-run.html)
- [Dags](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html)
- [TaskFlow](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/taskflow.html)
- [Testing Dags with dag.test()](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/debug.html)
- [Asset-Aware Scheduling](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/asset-scheduling.html)
- [Tasks](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html)
- [What Is Amazon Managed Workflows for Apache Airflow?](https://docs.aws.amazon.com/mwaa/latest/userguide/what-is-mwaa.html)
- [Managed Airflow overview](https://cloud.google.com/composer/docs/composer-3/composer-overview)
- [Assets](https://docs.dagster.io/guides/build/assets)
- [Partitioning assets](https://docs.dagster.io/guides/build/partitions-and-backfills/partitioning-assets)
- [Schedules](https://docs.dagster.io/guides/automate/schedules)
- [Declarative Automation](https://docs.dagster.io/guides/automate/declarative-automation)
- [Op retries](https://docs.dagster.io/guides/build/ops/op-retries)
- [Asset checks](https://docs.dagster.io/guides/test/asset-checks)
- [Dagster OSS deployment architecture](https://docs.dagster.io/deployment/oss/oss-deployment-architecture)
- [dbt (dagster-dbt)](https://docs.dagster.io/integrations/libraries/dbt)
- [Backfilling data](https://docs.dagster.io/guides/build/partitions-and-backfills/backfilling-data)
- [Dagster CLI](https://docs.dagster.io/api/dagster/cli)
- [About Dagster+](https://docs.dagster.io/deployment/dagster-plus)
- [Flows](https://docs.prefect.io/v3/concepts/flows)
- [Tasks](https://docs.prefect.io/v3/concepts/tasks)
- [Retries](https://docs.prefect.io/v3/how-to-guides/workflows/retries)
- [Deployments](https://docs.prefect.io/v3/concepts/deployments)
- [Schedules](https://docs.prefect.io/v3/concepts/schedules)
- [Work pools](https://docs.prefect.io/v3/concepts/work-pools)
- [Prefect server](https://docs.prefect.io/v3/concepts/server)