# Lakeflow Jobs: schedule a daily job with tasks, retries and a run date

A Lakeflow job is a set of tasks with the order they run in, a trigger such as a daily schedule, and settings for each task: its code, its parameters, its compute, its retries and the condition it runs on. Give the job a run_date parameter whose default is the day the run was triggered, pass it to every task, and set retries on the load. A failed run can then be repaired, which reruns only the tasks that failed and those after them.

- Page: https://tecyfy.com/data-engineering/course/orchestration/lakeflow-jobs
- Course: Data engineering course (https://tecyfy.com/data-engineering/course), Module 10: Orchestration, Lakeflow Jobs, a daily job on Databricks
- Parts of this module: Run dates, safe reruns and retries (https://tecyfy.com/data-engineering/course/orchestration); Lakeflow Jobs, a daily job on Databricks (https://tecyfy.com/data-engineering/course/orchestration/lakeflow-jobs); Backfills and corrections (https://tecyfy.com/data-engineering/course/orchestration/backfill); Apache Airflow, the same job as a DAG (https://tecyfy.com/data-engineering/course/orchestration/airflow); Build it: backfill a month, rerun the failed day (https://tecyfy.com/data-engineering/course/orchestration/build-it)

At the end of this part you can:

- Build the shop's daily load as a Lakeflow job with 2 tasks, a schedule and a run_date parameter, and read it as YAML
- Choose a task's retries, Run if condition and the job's maximum concurrent runs, knowing what each does to a run
- Repair a failed run, or run 1 past day again, by giving the job its date

[Part 1](/data-engineering/course/orchestration) built a daily load that takes its day from a run date and replaces that day in `orders_daily`, and a check that exits 1 when the day does not match its file. This part runs them as a Lakeflow job, the orchestrator built into Databricks. To rebuild part 1 from nothing, run:

```bash
python make_days.py
python load_day.py 2026-09-02
python check_day.py 2026-09-02
```

## The scripts on Databricks

A job runs the same 2 scripts. Upload `load_day.py` and `check_day.py` to a workspace folder, the order files to a volume such as `/Volumes/<catalog>/bronze/landing/orders/`, and change 3 things in each script:

1. The file path: `f"/Volumes/<catalog>/bronze/landing/orders/{day}.csv"` in place of `f"landing/orders/{day}.csv"`, with your catalog's name.
2. The table: a Unity Catalog name such as `silver.orders_daily`, in the schemas of [module 5](/data-engineering/course/unity-catalog), in place of `orders_daily`.
3. The session: the job's compute already has one, so the last lines become:

```python
if __name__ == "__main__":
    spark = SparkSession.builder.getOrCreate()  # the compute's own session
    print(load_day(spark, sys.argv[1], append="--append" in sys.argv))
```

`check_day.py` ends the same way, with `check_day(spark, sys.argv[1])` and its `sys.exit`. An exit code of 1 fails the task, as it would in any orchestrator.

## The job

In the workspace, **Jobs & Pipelines** creates a job. Add a task `load_day` of type Python script pointing at `load_day.py`, then a task `check_day` pointing at `check_day.py` that depends on `load_day`. Free Edition has only serverless compute, which is what a new task uses there. A job can be read as YAML from the menu next to **Run now**, **Switch to code version (YAML)**. Written as YAML, the shop's job is:

```yaml
resources:
  jobs:
    shop_daily:
      name: shop_daily
      schedule:
        quartz_cron_expression: "0 0 2 * * ?"  # seconds, minutes, hours: every day at 02:00
        timezone_id: UTC
        pause_status: UNPAUSED
      max_concurrent_runs: 1
      parameters:
        - name: run_date
          default: "{{job.trigger.time.iso_date}}"
      email_notifications:
        on_failure:
          - <your email>
      tasks:
        - task_key: load_day
          spark_python_task:
            python_file: /Workspace/Users/<you>/shop/load_day.py
            parameters: ["{{job.parameters.run_date}}"]
          environment_key: default
          max_retries: 2
          min_retry_interval_millis: 300000  # 5 minutes
        - task_key: check_day
          depends_on:
            - task_key: load_day
          spark_python_task:
            python_file: /Workspace/Users/<you>/shop/check_day.py
            parameters: ["{{job.parameters.run_date}}"]
          environment_key: default
      environments:
        - environment_key: default
          spec:
            environment_version: "2"
```

Module 11 keeps this file with the code and deploys it with a bundle. Each block is a choice.

## The run date

`run_date` is a job parameter, and its default is a dynamic value reference: `{{job.trigger.time.iso_date}}` is the date, in UTC, the run was triggered, so the scheduled run at 02:00 on 2 September gets `2026-09-02` and loads 1 September. A Python script task takes its parameters as a list of command-line arguments, and job parameters are not passed to it on their own, so each task names it: `["{{job.parameters.run_date}}"]` arrives as `sys.argv[1]`.

The default is only a default. **Run now with different parameters** starts a run with `run_date` set to any day, which is how you load 1 past day again, and part 3's backfill sets it for every run.

## Triggers

A job starts on 1 of these triggers, or by hand with **Run now**:

| Trigger | Starts a run |
| --- | --- |
| Scheduled | At times set by a Quartz cron expression, in a time zone |
| File arrival | When new files land in a Unity Catalog storage location |
| Table update | When source tables are updated |
| Continuous | Again as soon as the last run ends, so the job always runs |
| Model update (Beta) | When a Unity Catalog model or version changes |

A file a night suits a schedule. Quartz cron has a seconds field first, so 02:00 every day is `0 0 2 * * ?`; the [cron builder](/data-engineering/cron-builder) writes it from words. Keep the time zone UTC unless the business day is local: in a zone with daylight saving time, an hourly job can be skipped or look delayed when the clocks change. Databricks notes a scheduled run can start up to several minutes late, which is 1 more reason the day comes from the trigger's date and not from the clock when the script happens to run.

## Retries, and what runs after a failure

For most jobs Databricks does not retry a failed task unless you set retries; serverless jobs choose retries for you by default. `max_retries: 2` with `min_retry_interval_millis: 300000` starts a failed `load_day` again up to 2 times, 5 minutes after the start of the attempt that failed. A retry is safe only because part 1 made the load idempotent. If you also set a timeout, it applies to each attempt.

A task's **Run if** condition decides whether it runs once the tasks it depends on have ended, after their retries:

| Run if | The task runs when |
| --- | --- |
| All succeeded (the default) | Every dependency succeeded |
| At least one succeeded | Any dependency succeeded |
| None failed | No dependency failed, and at least 1 ran |
| All done | Every dependency has ended, however it ended |
| At least one failed | Any dependency failed |
| All failed | Every dependency failed |

When the condition is not met, the first 3 mark the task **Upstream failed**, and the last 2 mark it **Excluded**, which is skipped. `check_day` keeps the default, so it never checks a day that failed to load. A task that sends a message or cleans up after a failure would use **At least one failed**. Press the controls to see a failure travel through a job, with and without a retry:

(An interactive demo goes here on the page: https://tecyfy.com/data-engineering/course/orchestration/lakeflow-jobs)

## Concurrent runs and notifications

`max_concurrent_runs: 1`, the default for a new job, lets 1 run of the job go at a time. Databricks skips a run that would go over the limit; with queueing on, which is the default for jobs made in the interface, the run waits up to 48 hours instead. For a daily load, 1 at a time means a slow run is never overtaken by the next. Free Edition allows at most 5 job tasks to run at once across the whole account.

`email_notifications.on_failure` sends a message when a run fails. A job or a task can also notify on start, success, a run longer than a threshold or a streaming backlog, by email or to Slack, Microsoft Teams, PagerDuty or a webhook, once an administrator has set those up.

## Repair a failed run

When a run fails, open it and click **Repair run**. A repair reruns only the tasks that did not succeed and the tasks that depend on them; the ones that succeeded keep their results. If `load_day` used up its retries because the day's file landed late, the repair, started once the file is there, runs `load_day` and then `check_day`; in a longer job, the tasks before them that succeeded are not run again. Parameters typed into the repair dialog override the run's own, so `run_date` can be set explicitly to the day the run was for. Repair needs 2 or more tasks; a job of 1 task is started again with **Run now**.

## Check yourself

1. The load_day task fails twice and then succeeds on its last retry. check_day has the default Run if condition. What happens?

   A. check_day is marked Upstream failed, because load_day failed
   B. check_day runs, because its dependency finally succeeded
   C. check_day runs 3 times, once after each attempt
   D. The job fails, because a retry counts as a failure

### Answers

1. B. The default condition, All succeeded, looks at how each dependency ended, after its retries. load_day ended in success, so check_day runs once. Had the last retry failed too, check_day would be marked Upstream failed and not run.

## Questions

### What is the difference between a Lakeflow job's retries and a repair run?

Retries are automatic. A task that fails is started again up to its max_retries, with min_retry_interval_millis between the start of the failed attempt and the next, inside the same run. A repair run is something you start after a run has failed. It reruns only the tasks that failed and the tasks that depend on them, keeps the ones that succeeded, and lets you change parameters for it. Repair needs a job with 2 or more tasks; a 1-task job is run again with Run now.

### How do I pass a date to a Python script task in a Lakeflow job?

Python script tasks take their parameters as a JSON array of strings, passed as command-line arguments, and job parameters are not pushed down to them on their own. Add a job parameter such as run_date with the default {{job.trigger.time.iso_date}}, and set the task's parameters to ["{{job.parameters.run_date}}"]. The script reads it as sys.argv[1], and a run you start with a different run_date gets that date instead.

## Sources

- [Configure and edit Lakeflow Jobs](https://docs.databricks.com/aws/en/jobs/configure-job)
- [Configure and edit tasks in Lakeflow Jobs](https://docs.databricks.com/aws/en/jobs/configure-task)
- [Configure task parameters](https://docs.databricks.com/aws/en/jobs/task-parameters)
- [What is a dynamic value reference?](https://docs.databricks.com/aws/en/jobs/dynamic-value-references)
- [Run jobs on a schedule](https://docs.databricks.com/aws/en/jobs/scheduled)
- [Automating jobs with schedules and triggers](https://docs.databricks.com/aws/en/jobs/triggers)
- [Configure task dependencies](https://docs.databricks.com/aws/en/jobs/run-if)
- [Troubleshoot and repair job failures](https://docs.databricks.com/aws/en/jobs/repair-job-failures)
- [Add notifications on a job](https://docs.databricks.com/aws/en/jobs/notifications)
- [Tutorial: Run Python code on serverless compute](https://docs.databricks.com/aws/en/dev-tools/databricks-connect/python/tutorial-serverless)
- [Databricks Free Edition limitations](https://docs.databricks.com/aws/en/getting-started/free-edition-limitations)