Lakeflow Jobs: schedule a daily job with tasks, retries and a run date

Part 1 built a daily load that takes its day from a run date and replaces that day in orders_daily, and a check that exits 1 when the day does not match its file. This part runs them as a Lakeflow job, the orchestrator built into Databricks. To rebuild part 1 from nothing, run:

python make_days.py
python load_day.py 2026-09-02
python check_day.py 2026-09-02

The scripts on Databricks

A job runs the same 2 scripts. Upload load_day.py and check_day.py to a workspace folder, the order files to a volume such as /Volumes/<catalog>/bronze/landing/orders/, and change 3 things in each script:

1. The file path: f"/Volumes/<catalog>/bronze/landing/orders/{day}.csv" in place of f"landing/orders/{day}.csv", with your catalog's name.
2. The table: a Unity Catalog name such as silver.orders_daily, in the schemas of module 5, in place of orders_daily.
3. The session: the job's compute already has one, so the last lines become:

if __name__ == "__main__":
    spark = SparkSession.builder.getOrCreate()  # the compute's own session
    print(load_day(spark, sys.argv[1], append="--append" in sys.argv))

check_day.py ends the same way, with check_day(spark, sys.argv[1]) and its sys.exit. An exit code of 1 fails the task, as it would in any orchestrator.

The job

In the workspace, Jobs & Pipelines creates a job. Add a task load_day of type Python script pointing at load_day.py, then a task check_day pointing at check_day.py that depends on load_day. Free Edition has only serverless compute, which is what a new task uses there. A job can be read as YAML from the menu next to Run now, Switch to code version (YAML). Written as YAML, the shop's job is:

resources:
  jobs:
    shop_daily:
      name: shop_daily
      schedule:
        quartz_cron_expression: "0 0 2 * * ?"  # seconds, minutes, hours: every day at 02:00
        timezone_id: UTC
        pause_status: UNPAUSED
      max_concurrent_runs: 1
      parameters:
        - name: run_date
          default: "{{job.trigger.time.iso_date}}"
      email_notifications:
        on_failure:
          - <your email>
      tasks:
        - task_key: load_day
          spark_python_task:
            python_file: /Workspace/Users/<you>/shop/load_day.py
            parameters: ["{{job.parameters.run_date}}"]
          environment_key: default
          max_retries: 2
          min_retry_interval_millis: 300000  # 5 minutes
        - task_key: check_day
          depends_on:
            - task_key: load_day
          spark_python_task:
            python_file: /Workspace/Users/<you>/shop/check_day.py
            parameters: ["{{job.parameters.run_date}}"]
          environment_key: default
      environments:
        - environment_key: default
          spec:
            environment_version: "2"

Module 11 keeps this file with the code and deploys it with a bundle. Each block is a choice.

The run date

run_date is a job parameter, and its default is a dynamic value reference: {{job.trigger.time.iso_date}} is the date, in UTC, the run was triggered, so the scheduled run at 02:00 on 2 September gets 2026-09-02 and loads 1 September. A Python script task takes its parameters as a list of command-line arguments, and job parameters are not passed to it on their own, so each task names it: ["{{job.parameters.run_date}}"] arrives as sys.argv[1].

The default is only a default. Run now with different parameters starts a run with run_date set to any day, which is how you load 1 past day again, and part 3's backfill sets it for every run.

Triggers

A job starts on 1 of these triggers, or by hand with Run now:

A file a night suits a schedule. Quartz cron has a seconds field first, so 02:00 every day is 0 0 2 * * ?; the cron builder writes it from words. Keep the time zone UTC unless the business day is local: in a zone with daylight saving time, an hourly job can be skipped or look delayed when the clocks change. Databricks notes a scheduled run can start up to several minutes late, which is 1 more reason the day comes from the trigger's date and not from the clock when the script happens to run.

Retries, and what runs after a failure

For most jobs Databricks does not retry a failed task unless you set retries; serverless jobs choose retries for you by default. max_retries: 2 with min_retry_interval_millis: 300000 starts a failed load_day again up to 2 times, 5 minutes after the start of the attempt that failed. A retry is safe only because part 1 made the load idempotent. If you also set a timeout, it applies to each attempt.

A task's Run if condition decides whether it runs once the tasks it depends on have ended, after their retries:

When the condition is not met, the first 3 mark the task Upstream failed, and the last 2 mark it Excluded, which is skipped. check_day keeps the default, so it never checks a day that failed to load. A task that sends a message or cleans up after a failure would use At least one failed. Press the controls to see a failure travel through a job, with and without a retry:

Try itRun a nightly job

load_customers

WorksTimes out onceBreaks

Retries

None1

Run

1. load_ordersWaiting

   load_customersWaiting
2. clean_ordersWaiting
3. build_revenueWaiting
4. refresh_dashboardWaiting

The run’s log

1. Nothing yet

Concurrent runs and notifications

max_concurrent_runs: 1, the default for a new job, lets 1 run of the job go at a time. Databricks skips a run that would go over the limit; with queueing on, which is the default for jobs made in the interface, the run waits up to 48 hours instead. For a daily load, 1 at a time means a slow run is never overtaken by the next. Free Edition allows at most 5 job tasks to run at once across the whole account.

email_notifications.on_failure sends a message when a run fails. A job or a task can also notify on start, success, a run longer than a threshold or a streaming backlog, by email or to Slack, Microsoft Teams, PagerDuty or a webhook, once an administrator has set those up.

Repair a failed run

When a run fails, open it and click Repair run. A repair reruns only the tasks that did not succeed and the tasks that depend on them; the ones that succeeded keep their results. If load_day used up its retries because the day's file landed late, the repair, started once the file is there, runs load_day and then check_day; in a longer job, the tasks before them that succeeded are not run again. Parameters typed into the repair dialog override the run's own, so run_date can be set explicitly to the day the run was for. Repair needs 2 or more tasks; a job of 1 task is started again with Run now.