What does a data engineer do? Set up Databricks Free Edition and PySpark

This course teaches data engineering by building 1 thing: the data side of a small online shop. Its orders start as raw files with mistakes in them and end as clean tables, a dashboard and an answer from an AI assistant. Each module adds a piece. This one shows the whole pipeline, sets up the 2 places you will run it, and runs a first query in each.

What a data engineer builds

Data is made in one place and needed in another. The shop's orders are written by its checkout, customer details by its customer system, clicks by its website. The finance team wants revenue by country, the marketing team wants repeat customers, and the support assistant wants to answer "where is my order?" Nobody wants to read the checkout's database to get there.

A data engineer builds the pipelinesData PipelineA set of steps that moves data from where it is made to where it is used, cleaning and reshaping it on the way, run on a schedule or as data arrives. in between. A pipeline does the same few jobs, whatever the company:

• Get the data in, from files, databases, apps and streams, as it arrives and without losing or doubling any.
• Clean and shape it: fix types, drop or quarantine what is broken, keep the newest version of each record, join it to the rest. Modern platforms load the raw data first and transform it inside the warehouse, the pattern called ELTELTLoading raw data into the warehouse or lakehouse first and transforming it there, the reverse of the older ETL order, which transformed data before loading it..
• Model it into tables that answer many questions the same way, such as the star schema of module 1.
• Serve it to dashboards, people writing SQL, apps and AI.
• Keep it running: on a schedule, with tests and checks on the data, at a cost someone has agreed to.

The last job is most of the work. A pipeline that runs once is a script. A pipeline is data arriving late, twice or in a new shape, a table that grows from thousands of rows to billions, and a wrong number that the finance team finds before you do.

The pipeline this course builds

The course's modules, grouped in 4 stages, each add a piece to the shop's pipeline:

• Foundations. SQL and data modelling (module 1) designs the tables the reports read. PySpark DataFrames (module 2) cleans the raw orders. How Spark runs a job (module 3) shows why the same code can take a second or an hour.
• The lakehouse. Tables you can update and go back in time on (module 4), who may read them (module 5), files and databases loaded as they change (module 6), the bronze, silver and gold pipeline (module 7) and the dashboards and apps that read the result (module 8). A LakehouseLakehouseOne store for all of an organisation's data: cheap files in cloud storage, as in a data lake, with the tables, transactions and fast queries of a data warehouse on top. keeps all of it in open table formats on cheap storage.
• Production. Streams (module 9), schedules and retries (module 10), tests and deployment (module 11), and speed and cost (module 12).
• Data for AI. Documents and tables prepared for retrieval (module 13), then the whole pipeline built end to end (module 14).

The data engineering hub follows 1 order through these steps, with where each runs on Databricks.

WalkthroughData engineeringEvery AI answer about a business is only as good as the data under it. Follow an order, a document and a card payment from the moment they are made to the…

Where the code runs

Every module runs in 2 places, and you can use either.

Databricks Free Edition is a free Databricks workspace for learning, with serverless compute and storage included. It is the real platform: notebooks, Unity Catalog, pipelines, jobs, SQL warehouses and dashboards. It has limits, and you will meet some of them:

Your own computer runs open-source Apache Spark. It is quicker to start, works offline and shows you things serverless hides, such as the Spark UI in module 3. It has no Unity Catalog or Databricks pipelines, so those modules say what to run on Free Edition and what the open-source equivalent is.

Set up Databricks Free Edition

1. Go to the Free Edition sign-up page from the Databricks Free Edition docs and choose a way to sign up. Databricks makes a workspace for you.
2. In the sidebar, choose New, then Notebook. It connects to serverless compute on its own.
3. In the first cell, run spark.version. A session called spark is already there in every notebook, so you never build one on Databricks.

Set up PySpark on your computer

Check Java first. In a terminal:

java -version

It should print a version of 17, 21 or 25. If Java is missing or older, install a current JDK, such as Eclipse Temurin, and set JAVA_HOME to where it is installed.

Then make a folder for the course with its own Python environment, so the course's packages do not mix with anything else on your computer:

mkdir data-engineering-course
cd data-engineering-course
python3 -m venv .venv
source .venv/bin/activate        # on Windows: .venv\Scripts\activate
pip install "pyspark[sql]==4.2.0"

pyspark[sql] also installs pandas and PyArrow, which later modules use. Activate the environment each time you open a new terminal. It matters more than it looks: Spark runs parts of a job in separate Python processes, and it starts them with whichever python3 comes first on your path. Without the environment active, that can be an older Python on your computer, and the first query fails with an error from a "python worker".

Run a first query

Save this as check_setup.py in the course folder:

"""Check that Spark runs: print where it runs, then answer 1 question in Python and the same in SQL."""
import sys
from decimal import Decimal

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = SparkSession.builder.appName("check").getOrCreate()
print("Python", sys.version.split()[0], "| Spark", spark.version)
print("Catalog", spark.catalog.currentCatalog(), "| Schema", spark.catalog.currentDatabase())

orders = spark.createDataFrame(
    [(1001, "C01", Decimal("25.50")), (1002, "C02", Decimal("12.00")), (1004, "C01", Decimal("40.00"))],
    "order_id BIGINT, customer_id STRING, amount DECIMAL(10,2)",
)
(
    orders.groupBy("customer_id")
    .agg(F.count("*").alias("orders"), F.sum("amount").alias("spent"))
    .orderBy("customer_id")
    .show()
)

orders.createOrReplaceTempView("orders")
spark.sql("SELECT count(*) AS orders, sum(amount) AS revenue FROM orders").show()

Run it with python check_setup.py. Spark prints some start-up lines of its own, then:

Python 3.12.15 | Spark 4.2.0
Catalog spark_catalog | Schema default
+-----------+------+-----+
|customer_id|orders|spent|
+-----------+------+-----+
|        C01|     2|65.50|
|        C02|     1|12.00|
+-----------+------+-----+

+------+-------+
|orders|revenue|
+------+-------+
|     3|  77.50|
+------+-------+

Your Python version may differ; the rest should match. On Databricks, paste the same code into a cell without the SparkSession line. It prints Databricks' own Spark version, and your workspace's catalog and schema in place of spark_catalog and default, which is why later modules ask Spark for those names instead of typing them.

Python and SQL, 1 engine

The script asked 2 questions, 1 with DataFrame methods and 1 in SQL, about the same 3 orders. createOrReplaceTempView gave the DataFrame a name that SQL can use. Spark runs both on the same engine, so neither is the "real" one and neither is faster for being SQL or Python. Use SQL where a query says it best, as module 1 does, and Python where you want functions, tests and loops, as module 2 does.

The amounts are DECIMAL(10,2), not floating-point numbers, from the very first line. Money stored as a float picks up rounding errors, which a revenue report then adds up. The habit costs nothing now and saves a painful search later.

Build it: set up the project

The course's project lives in the folder you just made. By the end of the course it holds the scripts that clean, model and serve the shop's orders. For now, make sure it is ready:

1. data-engineering-course/ exists, with its .venv and check_setup.py.
2. python check_setup.py prints the 2 tables above with the environment active.
3. A Free Edition notebook runs the same code and prints your workspace's catalog.

If all 3 work, you are ready for module 1, which writes the SQL every later module uses. If you already write SQL, module 2 starts on PySpark.