# What does a data engineer do? Set up Databricks Free Edition and PySpark

A data engineer builds the pipelines that take data from where it is made, such as an app's database or a stream of events, to where it is used, such as a dashboard, a report or an AI assistant, checking, cleaning and shaping it into tables people can trust on the way. To practise for free, use Databricks Free Edition, a serverless, quota-limited Databricks workspace, and PySpark on your own computer, which needs Python 3.10 or later and Java 17, 21 or 25.

- Page: https://tecyfy.com/data-engineering/course/start-here
- Course: Data engineering course (https://tecyfy.com/data-engineering/course), Module 0: Start here: what a data engineer builds

At the end of this module you can:

- Say what a data engineer builds, and what each part of this course adds to 1 pipeline
- Set up Databricks Free Edition and know its limits
- Install PySpark on your own computer and check that it runs
- Run the same question in Python and in SQL on Spark

This course teaches data engineering by building 1 thing: the data side of a small online shop. Its orders start as raw files with mistakes in them and end as clean tables, a dashboard and an answer from an AI assistant. Each module adds a piece. This one shows the whole pipeline, sets up the 2 places you will run it, and runs a first query in each.

## What a data engineer builds

Data is made in one place and needed in another. The shop's orders are written by its checkout, customer details by its customer system, clicks by its website. The finance team wants revenue by country, the marketing team wants repeat customers, and the support assistant wants to answer "where is my order?" Nobody wants to read the checkout's database to get there.

A data engineer builds the [pipelines](https://tecyfy.com/concepts/data-pipeline) in between. A pipeline does the same few jobs, whatever the company:

- **Get the data in**, from files, databases, apps and streams, as it arrives and without losing or doubling any.
- **Clean and shape it**: fix types, drop or quarantine what is broken, keep the newest version of each record, join it to the rest. Modern platforms load the raw data first and transform it inside the warehouse, the pattern called [ELT](https://tecyfy.com/concepts/elt).
- **Model it** into tables that answer many questions the same way, such as the star schema of module 1.
- **Serve it** to dashboards, people writing SQL, apps and AI.
- **Keep it running**: on a schedule, with tests and checks on the data, at a cost someone has agreed to.

The last job is most of the work. A pipeline that runs once is a script. A pipeline is data arriving late, twice or in a new shape, a table that grows from thousands of rows to billions, and a wrong number that the finance team finds before you do.

## The pipeline this course builds

The course's modules, grouped in 4 stages, each add a piece to the shop's pipeline:

- **Foundations.** SQL and data modelling (module 1) designs the tables the reports read. PySpark DataFrames (module 2) cleans the raw orders. How Spark runs a job (module 3) shows why the same code can take a second or an hour.
- **The lakehouse.** Tables you can update and go back in time on (module 4), who may read them (module 5), files and databases loaded as they change (module 6), the bronze, silver and gold pipeline (module 7) and the dashboards and apps that read the result (module 8). A [Lakehouse](https://tecyfy.com/concepts/lakehouse) keeps all of it in open table formats on cheap storage.
- **Production.** Streams (module 9), schedules and retries (module 10), tests and deployment (module 11), and speed and cost (module 12).
- **Data for AI.** Documents and tables prepared for retrieval (module 13), then the whole pipeline built end to end (module 14).

The [data engineering hub](/data-engineering) follows 1 order through these steps, with where each runs on Databricks.

- Walkthrough: [Data engineering](https://tecyfy.com/data-engineering). Every AI answer about a business is only as good as the data under it. Follow an order, a document and a card payment from the moment they are made to the…

## Where the code runs

Every module runs in 2 places, and you can use either.

**Databricks Free Edition** is a free Databricks workspace for learning, with serverless compute and storage included. It is the real platform: notebooks, Unity Catalog, pipelines, jobs, SQL warehouses and dashboards. It has limits, and you will meet some of them:

| Limit on Free Edition | What it means here |
| --- | --- |
| Serverless compute only | No clusters to size; Spark runs through Spark Connect, as module 2 shows |
| 1 SQL warehouse, size 2X-Small | Enough for this course's dashboards |
| At most 5 job tasks running at once | Enough for the course's jobs |
| A daily compute quota | Over it, compute stops until the next day; your data is kept |
| No R or Scala | The course uses Python and SQL |
| Non-commercial use only | Learn and build here, not run an employer's work |

**Your own computer** runs open-source Apache Spark. It is quicker to start, works offline and shows you things serverless hides, such as the Spark UI in module 3. It has no Unity Catalog or Databricks pipelines, so those modules say what to run on Free Edition and what the open-source equivalent is.

## Set up Databricks Free Edition

1. Go to the Free Edition sign-up page from the [Databricks Free Edition docs](https://docs.databricks.com/aws/en/getting-started/free-edition) and choose a way to sign up. Databricks makes a workspace for you.
2. In the sidebar, choose **New**, then **Notebook**. It connects to serverless compute on its own.
3. In the first cell, run `spark.version`. A session called `spark` is already there in every notebook, so you never build one on Databricks.

## Set up PySpark on your computer

Check Java first. In a terminal:

```bash
java -version
```

It should print a version of 17, 21 or 25. If Java is missing or older, install a current JDK, such as [Eclipse Temurin](https://adoptium.net/), and set `JAVA_HOME` to where it is installed.

Then make a folder for the course with its own Python environment, so the course's packages do not mix with anything else on your computer:

```bash
mkdir data-engineering-course
cd data-engineering-course
python3 -m venv .venv
source .venv/bin/activate        # on Windows: .venv\Scripts\activate
pip install "pyspark[sql]==4.2.0"
```

`pyspark[sql]` also installs pandas and PyArrow, which later modules use. Activate the environment each time you open a new terminal. It matters more than it looks: Spark runs parts of a job in separate Python processes, and it starts them with whichever `python3` comes first on your path. Without the environment active, that can be an older Python on your computer, and the first query fails with an error from a "python worker".

## Run a first query

Save this as `check_setup.py` in the course folder:

```python
"""Check that Spark runs: print where it runs, then answer 1 question in Python and the same in SQL."""
import sys
from decimal import Decimal

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = SparkSession.builder.appName("check").getOrCreate()
print("Python", sys.version.split()[0], "| Spark", spark.version)
print("Catalog", spark.catalog.currentCatalog(), "| Schema", spark.catalog.currentDatabase())

orders = spark.createDataFrame(
    [(1001, "C01", Decimal("25.50")), (1002, "C02", Decimal("12.00")), (1004, "C01", Decimal("40.00"))],
    "order_id BIGINT, customer_id STRING, amount DECIMAL(10,2)",
)
(
    orders.groupBy("customer_id")
    .agg(F.count("*").alias("orders"), F.sum("amount").alias("spent"))
    .orderBy("customer_id")
    .show()
)

orders.createOrReplaceTempView("orders")
spark.sql("SELECT count(*) AS orders, sum(amount) AS revenue FROM orders").show()
```

Run it with `python check_setup.py`. Spark prints some start-up lines of its own, then:

```text
Python 3.12.15 | Spark 4.2.0
Catalog spark_catalog | Schema default
+-----------+------+-----+
|customer_id|orders|spent|
+-----------+------+-----+
|        C01|     2|65.50|
|        C02|     1|12.00|
+-----------+------+-----+

+------+-------+
|orders|revenue|
+------+-------+
|     3|  77.50|
+------+-------+
```

Your Python version may differ; the rest should match. On Databricks, paste the same code into a cell without the `SparkSession` line. It prints Databricks' own Spark version, and your workspace's catalog and schema in place of `spark_catalog` and `default`, which is why later modules ask Spark for those names instead of typing them.

## Python and SQL, 1 engine

The script asked 2 questions, 1 with DataFrame methods and 1 in SQL, about the same 3 orders. `createOrReplaceTempView` gave the DataFrame a name that SQL can use. Spark runs both on the same engine, so neither is the "real" one and neither is faster for being SQL or Python. Use SQL where a query says it best, as module 1 does, and Python where you want functions, tests and loops, as module 2 does.

The amounts are `DECIMAL(10,2)`, not floating-point numbers, from the very first line. Money stored as a float picks up rounding errors, which a revenue report then adds up. The habit costs nothing now and saves a painful search later.

## Build it: set up the project

The course's project lives in the folder you just made. By the end of the course it holds the scripts that clean, model and serve the shop's orders. For now, make sure it is ready:

1. `data-engineering-course/` exists, with its `.venv` and `check_setup.py`.
2. `python check_setup.py` prints the 2 tables above with the environment active.
3. A Free Edition notebook runs the same code and prints your workspace's catalog.

If all 3 work, you are ready for [module 1](/data-engineering/course/sql-and-data-modelling), which writes the SQL every later module uses. If you already write SQL, [module 2](/data-engineering/course/pyspark-dataframes) starts on PySpark.

## Check yourself

1. You install PySpark in a virtual environment and run python check_setup.py without activating the environment. Spark starts, prints its version, then fails at the first query with a Python worker error: TypeError: unsupported operand type(s) for |: 'type' and 'type'. What is wrong?

   A. The data in the script has the wrong types, so Spark cannot add them up
   B. PySpark 4.2 does not run on Python 3.12
   C. Spark started its Python workers with the computer's other, older python3, so activate the environment or set PYSPARK_PYTHON
   D. Java is missing, so the query cannot start

2. You have used up the day's compute quota on Databricks Free Edition in the middle of the afternoon. What happens to your work?

   A. Your compute stops for the rest of the day, and your notebooks, tables and settings are kept until it starts again
   B. Databricks deletes the tables you made that day
   C. Your workspace moves to a paid plan for the rest of the day
   D. Nothing; the quota only limits SQL warehouses

3. A friend wants to follow the course in Scala notebooks on Free Edition, and later run their employer's pipeline there. What do you tell them?

   A. Both work, since Free Edition has every feature of Databricks
   B. Scala works, but only on a SQL warehouse
   C. Scala works on serverless notebooks, and work pipelines need LinkedIn verification first
   D. Neither fits: Free Edition does not support R or Scala, and it is for non-commercial use only

### Answers

1. C. Spark runs some work in separate Python worker processes, and by default it starts them with python3 from your PATH. Without the environment activated, that is the computer's own Python, here 3.9, which cannot read code written for 3.10 and later. Java is there, since Spark started and printed its version; the data and the Python version are fine. Activate the environment, or point PYSPARK_PYTHON at its Python.
2. A. Free Edition shuts compute down for the rest of the day when the quota runs out, and for the rest of the month in extreme cases, but Databricks says your data and settings are not deleted. There is no paid overflow, and the quota covers all compute. Small data, like this course's, keeps you well inside it.
3. D. Databricks lists both limits for Free Edition: R and Scala are not supported, and it is for non-commercial use only. The course uses Python and SQL, which Free Edition runs. LinkedIn verification raises some usage limits but does not lift these.

## Questions

### What does a data engineer do?

A data engineer builds and runs the pipelines that move data from where it is made, such as an app's database, a payments service or a stream of clicks, to where it is used, such as dashboards, reports, machine learning and AI assistants. On the way the data is checked, cleaned, joined and shaped into tables, and most of the work is keeping that running every day as data arrives late, twice or in a new shape.

### Is Databricks Free Edition really free?

Yes. Free Edition is a no-cost Databricks workspace for learning and experimenting, with serverless compute and default storage. It is quota-limited, for non-commercial use only, has 1 SQL warehouse of size 2X-Small, at most 5 job tasks running at once and no R or Scala. It replaced Community Edition, which was retired in 2025.

### What do I need to run PySpark on my computer?

Python 3.10 or later and Java 17, 21 or 25 with JAVA_HOME set, because Spark runs on the Java virtual machine. Then pip install pyspark in a virtual environment, activate it, and start a session with SparkSession.builder.getOrCreate(). No cluster is needed; Spark uses your computer's cores.

### Should I learn PySpark or SQL for data engineering?

Both, and they are less different than they look. Spark runs SQL and DataFrame code on the same engine, so the same question gives the same plan either way. SQL is quicker for queries and models; Python is better for reusable steps, tests and anything a query cannot say. This course uses each where it fits.

## Sources

- [Databricks Free Edition](https://docs.databricks.com/aws/en/getting-started/free-edition)
- [Databricks Free Edition limitations](https://docs.databricks.com/aws/en/getting-started/free-edition-limitations)
- [Manage notebooks](https://docs.databricks.com/aws/en/notebooks/notebooks-manage)
- [Installation](https://spark.apache.org/docs/latest/api/python/getting_started/install.html)
- [Spark overview](https://spark.apache.org/docs/latest/)
- [Spark configuration, environment variables](https://spark.apache.org/docs/latest/configuration.html)
- [Spark SQL, DataFrames and Datasets guide](https://spark.apache.org/docs/latest/sql-programming-guide.html)
- [Serverless compute limitations](https://docs.databricks.com/aws/en/compute/serverless/limitations)
- [venv, creation of virtual environments](https://docs.python.org/3/library/venv.html)