Data engineering on Databricks and PySpark
A free data engineering course, from your first PySpark DataFrame to a lakehouse that feeds dashboards and AI.
The modules go in order, and each one is a guide with code you run, one exercise that adds to a final project and 3 questions to check yourself. The project is one shop's orders, taken from raw files to clean tables, a dashboard and an AI answer. Where Tecyfy already explains an idea, a calculator or a comparison, the module links it instead of explaining it again.
What a data engineer does
A data engineer builds the pipelinesData PipelineA set of steps that moves data from where it is made to where it is used, cleaning and reshaping it on the way, run on a schedule or as data arrives. that move data from where it is made, an app's database, a payments service, a stream of clicks, to where it is used: a dashboard, a report, a machine learning model or an AI assistant. On the way the data is checked, cleaned, joined and shaped into tables that people and programs can trust.
Most of the job is making that work every day without anyone watching. Data arrives late, twice or in a new shape. A job that took 5 minutes takes an hour once the table grows. A wrong number in a dashboard is found by the finance team before you. So a large share of the work is tests, checks on the data itself, schedules that recover from failure and a bill that stays in budget.
The course uses Databricks for the platform, because it holds every part in one place: Spark, tables, governance, pipelines, jobs and serving. The parts underneath are open source, Apache Spark, Delta Lake and Apache Iceberg, and each module says what the open-source version is and where another platform, such as Snowflake or BigQuery, does the same job differently.
The modules
Each module says what you can do at the end and links what Tecyfy already has on it. Mark a module done at the end of its guide, and this map keeps your place in this browser.
Choose what fits you, and the module to start with is marked below.
Foundations
What the job is, the SQL every pipeline rests on, and how to write and read Spark code.
Module 0: Start here: what a data engineer builds
You can: Say what a data engineer builds, set up Databricks Free Edition and PySpark on your own computer, and run a first query.
Read the guideModule 1: SQL and data modelling for analytics
You can: Write the window, join and aggregate queries pipelines need, design a star schema, and choose between slowly changing dimension type 1 and type 2.
Read the guideModule 2: PySpark DataFrames
You can: Read files with a schema, clean, join and aggregate them, write the result, write code that runs the same on Databricks serverless, and know when DuckDB or Polars is enough.
Read the guideWith it on Tecyfy
- Quiz: PySpark Basics
- Quiz: PySpark Aggregations
- Quiz: PySpark UDFs & Custom Functions: Essentials
- Interview: PySpark interview questions for Data Engineer roles
- Article: What’s New in Apache Spark 4.0.0: ANSI SQL, Spark Connect, Python and Streaming
- Article: Spark VARIANT vs JSON Strings: A Benchmark on 2 Million Events
Module 3: How Spark runs a job
You can: Read jobs, stages and tasks in the Spark UI, size shuffles and partitions, and say what adaptive query execution fixes on its own.
Read the guideWith it on Tecyfy
- Tool: Spark sizing calculator
- Tool: Spark shuffle partition calculator
- Concept: Shuffle and Data Skew
- Concept: Data Partitioning and Clustering
- Article: PySpark Joins: Broadcast, Sort-Merge and Shuffled Hash, and How Spark Chooses
- Article: Data Skew in PySpark: How to Find It and Fix It
- Quiz: PySpark Performance Tuning
The lakehouse
Tables you can update and trust, who may read them, how data arrives, the pipeline that cleans it and how people use the result.
Module 4: Tables: Parquet, Delta Lake and Iceberg
You can: Update a table with MERGE, go back to an earlier version, change a schema safely, cluster a table, and say when Iceberg and UniForm matter.
Read the guideModule 5: Unity Catalog and governance
You can: Organise data in catalogs, schemas and volumes, grant access, follow lineage, and keep personal data away from people who should not see it.
Read the guideModule 6: Getting data in
You can: Load new files as they land with Auto Loader, connect a SaaS app or database with Lakeflow Connect, and capture changes from a database.
Read the guideModule 7: Pipelines: medallion with declarative pipelines
You can: Build bronze, silver and gold tables with Spark Declarative Pipelines on Lakeflow, and stop bad rows with expectations.
Read the guideWith it on Tecyfy
- Concept: Medallion Architecture
- Article: Medallion Architecture in Databricks: Bronze, Silver and Gold Done Right
- Article: Spark Declarative Pipelines: A Practical Guide, from Delta Live Tables to Apache Spark
- Comparison: dbt vs Spark Declarative Pipelines
- Concept: Data Quality
- Interview: dbt interview questions for Data Engineer roles
Module 8: Serving data
You can: Give analysts a SQL warehouse, business users a dashboard and Genie, partners a share with Delta Sharing, and an app fast lookups with Lakebase.
Read the guide
Production
Data that never stops arriving, jobs that run on their own, changes you can test and ship, and a bill you can explain.
Module 9: Streaming
You can: Read a stream from Kafka with Structured Streaming, handle late data with event time and watermarks, write each record once, and decide when a stream is worth it.
Read the guideModule 10: Orchestration
You can: Schedule work with Lakeflow Jobs or Airflow, retry safely, make every task safe to run twice, and backfill a month of data.
Read the guideModule 11: Testing, CI/CD and bundles
You can: Unit test PySpark code, agree data contracts, monitor data quality, and ship a pipeline with Declarative Automation Bundles (formerly Databricks Asset Bundles).
Read the guideModule 12: Performance and cost
You can: Choose serverless or classic compute, find the slow step and fix it, and say what a pipeline costs a month before it runs.
Read the guide
Data for AI
The tables an AI answer is built from, and the whole pipeline end to end.
Module 13: Data for AI
You can: Prepare documents and tables for retrieval, keep a vector index in step with its table, and call a model from SQL.
Read the guideModule 14: Final project
You can: Build the order pipeline end to end, from raw files to a dashboard and an AI answer, and review it against a checklist.
Read the guide
After the course
Test what you know, then practise the questions data engineering interviews ask.
- InterviewPySpark interview questions for Data Engineer roles PySpark questions for Data Engineer roles, with answers, from easy to hard.
- InterviewDatabricks interview questions for Data Engineer roles Databricks questions for Data Engineer roles, with answers, from easy to hard.
- InterviewSQL interview questions for Data Engineer roles SQL questions for Data Engineer roles, with answers, from easy to hard.
- InterviewSystem design interview questions for Data Engineer roles System design questions for Data Engineer roles, with answers, from easy to hard.
- ScenarioThe revenue dashboard that is 17% too high April revenue on the sales dashboard is 17.2% above finance's figure, and the board meeting is at 14:00. Find the cause in the query, not in the warehouse.
- QuizPySpark Performance Tuning Master the key techniques for optimizing PySpark applications, improving execution speed, and handling large-scale data efficiently.
Questions
- What is data engineering?
- Data engineering is building and running the pipelines that collect data from the systems where it is made, check and clean it, and store it as tables that dashboards, reports, machine learning models and AI assistants can use. A data engineer writes the code, keeps it running every day and makes sure the numbers it produces can be trusted.
- Is this data engineering course free?
- Yes. Every module, exercise and question is free, with no account and no sign-up on Tecyfy. The code runs on PySpark on your own computer, which is free, or on Databricks Free Edition, which costs nothing and is meant for learning and personal projects, not commercial work.
- Do I need a Databricks account?
- Not for the Spark modules, which run on your computer with PySpark. The modules on Unity Catalog, Lakeflow pipelines, jobs and serving use Databricks features with no open-source version you can run locally, and Databricks Free Edition covers them at no cost. Free Edition runs serverless compute only and has a daily usage quota; the exercises use small files.
- What do I need to know before I start?
- Enough Python to write a function and run a script. Module 1 teaches the SQL the course uses, so you do not need it first, and you do not need any cloud account, Spark or Java experience. Module 0 sets up everything else.
- How long does the course take?
- About 2 hours a module, reading the guide and doing its exercise, so about 30 hours in all. You can start at any module that fits what you know; the map at the top of this page suggests where.
- Which version of Spark does the course use?
- Apache Spark 4.2, the current release, and the PySpark package of the same version, which needs Python 3.10 or later and Java 17, 21 or 25. Databricks serverless compute runs its own current Databricks Runtime, so the course's code avoids the few PySpark features serverless does not support.
Sources