What is a lakehouse?

For more than a decade, a large company's data has usually lived in 2 places. Everything landed first in a data lake: cheap files in cloud storage, kept raw, in any format. Then the part the business reported on was copied again into a data warehouse, a separate system with tidy tables and fast queries.

A lakehouse is the idea that you do not need the second copy. Keep the files in cheap cloud storage, and add the parts that made the warehouse worth paying for, on top of them.

Why 2 copies became a problem

The 2021 paper that named the pattern, written by 4 people at Databricks, lists what the 2-tier setup costs. Every copy from lake to warehouse is another job that can fail or bring in a bug. The warehouse is always behind the lake, with new data often taking days to arrive; the paper cites a survey in which 86% of analysts were working from out-of-date data. And machine learning fits neither side well. The warehouse holds only the curated slice, often in a format a training job cannot read directly, while the lake holds everything with no guarantees about it.

What sits on top of the files

The trick is a layer that makes a folder of files behave like a database table. The data stays in an open file format such as Parquet. Next to it sits a log that records every change to the table. Writers add to the log in a way that either fully succeeds or does not happen at all, so a reader never sees half a write. Each write makes a new version of the table, and older versions can still be queried.

This is what an open table formatOpen Table FormatA published way of storing a table as plain data files plus a log of changes, so many engines can read and safely write it. Delta Lake and Apache Iceberg are the best known. provides. Delta Lake is the default on Databricks, where every table is a Delta table unless you say otherwise. Apache Iceberg is another, and Databricks can convert Iceberg tables to Delta. Because the format and its log are published, more than one engine can read the same table, which is the "open" in the original paper.

A catalog then sits over the tables. On Databricks that is Unity Catalog, which controls who can read what and tracks lineageData LineageThe record of where a piece of data came from and everything it passed through, from source tables to the dashboards, models and AI answers built on it., from the source data through to the dashboards and models built on it.

Layers of quality

Raw files and finished tables now share one store, so teams usually mark how far each table can be trusted. The common pattern is the medallion architectureMedallion ArchitectureOrganising a lakehouse in 3 layers of rising quality: bronze for raw data, silver for cleaned and joined data, and gold for tables shaped for reports and AI.: bronze tables hold data as it arrived, silver tables hold it cleaned and joined, and gold tables are shaped for a report, a dashboard or an AI application. Databricks recommends it but does not require it.

Why it matters for AI

An AI system needs both kinds of data at once. A model that answers from company documents needs the raw text, the PDFs, tickets and pages a warehouse never held. An assistant that answers questions about sales needs the governed, agreed numbers a lake never guaranteed. A lakehouse keeps both under one set of permissions, so the documents feeding an embedding pipelineEmbedding PipelineThe pipeline that turns source documents into searchable vectors: it reads new and changed content, chunks it, embeds each chunk and keeps the vector index in step with the source. and the tables behind a dashboard follow the same rules.

It does not make data correct by itself. A lakehouse full of unchecked bronze data is still a lake. The gain is one copy, one set of rules and one place to check, so when an AI answer looks wrong you can trace which table it came from.