# What is a lakehouse? One store for all of an organisation's data: cheap files in cloud storage, as in a data lake, with the tables, transactions and fast queries of a data warehouse on top. - Page: https://tecyfy.com/concepts/lakehouse - Subject: Data engineering in the AI era (https://tecyfy.com/concepts) - Level: beginner - Also called: Data lakehouse For more than a decade, a large company's data has usually lived in 2 places. Everything landed first in a **data lake**: cheap files in cloud storage, kept raw, in any format. Then the part the business reported on was copied again into a **data warehouse**, a separate system with tidy tables and fast queries. A lakehouse is the idea that you do not need the second copy. Keep the files in cheap cloud storage, and add the parts that made the warehouse worth paying for, on top of them. ## Why 2 copies became a problem The 2021 paper that named the pattern, written by 4 people at Databricks, lists what the 2-tier setup costs. Every copy from lake to warehouse is another job that can fail or bring in a bug. The warehouse is always behind the lake, with new data often taking days to arrive; the paper cites a survey in which 86% of analysts were working from out-of-date data. And machine learning fits neither side well. The warehouse holds only the curated slice, often in a format a training job cannot read directly, while the lake holds everything with no guarantees about it. ## What sits on top of the files The trick is a layer that makes a folder of files behave like a database table. The data stays in an open file format such as Parquet. Next to it sits a log that records every change to the table. Writers add to the log in a way that either fully succeeds or does not happen at all, so a reader never sees half a write. Each write makes a new version of the table, and older versions can still be queried. This is what an [open table format](https://tecyfy.com/concepts/open-table-format) provides. Delta Lake is the default on Databricks, where every table is a Delta table unless you say otherwise. Apache Iceberg is another, and Databricks can convert Iceberg tables to Delta. Because the format and its log are published, more than one engine can read the same table, which is the "open" in the original paper. A catalog then sits over the tables. On Databricks that is Unity Catalog, which controls who can read what and tracks [lineage](https://tecyfy.com/concepts/data-lineage), from the source data through to the dashboards and models built on it. ## Layers of quality Raw files and finished tables now share one store, so teams usually mark how far each table can be trusted. The common pattern is the [medallion architecture](https://tecyfy.com/concepts/medallion-architecture): **bronze** tables hold data as it arrived, **silver** tables hold it cleaned and joined, and **gold** tables are shaped for a report, a dashboard or an AI application. Databricks recommends it but does not require it. ## Why it matters for AI An AI system needs both kinds of data at once. A model that answers from company documents needs the raw text, the PDFs, tickets and pages a warehouse never held. An assistant that answers questions about sales needs the governed, agreed numbers a lake never guaranteed. A lakehouse keeps both under one set of permissions, so the documents feeding an [embedding pipeline](https://tecyfy.com/concepts/embedding-pipeline) and the tables behind a dashboard follow the same rules. It does not make data correct by itself. A lakehouse full of unchecked bronze data is still a lake. The gain is one copy, one set of rules and one place to check, so when an AI answer looks wrong you can trace which table it came from. ## Related - [Data Lake](https://tecyfy.com/concepts/data-lake): Cheap storage that holds data of any kind in its raw form, usually as files in cloud object storage, to be cleaned and structured later. - [Data Warehouse](https://tecyfy.com/concepts/data-warehouse): A database built for analysis: cleaned, structured data from many systems, kept in one place and shaped for fast reports and dashboards. - [Open Table Format](https://tecyfy.com/concepts/open-table-format): A published way of storing a table as plain data files plus a log of changes, so many engines can read and safely write it. Delta Lake and Apache Iceberg are the best known. - [Medallion Architecture](https://tecyfy.com/concepts/medallion-architecture): Organising a lakehouse in 3 layers of rising quality: bronze for raw data, silver for cleaned and joined data, and gold for tables shaped for reports and AI. - [ELT](https://tecyfy.com/concepts/elt): Loading raw data into the warehouse or lakehouse first and transforming it there, the reverse of the older ETL order, which transformed data before loading it. - [Embedding Pipeline](https://tecyfy.com/concepts/embedding-pipeline): The pipeline that turns source documents into searchable vectors: it reads new and changed content, chunks it, embeds each chunk and keeps the vector index in step with the source. ## Sources - [Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics](https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf) - [What is a data lakehouse?](https://docs.databricks.com/aws/en/lakehouse/) - [What is Delta Lake in Databricks?](https://docs.databricks.com/aws/en/delta/) - [What is the medallion lakehouse architecture?](https://docs.databricks.com/aws/en/lakehouse/medallion)