# Unity Catalog explained: catalogs, schemas, tables and volumes

Unity Catalog is Databricks' catalog and access control for data and AI. Every table, view, volume, function and model has a 3-part name, catalog.schema.object: a catalog is the top unit, often 1 per environment or business area, and a schema groups the objects inside it. Unity Catalog records each object and decides who may use it. A managed table or volume leaves both its files and its access to Unity Catalog; an external one keeps its files where you put them, and Unity Catalog governs only the access.

- Page: https://tecyfy.com/data-engineering/course/unity-catalog
- Course: Data engineering course (https://tecyfy.com/data-engineering/course), Module 5: Unity Catalog and governance, Catalogs, schemas and the 3-part name
- Parts of this module: Catalogs, schemas and the 3-part name (https://tecyfy.com/data-engineering/course/unity-catalog); Privileges and grants (https://tecyfy.com/data-engineering/course/unity-catalog/privileges); Masking personal data (https://tecyfy.com/data-engineering/course/unity-catalog/personal-data); Lineage and audit logs (https://tecyfy.com/data-engineering/course/unity-catalog/lineage-and-audit); Build it: the shop's catalog (https://tecyfy.com/data-engineering/course/unity-catalog/build-it)

At the end of this part you can:

- Name any table on Databricks by its 3 parts, and say which catalog and schema a short name resolves to
- Say what managed and external mean for tables and volumes
- Lay out catalogs and schemas for environments and for the layers of a pipeline

Every table in this course so far had a short name, such as `orders`. On Databricks, every table, view, volume and function also has a full name with 3 parts, and Unity Catalog, Databricks' catalog, decides who may use it. This module lays out the shop's data in a catalog, grants access, masks personal data and follows where data goes.

## The 3-part name

Unity Catalog names everything as `catalog.schema.object`. A **catalog** is the top unit, often 1 for each environment or business area. A **schema** groups related objects inside a catalog. The **objects** are tables, views, volumes for files, functions and models. Each of these levels can be granted to people on its own, which [part 2](/data-engineering/course/unity-catalog/privileges) uses.

Spark on your computer has a catalog too, called `spark_catalog`, with the same parts. The scripts in this module use `delta_session.py` from [module 4](/data-engineering/course/delta-lake-and-iceberg) for their session. Save this as `catalog.py`:

```python
"""Make the shop's schemas and 1 table, then find the table by its 3-part name."""
from delta_session import delta_spark

spark = delta_spark()
catalog = spark.catalog.currentCatalog()
print("Catalog:", catalog, "| Schema:", spark.catalog.currentDatabase())

for schema in ["bronze", "silver", "gold"]:
    spark.sql(f"CREATE SCHEMA IF NOT EXISTS {catalog}.{schema}")

spark.sql(f"""
    CREATE TABLE IF NOT EXISTS {catalog}.silver.customers (
        customer_id STRING, name STRING, email STRING, country STRING
    ) USING DELTA
""")
spark.sql(f"""
    INSERT OVERWRITE {catalog}.silver.customers VALUES
        ('C01', 'Ana', 'ana.silva@example.com', 'ES'),
        ('C02', 'Ben', 'ben.byrne@example.org', 'IE'),
        ('C03', 'Chen', 'chen.li@example.net', 'DE')
""")

spark.sql(f"SHOW SCHEMAS IN {catalog}").show()
for table in spark.catalog.listTables(f"{catalog}.silver"):
    print(table.catalog, table.namespace, table.name, table.tableType)
spark.sql(f"SELECT customer_id, name, country FROM {catalog}.silver.customers ORDER BY customer_id").show()
```

Run it with `python catalog.py`:

```text
Catalog: spark_catalog | Schema: default
+---------+
|namespace|
+---------+
|   bronze|
|  default|
|     gold|
|   silver|
+---------+

spark_catalog ['silver'] customers MANAGED
+-----------+----+-------+
|customer_id|name|country|
+-----------+----+-------+
|        C01| Ana|     ES|
|        C02| Ben|     IE|
|        C03|Chen|     DE|
+-----------+----+-------+
```

The script never writes a catalog's name. It asks Spark for the current one, so the same code runs on your computer, where the answer is `spark_catalog`, and on Databricks, where it is your workspace's default catalog. A short name such as `silver.customers` is completed the same way, with the current catalog, and `customers` alone with the current schema too.

The difference is what stands behind the name. Locally, the catalog is a list of names in the `metastore_db` folder, and anyone who can open the folder can read every file. In Unity Catalog, a query goes through the name, and Unity Catalog checks the privileges of whoever runs it before the files are read.

## Managed or external

The table above is **managed**: you gave it a name, and the catalog chose where its files live. In Unity Catalog, a managed table leaves both its files and its access to Unity Catalog, in the storage of its catalog or schema. It is the default and the type Databricks recommends, and the type whose file sizes and upkeep Databricks can handle for you, as [module 4](/data-engineering/course/delta-lake-and-iceberg/optimize-and-vacuum) showed.

An **external** table keeps its files at a location you name, such as a folder another system also writes. Unity Catalog governs who can use it, and nothing more. Use it when the files must stay where they are.

Files that are not tables, such as the CSV files the shop receives, go in a **volume**, the same split for files: a managed volume lives in the catalog's storage, an external one points at a folder you already have. Module 2 put the shop's raw files in one, at `/Volumes/<catalog>/<schema>/raw`.

## Laying out catalogs and schemas

Databricks describes the catalog as the main unit for separating data, and suggests, for example, 1 catalog for production and 1 for development. Give them the same schemas inside, and the same code runs against either by changing only the catalog. A catalog can also be bound to the workspaces allowed to use it, so a production catalog is not reachable from a workspace for experiments. Data that needs tighter control, such as customers' personal details, can have its own catalog too.

Inside a catalog, this course uses 1 schema for each layer of the pipeline: `bronze` for data as it arrived, `silver` for cleaned tables and `gold` for what analysts and dashboards read. Module 7 builds those layers; this module decides who sees each.

## On Databricks

Every workspace on Unity Catalog has a default catalog, used when a query leaves the catalog out. In some workspaces it is the workspace catalog, which usually has the workspace's name and is open to all of the workspace's users. To see where you are, run:

```sql
SELECT current_catalog(), current_schema()
```

To run `catalog.py` in a notebook, leave out the import and the `delta_spark()` line: the notebook's `spark` is already connected to Unity Catalog. In **Catalog** in the sidebar, the new schemas and the `customers` table appear under your catalog, with their columns, their history and, from [part 4](/data-engineering/course/unity-catalog/lineage-and-audit), their lineage.

## Check yourself

1. In a Databricks notebook, you run SELECT * FROM customers. Which table does it read?

   A. The first table called customers in any catalog you can use
   B. The table customers in the current catalog and the current schema, which you can see with SELECT current_catalog() and current_schema()
   C. The table customers in the hive_metastore catalog, always
   D. It fails, because Unity Catalog requires all 3 parts in every query

2. The shop wants its development tables and its production tables kept apart, with different people allowed to change each. What layout does Databricks suggest?

   A. 1 catalog for production and 1 for development, each with the same schemas inside
   B. 1 schema called prod and 1 called dev in the same catalog, with everything else mixed in
   C. Separate metastores for each, in the same region
   D. A prefix on every table name, such as prod_orders and dev_orders

### Answers

1. B. A short name is completed with the current catalog and schema, the same rule Spark uses on your computer. Each workspace has a default catalog, and a notebook can change it with USE CATALOG and USE SCHEMA. Write all 3 parts in jobs that must not depend on where they run.
2. A. Databricks describes catalogs as the main unit for separating data, for example a catalog for production and one for development. The same schema names in each let the same code run against either, by changing only the catalog, and a catalog can be bound to the workspaces allowed to use it.

## Questions

### What is the difference between a catalog and a schema in Databricks?

A catalog is the top level of Unity Catalog's 3-part name and the main unit for separating data, such as production from development. A schema, also called a database, sits inside a catalog and groups tables, views, volumes, functions and models. The full name of a table is catalog.schema.table.

### What is the difference between a managed and an external table in Unity Catalog?

For a managed table, Unity Catalog handles both the access and the files, in the storage of its catalog or schema, and Databricks recommends managed tables. For an external table, you choose the storage location and Unity Catalog governs only who can use the table. The same split applies to volumes, which hold files rather than tables.

## Sources

- [What is Unity Catalog?](https://docs.databricks.com/aws/en/data-governance/unity-catalog/)
- [What are catalogs in Databricks?](https://docs.databricks.com/aws/en/catalogs/)
- [Manage the default catalog](https://docs.databricks.com/aws/en/catalogs/default)
- [Unity Catalog managed tables](https://docs.databricks.com/aws/en/tables/managed)
- [Predictive optimization for Unity Catalog managed tables](https://docs.databricks.com/aws/en/optimizations/predictive-optimization)
- [current_schema function](https://docs.databricks.com/aws/en/sql/language-manual/functions/current_schema)
- [pyspark.sql.Catalog](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/catalog.html)