Delta Sharing on Databricks: share tables with a partner using OpenSharing

Part 1 built sales, the shop's serving table. A partner, say the courier that delivers the shop's orders, wants the sold orders every day to plan its vans. You could email a CSV each morning, and then manage who has which copy. This part shares the table instead.

The protocol and its 2 names

Delta Sharing is an open protocol for sharing tables between companies, whatever platform each one uses. The open-source project, its Python library delta-sharing and its Spark connector use that name. Databricks' docs now call Databricks' sharing platform, built on it, OpenSharing, and so does Catalog Explorer.

3 objects make a share work:

• A share is a read-only collection of tables, views and other assets, kept in Unity Catalog. Adding a table to it copies nothing.
• A recipient stands for the company you share with. You grant it SELECT on shares, the only privilege a share has.
• The provider is you. You can add or remove tables, and revoke a recipient, at any time.

A recipient reads your current data each time they query, through short-lived links. Nothing is sent in advance, and there is no copy to fall out of date.

Share the table on Databricks

As the provider, in a SQL cell or the SQL editor, make a share and put sales in it:

CREATE SHARE IF NOT EXISTS courier_share COMMENT 'Sold orders for the courier';

ALTER SHARE courier_share
ADD TABLE <catalog>.shop.sales
COMMENT 'Sold orders, 1 row per order'
WITH HISTORY;

This needs the CREATE SHARE privilege on the metastore, and you must own the share to add to it. WITH HISTORY shares the table's history too, so the recipient can run time travel queries on it. It matters for open recipients: a Delta table shared with its full history and no partition filter can be read straight from its files, while a shared view or materialized view is computed on the provider's side for a recipient outside Databricks. PARTITION shares only part of a table, and AS gives it another name.

Then make the recipient. How depends on whether the courier uses Databricks:

• On Databricks, Databricks-to-Databricks sharing. Ask them for their metastore's sharing identifier, in the form <cloud>:<region>:<uuid>, and create CREATE RECIPIENT courier USING ID '<sharing-identifier>'. The share appears in their own catalog, under their own governance, and can also hold notebooks, volumes and models.
• Anywhere else, open sharing. Create CREATE RECIPIENT courier with no ID. Databricks makes a token and a credential file, and DESCRIBE RECIPIENT courier shows the activation link, from which the courier downloads the file, once.

Either way, grant the share:

GRANT SELECT ON SHARE courier_share TO RECIPIENT courier;

Send the activation link by a secure channel. The credential file is a secret: anyone who has it can read the share until the token expires or you rotate it. The provider side runs only on Databricks, so its SQL follows Databricks' docs.

Read it as the recipient

The courier needs the credential file and a client. To see what that is like without a Databricks account, the Delta Sharing project runs an open example server with sample tables, and publishes its credential file, open-datasets.share, in the project's README. Download it beside a new script, and install the library:

pip install delta-sharing

The file is JSON with 3 fields: shareCredentialsVersion, the server's endpoint and a bearerToken. A file from a real provider has the same shape. Save this as read_share.py:

"""Read a shared table as a recipient, with only the credential file the provider sent."""
import delta_sharing

profile = "open-datasets.share"
client = delta_sharing.SharingClient(profile)
for table in client.list_all_tables():
    print(f"{table.share}.{table.schema}.{table.name}")

# A table's address is the credential file, then # and share.schema.table.
trips = delta_sharing.load_as_pandas(f"{profile}#delta_sharing.default.nyctaxi_2019", limit=3)
print(trips[["pickup_datetime", "trip_distance", "fare_amount", "total_amount"]])

python read_share.py prints, with delta-sharing 1.4.2:

delta_sharing.default.COVID_19_NYT
delta_sharing.default.boston-housing
delta_sharing.default.flight-asa_2008
delta_sharing.default.lending_club
delta_sharing.default.nyctaxi_2019
delta_sharing.default.nyctaxi_2019_part
delta_sharing.default.owid-covid-data
      pickup_datetime  trip_distance  fare_amount  total_amount
0 2019-05-27 10:13:54          10.40          0.5           2.5
1 2019-09-01 07:51:00           4.70          0.5           0.0
2 2019-09-01 07:06:00           3.97          0.5           0.0

The first list is every table the credential can see, as share.schema.table. limit=3 asks for only 3 rows, the first the server sends, not a sample of the table: a shared table can be large, and a limit keeps a first look small. For the courier, the address would be config.share#courier_share.shop.sales and the result the sold orders. The same file works in Spark, Power BI and Tableau through their Delta Sharing connectors.

What to share, and what not

A share sends data outside the company, so decide its content as you would a contract:

• Share a serving table, not silver. sales has a stated grain, described columns and checks; silver can change shape when the pipeline does.
• Share only the columns they need. sales has no names or emails. If a table has them, share a view without them, or keep them masked, as module 5 did. Dynamic views that filter rows and columns per recipient are for Databricks recipients.
• Keep track of who has what. Each recipient is an object in Unity Catalog, and SHOW GRANTS TO RECIPIENT courier lists the shares it can read. REVOKE SELECT ON SHARE courier_share FROM RECIPIENT courier takes one away.

Part 5 serves the last kind of reader, an app that needs 1 row in milliseconds.