Databricks Zerobus Ingest vs Kafka: when to skip the message bus

Zerobus Ingest is a Databricks service that takes records from your producers and writes them straight into a Unity Catalog Delta LakeDelta LakeAn open table format that turns Parquet files in cloud storage into a table with transactions, a version history, schema checks, and updates and deletes, so a data lake behaves like a database. table, with no Kafka cluster in between. It replaces Kafka when Databricks is the only place the stream goes. It cannot replace Kafka when other systems read the same events, because Zerobus is write-only: nothing can consume from it.

That one line decides most cases. The rest of this page is about the cases it does not decide: what you give up when you drop the bus, how an existing producer switches, and what the limits and the bill look like.

What changes when the bus goes

A usual streaming path into Databricks has 3 parts you run: the producers, a Kafka cluster (brokers, topics, partitions, retention), and a job that reads the topic and writes the table, often Structured Streaming or a pipeline. Zerobus folds the last 2 into one serverless endpoint. You create the table, then push to it:

from zerobus.sdk.sync import ZerobusSdk
from zerobus.sdk.shared import TableProperties

sdk = ZerobusSdk(SERVER_ENDPOINT, DATABRICKS_WORKSPACE_URL)
stream = sdk.create_stream(CLIENT_ID, CLIENT_SECRET, TableProperties("main.default.air_quality"))
stream.ingest_record_offset({"device_name": "sensor-1", "temp": 22, "humidity": 55})
stream.close()

That is Databricks' own example. The table's schema is the contract: each record is checked against it, made durable, acknowledged in about 150 ms, and in the table about 5 seconds after it was sent. There are no partitions to choose; you scale by opening more streams, and the number of streams has no limit. Databricks ingested 1 PB into a single table in under 24 hours to show the ceiling, 12 GB a second sustained.

The endpoint speaks 5 protocols, so most producers need no new library:

Who reads the stream decides it

Kafka is two things at once: a pipe that carries records, and a log that many readers consume at their own pace. Zerobus is only the pipe, and only into Databricks. So ask who reads the events today, and who will next year.

If a fraud service, a search index or another warehouse reads the topic, Kafka stays. Databricks says the same in its docs: where Kafka serves readers outside the lakehouse, copy it in with a managed streaming connector. If Databricks is the only reader, the cluster is a pipe you pay to run, and Zerobus removes it.

Then check what the stream must do. Some needs move to the table, one needs nothing from either, and a few are beyond Zerobus:

Try itDo you still need Kafka for this stream?

Who reads the stream

Only DatabricksDatabricks and other systems

The stream must

·Nothing special·Replay from an offset·Keep keys and headers·Compressed batches·No duplicates·One order across producers·Records over 10 MB·Under a second end to end·Write to an external table

Producers

apps, devices, collectors, CDC

→

Zerobus Ingest endpoint

gRPC SDK, REST, OTLP, MQTT or Kafka protocol

serverless, no partitions

→

Delta table in Unity Catalog

in the table about 5 s after the send

The ones people trip on:

• Replay. With Kafka you rewind a consumer to an offset. With Zerobus the table is the log: replay by reading the Delta table as a stream from a starting version, or query an older version with time travel. It works, but it is a change to how your jobs restart, so plan it before you turn the cluster off.
• Duplicates. Delivery is at least once when the interface acknowledges durability and the client retries what was not acknowledged; on MQTT that means QoS 1, since QoS 0 is best-effort. The SDK replays records that were not acknowledged, so a record can land twice after a network fault. Put an id in each record and remove repeats in the next step, which an AUTO CDC flow does for you when the id is the key.
• Order. Records keep their order within one stream. Spread a producer over several streams and the order across them is gone. Kafka has the same rule for partitions, so a design that needs one global order needs a sequence number in either case.
• Latency. About 5 seconds to the table is fine for dashboards, features and alerts measured in minutes. When a decision must happen in under a second, Databricks points to real-time mode in Structured Streaming, which reads from a source such as Kafka.

Switching a Kafka producer without new code

The Kafka-compatible API lets an existing producer write to Zerobus with configuration changes only. The topic is the table's full name, and Zerobus checks each value against the table's schema:

from kafka import KafkaProducer

producer = KafkaProducer(
    bootstrap_servers="1234567890123456.zerobus.us-west-2.cloud.databricks.com:9092",
    security_protocol="SASL_SSL",
    sasl_mechanism="OAUTHBEARER",
    sasl_oauth_token_provider=ZerobusTokenProvider(),  # mints a table-scoped OAuth token
    acks="all",               # acknowledged only once durable
    compression_type=None,    # compressed batches are rejected
)
producer.send("main.default.air_quality", value=b'{"device_name": "sensor-1", "temp": 22, "humidity": 55}')
producer.flush()

The token provider exchanges a service principal's id and secret for an OAuthOAuthThe standard way to let one app act for you at another service without giving it your password; the app gets a limited, revocable access token instead. MCP servers use it to let AI agents sign in. token scoped to that one table; Databricks' docs give the full function. Tokens last an hour, so pass a callback, not a fixed string.

4 things behave differently from a real broker. Only the value is kept: keys, headers, partition and timestamp are dropped, so move anything you need into the JSON. Values must be JSON; for protobuf or Avro use an SDK. The whole endpoint is one logical partition, 0, which matters only to tools that inspect partitions. And the API is in Beta, with a default of 50,000 messages a second per workspace against 100,000 records a second per stream on the SDK.

The limits that matter

The partition rule catches tables partitioned by something fine-grained, such as device or customer: a busy stream touches more than 100 values in 5 seconds, which Zerobus does not support. Databricks recommends liquid clustering for these tables instead of Data Partitioning and ClusteringData Partitioning and ClusteringTwo ways to lay out a large table so a query reads only the files it needs: partitioning splits it by a column such as the date, clustering keeps similar values in the same files., with predictive optimization on so the clustering keeps up. The 28-day rule matters if a table is dropped or its permissions break while producers keep sending: after 28 days the held records are gone for good.

Kafka's own default record size is 1 MB, so 10 MB is roomier than most clusters; our article on Kafka's max message size covers why large records are better as files with a pointer in either system.

What it costs, and how to see it

Zerobus is billed on the Jobs Serverless SKU: in AWS US East that is $0.35 per DBU on the Premium plan and $0.45 on Enterprise. Databricks does not publish how many DBUs a gigabyte or a million records take, so no honest figure per GB exists yet. Measure it in a week of real traffic with the billing system table:

SELECT usage_date,
       product_features.lakeflow_connect.zerobus_request_type AS interface,
       SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE billing_origin_product = 'LAKEFLOW_CONNECT'
  AND product_features.lakeflow_connect.zerobus_request_type IS NOT NULL
GROUP BY ALL
ORDER BY usage_date;

Then set it against what the bus costs you now: the brokers or the managed Kafka bill, the job that copies topics into tables, and the time spent sizing partitions and upgrading clusters. Our Kafka sizing calculator gives the brokers a topic needs on Amazon MSK, and the Databricks cost calculator prices the job that reads it. If Kafka stays because others read it, how many partitions you need is the next question.

Moving one stream over

Move one stream at a time, not the platform:

1. Pick a stream that only Databricks reads, with records under 10 MB, written to a managed table.
2. Create the target table yourself; Zerobus never creates or alters tables. Use liquid clustering, not fine partitions.
3. Give a service principal USE CATALOG, USE SCHEMA, and SELECT and MODIFY on that table.
4. Point a copy of the producer at Zerobus (the Kafka-compatible settings above, or an SDK) and write to a new table beside the old one.
5. Compare counts and duplicates between the two tables for a few days, and test how your downstream jobs restart from the table instead of an offset.
6. Turn off the topic's consumer job, then the topic.

Zerobus has been public since its preview in October 2025. Since then Databricks has made streaming tables and Apache Arrow generally available as targets and formats, added PrivateLink, and put ingestion into default storage in preview, so check the docs for the target you plan before you commit. For where streams fit in a data platform, see the StreamingStreamingGetting a model's answer in small pieces as it is written, instead of all at once at the end, so people see text start sooner and long answers do not hit a timeout. concept, and for the brokers you might keep, Kafka vs Kinesis vs Pub/Sub.