# Kafka vs Kinesis vs Pub/Sub: how they differ and which to choose

Kafka, Kinesis Data Streams and Pub/Sub all carry events from the services that write them to the services that read them. Kafka is an open-source log that you run yourself or buy managed (Amazon MSK, Google's Managed Service for Apache Kafka and other vendors); topics are split into partitions, events are kept for as long as you set (7 days by default, or forever), any number of consumer groups can read them again, and transactions give exactly-once processing between Kafka topics. Kinesis Data Streams is AWS's own stream, split into shards of 1 MB a second of writes each, kept 24 hours by default and up to 365 days. Pub/Sub is Google Cloud's messaging service with no partitions to size; it hands each message to a subscriber and redelivers it until it is acknowledged. Choose Kafka for an open, portable log with many readers and long replay, Kinesis on AWS without running brokers, and Pub/Sub on Google Cloud, especially to feed many services or BigQuery.

- Page: https://tecyfy.com/data-engineering/kafka-vs-kinesis-vs-pub-sub
- Part of: Data engineering (https://tecyfy.com/data-engineering)

- **Apache Kafka** (Apache Software Foundation; managed by Amazon MSK, Google Cloud and other vendors, https://kafka.apache.org/43/getting-started/introduction/): An open-source event streaming platform that stores events in partitioned, replicated topics, for as long as you set, and lets any number of consumers read them. Install: `Run brokers yourself, or create a cluster on Amazon MSK or Google's Managed Service for Apache Kafka`
- **Amazon Kinesis Data Streams** (Amazon Web Services, https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html): An AWS service that stores a stream of records in shards, each with a fixed capacity, scaled for you in on-demand mode or by the number of shards you set. Install: `aws kinesis create-stream --stream-name orders --stream-mode-details StreamMode=ON_DEMAND`
- **Google Cloud Pub/Sub** (Google Cloud, https://cloud.google.com/pubsub/docs/overview): A Google Cloud messaging service where publishers send messages to a topic and each subscription receives every message, delivered and redelivered until it is acknowledged. Install: `gcloud pubsub topics create orders`

All 3 take events from the services that write them and hand them to the services that read them, so neither side waits for the other. The difference is what they keep and how they split the work. Kafka and Kinesis are logs: events are kept in order, split into partitions or shards, and each reader keeps its place. Pub/Sub is closer to a queue that copies each message to every subscription and tracks each message until a reader acknowledges it.

## What each one is

**Apache Kafka** stores events in topics, and each topic is split into partitions spread over the brokers. An event's key picks its partition, so all events for 1 customer or 1 country stay in order. Reading does not delete anything: events stay for the retention you set, and every consumer group keeps its own offset in each partition, so a new team can read the whole history. You run the brokers, or a managed service such as Amazon MSK or Google's runs them. Kafka Connect moves data in and out, and Kafka Streams processes it.

**Amazon Kinesis Data Streams** has the same shape as a log on AWS, with shards in place of partitions. Each shard takes up to 1 MB or 1,000 records a second of writes, and in on-demand mode AWS adds shards as traffic grows; in provisioned mode you set the count and pay per shard hour. Records are kept 24 hours unless you pay to keep them longer, up to a year. Readers either share each shard's 2 MB a second of reads or register for enhanced fan-out and get their own.

**Google Cloud Pub/Sub** has no partitions or shards to size. Publishers send to a topic, every subscription gets its own copy of each message, and Pub/Sub leases each message to 1 subscriber client at a time and redelivers it if it is not acknowledged. That makes it a natural fit for spreading work over any number of workers, for pushing messages to services over HTTP, and for writing them straight into BigQuery. The trade is that order is off by default and must be asked for with ordering keys.

## Side by side

| | Apache Kafka | Amazon Kinesis Data Streams | Google Cloud Pub/Sub |
| --- | --- | --- | --- |
| **What it is** | An open-source distributed log. You run the brokers, or a managed service runs them for you. | A managed AWS service. There are no servers to run; you choose on-demand or provisioned capacity. | A managed Google Cloud service. There are no servers, partitions or shards to size. |
| **How it scales** | Partitions. A topic is split across brokers, and the partition count caps how many consumers in a group share the work. | Shards, each up to 1 MB or 1,000 records a second of writes and 2 MB a second of reads. On-demand mode adds shards as traffic grows. | Per message. Pub/Sub leases each message to a subscriber client, so the number of readers is not tied to partitions. |
| **Order** | Kept within a partition. Events with the same key go to the same partition and are read in the order written. | Records with the same partition key go to the same shard. For strictly increasing order, write serially and pass SequenceNumberForOrdering. | None by default. With message ordering on, messages with the same ordering key are delivered in order, up to 1 MB a second per key. |
| **Duplicates** | The idempotent producer stops duplicates from retries. Transactions with read_committed consumers give exactly-once from topic to topic. | Records can arrive more than once, from producer and consumer retries. Put a unique key in each record and remove duplicates. | At least once by default. Exactly-once delivery is available on pull subscriptions, in one region. |
| **How long events are kept** | 7 days by default (retention.ms), any time you set, or forever. Compacted topics keep the latest value for each key. | 24 hours by default, up to 365 days at extra cost. | Unacknowledged messages 7 days by default. A topic can keep every message up to 31 days. |
| **Reading again** | Any consumer can go back to an earlier offset, and a new consumer group can read the topic from the start. | A reader can start from any record still kept, by sequence number, by time or from the oldest (TRIM_HORIZON). | Seek a subscription to a time or a snapshot. Replaying acknowledged messages needs topic retention or a subscription that keeps them. |
| **Many readers** | Any number of consumer groups, each with its own position in each partition. | Shared readers split 2 MB a second per shard. Enhanced fan-out gives each registered consumer its own 2 MB a second per shard, up to 20 (50 in On-demand Advantage). | Any number of subscriptions to a topic (up to 10,000), pulled, pushed to an HTTP endpoint, or written straight to BigQuery or Cloud Storage. |
| **Largest message** | About 1 MB by default (max.message.bytes), which you can raise. | 10 MiB. | 10 MB. |
| **Price** | The brokers and their disks, yourself or through a managed service's price. | Per GB written and read in on-demand mode, or per shard hour in provisioned mode, plus extended retention. Not in the AWS Free Tier. | $40 per TiB of messages after the first 10 GiB each month, plus $0.27 per GiB-month for retained messages. |

## Which one fits

- **A stream that many teams read and replay over weeks or months:** Apache Kafka. Kafka keeps events for as long as you set, every consumer group keeps its own position, and compacted topics hold the latest value for each key with no end date.
- **Events between services on AWS without running servers:** Amazon Kinesis Data Streams. On-demand streams need no capacity planning and scale with the traffic, and data delivery writes the stream to Apache Iceberg tables or S3 with no consumer code.
- **Events between services on Google Cloud, or straight into BigQuery:** Google Cloud Pub/Sub. There are no partitions to size, push subscriptions call your services, and a BigQuery subscription writes messages into a table with no code.
- **The same pipeline on more than 1 cloud or on your own servers:** Apache Kafka. Kafka is open source and runs on your hardware and as managed services on AWS and Google Cloud, so the same producers and consumers move with you.
- **Exactly-once processing from stream to stream:** Apache Kafka. Kafka's transactional producer writes the output and the consumer's committed offset together, and read_committed consumers see only committed results. Kinesis and Pub/Sub leave duplicate handling to the reader.
- **A work queue where each message is handled by any free worker:** Google Cloud Pub/Sub. Pub/Sub leases each message to a worker and redelivers it if it is not acknowledged in time, so the number of workers is not limited by partitions or shards.

## The same stream in each

The same small stream in each: a shop writes 5 orders keyed by country, and a reader adds up revenue by country. Each file was run on a laptop, Kafka 4.4 in KRaft mode, Kinesis on the moto mock and Pub/Sub on Google's emulator, and all 3 printed DE 222.0, FR 80.5 and IN 15.25.

### Apache Kafka: `kafka_orders.py`

```python
import json
from confluent_kafka import Consumer, Producer
from confluent_kafka.admin import AdminClient, NewTopic
from orders import ORDERS

BROKERS = "localhost:9092"
admin = AdminClient({"bootstrap.servers": BROKERS})
admin.create_topics([NewTopic("orders", num_partitions=3, replication_factor=1)])["orders"].result()

# The key picks the partition: one country's orders stay in order.
producer = Producer({"bootstrap.servers": BROKERS, "enable.idempotence": True})
for order in ORDERS:
    producer.produce("orders", key=order["country"], value=json.dumps(order))
producer.flush()

# A consumer group: Kafka stores how far "revenue" has read, per partition.
consumer = Consumer({
    "bootstrap.servers": BROKERS,
    "group.id": "revenue",
    "auto.offset.reset": "earliest",
    "enable.auto.commit": False,
})
consumer.subscribe(["orders"])
revenue, seen = {}, 0
while seen < len(ORDERS):
    msg = consumer.poll(5.0)
    if msg is None or msg.error():
        continue
    order = json.loads(msg.value())
    revenue[order["country"]] = revenue.get(order["country"], 0) + order["amount"]
    seen += 1
    consumer.commit(msg)  # after the work, so a crash re-reads rather than skips
consumer.close()
print(dict(sorted(revenue.items())))
```

The key decides the partition, so the 3 German orders landed in 1 partition, in order. The consumer group revenue commits its position after each order is counted, so a crash reads an order again rather than skipping it, and a second group could read the whole topic again from the start. Run with confluent-kafka 2.16.

### Amazon Kinesis Data Streams: `kinesis_orders.py`

```python
import json
import boto3
from orders import ORDERS

kinesis = boto3.client("kinesis", region_name="eu-central-1")
kinesis.create_stream(StreamName="orders", StreamModeDetails={"StreamMode": "ON_DEMAND"})
kinesis.get_waiter("stream_exists").wait(StreamName="orders")

# The partition key picks the shard: one country's orders stay in order.
kinesis.put_records(StreamName="orders", Records=[
    {"PartitionKey": order["country"], "Data": json.dumps(order).encode()} for order in ORDERS
])

# Read every shard from the oldest record still kept. A real consumer uses the
# Kinesis Client Library, which keeps each shard's checkpoint in DynamoDB.
revenue = {}
for shard in kinesis.list_shards(StreamName="orders")["Shards"]:
    iterator = kinesis.get_shard_iterator(
        StreamName="orders", ShardId=shard["ShardId"], ShardIteratorType="TRIM_HORIZON"
    )["ShardIterator"]
    for record in kinesis.get_records(ShardIterator=iterator)["Records"]:
        order = json.loads(record["Data"])
        revenue[order["country"]] = revenue.get(order["country"], 0) + order["amount"]
print(dict(sorted(revenue.items())))
```

An on-demand stream needs no shard count; the mock started this one with 4. The partition key places each order on a shard. This reader walks every shard itself; in production the Kinesis Client Library does that, sharing shards between workers and keeping each shard's checkpoint in DynamoDB. Run with boto3 1.43.

### Google Cloud Pub/Sub: `pubsub_orders.py`

```python
import json
from google.cloud import pubsub_v1
from orders import ORDERS

PROJECT = "shop-local"
publisher = pubsub_v1.PublisherClient(
    publisher_options=pubsub_v1.types.PublisherOptions(enable_message_ordering=True)
)
subscriber = pubsub_v1.SubscriberClient()
topic = publisher.create_topic(name=publisher.topic_path(PROJECT, "orders")).name
# A subscription is a copy of the topic for one reader. Create it before publishing:
# it only receives messages published after it exists.
subscription = subscriber.create_subscription(request={
    "name": subscriber.subscription_path(PROJECT, "revenue"),
    "topic": topic,
    "enable_message_ordering": True,
}).name

# The ordering key keeps one country's orders in order; without one, there is no order.
for order in ORDERS:
    publisher.publish(topic, json.dumps(order).encode(), ordering_key=order["country"]).result()

revenue, seen = {}, 0
while seen < len(ORDERS):
    response = subscriber.pull(request={"subscription": subscription, "max_messages": 10})
    for received in response.received_messages:
        order = json.loads(received.message.data)
        revenue[order["country"]] = revenue.get(order["country"], 0) + order["amount"]
        seen += 1
    # Acknowledge after the work; a message not acknowledged in time is delivered again.
    if response.received_messages:
        subscriber.acknowledge(request={
            "subscription": subscription,
            "ack_ids": [m.ack_id for m in response.received_messages],
        })
print(dict(sorted(revenue.items())))
```

No partitions or shards anywhere. The subscription is made before publishing because it only receives messages published after it exists, and ordering is switched on twice, on the publisher and on the subscription. A message that is not acknowledged in time is delivered again. Run with google-cloud-pubsub 2.43 against the emulator (PUBSUB_EMULATOR_HOST).

The Kafka and Kinesis files look alike because the model is alike: a key picks the partition or shard, and order holds within it. They differ in who keeps the reader's place. Kafka stores each consumer group's offsets itself, so the consumer commits and moves on. Kinesis stores nothing for the reader; the Kinesis Client Library keeps checkpoints in a DynamoDB table, and a hand-written reader like this one starts from wherever it is told.

The Pub/Sub file has no partitions at all. It creates a subscription before the first message is published, because a new subscription starts empty, and asks for ordering in 2 places. Each message is acknowledged after it is counted, and anything not acknowledged in time comes back.

## Things that trip people up

- **Order is per key, never per topic.** Kafka orders within a partition, Kinesis within a shard, Pub/Sub within an ordering key. If 2 events must stay in order, give them the same key.
- **Every reader must handle duplicates.** Retries on either side can deliver an event twice in all 3 (exactly-once in Kafka and Pub/Sub has conditions). Make the write at the end [idempotent](https://tecyfy.com/concepts/idempotent-pipeline), with an id in each event.
- **A hot key fills 1 partition or shard.** If 1 key carries most of the traffic, adding partitions or shards does not help, because that key still lands on 1 of them. Kinesis on-demand mode does not split a single key beyond a shard's 1 MB a second.
- **A Pub/Sub subscription made today does not see yesterday.** It starts with messages published after it exists. To read the past, keep messages on the topic and seek.
- **Kinesis keeps 24 hours unless you pay for more.** A reader down for a weekend loses data on a stream with the default retention.

## Sizing it

For Kafka, the number of partitions comes from the busiest traffic and what 1 partition can take from producers and give a consumer, and the brokers from disk, network and copies. The [Kafka sizing calculator](/data-engineering/kafka-sizing) does those sums for Amazon MSK or your own hardware. What happens after the stream, when events land in tables, is the [change data capture](https://tecyfy.com/concepts/change-data-capture) and [watermark](https://tecyfy.com/concepts/watermark) side of the story.

## Questions

### What is the difference between Kafka, Kinesis and Pub/Sub?

Kafka is an open-source log you run or buy managed, scaled by partitions, keeping events as long as you set. Kinesis Data Streams is AWS's managed stream, scaled by shards and keeping records 24 hours by default, up to 365 days. Pub/Sub is Google Cloud's managed messaging service, with no partitions or shards; it delivers each message to every subscription and redelivers it until it is acknowledged.

### Is Kinesis the same as Kafka?

No. Both store an ordered stream split into parts (partitions in Kafka, shards in Kinesis) and let several applications read it, but Kinesis is an AWS service with its own API, and Kafka is open source. On AWS, Amazon MSK runs Kafka itself, for teams that want Kafka's protocol and tools without running the brokers.

### Does Pub/Sub guarantee message order?

Not by default. Turn on message ordering on the subscription and publish with an ordering key, and messages with the same key arrive in the order Pub/Sub received them. The messages must be published in the same region, and each ordering key is limited to 1 MB a second.

### Which one gives exactly-once?

Kafka, between Kafka topics, with the idempotent producer and transactions read by read_committed consumers. Pub/Sub offers exactly-once delivery on pull subscriptions when subscribers connect in the same region. Kinesis delivers at least once, so readers remove duplicates with a key in each record. In all 3, writing to an outside system exactly once needs an idempotent write there.

### How long do Kafka, Kinesis and Pub/Sub keep messages?

Kafka keeps them 7 days by default, and a topic can keep them for any time or forever. Kinesis keeps records 24 hours by default and up to 365 days. Pub/Sub keeps unacknowledged messages 7 days by default, and a topic with retention keeps every message up to 31 days.

### How many Kafka partitions or Kinesis shards do I need?

Divide the busiest traffic by what 1 partition or shard can take. A Kinesis shard takes 1 MB or 1,000 records a second of writes; a Kafka partition's limit depends on your brokers and consumers, so measure it. The Kafka sizing calculator does the sums for partitions and brokers.

## Sources

- [Introduction](https://kafka.apache.org/43/getting-started/introduction/)
- [Topic configs](https://kafka.apache.org/43/configuration/topic-configs/)
- [Design](https://kafka.apache.org/43/design/design/)
- [What is Amazon MSK?](https://docs.aws.amazon.com/msk/latest/developerguide/what-is-msk.html)
- [What is Pub/Sub?](https://cloud.google.com/pubsub/docs/overview)
- [Amazon Kinesis Data Streams terminology and concepts](https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html)
- [Quotas and limits](https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html)
- [Choose the right mode to stream in](https://docs.aws.amazon.com/streams/latest/dev/how-do-i-size-a-stream.html)
- [GetShardIterator](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetShardIterator.html)
- [Develop enhanced fan-out consumers with dedicated throughput](https://docs.aws.amazon.com/streams/latest/dev/enhanced-consumers.html)
- [Handle duplicate records](https://docs.aws.amazon.com/streams/latest/dev/kinesis-record-processor-duplicates.html)
- [PutRecord](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html)
- [Amazon Kinesis Data Streams pricing](https://aws.amazon.com/kinesis/data-streams/pricing/)
- [Subscription overview](https://cloud.google.com/pubsub/docs/subscription-overview)
- [Order messages](https://cloud.google.com/pubsub/docs/ordering)
- [Exactly-once delivery](https://cloud.google.com/pubsub/docs/exactly-once-delivery)
- [Replay and purge messages with seek](https://cloud.google.com/pubsub/docs/replay-overview)
- [Pub/Sub quotas and limits](https://cloud.google.com/pubsub/quotas)
- [Pub/Sub pricing](https://cloud.google.com/pubsub/pricing)