How many partitions should a Kafka topic have?

At least the larger of t ÷ p and t ÷ c, where t is the topic's throughput, p what one partition takes from producers and c what one consumer reads from one partition. At 50 MB/s planned to double, with 10 MB/s a partition from producers and 5 MB/s to a consumer, that is 20. Measure p and c on your own setup: Confluent finds producers reach tens of MB a second on a partition, while a consumer's speed is its own work.

How much throughput can one Kafka partition handle?

Producers often reach tens of MB a second on one partition, depending on batching, compression, the acks setting and the copies. A consumer reads a partition only as fast as its own processing allows. On Amazon MSK Express brokers one partition takes at most 15 MB a second.

Can I add partitions to a Kafka topic later?

Yes, but with keyed messages it changes which partition a key goes to, so messages for one key can arrive out of order across the change. Confluent's advice is to set the partitions for the traffic you expect in 1 or 2 years, start with a small cluster and add brokers as traffic grows.

How many consumers can a Kafka consumer group have?

As many as it likes, but only one consumer in a group reads each partition, so a group with more consumers than the topic has partitions leaves some idle. A topic of 20 partitions keeps at most 20 consumers of a group busy.

How do I calculate Kafka disk space?

Multiply the average throughput by the retention time and the number of copies. 50 MB/s for Kafka's default 168 hours in 3 copies is 90.7 TB. Spread over 9 brokers that is 10.1 TB each, and 11.9 TB of disk to stay under 85 percent full.

How many Kafka brokers do I need?

Enough for the largest of 4 limits: disk writes (t × r ÷ brokers), network out (t × (groups + r − 1) ÷ brokers), disk space and partitions per broker, with one broker for each copy at least. AWS keeps throughput at 80 percent of a broker's limit and adds room to lose a broker or a zone. The example needs 9 MSK m7g.xlarge brokers, or 6 Express m7g.large brokers.

What replication factor should Kafka use?

3 for production. AWS warns that 1 copy can take partitions offline during a rolling update and 2 can lose data, and advises min.insync.replicas of 2 so a topic with 3 copies keeps taking writes while one is offline. Kafka's own default.replication.factor is 1, so set it. MSK Express brokers always keep 3.

How many partitions can a Kafka broker hold?

Confluent suggests 2,000 to 4,000 per broker, copies counted, so a failed broker's partitions find new leaders quickly. Amazon MSK recommends 1,000 on m7g.large and m7g.xlarge Standard brokers, 2,000 on m7g.2xlarge and 4,000 from m7g.4xlarge up; Express brokers hold more, up to 20,000 on m7g.16xlarge.

What is the difference between MSK Express and Standard brokers?

Express brokers come with storage that grows on its own and a set throughput per broker, from 15.6 MB a second in and 31.2 out on m7g.large to 500 in and 1,000 out on m7g.16xlarge, always with 3 copies in 3 zones. Standard brokers use EBS volumes you size, and their throughput is set by the volume, the instance's EBS link and its network.

How many partitions and brokers your Kafka topic needs

50,000 messages a second of 1,000 bytes is 50 MB/s. Planned for twice that, it needs 20 partitions, and kept for 7 days in 3 copies it fills 90.7 TB of disk: 9 MSK m7g.xlarge brokers.

Put in the messages a second at the busiest time, their size, how long Kafka keeps them, how many copies and how many consumer groups read them. See the partitions the producers and the consumers need, the disk the messages fill, the network each broker sends, and the brokers that carry it, with the limit that decides. Choose Amazon MSK Express or Standard broker sizes, or put in your own.

Size your topic Questions

The traffic

Messages a second, at the busiest

a second

Average message size

bytes

As sent, after the producer compresses its batches.

Keep messages for

Kafka's default, 168 hours.

Consumer groups reading it all

groups

AWS advises at least 3: with 1 a partition goes offline in a rolling update, with 2 data can be lost.

The partitions

One partition takes from producers

MB/s

Confluent finds tens of MB/s on one partition; measure yours.

One consumer reads from a partition

MB/s

How fast your consumer's own work goes. Measure it.

Traffic in 1 to 2 years

×2

Partitions are set for this, so keys need not move later.

Average traffic, as a share of the busiest

100percent

For the disk: a quiet night fills less of it.

The brokers

Standard broker size

MSK provisions storage throughput from m7g.2xlarge up. This size writes 156 MB/s, its EBS link's limit.

Just the brokers the limits need, with no room for one down for an update.

The answer

50,000 messages a second of 1,000 bytes is 50 MB/s. Create 20 partitions: the consumers, at 5 MB/s a partition, need them for 100 MB/s. Run 9 m7g.xlarge brokers; disk space sets the count.

Partitions: 20; 60 with every copy
Brokers: 9; m7g.xlarge
Kept on disk: 90.7 TB; 3 copies, 7 days

What sets the brokers

Each limit on its own, at 80 percent of what a broker sustains and disks at most 85 percent full. The largest decides.

Brokers each limit needs
Limit	A broker takes	Brokers it needs
Disk writes	125 MB/s	1.2
Network out	188 MB/s	1.07
Partitions per broker	1,000 partitions	0.06
Disk space	15 TB	6.07
One broker per copy	1 copy	3

The limits need 7. Brokers go evenly in 3 zones. So 9.

Each of the 9 brokers

Written to disk: 16.7 MB/s; Its share plus copies from the others
Network out: 22.2 MB/s; To each group and each follower
Partitions: 7; Leaders and copies
Kept: 10.1 TB; Its partitions, every copy
Disk to give it: 11.9 TB; Kept at 85 percent full

Where the partitions come from

Partitions the producers and the consumers need
Side	A partition	For 100 MB/s
Producers	10 MB/s	10
Consumers	5 MB/s	20

A consumer group runs at most 20 consumers on this topic, one for each partition; more sit idle. On average 50 MB/s arrives, 4.32 TB a day before copies. The sums assume producers spread messages evenly, every broker holds the same partitions and consumers read the newest messages. Test the cluster with your own traffic before you rely on it.

How the calculator works it out

A partition is the unit of parallelism in Kafka: producers write to partitions side by side, and within a consumer group each partition goes to one consumer. Confluent's rule is to measure what one partition takes from producers (p) and what one consumer reads from one partition (c), then create at least the larger of t ÷ p and t ÷ c partitions for a throughput t. In the example, 100 MB/s needs 10 partitions at 10 MB/s a partition for the producers and 20 at 5 MB/s for the consumers, so 20.

The partitions are set for the traffic a year or 2 ahead. Kafka sends every message with the same key to the same partition, which keeps one key's messages in order. Add partitions later and keys map to other partitions, so Confluent advises creating a few more at the start and adding brokers as traffic grows.

Each broker writes to disk what producers send it plus a copy of what the other brokers lead, for t × r ÷ brokers with r copies, and sends a copy to every consumer group and every follower, for t × (groups + r − 1) ÷ brokers. AWS's right-sizing guide keeps both at 80 percent of what a broker sustains. The disk holds the average traffic for the retention time in every copy: 90.7 TB in the example, kept under the 85 percent full at which MSK suggests an alarm.

The brokers needed are the largest of these limits, the partitions each broker should hold and one broker for each copy. In the example disk space decides: 7 brokers would hold it, and MSK places brokers evenly in 3 availability zones, so 9, each keeping 10.1 TB.

Questions

How many partitions should a Kafka topic have?: At least the larger of t ÷ p and t ÷ c, where t is the topic's throughput, p what one partition takes from producers and c what one consumer reads from one partition. At 50 MB/s planned to double, with 10 MB/s a partition from producers and 5 MB/s to a consumer, that is 20. Measure p and c on your own setup: Confluent finds producers reach tens of MB a second on a partition, while a consumer's speed is its own work.
How much throughput can one Kafka partition handle?: Producers often reach tens of MB a second on one partition, depending on batching, compression, the acks setting and the copies. A consumer reads a partition only as fast as its own processing allows. On Amazon MSK Express brokers one partition takes at most 15 MB a second.
Can I add partitions to a Kafka topic later?: Yes, but with keyed messages it changes which partition a key goes to, so messages for one key can arrive out of order across the change. Confluent's advice is to set the partitions for the traffic you expect in 1 or 2 years, start with a small cluster and add brokers as traffic grows.
How many consumers can a Kafka consumer group have?: As many as it likes, but only one consumer in a group reads each partition, so a group with more consumers than the topic has partitions leaves some idle. A topic of 20 partitions keeps at most 20 consumers of a group busy.
How do I calculate Kafka disk space?: Multiply the average throughput by the retention time and the number of copies. 50 MB/s for Kafka's default 168 hours in 3 copies is 90.7 TB. Spread over 9 brokers that is 10.1 TB each, and 11.9 TB of disk to stay under 85 percent full.
How many Kafka brokers do I need?: Enough for the largest of 4 limits: disk writes (t × r ÷ brokers), network out (t × (groups + r − 1) ÷ brokers), disk space and partitions per broker, with one broker for each copy at least. AWS keeps throughput at 80 percent of a broker's limit and adds room to lose a broker or a zone. The example needs 9 MSK m7g.xlarge brokers, or 6 Express m7g.large brokers.
What replication factor should Kafka use?: 3 for production. AWS warns that 1 copy can take partitions offline during a rolling update and 2 can lose data, and advises min.insync.replicas of 2 so a topic with 3 copies keeps taking writes while one is offline. Kafka's own default.replication.factor is 1, so set it. MSK Express brokers always keep 3.
How many partitions can a Kafka broker hold?: Confluent suggests 2,000 to 4,000 per broker, copies counted, so a failed broker's partitions find new leaders quickly. Amazon MSK recommends 1,000 on m7g.large and m7g.xlarge Standard brokers, 2,000 on m7g.2xlarge and 4,000 from m7g.4xlarge up; Express brokers hold more, up to 20,000 on m7g.16xlarge.
What is the difference between MSK Express and Standard brokers?: Express brokers come with storage that grows on its own and a set throughput per broker, from 15.6 MB a second in and 31.2 out on m7g.large to 500 in and 1,000 out on m7g.16xlarge, always with 3 copies in 3 zones. Standard brokers use EBS volumes you size, and their throughput is set by the volume, the instance's EBS link and its network.

Put this calculator on your own site

Copy the code into any web page: a blog post, a course page or your team's wiki. The calculator runs in a frame and shows the same figures as this page, so when the prices change here they change on your page too. It is free and needs no account, and nothing a reader types is sent anywhere.

The code opens the calculator with the settings you have chosen above.

The short script at the end lets the frame grow and shrink with the calculator. Where a page cannot take scripts, choose a fixed height and the frame scrolls.

<iframe src="https://tecyfy.com/embed/kafka-sizing" title="Kafka sizing calculator" width="100%" height="2800" style="border:0;display:block" loading="lazy" allow="clipboard-write"></iframe>
<p><a href="https://tecyfy.com/data-engineering/kafka-sizing">Kafka sizing calculator</a> by Tecyfy</p>
<script>addEventListener("message",function(e){if(e.origin!=="https://tecyfy.com"||!e.data||e.data.type!=="tecyfy:height")return;var h=Number(e.data.height);if(!(h>0))return;document.querySelectorAll("iframe").forEach(function(f){if(f.contentWindow===e.source)f.style.height=Math.min(h,20000)+"px"})})</script>

Sources