Kafka max message size: the 2 limits that reject a record

Kafka's largest message is about 1 MB by default, and 2 settings decide whether a bigger record gets through: the producer's max.request.size, checked on the record before compression, and the broker's message.max.bytes (or the topic's max.message.bytes), checked on the batch after compression. Raise the one that rejects your record, and only as far as you need.

Older answers list 4 settings that "must agree", adding the consumer's and the replicas' fetch sizes. Those 2 no longer block anything: on clients from Kafka 0.10.2 on, a consumer or follower that asks for less than the first batch waiting for it still gets that batch, so it keeps making progress.

The settings and their defaults

From the Kafka 4.3 configuration reference:

2 more producer settings can bite on very large records: buffer.memory (32 MiB), the producer's memory for records waiting to be sent, and the broker's socket.request.max.bytes (100 MiB), the largest request it reads off the network.

Which error you see, and what it means

The producer throws RecordTooLargeException in both cases, with different words:

• "The message is 3145800 bytes when serialized which is larger than 1048576, which is the value of the max.request.size configuration." The producer refused it and nothing was sent. Raise max.request.size on the producer.
• "The request included a message larger than the max message size the server will accept." The broker refused the batch. Raise max.message.bytes on the topic.

Compression changes which limit you hit

The producer compresses whole batches, so its own limit sees the record at full size, while the broker sees it compressed. A worked example: an order event with an embedded document, 3 MB of JSON, which compresses with zstd to 600 KB. Your own ratio will differ, so measure it.

So with compression on, only the producer setting needs to change for this record. Without it, both do. Compression is off by default (compression.type=none).

Try itWhere a large Kafka record stops

Record size

3.0 MiB

Compression ratio (1 = off)

5×

Producer max.request.size (MiB)

1 MiB

Topic max.message.bytes (MiB)

1 MiB

1. Producer: max.request.size3.0 MiB before compression vs 1.0 MiBRejected
2. Broker: max.message.bytes614 KiB after compression vs 1.0 MiBNever reached
3. Consumers and replicasAn oversized first batch is still returnedNever reached

Raise the limit on one topic, not the cluster

Change the topic that needs it, so a bad producer on another topic still gets stopped at 1 MB:

kafka-configs.sh --bootstrap-server broker:9092 \
  --alter --entity-type topics --entity-name order-documents \
  --add-config max.message.bytes=4194304

And the producer, here in Python with confluent-kafka:

from confluent_kafka import Producer

producer = Producer({
    "bootstrap.servers": "broker:9092",
    "compression.type": "zstd",
    "message.max.bytes": 4194304,   # librdkafka's name for the producer's request limit
})

The Java client calls the same producer limit max.request.size:

props.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, 4194304);
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "zstd");

Clients built on librdkafka (Python, Go, .NET) use message.max.bytes for the producer's limit, with a default of 1,000,000 bytes, and the name is easy to confuse with the broker setting. Check your client's own reference.

Consumers need no change to read the larger records. Raising max.partition.fetch.bytes to at least the largest record still helps throughput, because a consumer that gets one oversized batch per request fetches less of everything else.

On Amazon MSK, set message.max.bytes in a custom cluster configuration, or max.message.bytes on the topic as above.

The better answer for big payloads

A raised limit makes large records possible, not cheap. Every record is stored once per replica for the topic's whole retention, a big record holds broker and client memory while it moves, and one slow consumer of large records holds up its partition. For files, images and documents, the usual pattern is to put the payload in object storage and send a small message that points to it:

{"order_id": 88231, "document": "s3://orders-docs/88231.pdf", "sha256": "9f2c...", "bytes": 3145800}

The consumer reads the pointer and fetches the file. The topic stays small and fast, and the file is stored once.

What to do

1. Read the error's wording to see whether the producer or the broker refused the record.
2. Turn on compression.type=zstd or lz4 first; it may be enough for the broker.
3. Raise max.request.size on the producer and max.message.bytes on that topic only, to the largest record you expect plus room.
4. For anything that is really a file, send a pointer instead.

Record size also changes how many partitions and brokers a topic needs. Our Kafka sizing calculator takes your traffic in MB per second and gives partitions, disk and brokers, and how many Kafka partitions explains the rule behind it. If you are still choosing a platform, see Kafka vs Kinesis vs Pub/Sub. Kafka itself is a StreamingStreamingGetting a model's answer in small pieces as it is written, instead of all at once at the end, so people see text start sooner and long answers do not hit a timeout. log.