Payload is text or avro? Neither, it's just bytes, how you serialise your payload data is up to you.
> Client libraries might have a hard-time of keeping up with ever-changing protocol specification. Therefore upgrades might also be challenging.
Kafka maintains backwards compatibility for old clients for a very long time. They've only just (Kafka 3.0) deprecated the protocol released in Kafka 0.8.
Choose a client library that has active maintainers, you'll be golden. If your client lib of choose
> Message size limit is 1MB.
It defaults to this, but it can be set much higher if needed.
Lastly - Kafka's not really a message queue. Apache Pulsar offers far more MQ like semantics though.
It's set to 1MB (and some additional bytes for log overhead) because it was found (by LinkedIn? I think?) to be a reasonable default for throughput. Technically you can set it up to anywhere near MAX_INT bytes, would be interesting to see what happens when you send a 1GiB message. :D You'd most likely need to tune other properties depending on how big you're going.
And message size, just to be clear, relates to a message from a producer which can contain 1 to N records, often compressed to reduce network contention. The 1MB limit applies to the bytes which are received by the broker, so you can send batches of records which are >1MB if you're compressing them.
Compression imposes some CPU cost on producers and consumers, but unless you've specifically configured a topic to use a mismatching compression algorithm (this defaults to "just accept whatever algorithm the producer" used), there's no cost on the broker, it's written as is, and then consumers pull it still compressed.
A good old fashioned database queue usually does it for me. It's easy for the DBA to monitor, Ops to run reports on, not a new program, nor another service to pay for and doesn't add additional proprietary lock-in. Plus, devs already have it setup and running.
6 comments
[ 3.2 ms ] story [ 26.6 ms ] threadThat's the complete opposite of my experience.
Message retrieval is push? Nope, definitely pull.
Payload is text or avro? Neither, it's just bytes, how you serialise your payload data is up to you.
> Client libraries might have a hard-time of keeping up with ever-changing protocol specification. Therefore upgrades might also be challenging.
Kafka maintains backwards compatibility for old clients for a very long time. They've only just (Kafka 3.0) deprecated the protocol released in Kafka 0.8.
Choose a client library that has active maintainers, you'll be golden. If your client lib of choose
> Message size limit is 1MB.
It defaults to this, but it can be set much higher if needed.
Lastly - Kafka's not really a message queue. Apache Pulsar offers far more MQ like semantics though.
It defaults to this, but it can be set much higher if needed.
What is it? I was surprised to hear it was this low.
And message size, just to be clear, relates to a message from a producer which can contain 1 to N records, often compressed to reduce network contention. The 1MB limit applies to the bytes which are received by the broker, so you can send batches of records which are >1MB if you're compressing them.
Compression imposes some CPU cost on producers and consumers, but unless you've specifically configured a topic to use a mismatching compression algorithm (this defaults to "just accept whatever algorithm the producer" used), there's no cost on the broker, it's written as is, and then consumers pull it still compressed.