naheedmk · June 21, 2019 04:12
diff --git a/kafka performance benchmark b/kafka performance benchmark
 Kafka benchmarks summary prepared from:
 https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
 https://gist.github.com/jkreps/c7ddb4041ef62a900e6c

 3 nodes, no replication, topic with 6 partitions, small messages 100bytes, single producer
 8.2 lakh/sec
 78 mb/s
 We don't need this one.

 3 nodes, 3 replica, small messages 100bytes, single ack, batch size = 8196
 7.8 lakh/sec
 75 mb/s

 3 nodes, 3 replica, small messages 100bytes, majority ack, batch size is 64000 i.e produer is using 64k*6*100 = 36MB buffer.
 4.2 lakh/sec
 40 mb/sec
 This is the configuration we want if we don't want to loose any data. So I am only interested in this.
 We use RAID 1, 7200 rpm hdd disk which has around 100 mb/sec sequential write.
 But in RAID 1, write performance degrades.
 So achieving 40-50 mb/sec is good enough.
 If node is using 1Gbps network, 125 MB/s is the maximum transfer rate from producer to single broker node.
 We can do this load test with batch size of 64, 124, 256, 512, 1024 and see the results.
 And we can do this test for different message sizes 100b, 200b, 500b, 1kb along with different batch sizes.


 3 nodes, 3 replica, small messages 100bytes, single ack, 3 producer
 20.2 lakh/sec
 193 mb/s


 3x replicated data, single thread consumer, 6 partitions
 9 lakh/s
 89 mb/s

 3x replicated data, 3 thread consumer, 6 partitions
 26  lakh/s
 249 mb/s

 3x replication, single ack, one producer and one consumer
 7.9 lakh/sec
 75 mb/sec

 end to end latency
 2 ms (median)
 3 ms (99th percentile)
 14 ms (99.9th percentile)


 In kafka performance doesn't degrades with data size.

 With higher message size mess/s will decrease but mb/sec may improve. With small message size batch size needs to be higher and with bigger message size batch size needs to be lower.
 https://content.linkedin.com/content/dam/engineering/en-us/blog/migrated/size_vs_record_throughput.png
 https://content.linkedin.com/content/dam/engineering/en-us/blog/migrated/size_vs_mb_throughput.png

 We are using dual quad-core Intel Xeon machines with 24GB of memory.
 You need sufficient memory to buffer active readers and writers. You can do a back-of-the-envelope estimate of memory needs by assuming you want to be able to buffer for 30 seconds and compute your memory need as write_throughput*30.
	Kafka benchmarks summary prepared from:
	https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
	https://gist.github.com/jkreps/c7ddb4041ef62a900e6c

	3 nodes, no replication, topic with 6 partitions, small messages 100bytes, single producer
	8.2 lakh/sec
	78 mb/s
	We don't need this one.

	3 nodes, 3 replica, small messages 100bytes, single ack, batch size = 8196
	7.8 lakh/sec
	75 mb/s

	3 nodes, 3 replica, small messages 100bytes, majority ack, batch size is 64000 i.e produer is using 64k6100 = 36MB buffer.
	4.2 lakh/sec
	40 mb/sec
	This is the configuration we want if we don't want to loose any data. So I am only interested in this.
	We use RAID 1, 7200 rpm hdd disk which has around 100 mb/sec sequential write.
	But in RAID 1, write performance degrades.
	So achieving 40-50 mb/sec is good enough.
	If node is using 1Gbps network, 125 MB/s is the maximum transfer rate from producer to single broker node.
	We can do this load test with batch size of 64, 124, 256, 512, 1024 and see the results.
	And we can do this test for different message sizes 100b, 200b, 500b, 1kb along with different batch sizes.


	3 nodes, 3 replica, small messages 100bytes, single ack, 3 producer
	20.2 lakh/sec
	193 mb/s


	3x replicated data, single thread consumer, 6 partitions
	9 lakh/s
	89 mb/s

	3x replicated data, 3 thread consumer, 6 partitions
	26 lakh/s
	249 mb/s

	3x replication, single ack, one producer and one consumer
	7.9 lakh/sec
	75 mb/sec

	end to end latency
	2 ms (median)
	3 ms (99th percentile)
	14 ms (99.9th percentile)


	In kafka performance doesn't degrades with data size.

	With higher message size mess/s will decrease but mb/sec may improve. With small message size batch size needs to be higher and with bigger message size batch size needs to be lower.
	https://content.linkedin.com/content/dam/engineering/en-us/blog/migrated/size_vs_record_throughput.png
	https://content.linkedin.com/content/dam/engineering/en-us/blog/migrated/size_vs_mb_throughput.png

	We are using dual quad-core Intel Xeon machines with 24GB of memory.
	You need sufficient memory to buffer active readers and writers. You can do a back-of-the-envelope estimate of memory needs by assuming you want to be able to buffer for 30 seconds and compute your memory need as write_throughput*30.
No results found