Jack Vanlightly

This is the first in an ongoing ad-hoc series of posts on Apache Kafka performance. I have no general direction, I'll just share interesting insights based on the performance testing I do on Apache Kafka.Recently I was curious to see if there was any gene

Yesterday I published some benchmarks of Hardwood 1.0 on my Threadripper. Someone suggested I run the One Billion Row Challenge too, to see how it does, so here it is!Gunnar Morling ran the original benchmarks on an EPYC 7502P, Zen 2, 32 cores with 128 GB

Hardwood is a minimal-dependency Java library for reading Parquet files. It currently has row-reader and columnar-reader APIs, with Parquet writing planned for the future.Gunnar Morling, Hardwood's author, published some initial benchmarks in the v1.0 ann

In this post we're going to see how share.acquire.mode=record_limit combined with:fewer consumers than partitionsand various cases of "partition skew"…can result in subpar performance with share groups. I stumbled on these issues when running large sets o

The lines between transactional systems, analytical systems, hybrid systems, and shared storage architectures are getting blurry. This post proposes a small taxonomy for describing the different ways systems, workloads, storage tiers, visibility, and dura

"Perfection is finally attained not when there is no longer anything to add, but when there is no longer anything to take away." - Antoine de Saint-ExupéryAI gives us an unprecedented ability to add. The danger is that we begin to mistake accumulation for

All tests were executed against Kafka 4.3.0 using Dimster. In the last post Broker-Visible vs Client-Local Parallelism we looked at two ways of scaling Kafka consumption. The final unit of parallelism can be visible to the broker, as consumers, or it can

This post is a little side-quest from my "Kafka Share Groups and Parallelizing Consumption" series.My "Kafka Share Groups and Parallelizing Consumption" series (part 1, part 2) has been laser focused on how different configurations and behaviors affect pa

All tests were executed against Kafka 4.3.0 using Dimster. In the last post we used simulated consumer processing time to reveal how important it is to set an appropriate value for max.poll.records to ensure the consumer parallelism that we expect. With a

All tests were executed against Kafka 4.2.0 using Dimster (and also validated against 4.3.0). In the last post we measured the overhead that the mechanics of share groups adds, and saw that it is pretty small. Likewise we saw that raw throughput was also