Reinventing Kafka on object storage (Interview)

2224.521 - 2247.147 Ryan Worrell

There are a lot of different ways to decide which shard you're going to write the data to. But let's just say, for now, you do it by hashing the key of the message and then routing it to the shard based on the hash of that key. So if you have the record with the same key, you'll end up going to that same broker every time that owns that partition. So that's how it works in the open source product.