Improving Apache Kafka Scalability and Elasticity with Tiered Storage

Streaming Audio: Apache Kafka® & Real-Time Data

Treść dostarczona przez Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka®. Cała zawartość podcastów, w tym odcinki, grafika i opisy podcastów, jest przesyłana i udostępniana bezpośrednio przez Confluent, founded by the original creators of Apache Kafka® and Founded by the original creators of Apache Kafka® lub jego partnera na platformie podcastów. Jeśli uważasz, że ktoś wykorzystuje Twoje dzieło chronione prawem autorskim bez Twojej zgody, możesz postępować zgodnie z procedurą opisaną tutaj https://pl.player.fm/legal.

2+ y ago 29:32

MP3•Źródło odcinka

What happens when you need to store more than a few petabytes of data? Rittika Adhikari (Software Engineer, Confluent) discusses how her team implemented tiered storage, a method for improving the scalability and elasticity of data storage in Apache Kafka®. She also explores the motivating factors for building it in the first place: cost, performance, and manageability.
Before Tiered Storage, there was no real way to retain Kafka data indefinitely. Because of the tight coupling between compute and storage, users were forced to use different tools to access cold and hot data. Additionally, the cost of re-replication was prohibitive because Kafka had to process large amounts of data rather than small hot sets.
As a member of the Kafka Storage Foundations team, Rittika explains to Kris Jenkins how her team initially considered a Kafka data lake but settled on a more cost-effective method – tiered storage. With tiered storage, one tier handles elasticity and throughput for long-term storage, while the other tier is dedicated to high-cost, low-latency, short-term storage. Before, re-replication impacted all brokers, slowing down performance because it required more replication cycles. By decoupling compute and storage, they now only replicate the hot set rather than weeks of data.
Ultimately, this tiered storage method broke down the barrier between compute and storage by separating data into multiple tiers across the cloud. This allowed for better scalability and elasticity that reduced operational toil.
In preparation for a broader rollout to customers who heavily rely on compacted topics, Rittika’s team will be implementing tier compaction to support tiering of compacted topics. The goal is to have the partition leader perform compaction. This will substantially reduce compaction costs (CPU/disk) because the number of replicas compacting is significantly smaller. It also protects the broker resource consumption through a new compaction algorithm and throttling.
EPISODE LINKS

Rozdziały

1. Intro (00:00:00)

2. Motivating factors behind Tiered Storage (00:02:22)

3. What is Tiered Storage? (00:04:25)

4. How does it work? (00:06:05)

5. How does it impact performance? (00:11:16)

6. Evolution of Confluent Tiered Storage (00:13:57)

7. Tiered Compaction (00:19:25)

8. Kafka Tiered Storage (00:20:17)

9. Getting started with Confluent Tiered Storage (00:23:17)

10. What is the impact for end users? (00:24:26)

11. It's a wrap! (00:27:17)

265 odcinków