DeBeAndo

Here is what 𝗔𝘁-𝗟𝗲𝗮𝘀𝘁/𝗠𝗼𝘀𝘁 𝗢𝗻𝗰𝗲 𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴 in 𝗞𝗮𝗳𝗸𝗮 is. Kafka is an extremely important Distributed Messaging System to understand as other similar systems are borrowing fundamental concepts from it, last time we covered Reading Data. Today we talk about At-Least Once and At-Most Once Processing. Why do you need to understand the concept and how it might impact your Stream Processing Application? 𝗦𝗼𝗺𝗲 𝗿𝗲𝗳𝗿𝗲𝘀𝗵𝗲𝗿𝘀: ➡️ Data is stored in Topics that can be compared to tables in Databases. ➡️ Messages in the Topics are called Records. ➡️ Topics are composed of Partitions. ➡️ Each Record has an offset assigned to it which denotes its place in the Partition. ➡️ Clients reading Data from Kafka are called Consumers. ➡️ Consumer Group is a logical collection of clients that read data from a Kafka Topic and share the state. 𝗥𝗲𝗯𝗮𝗹𝗮𝗻𝗰𝗲: ➡️ Consumers commit their state into a Kafka Topic named __consumer_offsets. ➡️ State is denoted by an offset that a Consumer is currently reading at in a given Partition. ➡️ There are different strategies when a commit happens. E.g. every n seconds, on-demand. ➡️ If a Consumer fails or the Consumer Group is restarted, a rebalance will be initiated. ➡️ New Consumers will launch and take the place of the old ones. ➡️ Consumers start Reading Data from the last committed state per Partition. There are different scenarios that can unfold when a Consumer restarts. Let’s take an easy example where you read data from Kafka, buffer it in memory and write a batch into an Object Storage. 𝗔𝘁-𝗟𝗲𝗮𝘀𝘁 𝗢𝗻𝗰𝗲 𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴: ➡️ You set an offset commit strategy to on-demand. 1️⃣ Read the entire batch. 2️⃣ Serialize and write it into object storage. 3️⃣ Wait for a successful write. 4️⃣ Commit offsets. ❗️If Consumer fails before committing offsets after a successful Write new consumer will start from the last committed offset and writes duplicate Data to Object Storage. ✅ This is usually ok for Batch Systems as deduplication is performed during ETL. 𝗔𝘁-𝗠𝗼𝘀𝘁 𝗢𝗻𝗰𝗲 𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴: ➡️ You commit offsets in small time intervals without waiting for a successful write to Object Storage. ❗️If Consumer fails before a successful Write into an Object Storage new consumer will start from the last committed offset and it will result in some Data not being processed. ✅ This is usually ok for Logging Systems as losing small amounts of data does not affect the System in Critical ways. Have you run into any problems with described behaviours? Let me know in the comment section 👇 -------- Follow me to upskill in #MLOps, #MachineLearning, #DataEngineering, #DataScience and overall #Data space. Also hit 🔔to stay notified about new content. 𝗗𝗼𝗻’𝘁 𝗳𝗼𝗿𝗴𝗲𝘁 𝘁𝗼 𝗹𝗶𝗸𝗲 💙, 𝘀𝗵𝗮𝗿𝗲 𝗮𝗻𝗱 𝗰𝗼𝗺𝗺𝗲𝗻𝘁! Join a growing community of Data Professionals by subscribing to my 𝗡𝗲𝘄𝘀𝗹𝗲𝘁𝘁𝗲𝗿.