Kafka scenario questions:

1. What is consumer group in kafka?

Consumer Groups: Consumers subscribing to one or more topics can be organized
into consumer groups. Each message in a topic partition is delivered to only one
consumer within the consumer group. This enables parallel processing and load
distribution.
2. What is offset?
Offsets: Each message within a partition is assigned a unique offset. Offsets are used
to track the progress of consumers. Consumers can specify an offset to start
consuming messages from a particular point in the topic.
3. Use cases of Kafka?
Use Cases: Kafka topics are versatile and can be used for various use cases, including
event streaming, log aggregation, real-time analytics, data integration, and more.
2. if you have any issue with message in kafka?
Message Not Being Consumed:
Cause: Consumer code might not be correctly subscribing to the appropriate topic or
partition. It could also be due to a consumer group misconfiguration.
Solution: Verify that your consumer code is correctly subscribing to the intended
topic and partition. Check consumer group configurations to ensure they are set up
correctly.
Message Processing Errors:

Cause: Messages might contain unexpected data or errors might occur during
message processing.
Solution: Implement proper error handling in your consumer code. Log error details
and potentially move problematic messages to an error queue
Kafka Producer Failures:

Cause: Errors in the producer code, network issues, or Kafka broker failures can
result in messages not being sent.
Solution: Properly handle exceptions in your producer code, implement retries, and
ensure your Kafka brokers are healthy.
Kafka Broker Failures:

Cause: If Kafka brokers are down, messages cannot be produced or consumed.

Solution: Monitor Kafka cluster health and set up proper replication for fault
tolerance.
1. How to implement Kafka retry mechanism?
we can use the RetryableTopic annotation to configure a more robust
strategy to handle failed messages. For example, we can send the
failed message to the Dead Letter Queue, limit the number of
retries, define timeout, exclude fatal exception reprocessing, etc.

A blocking retry enables the consumer to attempt consuming a

message again if the initial attempt fails due to a temporary error.
The consumer waits a certain amount of time, known as the retry
backoff period, before trying to consume the message again.

https://www.baeldung.com/spring-retry-kafka-
consumer#:~:text=To%20implement%20a%20retry
%20logic,unacknowledged%20messages%20to%20another
%20consumer

https://betterprogramming.pub/spring-boot-kafka-non-blocking-retries-a-hands-on-
tutorial-a0c425acc3dd

2. acknowledgment in kafka how it works?

In Apache Kafka, acknowledgment (ack) plays a crucial role in ensuring the reliability
of message delivery. Kafka uses a system of acknowledgments to manage the
process of confirming that messages have been successfully sent and processed. This
mechanism helps prevent data loss and ensures that messages are reliably
transferred between producers and consumers.
Producers and Message Acknowledgment:

When a producer sends a message to a Kafka topic, it can specify an

acknowledgment requirement. There are three common acknowledgment modes:

acks=0: The producer does not wait for an acknowledgment from the broker after
sending the message. This offers the least reliability, as the producer assumes the
message was successfully sent even if it was not.

acks=1: The producer waits for an acknowledgment from the leader replica of the
partition where the message was sent. Once the leader receives the message and
persists it to disk, it sends an acknowledgment back to the producer. This mode
offers moderate reliability.

acks=all (or acks=-1): The producer waits for acknowledgments from all in-sync
replicas (ISRs) of the partition. This mode provides the highest level of reliability, as
the message is considered successfully sent only when it has been replicated to all
ISRs.

The acknowledgment is sent back to the producer in the form of a future response.
The producer can then decide whether to send more messages or wait for additional
acknowledgments.

Consumers and Message Acknowledgment:

In Kafka, consumers are responsible for acknowledging the consumption of

messages from a topic. Once a consumer processes a message and is confident that
it has been successfully handled, it should acknowledge the message to Kafka. This
acknowledges to the broker that the message has been consumed and can be safely
marked as processed.
Kafka offers two acknowledgment modes for consumers:

enable.auto.commit=true: In this mode, Kafka automatically manages the

acknowledgment of consumed messages at predefined intervals. This approach is
convenient but can lead to a slight risk of duplicate processing if a consumer crashes
before acknowledging the message.

enable.auto.commit=false: In this mode, the consumer manually controls when

acknowledgments are sent. Consumers can acknowledge messages individually or in
batches after successful processing. This mode provides more control over
acknowledgment timing and reduces the risk of duplicate processing.

3. Offsets and Acknowledgment in Kafka?

Kafka uses the concept of "offsets" to track the progress of consumers within each
partition. When a consumer acknowledges a message, it essentially acknowledges
the processing of messages up to a certain offset. This offset is then used to
determine where the consumer should resume reading from after a restart or
failure.

Acknowledgment in Kafka is essential for maintaining data integrity and ensuring

reliable message processing across producers and consumers. It allows Kafka to
provide strong guarantees of message delivery and consumption while
accommodating various levels of reliability based on your application's
requirements.

4. While processing the messages at one offset if system goes down then what
happens?
The message processing will resume from particular offset where it left processing.
5. Error handling for producers in kafka?
Error handling for producers in Kafka involves implementing strategies to handle
various types of errors that can occur while producing messages to Kafka topics.
These errors can include network issues, broker unavailability, message serialization
failures, and more.
Retry Mechanisms:
Implement retry mechanisms to handle transient errors. When a send operation
fails, you can retry sending the message after a certain interval.
Error Logging:
Log detailed error messages and stack traces when a send operation fails. This helps
in diagnosing the root cause of the error and makes troubleshooting easier.
Handle Failed Sends:
If a send operation consistently fails after multiple retries, you need to decide how to
handle such cases. You might choose to log the message, alert a monitoring system,
or implement a dead-letter queue for further analysis

6. How to acknowledge errors in Kafka?

Acknowledge Errors:
Kafka producers provide acknowledgment mechanisms to confirm whether a
message was successfully sent to the broker. Pay attention to acknowledgment
settings (acks) and handle different acknowledgment scenarios:

acks=0: Messages are sent without waiting for acknowledgment. No guarantee of

delivery, and no error is reported even if the send fails.
acks=1: Messages are sent and acknowledged by the leader. A failure in sending is
reported with an exception.
acks=all: Messages are sent and acknowledged by all in-sync replicas. Provides
stronger durability guarantees.
Implement Error Handlers:
Define custom error handlers that encapsulate the logic for handling specific types of
errors. This can help keep your code clean and maintainable.
7. Error handling for consumers in kafka?
https://www.geeksforgeeks.org/exception-handling-in-apache-kafka/
Error handling for Kafka consumers is essential to ensure that your application can
gracefully handle various types of errors that may arise while consuming messages
from Kafka topics.
Exception Handling:
Use try-catch blocks to handle exceptions that can occur during message processing.
This includes network errors, deserialization exceptions, and business logic errors.

Log Detailed Errors:

Log error messages and stack traces for exceptions that occur during message
processing. Include contextual information such as the topic, partition, and offset of
the message.
Error Handling in Transactions:
If you're using transactions with Kafka consumers, ensure that you handle exceptions
and transaction rollbacks appropriately to maintain data consistency.

8. why is Kafka so popular? And what makes it such a popular

choice for companies?
Scalability: The scalability of a system is determined by how well it can maintain its
performance when exposed to changes in application and processing demands.
Apache Kafka has a distributed architecture capable of handling incoming messages
with higher volume and velocity. As a result, Kafka is highly scalable without any
downtime impact.

High Throughput: Apache Kafka is able to handle thousands of messages per second.
Messages coming in at a high volume or a high velocity or both will not affect the
performance of Kafka.

Low Latency: Latency refers to the amount of time taken for a system to process a
single event. Kafka offers a very low latency, which is as low as ten milliseconds.

Fault Tolerance: By using replication, Kafka can handle failures at nodes in a cluster
without any data loss. Running processes, too, can remain undisturbed. The
replication factor determines the number of replicas for a partition. For a replication
factor of ‘n,’ Kafka guarantees a fault tolerance for up to n-1 servers in the Kafka
cluster.
Reliability: Apache Kafka is a distributed platform with very high fault tolerance,
making it a very reliable system to use.

Durability: Data present on the Kafka cluster is allowed to remain persistent more on
the cluster than on the disk. This ensures that Kafka’s data remains durable.

9. What is a Kafka Cluster?

A Kafka cluster is a distributed system composed of multiple Kafka brokers working
together to handle the storage and processing of real-time streaming data. It
provides fault tolerance, scalability, and high availability for efficient data streaming
and messaging in large-scale applications.

10. What is topic?

Topics serve as channels or categories to which producers publish messages, and
from which consumers subscribe to receive messages.
Kafka follows a publish-subscribe model. Producers publish (send) messages to one
or more topics, and consumers subscribe to (consume) messages from one or more
topics.
11. What is partition?
Partitions: Each topic can be divided into partitions. Partitions are the basic unit of
parallelism and scalability in Kafka. They allow data to be spread across multiple
brokers and processed concurrently.
12. What is replication?
Replication: Kafka provides data replication for fault tolerance. Each partition can
have multiple replicas, with one replica designated as the leader and the others as
followers. The leader handles read and write requests, while followers replicate data
for backup.
13. What is retention policy in kafka?
Retention Policy: Topics have a retention policy that determines how long messages
are retained in a topic. Messages can be retained based on time or size. This
retention policy helps in managing data storage.

14. Kafka Broker?

A Kafka broker is a core component of the Apache Kafka platform. It is responsible
for handling and managing the storage, distribution, and processing of messages or
records in Kafka.
Server Instance: A Kafka broker is an instance of the Kafka server that runs on a
physical machine or a virtual server. Multiple brokers are typically deployed in a
Kafka cluster to provide scalability and high availability.
Storage: Brokers store messages (also called records) in topics. Each broker stores
one or more partitions of each topic. Partitions are the basic unit of storage and
parallelism in Kafka.
Partition Leader and Followers: For each partition, one broker is designated as the
leader and the others as followers. The leader handles all read and write operations
for the partition, while followers replicate the data for fault tolerance.

15. What is Kafka cluster?

In Apache Kafka, a cluster refers to a group of Kafka brokers that work together to
provide a distributed and fault-tolerant messaging system. A Kafka cluster is a
fundamental building block of the Kafka architecture and is designed to handle the
storage, distribution, and processing of messages across multiple machines or
servers
Multiple Brokers: A Kafka cluster consists of multiple Kafka brokers. Each broker is an
instance of the Kafka server and runs on a separate machine or server.
Distributed and Fault-Tolerant: Kafka clusters are designed for both distribution and
fault tolerance. Data is distributed across multiple brokers, and each partition of a
topic is replicated across multiple brokers for fault tolerance.
16. consumer rebalanacing mechanism kafka?
Consumer rebalancing is a mechanism in Apache Kafka that ensures an even
distribution of partitions across consumer instances within a consumer group. When
consumer instances are added, removed, or fail, Kafka's rebalancing mechanism
redistributes the partitions to maintain load balance and allow each consumer
instance to process a roughly equal number of partitions.
17. Partitions and Consumers?
Kafka topics are divided into partitions. Each partition can be consumed by only one
consumer instance at a time. Multiple consumer instances within the same group
can consume different partitions of the same topic.
18. what happens when a consumer fails in kafka?
When a consumer fails in Apache Kafka, it means that the consumer instance
becomes unresponsive or crashes. Kafka is designed to handle consumer failures
gracefully while minimizing data loss and ensuring that data processing continues
smoothly
19. Heartbeat in Kafka?
In Apache Kafka, a heartbeat is a periodic signal sent by a Kafka consumer to the
Kafka broker to indicate that the consumer is alive and actively processing messages.
Heartbeats play a crucial role in maintaining the health and status of consumer
instances within a consumer group.
Heartbeat Mechanism: Each consumer instance sends heartbeats to the group
coordinator at regular intervals. The interval is defined by the heartbeat.interval.ms
configuration property, which specifies how often the consumer should send
heartbeats.
The heartbeat serves as a liveness check for the consumer. It lets the group
coordinator know that the consumer is still operational and actively processing
messages. If the coordinator doesn't receive a heartbeat within a certain time frame
(configured by the session.timeout.ms property), it assumes that the consumer has
failed.
20. What triggers a consumer group rebalance in kafka?
If a consumer's heartbeat is not received by the coordinator within the session
timeout period, the coordinator considers the consumer as failed. This triggers a
group rebalance, during which partitions are reassigned to other active consumers in
the group.
21. What happens when a consumer is down in Kafka?
When a consumer goes down or becomes unresponsive in Apache Kafka, it triggers a
sequence of actions and mechanisms to ensure that message processing continues
smoothly and that data integrity is maintained.
Heartbeat Missed: Kafka consumers send periodic heartbeats to the group
coordinator to indicate their liveness. If a consumer goes down or becomes
unresponsive, it stops sending heartbeats.
22. What is rebalancing in kafka?
A rebalance is a process in which the partitions that were being processed by the
failed consumer are redistributed among the remaining active consumers within the
same consumer group.

Partition Reassignment: Partitions that were previously assigned to the failed

consumer are reassigned to other consumers in the group. This ensures that no
partitions remain unprocessed and that the workload is distributed evenly.

New Leader Election: If the failed consumer was the leader for any partitions, a new
leader for those partitions is elected among the followers. This ensures that data
processing can continue without interruption.
23. What is parallel processing in Kafka?
Parallel Processing:
Kafka's partition assignment ensures that each partition is assigned to only one
consumer instance within a consumer group. This allows multiple consumer
instances to process different partitions concurrently, achieving parallel processing.
Scaling: If you want to further increase parallelism, you can add more consumer
instances to the same consumer group. Kafka will automatically reassign partitions
to the new instances during rebalancing.
24. what is exactly once in Kafka?
Message will be delivered gauranteed and without duplicates, even in failures.

Transactional Messaging:
Kafka introduced support for transactions, allowing producers to send messages to
multiple partitions within a single atomic transaction. This ensures that either all
messages from the transaction are written to partitions, or none are written.

While "Exactly Once" semantics is a powerful guarantee, it comes with some

performance overhead due to the additional coordination and transaction
management. Depending on your use case, you might choose to use "Exactly Once"
semantics when data integrity and duplication prevention are critical

25. What is atleast once in Kafka?

This means that a message may be delivered to a consumer more than once but will
not be lost.
26. What is at most once in Kafka?
Message wont be duplicated, can be lost.
27. Idempotent consumers?
Configure your consumer instances to be idempotent. This means that even if a
consumer processes the same message multiple times, it has the same effect as
processing it once. Implementing idempotent processing helps prevent issues when
messages are consumed more than once.
Idempotent Processing Logic:
Design your consumer's processing logic to be idempotent. This means that
processing the same message multiple times should have the same outcome as
processing it once. This is essential for preventing unintended side effects due to
duplicates.
28. Idempotent producers?
Configure Idempotence:
To enable idempotence in a Kafka producer, set the enable.idempotence
configuration property to true in the producer's configuration. This property ensures
that the producer sends messages in a way that prevents duplicates.
Acknowledge Configuration:
Configure the acks configuration property based on your requirements. For
idempotent producers, using acks=all ensures that the producer waits for
acknowledgment from all replicas before considering a message as successfully sent.

29. When you have a consumer, and a given Kafka topic and an application that is
going to consume off of that topic, and you deploy your service into multiple
instances across different regions, how do you get them to not consume a message
more than once?
When you have a Kafka consumer deployed across multiple instances in different
regions and you want to ensure that messages are not consumed more than once
(achieving "Exactly Once" or "At Least Once" semantics)
Organize your consumer instances into a single consumer group. Kafka's consumer
group mechanism ensures that each partition is consumed by only one consumer
instance at a time. This avoids duplication of message consumption.
Auto Commit Offsets:
Use manual offset management and disable auto-commit of offsets. This gives you
more control over when offsets are committed, ensuring that offsets are committed
only after messages are successfully processed.

30. Auto vs manual offset commit?

Auto-Commit vs. Manual Commit:
Kafka consumers can be configured to commit offsets automatically or manually.
With auto-commit, the consumer periodically commits offsets in the background.
With manual commit, the consumer explicitly decides when to commit offsets.

31. How to handle If a consumer fails before committing an offset?

If a consumer fails before committing an offset, Kafka will reassign the uncommitted
partitions to other consumers during rebalancing. When the consumer restarts, it
starts processing from the last committed offset.
32. .Suppose, by design, you have a pipeline where consumers take 10 mins to
process information. How would I go about that? What are the issues?
If you have a Kafka consumer pipeline where each consumer takes a significant
amount of time, such as 10 minutes, to process information from messages, there
are several considerations and potential issues to be aware of.
Designing a data pipeline where consumers take 10 minutes to process information
is certainly feasible, but it comes with some considerations and potential issues to
address. Here's how you can approach it:

1. Parallel Processing: To handle the processing time of 10 minutes per item, you'll
likely need to parallelize your consumer instances. You can have multiple consumer
instances running in your consumer group to process messages concurrently. This
approach can help you achieve better throughput.

2. Scaling: Depending on the volume of incoming data and the complexity of

processing, you may need to scale your consumer group dynamically. Consider using
auto-scaling mechanisms to add or remove consumer instances as needed to
maintain processing efficiency.

3. Load Balancing: Implement load balancing to distribute messages evenly among

consumer instances. Kafka's partitioning mechanism helps with this. Ensure that the
partitions are distributed across consumers in a balanced way to make the best use
of resources.

33. Suppose, by design, you have a pipeline where consumers take 10 mins to
process information. How would I go about that? What are the issues?

Configure the consumer group's rebalancing and session timeouts, as well as

heartbeat intervals, carefully. Long processing times can increase the likelihood of
false failure detections if heartbeats are not sent frequently enough.

Ensure that offset management is handled properly. Commit offsets only after
processing is successfully completed. If offsets are committed too early, you might
process messages multiple times if a consumer crashes during processing.

34. Order of message processing in Kafka?

Kafka guarantees message order within a partition.
35. What if the consumer fails to consume the messages?
Offset tracking:
Kafka consumers maintain an offset to track the last message they successfully
processed. When a consumer fails, it can use this offset to resume processing from
where it left off once it recovers.
Consumer Group and Load Balancing:
If you're using consumer groups (multiple consumers within a group consuming from
the same topic), Kafka will automatically reassign partitions to other consumers in
the group if one consumer fails. This helps distribute the workload and ensures that
messages are still being processed.
Dead letter queue.
36. How to ensure of the messages are not lost?
Replication Factor:
Configure a replication factor greater than 1 for your Kafka topics. This means that
each partition's data is replicated across multiple brokers. If a broker fails, the data is
still available on other replicas.
Min In-Sync Replicas (ISR):
Set the min.insync.replicas configuration. This specifies the minimum number of in-
sync replicas that must acknowledge the message before it's considered committed.
This helps ensure data durability.

acks Configuration:
Set the acks configuration when producing messages. Values such as all or -1 ensure
that the leader and all replicas have acknowledged the message before the producer
receives an acknowledgment.

34. what are the things to consider when consumer consuming

messages from kafka ?
Consumer Group: Kafka consumers typically belong to consumer groups. Each group
is responsible for processing a subset of the topics' partitions.
Offset Management: Kafka keeps track of the consumer's progress in the form of
offsets.
Error Handling: Implement robust error handling and retry mechanisms.
Serialization and Deserialization: Ensure that your messages are serialized and
deserialized efficiently

35. when does kafka perform partition rebalancing?

In Apache Kafka, partition rebalancing occurs when there are changes to the
consumer group's membership. Specifically, it happens under the following
circumstances:

Consumer Group Startup: When a consumer group starts up, either because new
consumers are joining or because it's the first time the group is being created, Kafka
performs partition assignment to distribute the partitions among the consumers.
This is the initial rebalance.

Consumer Joining or Leaving: Whenever a consumer joins or leaves a consumer

group, a rebalance is triggered. This can happen due to various reasons, such as a
new consumer instance starting, an existing instance crashing, or a consumer
manually leaving the group.

Partitions Added or Removed: If the number of partitions in a topic changes (e.g.,

partitions are added or removed), Kafka will trigger a rebalance to ensure that the
new partitions are evenly distributed among the consumers.

Q: what happens if retention period is over in kafka and message is still not
processed?
Consumer Lag: If a message has not been consumed or processed by any consumer
within the retention period, it creates a consumer lag. Consumer lag occurs when
consumers fall behind the latest messages in a Kafka topic. This lag represents the
gap between the latest offset (position) in the topic and the offset up to which the
consumer has processed messages.
This situation is one reason why replication is crucial in Kafka clusters to ensure data
durability and prevent data loss in case of broker failures.
Q: in kafka how can you ensure that the message is consumed exactly once?
Kafka provides a configuration called enable.idempotence for consumers. Setting this
property to true ensures that Kafka consumers are idempotent. Idempotent
consumers prevent duplicate processing of records even if the consumer's
acknowledgment (commit) fails and it has to retry.

Properties props = new Properties();

props.put("bootstrap.servers", "your_kafka_bootstrap_servers");
props.put("key.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
props.put("enable.idempotence", "true");

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);