Top 20 Kafka Interview Questions & Answers

1. What is Kafka?

Kafka is an open-source message broker project created by the Apache Software Foundation written in Scala, where the design is heavily inclined by transaction logs. It is basically a distributed publish-subscribe messaging system.

2. List the various components in Kafka.

The major components of Kafka are:

• Topic – a stream of messages belonging to the same type
• Consumer – that subscribes to various topics and pulls data from the brokers.
• Brokers – a set of servers where the publishes messages are stored
• Producer – that can publish messages to a topic

3. Explain the role of the offset.

Messages contained in the panels are allotted a unique ID number that is known as offset. It is used to identify every message within the panel.

4. What is a Consumer Group?

Consumer Group is a concept exclusive to Kafka.  Every Kafka consumer group consists of one or more consumers that jointly consume a set of subscribed topics.

5. What is the role of the Zookeeper?

The role of Zookeeper is to store offsets of messages consumed for a particular topic and partition by a specific Consumer Group.

6. Is it possible to use Kafka without Zookeeper?

No, it is not possible to avoid Zookeeper and connect directly to the Kafka server. If, in a case, Zookeeper is down, you cannot service any client request.

7. Explain the concept of Leader and Follower.

Every partition in Kafka has one server which plays the role of a Leader, and none or more servers that act as Followers. The Leader performs the task of all read and write requests for the partition, while the role of the Followers is to passively replicate the leader. In the event of the Leader failing, one of the Followers will take on the role of the Leader.

8. What roles do Replicas and the ISR play?

• Replicas are essentially a list of nodes that replicate the log for a particular partition irrespective of whether they play the role of the Leader.
• ISR stands for In-Sync Replicas. It is essentially a set of message replicas that are synced to the leaders.

9. Why are Replications critical in Kafka?

Replication confirms that published messages are not vanished and can be consumed in the event of any machine error, program error or frequent software upgrades.

10. If a Replica stays out of the ISR for a long time, what does it signify?

It signifies that the Follower is no able to fetch data as fast as data accumulated by the Leader.

11. What is the process for starting a Kafka server?

Since Kafka uses Zookeeper, it is important to initialize the Zookeeper server, and then fire up the Kafka server.

• To start the Zookeeper server: > bin/zookeeper-server-start.sh config/zookeeper.properties
• Next, to start the Kafka server: > bin/kafka-server-start.sh config/server.properties

12. How do you define a Partitioning Key?

Within the Producer, Partitioning Key is used to indicate the destination partition of the message. By default, a hashing-based Partitioner is used to determine the partition ID given the key.

13. In the Producer, when does QueueFullException occur?

QueueFullException typically occurs when the Producer tries to send messages at a pace that the Broker cannot handle. Since the Producer doesn’t block, users will need to add enough brokers to collaboratively handle the load.

14. Explain the role of the Kafka Producer API.

The role of Kafka’s Producer API is to wrap the two producers – kafka.producer.SyncProducer and the kafka.producer.async.AsyncProducer. The aim is to expose all the producer functionality through a single API to the client.

15. What is the main difference between Kafka and Flume?

Even though both Kafka and Flume are used for real-time processing, Kafka is accessible and ensures message stability.

16. Describe partitioning key?

Its role is to specify the target divider of the memo, within the producer. Usually, a hash-oriented divider concludes the divider ID according to the given factors. Consumers also use the tailored Partitions.

17. Inside the manufacturer, when does the QueueFullException emerge?

QueueFullException naturally happens when the manufacturer tries to propel communications at a speed which Broker can’t grip. Consumers need to insert sufficient brokers to collectively grip the amplified load since the Producer doesn’t block.

18. Can Kafka be utilized without Zookeeper?

It is impossible to use Kafka without Zookeeper because it is not practicable to go around Zookeeper and attach in a straight line to the server. If the Zookeeper is down for a number of causes, then we will not be able to serve any customer demand.

19. Elaborate Kafka architecture.

A cluster contains multiple brokers since it is a distributed system. Topic in the system will get divided into various partitions and each broker store one or more of those partitions so that multiple producers and consumers can publish and retrieve messages at the same time.

20. What do you know about partitioning key?

A partition key can be precise to point to the aimed division of a communication, in Kafka producer. Usually, a hash-oriented divider concludes the division id with the input and people uses modified divisions also.