Kafka - Zookeeper configuration

https://www.youtube.com/watch?v=SxHsnNYxcww

Basically there are 2 types of zookeeper configurations:

1. Single Node - Only 1 Zookeeper server - Single point of failure
2. Zookeeper Ensemble - cluster of zookeeper nodes - more robust and no single point of failure.

In Zookeeper Ensemble case, even if one of the zookeeper node is down still the zookeeper can maintain the cluster state because of the remaining zookeeper servers running on other nodes.

Setting Zookeeper Ensemble - Kafka 3 brokers setup.

changes in zookeeper side:

mach1 - in /usr/lib/zookeeper/conf/zoo1.cfg

server.1=mach1:2888:3888
server.2=mach2:2889:3889
server.3=mach3:2890:3890

Here, the above 3 entries will specify the Ensemble of zookeeper servers cluster.

Start zookeeper server with this config file,
zookeeper-server-start.sh config/zoo1.cfg

update the same numbering sequence in "dataDir" property as this will help in numbering the zookeeper ensemble cluser.

clientPort=2181 . This port represents the clients to connect to this zookeeper server.

The entries in the config file would be same in other 2 machines apart from clientPort.
They might have clientPort=2182 and clientPort=2183 respectively.

Changes in kafka side:

mach1 in /opt/kafka-2.11-0.10.1.1/config/server.properties specify the client connection ports of all the 3 zookeeper server in Ensemble

zookeeper.connect=mach1IP:2181,mach2IP:2182,mach3IP:2183

configure the server.properties file all the kafka brokers and remember to have different port port and Brokerids for each of them.

Start kafka with this server.properties file. Now this kafka cluster state is being watched by 3 zookepers Ensemble

Eg: bin/kafka-topics.sh --list --zookeeper xvzw160.xdev.motive.com:2181,xvzw161.xdev.motive.com:2181,xvzw162.xdev.motive.com:2181

/bin/kafka-console-producer --broker-list kafka02.example.com:9092,kafka03.example.com:9092 --topic t

Note: In case of Single node zookeeper the zookeeper.connect will have only 1 zookeeper server entry.