Overview

The rise of micro-services brings another level of software architecture, which is a event driven architecture. One of the tools out there to support this mechanism is Apache Kafka. Today’s article will speak about how to install Kafka in the server by using Ansible Playbook.

Confluent Kafka Playbook

This playbook will install Confluent Kafka into 3 cluster nodes. Each node will contain one Kafka broker and one Zookeeper instance. They will in sync one another.

For the installation, I will use Kafka from https://www.confluent.io, or for more clarity I will call it as Confluent Kafka. The Confluent itself provides two version, enterprise and open source. This playbook will use the latter version, which is an open source one.

Playbook Structure

The playbook only contains one role, which is confluent-kafka. Basically this roles will install and configure 4 services:

Zookeeper
Apache Kafka
Schema Registry
REST Proxy

The roles itself will automatically install open-jdk-8 as well. You can disable the java installation, if you already installed JDK in your target machine, by setting variable java_install into false inside roles/confluent-kafka/defaults/main.yml.

Registering Hosts for Clustering

Because some setup need to be align with properties configuration, there are some tricks to get all hosts into Ansible variables. For example, in server.properties there are some configuration to list all zookeeper instances. To address this configuration challenges, I set up a group called kafka_broker_nodes, which contains the 3 nodes to be configured.

Please take a note with ansible_host variable that are listed in all nodes. This variable values will be used by the playbook to be manipulated via Jinja filter. Therefore, we can trick the properties configuration for all the services.

Execute

 // make sure you point to the correct host. See hosts files.
 ansible-playbook site.yml -i hosts_local

1 2	// make sure you point to the correct host. See hosts files. ansible-playbook site.yml -i hosts_local

Conclusion

That is it. By using this playbook, we can setup Apache Kafka plus Zookeeper into three nodes in one single execution. One more information, I setup the playbook to run only on Redhat/CentOS 7 and using Ansible version 2.4

All the codes are in my repository. Whenever any of you want to enhance the installation, such as to be able to execute in Debian distribution or create a user instead of root to run all the services, I am more than welcome.

Author: ru rocker

I have been a professional software developer since 2004. Java, Python, NodeJS, and Go-lang are my favorite programming languages. I also have an interest in DevOps. I hold professional certifications: SCJP, SCWCD, PSM 1, AWS Solution Architect Associate, and AWS Solution Architect Professional. View all posts by ru rocker

How to Install Confluent Kafka Cluster by using Ansible