The rise of micro-services brings another level of software architecture, which is a event driven architecture. One of the tools out there to support this mechanism is Apache Kafka. Today’s article will speak about how to install Kafka in the server by using Ansible Playbook.
Confluent Kafka Playbook
This playbook will install Confluent Kafka into 3 cluster nodes. Each node will contain one Kafka broker and one Zookeeper instance. They will in sync one another.
For the installation, I will use Kafka from https://www.confluent.io, or for more clarity I will call it as Confluent Kafka. The Confluent itself provides two version, enterprise and open source. This playbook will use the latter version, which is an open source one.
The playbook only contains one role, which is
confluent-kafka. Basically this roles will install and configure 4 services:
- Apache Kafka
- Schema Registry
- REST Proxy
The roles itself will automatically install
open-jdk-8 as well. You can disable the java installation, if you already installed JDK in your target machine, by setting variable
Registering Hosts for Clustering
Because some setup need to be align with properties configuration, there are some tricks to get all hosts into Ansible variables. For example, in
server.properties there are some configuration to list all zookeeper instances. To address this configuration challenges, I set up a group called
kafka_broker_nodes, which contains the 3 nodes to be configured.
Please take a note with
ansible_host variable that are listed in all nodes. This variable values will be used by the playbook to be manipulated via Jinja filter. Therefore, we can trick the properties configuration for all the services.
// make sure you point to the correct host. See hosts files.
ansible-playbook site.yml -i hosts_local
That is it. By using this playbook, we can setup Apache Kafka plus Zookeeper into three nodes in one single execution. One more information, I setup the playbook to run only on Redhat/CentOS 7 and using Ansible version 2.4
All the codes are in my repository. Whenever any of you want to enhance the installation, such as to be able to execute in Debian distribution or create a user instead of root to run all the services, I am more than welcome.