Step 1 – Installing Java
-------------------------------------
sudo apt update
sudo apt install default-jdk
Verify the current active Java version.
java --version
openjdk version "11.0.9.1" 2020-11-04
OpenJDK Runtime Environment (build 11.0.9.1+1-Ubuntu-0ubuntu1.20.04)
OpenJDK 64-Bit Server VM (build 11.0.9.1+1-Ubuntu-0ubuntu1.20.04, mixed mode, sharing)
Step 2 – Download Latest Apache Kafka
-------------------------------------
wget http://www-us.apache.org/dist/kafka/2.7.0/kafka_2.13-2.7.0.tgz
Then extract the archive file
tar xzf kafka_2.13-2.7.0.tgz
mv kafka_2.13-2.7.0 /usr/local/kafka
Step 3 – Creating Systemd Unit Files
-------------------------------------
vim /etc/systemd/system/zookeeper.service
And add the following content:
[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target
[Service]
Type=simple
ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
ExecStop=/usr/local/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal
[Install]
WantedBy=multi-user.target
Save the file and close it.
Next, to create a systemd unit file for the Kafka service:
vim /etc/systemd/system/kafka.service
Add the below content. Make sure to set the correct JAVA_HOME path as per the Java installed on your system.
[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service
[Service]
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/java-1.11.0-openjdk-amd64"
ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
ExecStop=/usr/local/kafka/bin/kafka-server-stop.sh
[Install]
WantedBy=multi-user.target
Save file and close.
Reload the systemd daemon to apply new changes.
systemctl daemon-reload
Step 4 – Start Kafka and Zookeeper Service
-------------------------------------
sudo systemctl start zookeeper
Now start the Kafka server and view the running status:
sudo systemctl start kafka
sudo systemctl status kafka
Kafka service on ubuntu
All done.
Step 5 – Create a Topic in Kafka
-------------------------------------
cd /usr/local/kafka
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testTopic
Created topic testTopic.
The replication-factor describes how many copies of data will be created. As we are running with a single instance keep this value 1.
Set the partitions options as the number of brokers you want your data to be split between. As we are running with a single broker keep this value 1.
You can create multiple topics by running the same command as above. After that, you can see the created topics on Kafka by the running below command:
bin/kafka-topics.sh --list --zookeeper localhost:2181
[output]
testTopic
Alternatively, instead of manually creating topics you can also configure your brokers to auto-create topics when a non-existent topic is published to.
Step 6 – Send and Receive Messages in Kafka
-------------------------------------
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic testTopic
>Welcome to kafka
>This is my first topic
>
Step 7 – Using Kafka Consumer
-------------------------------------
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic testTopic --from-beginning
Welcome to kafka
This is my first topic