Apache Kafka is a fault tolerant publish-subscribe streaming platform that lets you process streams of records as they occur. This Kafka Quickstart Tutorial walks through the steps needed to get Apache Kafka up and running on a single Linux/Unix machine. In this tutorial we use
Running Kafka requires Java. You can install Java from the command line using:
# installing Java 8 apt-get install openjdk-8-jdk
Downloading Apache Kafka
Download the desired version of Apache Kafka from a mirror site found at
https://www.apache.org/dyn/closer.cgi?path=/kafka/. You can download Kafka through a browser or using
# download Kafka with wget wget http://apache.cs.utah.edu/kafka/0.10.2.0/kafka_2.11-0.10.2.0.tgz
Extracting Kafka Download Contents
In order to start using Kafka, you will need to extract the contents of the downloaded tarball. Using this command will extract the contents of the downloaded
kafka_2.11-0.10.2.0.tgz tarball into the current directory:
# extracting Kafka tarball tar -xzf kafka_2.11-0.10.2.0.tgz
The extracted contents of this Kafka version will be a directory called
kafka_2.11-0.10.2.0. We will consider this directory
KAFKA_HOME. It is helpful to export the
KAFKA_HOME variable as part of
.bash_profile. This way it will be available to you whenever a new shell session is started.
From here you should be ready to start using Apache Kafka.
Kafka uses Zookeeper, so you must start a Zookeeper server before starting Kafka. Zookeeper comes as part of the Kafka download that was previously extracted, so there is no need to install it separately. You can start Zookeeper with the following commands:
# starting Zookeeper from within KAFKA_HOME cd $KAFKA_HOME bin/zookeeper-server-start.sh config/zookeeper.properties
Starting the Kafka Server
Now that a Zookeeper server is running, we can start our Kafka server. Zookeeper will be running in the foreground in the window it was started from. To start the Kafka server open a new terminal window and use the following commands:
# starting Kafka from within KAFKA_HOME cd $KAFKA_HOME $KAFKA_HOME/bin/kafka-server-start.sh config/server.properties
Creating a Topic
Now that our Zookeeper and Kafka servers are running, we can now start using them. In a separate terminal window, use the command below to create the topic
tutorial-test with only one partition and only one replica:
# creating tutorial-test topic $KAFKA_HOME/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic tutorial-test
We can verify that the topic was created by listing all Kafka topics:
# listing Kafka topics $KAFKA_HOME/bin/kafka-topics.sh --list --zookeeper localhost:2181
Now that we have created a topic, we can now send messages to it. The
kafka-console-producer will take messages from standard input and publish these messages to Kafka. Pressing
<Enter> on your keyboard will send a message.
Here is an example of sending a message with the
# sending messages to Kafka $KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic tutorial-test My first message!
By starting a consumer we can consume messages sent to a Kafka topic. The
kafka-console-consumer is included with the code we downloaded, and will read messages from a Kafka topic and write them to standard output. Adding the
--from-beginning option will instruct the program to consume all available messages from the beginning of the topic.
This consumer will continue to consume messages as they are sent the the
tutorial-test topic from our producer (in our example the
# consuming messages from Kafka $KAFKA_HOME/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic tutorial-test --from-beginning My first message!