Apache Kafka is a fault tolerant publish-subscribe streaming platform that lets you process streams of records as they occur. This Kafka Quickstart Tutorial walks through the steps needed to get Apache Kafka up and running on a single Linux/Unix machine. In this tutorial we use Ubuntu
and Kafka 0.10.2.0
.
Installing Java
Running Kafka requires Java. You can install Java from the command line using:
# installing Java 8
apt-get install openjdk-8-jdk
Downloading Apache Kafka
Download the desired version of Apache Kafka from a mirror site found at https://www.apache.org/dyn/closer.cgi?path=/kafka
/. You can download Kafka through a browser or using wget
:
# download Kafka with wget
wget http://apache.cs.utah.edu/kafka/0.10.2.0/kafka_2.11-0.10.2.0.tgz
Extracting Kafka Download Contents
In order to start using Kafka, you will need to extract the contents of the downloaded tarball. Using this command will extract the contents of the downloaded kafka_2.11-0.10.2.0.tgz
tarball into the current directory:
# extracting Kafka tarball
tar -xzf kafka_2.11-0.10.2.0.tgz
The extracted contents of this Kafka version will be a directory called kafka_2.11-0.10.2.0
. We will consider this directory KAFKA_HOME
. It is helpful to export the KAFKA_HOME
variable as part of .bashrc
or .bash_profile
. This way it will be available to you whenever a new shell session is started.
From here you should be ready to start using Apache Kafka.
Starting Zookeeper
Kafka uses Zookeeper, so you must start a Zookeeper server before starting Kafka. Zookeeper comes as part of the Kafka download that was previously extracted, so there is no need to install it separately. You can start Zookeeper with the following commands:
# starting Zookeeper from within KAFKA_HOME
cd $KAFKA_HOME
bin/zookeeper-server-start.sh config/zookeeper.properties
Starting the Kafka Server
Now that a Zookeeper server is running, we can start our Kafka server. Zookeeper will be running in the foreground in the window it was started from. To start the Kafka server open a new terminal window and use the following commands:
# starting Kafka from within KAFKA_HOME
cd $KAFKA_HOME
$KAFKA_HOME/bin/kafka-server-start.sh config/server.properties
Creating a Topic
Now that our Zookeeper and Kafka servers are running, we can now start using them. In a separate terminal window, use the command below to create the topic tutorial-test
with only one partition and only one replica:
# creating tutorial-test topic
$KAFKA_HOME/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic tutorial-test
We can verify that the topic was created by listing all Kafka topics:
# listing Kafka topics
$KAFKA_HOME/bin/kafka-topics.sh --list --zookeeper localhost:2181
Producing Messages
Now that we have created a topic, we can now send messages to it. The kafka-console-producer
will take messages from standard input and publish these messages to Kafka. Pressing <Enter>
on your keyboard will send a message.
Here is an example of sending a message with the kafka-console-producer
:
# sending messages to Kafka
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic tutorial-test
My first message!
Consuming Messages
By starting a consumer we can consume messages sent to a Kafka topic. The kafka-console-consumer
is included with the code we downloaded, and will read messages from a Kafka topic and write them to standard output. Adding the --from-beginning
option will instruct the program to consume all available messages from the beginning of the topic.
This consumer will continue to consume messages as they are sent the the tutorial-test
topic from our producer (in our example the kafka-console-producer
):
# consuming messages from Kafka
$KAFKA_HOME/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic tutorial-test --from-beginning
My first message!