Apache Kafka is a fault tolerant publish-subscribe streaming platform that lets you process streams of records as they occur. This Kafka Quickstart Tutorial walks through the steps needed to get Apache Kafka up and running on a single Linux/Unix machine. In this tutorial we use Ubuntu and Kafka 0.10.2.0.

Installing Java

Running Kafka requires Java. You can install Java from the command line using:

# installing Java 8
apt-get install openjdk-8-jdk

Downloading Apache Kafka

Download the desired version of Apache Kafka from a mirror site found at https://www.apache.org/dyn/closer.cgi?path=/kafka/. You can download Kafka through a browser or using wget:

# download Kafka with wget
wget http://apache.cs.utah.edu/kafka/0.10.2.0/kafka_2.11-0.10.2.0.tgz

Extracting Kafka Download Contents

In order to start using Kafka, you will need to extract the contents of the downloaded tarball. Using this command will extract the contents of the downloaded kafka_2.11-0.10.2.0.tgz tarball into the current directory:

# extracting Kafka tarball
tar -xzf kafka_2.11-0.10.2.0.tgz

The extracted contents of this Kafka version will be a directory called kafka_2.11-0.10.2.0. We will consider this directory KAFKA_HOME. It is helpful to export the KAFKA_HOME variable as part of .bashrc or .bash_profile. This way it will be available to you whenever a new shell session is started.

From here you should be ready to start using Apache Kafka.

Starting Zookeeper

Kafka uses Zookeeper, so you must start a Zookeeper server before starting Kafka. Zookeeper comes as part of the Kafka download that was previously extracted, so there is no need to install it separately. You can start Zookeeper with the following commands:

# starting Zookeeper from within KAFKA_HOME
cd $KAFKA_HOME
bin/zookeeper-server-start.sh config/zookeeper.properties

Starting the Kafka Server

Now that a Zookeeper server is running, we can start our Kafka server. Zookeeper will be running in the foreground in the window it was started from. To start the Kafka server open a new terminal window and use the following commands:

# starting Kafka from within KAFKA_HOME
cd $KAFKA_HOME
$KAFKA_HOME/bin/kafka-server-start.sh config/server.properties

Creating a Topic

Now that our Zookeeper and Kafka servers are running, we can now start using them. In a separate terminal window, use the command below to create the topic tutorial-test with only one partition and only one replica:

# creating tutorial-test topic
$KAFKA_HOME/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic tutorial-test 

We can verify that the topic was created by listing all Kafka topics:

# listing Kafka topics
$KAFKA_HOME/bin/kafka-topics.sh --list --zookeeper localhost:2181

Producing Messages

Now that we have created a topic, we can now send messages to it. The kafka-console-producer will take messages from standard input and publish these messages to Kafka. Pressing <Enter> on your keyboard will send a message.

Here is an example of sending a message with the kafka-console-producer:

# sending messages to Kafka
$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic tutorial-test
My first message!

Consuming Messages

By starting a consumer we can consume messages sent to a Kafka topic. The kafka-console-consumer is included with the code we downloaded, and will read messages from a Kafka topic and write them to standard output. Adding the --from-beginning option will instruct the program to consume all available messages from the beginning of the topic.

This consumer will continue to consume messages as they are sent the the tutorial-test topic from our producer (in our example the kafka-console-producer):

# consuming messages from Kafka
$KAFKA_HOME/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic tutorial-test --from-beginning
My first message!