Apache Kafka

Apache Kafka can be used to asynchronously process vast amounts of data, but it can be difficult to get started. This 1-Click App automatically configures a single-node Kafka cluster, so you can spend less time configuring it and more time developing the software that uses it.

Note: This app should not be used in production environments. Since it consists of a single-node cluster, it does not offer high availability, scaling, or any form of data-loss prevention. This app serves to provide an easy way to get started in development environments.

Software Included

Package Version License
Apache Kafka 3.1.0 Apache 2.0
OpenJDK 11 GPL-2.0

Creating an App using the Control Panel

Click the Deploy to DigitalOcean button to create a Droplet based on this 1-Click App. If you aren’t logged in, this link will prompt you to log in with your DigitalOcean account.

Deploy to DO

Creating an App using the API

In addition to creating a Droplet from the Apache Kafka 1-Click App using the control panel, you can also use the DigitalOcean API. As an example, to create a 4GB Apache Kafka Droplet in the SFO2 region, you can use the following curl command. You need to either save your API access token) to an environment variable or substitute it in the command below.

curl -X POST -H 'Content-Type: application/json' \
         -H 'Authorization: Bearer '$TOKEN'' -d \
        '{"name":"choose_a_name","region":"sfo2","size":"s-2vcpu-4gb","image": "sharklabs-apachekafka"}' \

Getting Started After Deploying Apache Kafka


Here’s a short example that uses kcat, an open-source command-line producer and consumer for Apache Kafka, to produce and consume events from your local machine. Because the server’s certificate authority (CA) certificate is self-signed, the CA certificate must be explicitly trusted by the client to properly verify the server’s identity:

  1. Copy the example configuration and CA root certificate from your new Droplet to your local machine
scp root@your_droplet_public_ipv4:'/opt/kafka/one-click-ssl/{ca.crt,example.librdkafka.config}' .
  1. Produce a message
echo 'Hello, TLS!' | kcat -P -b your_droplet_public_ipv4:9093 \
  -F ./example.librdkafka.config \
  -X ssl.ca.location=./ca.crt \
  -t quickstart
  1. Consume the message
kcat -C -b your_droplet_public_ipv4:9093 \
  -F ./example.librdkafka.config \
  -X ssl.ca.location=./ca.crt \
  -t quickstart

Exit the consumer by pressing Ctrl+c.

Additional Details

Your Kafka server is configured with two listeners:

  • - PLAINTEXT and no authentication required
  • - TLS/SSL using a self-signed certificate and SCRAM-SHA-256 authentication

PLAINTEXT connections are an easy way to get started, but transmitting PLAINTEXT data, especially login credentials, over the public internet leaves your information vulnerable. For security purposes, PLAINTEXT connections can only be established from inside the server:

  1. Connect to your Droplet using SSH
ssh root@your_droplet_public_ipv4
  1. Create a topic
/opt/kafka/bin/kafka-topics.sh \
  --create \
  --topic quickstart \
  --bootstrap-server localhost:9092
  1. Produce your first event
echo 'Hello, World!' | /opt/kafka/bin/kafka-console-producer.sh \
  --topic quickstart \
  --bootstrap-server localhost:9092
  1. Consume the event
/opt/kafka/bin/kafka-console-consumer.sh \
  --from-beginning \
  --max-messages 1 \
  --topic quickstart \
  --bootstrap-server localhost:9092

The configuration for this Kafka broker can be found in


The /opt/kafka/one-click-ssl directory contains the following:

File Contents
.keystore_password Password for Java KeyStores in this directory
ca.crt Self-signed CA root certificate
ca.key CA private key
example.librdkafka.config Client authentication configuration
kafka.crt Kafka server certificate signed by CA
kafka.keystore.jks Java KeyStore containing ca.crt and kafka.crt
kafka.truststore.jks Java KeyStore containing ca.crt