Apache Kafka

Apache Kafka can be used to asynchronously process vast amounts of data, but it can be difficult to get started. This 1-Click App automatically configures a single-node Kafka cluster, so you can spend less time configuring it and more time developing the software that uses it.

Note: This app should not be used in production environments. Since it consists of a single-node cluster, it does not offer high availability, scaling, or any form of data-loss prevention. This app serves to provide an easy way to get started in development environments.

Software Included

Creating an App using the Control Panel

Click the Deploy to DigitalOcean button to create a Droplet based on this 1-Click App. If you aren’t logged in, this link will prompt you to log in with your DigitalOcean account.

Creating an App using the API

In addition to creating a Droplet from the Apache Kafka 1-Click App using the control panel, you can also use the DigitalOcean API. As an example, to create a 4GB Apache Kafka Droplet in the SFO2 region, you can use the following curl command. You need to either save your API access token) to an environment variable or substitute it in the command below.

curl -X POST -H 'Content-Type: application/json' \
         -H 'Authorization: Bearer '$TOKEN'' -d \
        '{"name":"choose_a_name","region":"sfo2","size":"s-2vcpu-4gb","image": "sharklabs-apachekafka"}' \
        "https://api.digitalocean.com/v2/droplets"

Getting Started After Deploying Apache Kafka

Quickstart

Here’s a short example that uses kcat, an open-source command-line producer and consumer for Apache Kafka, to produce and consume events from your local machine. Because the server’s certificate authority (CA) certificate is self-signed, the CA certificate must be explicitly trusted by the client to properly verify the server’s identity:

1. Copy the example configuration and CA root certificate from your new Droplet to your local machine

scp root@your_droplet_public_ipv4:'/opt/kafka/one-click-ssl/{ca.crt,example.librdkafka.config}' .

2. Produce a message

echo 'Hello, TLS!' | kcat -P -b your_droplet_public_ipv4:9093 \
  -F ./example.librdkafka.config \
  -X ssl.ca.location=./ca.crt \
  -t quickstart

3. Consume the message

kcat -C -b your_droplet_public_ipv4:9093 \
  -F ./example.librdkafka.config \
  -X ssl.ca.location=./ca.crt \
  -t quickstart

Exit the consumer by pressing Ctrl+c.

Additional Details

Your Kafka server is configured with two listeners:

• 127.0.0.1:9092 - PLAINTEXT and no authentication required
• 0.0.0.0:9093 - TLS/SSL using a self-signed certificate and SCRAM-SHA-256 authentication

PLAINTEXT connections are an easy way to get started, but transmitting PLAINTEXT data, especially login credentials, over the public internet leaves your information vulnerable. For security purposes, PLAINTEXT connections can only be established from inside the server:

1. Connect to your Droplet using SSH

ssh root@your_droplet_public_ipv4

2. Create a topic

/opt/kafka/bin/kafka-topics.sh \
  --create \
  --topic quickstart \
  --bootstrap-server localhost:9092

3. Produce your first event

echo 'Hello, World!' | /opt/kafka/bin/kafka-console-producer.sh \
  --topic quickstart \
  --bootstrap-server localhost:9092

4. Consume the event

/opt/kafka/bin/kafka-console-consumer.sh \
  --from-beginning \
  --max-messages 1 \
  --topic quickstart \
  --bootstrap-server localhost:9092

The configuration for this Kafka broker can be found in

/opt/kafka/config/server.properties.

The /opt/kafka/one-click-ssl directory contains the following: