How to Create Kafka Clusters

Validated on 8 Dec 2022 • Last edited on 29 Oct 2024

Kafka is an open-source distributed event and stream-processing platform built to process demanding real-time data feeds. It is inherently scalable, with high throughput and availability.

Create a Database Cluster Using the CLI

Note
To create a database using doctl, you need to provide values for the --engine, --region, and --size flags. Use the doctl databases options engines, doctl databases options regions, and doctl databases options slugs commands, respectively, to get a list of available values.
How to Create a Database Using the DigitalOcean CLI

  1. Install doctl, the DigitalOcean command-line tool.

  2. Create a personal access token and save it for use with doctl.

  3. Use the token to grant doctl access to your DigitalOcean account.

              doctl auth init

  4. Finally, run doctl databases create. Basic usage looks like this, but you can read the usage docs for more details:

                doctl databases create <name> [flags]

    The following example creates a database cluster named example-database in the nyc1 region with a single 1 GB node:

                  doctl databases create example-database --region nyc1 --size db-s-1vcpu-1gb --num-nodes 1

Create a Database Cluster Using the API

Note
To create a database using the API, you need to provide values for the engine, region, and size fields, which specify the database’s engine, its datacenter, and its configuration (number of CPUs, amount of RAM, and hard disk space). Use the /v2/databases/options endpoint to get a list of available values.
How to Create a Database Using the DigitalOcean API

  1. Create a personal access token and save it for use with the API.

  2. Send a POST request to https://api.digitalocean.com/v2/databases

    cURL

    Using cURL:

                    curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  -d '{"name": "backend", "engine": "pg", "version": "14", "region": "nyc3", "size": "db-s-2vcpu-4gb", "num_nodes": 2, "storage_size_mib": 61440, "tags": ["production"]}' \
  "https://api.digitalocean.com/v2/databases"

    Go

    Using Godo, the official DigitalOcean V2 API client for Go: 

                    import (
    "context"
    "os"

    "github.com/digitalocean/godo"
)

func main() {
    token := os.Getenv("DIGITALOCEAN_TOKEN")

    client := godo.NewFromToken(token)
    ctx := context.TODO()

    createRequest := &godo.DatabaseCreateRequest{
        Name:       "backend",
        EngineSlug: "pg",
        Version:    "14",
        Region:     "nyc3",
        SizeSlug:   "db-s-2vcpu-4gb",
        NumNodes:   2,
        StorageSizeMiB : 61440,
    }

    cluster, _, err := client.Databases.Create(ctx, createRequest)
}

    Python

                    import os
from pydo import Client

client = Client(token=os.environ.get("DIGITALOCEAN_TOKEN"))

create_req = {
  "name": "backend",
  "engine": "pg",
  "version": "14",
  "region": "nyc3",
  "size": "db-s-2vcpu-4gb",
  "num_nodes": 2,
  "storage_size_mib": 61440,
  "tags": [
    "production"
  ]
}

create_resp = client.databases.create_cluster(body=create_req)

Create a Database Cluster using the Control Panel

You can create a Kafka database cluster at any time from the Create menu by selecting Databases. This takes you to the Create a Database page.

The Create menu

Choose a datacenter

In the Choose a datacenter section, select the datacenter for your database cluster.

The datacenter selection portion of the databases create page

This page lists the datacenters in which you currently have the most resources. The number of resources you have in each datacenter is listed to the right as X resources. Hover over this text to see the specific resources you have in that datacenter.

For the best performance, create your database in the same datacenter as your other DigitalOcean resources.

Note
Each region has one or more datacenters, each with a different VPC network. By including resources in the same datacenter, they all gain access to the same private networking interface, which reduces lag and keeps traffic between them from being routed over the public internet.

Choose a database engine

In the Choose a database engine section, choose Kafka.

The database engine selection portion of the databases create page

We currently support Kafka v3.6 and v3.7. You cannot change the Kafka version after creating a cluster.

Choose a cluster configuration

In the Choose a cluster configuration section, select the machine type and the number and size of the database nodes. If you select the Shared CPU plan, you also need to select a CPU option. If you select a Dedicated CPU plan, you can customize the number of nodes in the cluster to 3, 6, 9, or 15 nodes. Each option lists its combined monthly cost, equivalent hourly cost, and node specifications. For more options, click See all plans.

The Choose a cluster configuration section of the Create a database page displaying the node size and monthly cost

After creation, you can increase the number or size of database nodes at any time. However, you cannot downsize nodes to have less storage than the current storage the cluster is currently using.

Because Kafka is more resource-intensive than other engines, we recommend using shared CPU plans for development and testing workloads and using dedicated CPU plans for production workloads.

Choose a storage size

In the Choose a storage size section, you can increase your storage by clicking the up arrow or entering your desired storage amount, in increments of 10 GB, up to the maximum listed in the storage range. Additional storage you add to the cluster costs $0.20 per GB per month.

You can increase your storage at any time. You cannot reduce the amount of storage in a cluster below the amount of storage you are currently using in your cluster.

Finalize and Create

In the last section, Finalize and Create, choose the name for the cluster, the project to add it to, and any tags you want to use.

The Finalize and Create section of the Create a database page

There are three sub-headers in this section:

  • Choose a name: You can leave the automatically-generated name for the database or choose a custom name. Names must be unique, be between 3 and 63 characters long, and only contain alphanumeric characters, dashes, and periods.

  • Select a project: You can leave the default project or choose another one.

  • Tags: You can add a tag by typing it into the text box and pressing enter. Tags can only contain letters, numbers, colons, dashes, and underscores.

After creation, you can always edit the database’s tags or move it to another project; however, its name cannot be changed.

When you’re ready, click the Create a Database Cluster button.

Clusters typically take five minutes or more to provision, but you can complete important configuration tasks such as restricting inbound connections while you wait.

Note
DigitalOcean Kafka does not currently support the native importing of existing Kafka databases. However, you can use Kafka MirrorMaker to replicate existing data to your DigitalOcean Kafka cluster.
Control Panel Blog Pricing Careers Terms Privacy Status API Docs Tutorials Support