How to Monitor Kafka Database Performance

Kafka is an open-source distributed event and stream-processing platform built to process demanding real-time data feeds. It is inherently scalable, with high throughput and availability.

DigitalOcean Managed Databases include metrics visualizations so you can monitor performance and health of your database cluster.

Cluster metrics monitor the performance of the nodes in a database cluster. Cluster metrics cover primary and standby nodes; metrics for each read-only node are displayed independently. This data can help guide capacity planning and optimization. You can also set up alerting on cluster metrics.

Database metrics monitor the performance of the database itself. This data can help assess the health of the database, pinpoint performance bottlenecks, and identify unusual use patterns that may indicate an application bug or security breach.

There are two groups of Kafka metrics: main server metrics, which are metrics on all databases in the cluster, and database metrics, which are metrics on individual database performance.

View Kafka Metrics

To view performance metrics for a Kafka database cluster, click the name of the database to go to its Overview page, then click the Insights tab.

The Insights tab of a Managed Database cluster

The Select object drop-down menu lists the cluster itself and all of the databases in the cluster. Choose the database to view its metrics.

In the Select Period drop-down menu, you can choose a time frame for the x-axis of the graphs, ranging from 1 hour to 30 days. Each line in the graphs displays about 300 data points.

By default, the summary to the right shows the most recent metrics values. When you hover over a different time in a graph, the summary displays the values from that time instead.

You may notice gaps in your metrics data from outages, platform maintenance, or a database failover or migration. You can check DigitalOcean’s status page for outages, review the cluster maintenance window, visit the cluster’s Settings > Logs (or Logs & Queries) page to look for failovers and migrations.

If you recently provisioned the cluster or changed its configuration, it may take a few minutes for the metrics data to finish processing before you see it on the Insights page.

If you have 200 or more databases on a single cluster, you may be unable to retrieve their metrics. If you reach this limit, create any additional databases in a new cluster.

Kafka Main Server Metrics Details

Kafka-specific main server metrics include:

  • Log Size (Largest Topics)
  • Disk I/O
  • Server - Messages per Second
  • Server - Incoming Messages - Count
  • Server - Bytes In and Out
  • Network Requests per Operation
  • Controller Offline Partitions

Main server metrics are displayed in the same view as cluster performance metrics.

Log Size (Largest Topics)

The log size plot presents the log size for each of your cluster’s largest Topics.

Disk I/O

The disk I/O plot presents the overall amount of data being written to and read from all nodes in the cluster.

Server - Messages Per Second

The messages-per-second plot presents the messages per second per node in the cluster.

Server - Incoming Messages - Count

The incoming messages plot presents the total number of messages received by the cluster by all nodes in the cluster.

Server - Bytes In and Out

The bytes-in-and-out plot presents the amount of bytes being sent and received by the cluster, organized into client bytes and replication bytes.

Network Requests Per Operation

The network requests per operation plot presents the amount of network requests across all nodes in the cluster for each of the following operations: FetchConsumer, FetchFollower, and Produce.

Controller Offline Partitions

The controller offline partitions plot presents the number of offline partitions per node in the cluster.