How to Forward Logs from a Kafka Cluster

Validated on 11 Jun 2024 • Last edited on 6 Aug 2024

Kafka is an open-source distributed event and stream-processing platform built to process demanding real-time data feeds. It is inherently scalable, with high throughput and availability.

Log forwarding allows you to transmit log data from any number of sources to a centralized database cluster. You can do so by creating and managing log sinks for your database clusters via the control panel or the API. Kafka supports forwarding to OpenSearch, Elasticsearch, and Rsyslog. You can 1-Click deploy these tools to a Droplet from the Databases section of the DigitalOcean Marketplace.

Create a Log Sink Using the API

To forward logs from managed database clusters, you need to create a log sink using the DigitalOcean API.

Note
To create a log sink using the API, you need to provide values for the sink_name, sink_type, and config fields, which specify the name of the log sink, its type (rsyslog, elasticsearch, or opensearch), and your desired URL or server/port configuration, respectively.
How to Create a Log Sink Using the DigitalOcean API

  1. Create a personal access token and save it for use with the API.

  2. Send a POST request to https://api.digitalocean.com/v2/databases/{database_cluster_uuid}/logsink

    cURL

    Using cURL:

                    curl -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  -d '{"sink_name": "logsink", "sink_type": "rsyslog", "config": {"server": "192.168.10.1", "port": 514, "tls": false, "format": "rfc5424"}}' \
  "https://api.digitalocean.com/v2/databases/9cc10173-e9ea-4176-9dbc-a4cee4c4ff30/logsink"

Forward Logs Using the Control Panel

To forward logs from a database cluster in the control panel, go to the Databases page, then select your database cluster. Click the name of the cluster to go to its Overview page, then click the Settings tab.

Cluster settings page.

On the Settings page, in the Log forwarding section, click Edit. Select the service you want to forward logs to.

Log forwarding service selection screen.

For OpenSearch and Elasticsearch, enter your destination name, endpoint URL, and index prefix. You can also click Show advanced configuration options to edit the log sink’s maximum days to store logs, request timeout (in seconds), and CA certificate for authentication. To confirm your changes, click Add destination.

For Rsyslog, enter your destination name, endpoint URL, endpoint port, and message format (RFC5424, RFC3164, or custom). You can also click Show advanced configuration options to edit the log sink’s structured data, CA certificate, client key, and client certificate. To confirm your changes, click Add destination.

Forward logs to OpenSearch popup.

Customize Rsyslog Format

If you choose a custom message format for your log sink to Rsyslog, you can use the following tags in limited Rsyslog style templating (%tag%): HOSTNAME, app-name, msg, msgid, pri, procid, structured-data, timestamp, and timestamp:::date-rfc3339.

For a list of examples, see the official Rsyslog documentation.

Control Panel Blog Pricing Careers Terms Privacy Status API Docs Tutorials Support