Kubernetes Monitoring Stack

Prometheus is a very popular and comprehensive systems monitoring solution. It is a standalone open source project and maintained independently of any company. A big plus is the capability to monitor Kubernetes clusters as well, which tend to be pretty complex in nature. Prometheus eases the operational tasks that are required in setting up a monitoring stack.

The kube-prometheus-stack is meant for cluster monitoring, so it is pre-configured to collect metrics from all Kubernetes components. In addition to that it delivers a default set of dashboards and alerting rules. Many of the useful dashboards and alerts come from the kubernetes-mixin project.

The kube-prometheus-stack consists of three main components:

• Prometheus Operator, for spinning up and managing Prometheus instances in your DOKS cluster.
• Grafana, for visualizing metrics and plot data using stunning dashboards.
• Alertmanager, for configuring various notifications (e.g. PagerDuty, Slack, email, etc) based on various alerts received from the Prometheus main server.

Prometheus follows a pull model when it comes to metrics gathering, meaning that it expects a /metrics endpoint to be exposed by the service in question for scraping. For every metric that is being fetched by Prometheus, a time series database is used to store the data points.

Grafana helps you gather data points from the Prometheus time series database, and plot everything using beautiful graphs grouped into dashboards. You can also perform queries using the PromQL language. To persist all the data (metrics and various settings), you need to allocate block storage for both Prometheus and Grafana instances via Persistent Volumes (or PVs).

The Alertmanager component handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or Slack. It also takes care of silencing and inhibition of alerts.

Please make sure to visit the official documentation page for each of the components to learn more:

• Prometheus, to learn more about all the available features, as well as various configuration options.
• Prometheus Operator, which provides useful information on how to use the operator.
• Alertmanager, to learn more about Alertmanager and integrations with various notification platforms.

Notes:

• DigitalOcean is using Helm v3 to deploy Prometheus to your DOKS cluster.
• This chart was formerly named prometheus-operator chart, now renamed to more clearly reflect that it installs the kube-prometheus project stack, within which Prometheus Operator is only one component.
• The Prometheus stack 1-Click App also includes a $1/month block storage for both Grafana and Prometheus time series database (two PVs of 5GB each, to start with).

Software Included

Creating an App using the Control Panel

Click the Deploy to DigitalOcean button to install a Kubernetes 1-Click Application. If you aren’t logged in, this link will prompt you to log in with your DigitalOcean account.

Creating an App using the API

In addition to creating Kubernetes Monitoring Stack using the control panel, you can also use the DigitalOcean API. As an example, to create a 3 node DigitalOcean Kubernetes cluster made up of Basic Droplets in the SFO2 region, you can use the following doctl command. You need to authenticate with doctl with your API access token) and replace the $CLUSTER_NAME variable with the chosen name for your cluster in the command below.

doctl kubernetes clusters create --size s-4vcpu-8gb $CLUSTER_NAME --1-clicks monitoring

Getting Started After Deploying Kubernetes Monitoring Stack

How to Connect to Your Cluster

Follow these instructions to connect to your cluster with kubectl and doctl. Additional instructions for connecting to your cluster are included in the DigitalOcean Control Panel.

How to confirm that Prometheus monitoring stack is running

First, check if the Helm installation was successful, by running below command:

helm ls -n kube-prometheus-stack

The output looks similar to (the STATUS column value should be deployed):

NAME                    NAMESPACE               REVISION  UPDATED               STATUS    CHART                         APP VERSION
kube-prometheus-stack   kube-prometheus-stack   1         2022-02-16 16:02:48   deployed  kube-prometheus-stack-55.7.0  0.70.0

Next, verify if the Prometheus stack Pods are up and running:

kubectl get pods -n kube-prometheus-stack

The output looks similar to (all Pods should be in a READY state, and STATUS should be Running):

NAME                                                        READY   STATUS    RESTARTS   AGE
alertmanager-kube-prometheus-stack-alertmanager-0           2/2     Running   0          8m24s
kube-prometheus-stack-grafana-6f6fbc5cd9-7zbnl              3/3     Running   0          8m30s
kube-prometheus-stack-kube-state-metrics-596b9c6b55-zxpmn   1/1     Running   0          8m30s
kube-prometheus-stack-operator-7bb8679c95-mpdms             1/1     Running   0          8m30s
kube-prometheus-stack-prometheus-node-exporter-vzb5m        1/1     Running   0          8m30s
kube-prometheus-stack-prometheus-node-exporter-xz44k        1/1     Running   0          8m30s
prometheus-kube-prometheus-stack-prometheus-0               2/2     Running   0          8m24s

Accessing Prometheus Web Panel

You can access Prometheus web console by port forwarding the kube-prometheus-stack-prometheus service:

kubectl port-forward svc/kube-prometheus-stack-prometheus 9090:9090 -n kube-prometheus-stack

Next, launch a web browser of your choice, and enter the following URL: http://localhost:9090. To see what targets were discovered by Prometheus, please navigate to http://localhost:9090/targets.

Accessing Grafana Web Panel

You can connect to Grafana (default credentials: admin/prom-operator), by port forwarding the kube-prometheus-stack-grafana service:

kubectl port-forward svc/kube-prometheus-stack-grafana 3000:80 -n kube-prometheus-stack

Next, launch a web browser of your choice, and enter the following URL: http://localhost:3000. You can take a look around, and see what dashboards are available for you to use from the kubernetes-mixin project as an example, by navigating to the following URL: http://localhost:3000/dashboards?tag=kubernetes-mixin.

Tweaking Helm Chart Values

The kube-prometheus-stack provides some custom values to start with. Please have a look at the values file from the main GitHub repository (explanations are provided inside, where necessary).

You can always inspect all the available options, as well as the default values for the kube-prometheus-stack Helm chart by running below command:

helm show values prometheus-community/kube-prometheus-stack --version 55.7.0

After tweaking the Helm values file (values.yml) according to your needs, you can always apply the changes via helm upgrade command, as shown below:

helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack --version 55.7.0 \
  --namespace kube-prometheus-stack \
  --values values.yml

Configuring Service Monitors for Prometheus

To monitor applications in your cluster, you usually define a so called ServiceMonitor CRD. This is a custom resource definition provided by the Prometheus Operator, which helps you in the process of adding new services that need to be monitored.

A typical ServiceMonitor configuration looks like below:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-app
  labels:
    team: frontend
spec:
  selector:
    matchLabels:
      app: example-app
  endpoints:
  - port: web

Explanations for the above configuration:

• spec.selector.matchLabels.app: Tells ServiceMonitor what application to monitor, based on a label.
• spec.endpoints.port: A reference to the port label used by the application that needs monitoring.

The kube-prometheus-stack Helm values file provided in the GitHub marketplace repository, contains a dedicated section (named additionalServiceMonitors) where you can define a list of additional services to monitor. Below snippet is setting up Nginx Ingress Controller monitoring as an example:

additionalServiceMonitors:
  - name: "ingress-nginx-monitor"
    selector:
      matchLabels:
        app.kubernetes.io/name: ingress-nginx
    namespaceSelector:
      matchNames:
        - ingress-nginx
    endpoints:
      - port: "metrics"

After adding required services to monitor, you need to upgrade the stack via the helm upgrade command, in order to apply the changes:

helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --version 55.7.0 \
  --namespace kube-prometheus-stack \
  --values values.yml

You can also check the full list of available CRDs which you can use to control the Prometheus Operator, by visiting the official GitHub documentation page.

Upgrading Kubernetes Prometheus Stack

You can check what versions are available to upgrade, by navigating to the kube-prometheus-stack official releases page from GitHub. Alternatively, you can also use ArtifactHUB, which provides a more rich and user friendly interface.

Then, to upgrade the stack to a newer version, please run the following command (make sure to replace the <> placeholders first):

helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --version <KUBE_PROMETHEUS_STACK_NEW_VERSION> \
  --namespace kube-prometheus-stack \
  --values <YOUR_HELM_VALUES_FILE>

See helm upgrade for command documentation.

Also, please make sure to check the official recommendations for various upgrade paths, from an existing release to a new major version of the Prometheus stack.

Uninstalling Kubernetes Prometheus Stack

To delete your installation of kube-prometheus-stack, please run the following Helm command:

helm uninstall kube-prometheus-stack -n kube-prometheus-stack

Note:

Above command will delete all the associated Kubernetes resources installed by the kube-prometheus-stack Helm chart, except the namespace itself. To delete the kube-prometheus-stack namespace as well, please run below command:

kubectl delete ns kube-prometheus-stack

Additional Resources

You can visit the Starter Kit set of guides provided by DigitalOcean for further study. Specifically for Prometheus, you can access the following content:

• Configuring Prometheus and Grafana.
• Getting to know PromQL.
• Visualizing Metrics using Grafana.
• Best practices for Prometheus PV sizing.

To further enrich your experience, you can also visit the media links from the official Prometheus documentation site.