How to Add Volumes to Kubernetes Clusters

Validated on 25 Feb 2021 • Last edited on 15 Feb 2024 cluster networking | worker node | pod

DigitalOcean Kubernetes (DOKS) is a managed Kubernetes service. Deploy Kubernetes clusters with a fully managed control plane, high availability, autoscaling, and native integration with DigitalOcean Load Balancers and volumes. You can add node pools using shared and dedicated CPUs, and NVIDIA H100 GPUs in a single GPU or 8 GPU configuration. DOKS clusters are compatible with standard Kubernetes toolchains and the DigitalOcean API and CLI.

When you need to write and access persistent data in a Kubernetes cluster, you can create and access DigitalOcean Volumes Block Storage by creating a PersistentVolumeClaim (PVC) as part of your deployment. This guide shows how to add volumes to your cluster using the Kubernetes command line tool, kubectl. To learn more about kubectl, see Overview of kubectl.

The claim can allow cluster workers to read and write database records, user-generated website content, log files, and other data that should persist after a process has completed.

When managing persistent volume claims:

  • Deleting a deployment will not automatically delete any PVCs that have been created. You’ll have to remove those manually with kubectl delete pvc.

  • If the volume is deleted before the PVC API object is removed, it may be in an inconsistent state and attempts to remove the PVC will stall or fail. See the troubleshooting instructions for a fix to try in this case.

  • If a PVC by the same name already exists, you will get an error message similar to the following:

    Error from server (AlreadyExists): error when creating "pvc.yml":
persistentvolumeclaims "csi-pvc" already exists

    Since the volume exists, it cannot be created. The existing volume will be mounted instead.

  • Volumes created in the control panel or via the API cannot be used by your Kubernetes clusters. You must create volumes within Kubernetes in order for your PVCs to use them.

Create a Configuration File

We recommend making pods that reference volumes owned by a StatefulSet. This section shows how to create a StatefulSet to use a PVC as a volume for a pod.

The StatefulSet config file can look like this:

    
        
            
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: my-csi-app-set
spec:
  selector:
    matchLabels:
      app: mypod
  serviceName: "my-frontend"
  replicas: 1
  template:
    metadata:
      labels:
        app: mypod
    spec:
      containers:
      - name: my-frontend
        image: busybox
        args:
        - sleep
        - infinity
        volumeMounts:
        - mountPath: "/data"
          name: csi-pvc
  volumeClaimTemplates:
  - metadata:
      name: csi-pvc
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi
      storageClassName: do-block-storage

The configuration example has:

  1. A pod template that defines how the pod gets created and the image the container uses. This example adds a pod based on the Linux BusyBox image, uses the volume named csi-pvc, and mounts it within the container at /data on the filesystem.

  2. A volumeClaimTemplates that is responsible for locating the volume by name csi-pvc. If a volume by that name does not exist, one will be created. If one already exists, then the existing volume will be mounted on the first object. This example creates a 5 GB volume that will be available to the cluster by the name csi-pvc.

The three highlighted values, name, accessModes, and storage can be customized as follows:

  1. The name must be lowercase alphanumeric values and dashes only and unique within the cluster. Within these constraints, you can name it whatever you want.

  2. accessModes must be set to ReadWriteOnce. The other parameters, ReadOnlyMany and ReadWriteMany, are not supported by DigitalOcean volumes. See the Kubernetes documentation for more about accessModes.

  3. The storage value specifies the size of the volume and can be customized to meet your needs. DigitalOcean storage values can range from 1 GB to 10,000 GB.

Use kubectl apply to create the StatefulSet with the pod and the mounted volume.

You can resize volumes through Kubernetes if the DOKS version is recent enough. To resize a volume, update the storage value of the PVC object to a new target size using kubectl edit pvc <your-pvc-name>. Alternatively, you can run:

kubectl patch pvc <your-pvc-name> -p '{ "spec": { "resources": { "requests": { "storage": "<new-size>" }}}}'

It may take a few minutes for the volume to resize or you may need to restart the application for the resize to become effective. To verify that the volume has resized, check its capacity value in the volumes list or cluster’s Kubernetes dashboard.

Note
Volumes can only be increased in size, but never decreased.

Billing for the volume begins when the object is successfully created. To end billing, you must explicitly delete the volume. Remove the PVC from your cluster before deleting the volume.

Show Volumes

Once you apply the config file to a deployment, you can see the volumes in the Resources tab of your cluster in the control panel.

Within the cluster, volumes will be identified by their names as defined in the name parameter. In the example above, the name is csi-pvc.

Regardless of what you set this name to be, the name of the volume on DigitalOcean will begin with pvc- and end with a unique identifying number, something like pvc-0213ed0abexample.

Alternatively, you can list the storage volumes associated with a cluster with the get pv command:

kubectl get pv

The output looks something like:

NAME                   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM             STORAGECLASS       REASON    AGE
pvc-0213ed0abexample   5Gi        RWO            Delete           Bound     default/csi-pvc   do-block-storage             11s

Setting Permissions on Volumes

By default, the filesystem owner of a volume is root:root. If a Pod is running as a non-root user and needs to create files or directories on the volume, this will fail due to insufficient or incorrect permissions. However, the following mountOptions settings are not supported by DigitalOcean Kubernetes:

    
        
            
mountOptions:
- dir_mode=0777
- file_mode=0777

The solution is to create a temporary container to change the permissions/ownership of the volume’s filesystem using initContainers.

We’re going to reuse the same PVC config as described in the example above and change the pod definition.

Define the Pod

The next example persists data to a Postgres database using the volume you created. The pod’s resource definition might look like:

    
        
            
---
apiVersion: v1
kind: Pod
metadata:
  name: my-csi-app
spec:
  containers:
    - name: my-db
      image: postgres:latest
      volumeMounts:
      - mountPath: "/var/lib/postgresql"
        name: my-do-volume
  initContainers:
  - name: pgsql-data-permission-fix
    image: busybox
    command: ["/bin/chmod","-R","777", "/data"]
    volumeMounts:
    - name: my-do-volume
      mountPath: /data
  volumes:
    - name: my-do-volume
      persistentVolumeClaim:
        claimName: csi-pvc

This adds a pod called my-csi-app based on the latest postgres image that names the csi-pvc volume my-do-volume and mounts it within the container at /data on the filesystem. This also creates an initContainer that temporarily mounts the volume and changes the file permissions for the specified path to 777. The initContainer then deletes itself. This all happens before the volume is mounted to the container. If you use securityContext in the YAML file for your Pod, you can use chown $userid instead of chmod 777.

Example:

    
        
            
securityContext:
  runAsUser: 1000
  fsGroup: 2000

Check the Volume Permissions

Once the cluster has been created, you can confirm the permissions were correct by checking the log with kubectl:

kubectl logs my-csi-app

The output should look like the following:

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Troubleshooting

Warning
In addition to the cluster’s Resources tab, cluster resources (worker nodes, load balancers, and volumes) are also listed outside the Kubernetes page in the DigitalOcean Control Panel. If you rename or otherwise modify these resources in the control panel, you may render them unusable to the cluster or cause the reconciler to provision replacement resources. To avoid this, manage your cluster resources exclusively with kubectl or from the control panel’s Kubernetes page.

As mentioned above, if the volume is removed manually before the PVC API object is removed with kubectl, this can cause issues. For instance, it can cause the PVC deletion to hang and never complete. If this happens, you can try the following:

    
        
            
kubectl get volumeattachments

The output will look something like this:

NAME           CREATED AT
$VOLUME_NAME   2019-03-08T21:58:24Z

Use your volume’s name, displayed by the previous command, in the commands below to gather information you’ll need to try to fix the issue.

    
        
            
kubectl describe volumeattachments $VOLUME_NAME
kubectl edit volumeattachment $VOLUME_NAME

The edit command above will allow us to edit the PVC using a text editor. Remove the following from the volume attachment metadata section, and save your changes:

finalizers:
  external-attacher/dobs-csi-digitalocean-com

Now, you can try removing the PVC:

    
        
            
kubectl delete pvc csi-pvc

If those steps don’t work, you can open a ticket with support.

References

For more about managing persistent volumes see:

Control Panel Blog Pricing Careers Terms Privacy Status API Docs Tutorials Support