# How to Add Volumes to Kubernetes Clusters

DigitalOcean Kubernetes (DOKS) is a Kubernetes service with a fully managed control plane, high availability, and autoscaling. DOKS integrates with standard Kubernetes toolchains and DigitalOcean’s load balancers, volumes, CPU and GPU Droplets, API, and CLI.

To write and access persistent data in a Kubernetes cluster, you can create and access [DigitalOcean Volumes Block Storage](https://docs.digitalocean.com/products/volumes/index.html.md) or [DigitalOcean NFS Storage](https://docs.digitalocean.com/products/kubernetes/how-to/use-nfs-storage/index.html.md).

To use Volumes Block Storage, create a [`PersistentVolumeClaim`](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) (PVC) as part of your [deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/). This guide shows how to add volumes to your cluster using the Kubernetes command line tool, `kubectl`. To learn more about `kubectl`, see [Overview of `kubectl`](https://kubernetes.io/docs/reference/kubectl/overview/).

The claim can allow cluster workers to read and write database records, user-generated website content, log files, and other data that should persist after a process has completed.

When managing persistent volume claims:

- Deleting a deployment will not automatically delete any PVCs that have been created. You’ll have to remove those manually with [`kubectl delete pvc`](https://kubernetes.io/docs/reference/kubectl/cheatsheet/#deleting-resources).
- If the volume is deleted before the PVC API object is removed, it may be in an inconsistent state and attempts to remove the PVC will stall or fail. See the [troubleshooting instructions](#troubleshooting) for a fix to try in this case.
- If a PVC by the same name already exists, you will get an error message similar to the following:

  ```
  Error from server (AlreadyExists): error when creating "pvc.yml":
  persistentvolumeclaims "csi-pvc" already exists
  ```

  Since the volume exists, it cannot be created. The existing volume will be mounted instead.
- Volumes created in the control panel or via the API cannot be used by your Kubernetes clusters. You must create volumes within Kubernetes in order for your PVCs to use them.

## Create a Configuration File

We recommend making pods that reference volumes [owned by a StatefulSet](https://docs.digitalocean.com/support/clusterlint-error-fixes/index.html.md#dobs-pod-owner). This section shows how to create a StatefulSet to use a PVC as a volume for a pod.

The StatefulSet config file can look like this:

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: my-csi-app-set
spec:
  selector:
    matchLabels:
      app: mypod
  serviceName: "my-frontend"
  replicas: 1
  template:
    metadata:
      labels:
        app: mypod
    spec:
      containers:
      - name: my-frontend
        image: busybox
        args:
        - sleep
        - infinity
        volumeMounts:
        - mountPath: "/data"
          name: csi-pvc
  volumeClaimTemplates:
  - metadata:
      name: csi-pvc
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi
      storageClassName: do-block-storage
```

The configuration example has:

1. A pod `template` that defines how the pod gets created and the image the container uses. This example adds a pod based on the Linux [BusyBox image](https://busybox.net/), uses the volume named `csi-pvc`, and mounts it within the container at `/data` on the filesystem.
2. A `volumeClaimTemplates` that is responsible for locating the volume by name `csi-pvc`. If a volume by that name does *not* exist, one will be created. If one already exists, then the existing volume will be mounted on the first object. This example creates a 5 GB volume that will be available to the cluster by the name `csi-pvc`.

The three highlighted values, `name`, `accessModes`, and `storage` can be customized as follows:

1. The `name` must be lowercase alphanumeric values and dashes only and unique within the cluster. Within these constraints, you can name it whatever you want.
2. `accessModes` must be set to `ReadWriteOnce`.`ReadWriteMany` is supported for [DigitalOcean NFS shares](https://docs.digitalocean.com/products/kubernetes/how-to/use-nfs-storage/index.html.md) and `ReadOnlyMany` is not supported by DigitalOcean volumes. See the Kubernetes documentation for more about [`accessModes`](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes).
3. The `storage` value specifies the size of the volume and can be customized to meet your needs. DigitalOcean storage values can range from 1 GB to 10,000 GB.

Use `kubectl apply` to create the StatefulSet with the pod and the mounted volume.

You can resize volumes through Kubernetes [if the DOKS version is recent enough](https://docs.digitalocean.com/products/kubernetes/details/volume-features/index.html.md). To resize a volume, update the `storage` value of the PVC object to a new target size using `kubectl edit pvc <your-pvc-name>`. Alternatively, you can run:

```
kubectl patch pvc <your-pvc-name> -p '{ "spec": { "resources": { "requests": { "storage": "<new-size>" }}}}'
```

It may take a few minutes for the volume to resize or you may need to restart the application for the resize to become effective. To verify that the volume has resized, check its capacity value in the [volumes list](#show-volumes) or [cluster’s Kubernetes dashboard](https://docs.digitalocean.com/products/kubernetes/getting-started/quickstart/index.html.md#kubernetes-dashboard).

**Note**: Volumes can only be increased in size, but never decreased.

Billing for the volume begins when the object is successfully created. To end billing, you must [explicitly delete the volume](#delete). Remove the PVC from your cluster before deleting the volume.

## Show Volumes

Once you apply the config file to a deployment, you can see the volumes in the [**Resources** tab](https://cloud.digitalocean.com/kubernetes/clusters) of your cluster in the control panel.

Within the cluster, volumes will be identified by their names as defined in the `name` parameter. In the example above, the name is `csi-pvc`.

Regardless of what you set this name to be, the name of the volume on DigitalOcean will begin with `pvc-` and end with a unique identifying number, something like `pvc-0213ed0abexample`.

Alternatively, you can list the storage volumes associated with a cluster with the `get pv` command:

```
kubectl get pv
```

The output looks something like:

```
NAME                   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM             STORAGECLASS       REASON    AGE
pvc-0213ed0abexample   5Gi        RWO            Delete           Bound     default/csi-pvc   do-block-storage             11s
```

## Setting Permissions on Volumes

By default, the filesystem owner of a volume is `root:root`. If a Pod is running as a non-root user and needs to create files or directories on the volume, this will fail due to insufficient or incorrect permissions. However, the following `mountOptions` settings are not supported by DigitalOcean Kubernetes:

```yaml
mountOptions:
- dir_mode=0777
- file_mode=0777
```

The solution is to create a temporary container to change the permissions/ownership of the volume’s filesystem using [`initContainers`](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/).

We’re going to reuse the same PVC config as described in the example above and change the pod definition.

### Define the Pod

The next example persists data to a Postgres database using the volume you created. The pod’s resource definition might look like:

```yaml
---
apiVersion: v1
kind: Pod
metadata:
  name: my-csi-app
spec:
  containers:
    - name: my-db
      image: postgres:latest
      volumeMounts:
      - mountPath: "/var/lib/postgresql"
        name: my-do-volume
  initContainers:
  - name: pgsql-data-permission-fix
    image: busybox
    command: ["/bin/chmod","-R","777", "/data"]
    volumeMounts:
    - name: my-do-volume
      mountPath: /data
  volumes:
    - name: my-do-volume
      persistentVolumeClaim:
        claimName: csi-pvc
```

This adds a pod called `my-csi-app` based on the latest `postgres` image that names the `csi-pvc` volume `my-do-volume` and mounts it within the container at `/data` on the filesystem. This also creates an `initContainer` that temporarily mounts the volume and changes the file permissions for the specified path to 777. The `initContainer` then deletes itself. This all happens before the volume is mounted to the container. If you use `securityContext` in the YAML file for your Pod, you can use `chown $userid` instead of `chmod 777`.

#### Example:

```yaml
securityContext:
  runAsUser: 1000
  fsGroup: 2000
```

### Check the Volume Permissions

Once the cluster has been created, you can confirm the permissions were correct by checking the log with `kubectl`:

```command
kubectl logs my-csi-app
```

The output should look like the following:

```
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128 MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
```

## Troubleshooting

**Warning**: In addition to the cluster’s **Resources** tab, cluster resources (worker nodes, load balancers, and volumes) are also listed outside the Kubernetes page in the DigitalOcean Control Panel. If you rename or otherwise modify these resources in the control panel, you may render them unusable to the cluster or cause the reconciler to provision replacement resources. To avoid this, manage your cluster resources exclusively with `kubectl` or from the control panel’s Kubernetes page.

As mentioned above, if the volume is removed manually before the PVC API object is removed with `kubectl`, this can cause issues. For instance, it can cause the PVC deletion to hang and never complete. If this happens, you can try the following:

```shell
kubectl get volumeattachments
```

The output will look something like this:

```
NAME           CREATED AT
$VOLUME_NAME   2019-03-08T21:58:24Z
```

Use your volume’s name, displayed by the previous command, in the commands below to gather information you’ll need to try to fix the issue.

```shell
kubectl describe volumeattachments $VOLUME_NAME
kubectl edit volumeattachment $VOLUME_NAME
```

The `edit` command above will allow us to edit the PVC using a text editor. Remove the following from the volume attachment `metadata` section, and save your changes:

```
finalizers:
  external-attacher/dobs-csi-digitalocean-com
```

Now, you can try removing the PVC:

```shell
kubectl delete pvc csi-pvc
```

If those steps don’t work, you can [open a ticket](https://cloudsupport.digitalocean.com) with support.

## References

For more about managing persistent volumes see:

- [Other Kubernetes Components](https://www.digitalocean.com/community/tutorials/an-introduction-to-kubernetes#other-kubernetes-components) in the DigitalOcean Community’s [Introduction to Kubernetes](https://www.digitalocean.com/community/tutorials/an-introduction-to-kubernetes).
- [Kubernetes Objects](https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/) in the official [Kubernetes Concepts guide](https://kubernetes.io/docs/concepts/)
- [Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) in the official [Kubernetes Storage guide](https://kubernetes.io/docs/concepts/storage/volumes/)