In this week’s blog, {code} team member Vladimir Vivien explains one of the critical features of Kubernetes – running stateful workloads – and outlines step-by-step how to use dynamic provisioning with persistent storage in Kubernetes.

The ability for Kubernetes to run stateful workloads has become one of the most critical features for the open source container orchestrator. There are categories of containerized workloads such as legacy applications, key-value stores, databases, batch systems, etc. that depend on the availability of persistent storage to run properly. If a database container dies or gets restarted on a new cluster node, the expectation is that these events would not cause data to be lost.

So how does Kubernetes handle this scenario? As you will see in this post, Kubernetes uses a plugin model, along with other components, to implement persistent storage for workload running on a cluster.

Volume Plugins

Kubernetes uses the notion of a volume plugin which is a vendor-provided glue code that orchestrates storage operations between the Kubernetes cluster and the backing storage platform. Kubernetes is distributed with several of these volume plugins, including one for Dell EMC ScaleIO.  Having the plugins being part of the codebase reduces the administrative steps required for setup and configure persistent storage.

You may remember us talking about a Kubernetes integration with FlexREX adds Storage Options to Kubernetes, which uses the FlexVolume interface for Kubernetes and where that fits alongside Kubernetes Adds Native ScaleIO Support. Dynamic Provisioning is a feature only available with native volume plugins and will be examined in this article.


To support persistent application workloads, Kubernetes offers the PersistentVolume (PV) primitive. This describes network attached storage, such as NFS, iSCSI, or a provider-specific platform like EBS, that remains persistent regardless of how and when it is consumed by its intended workload. The lifecycle is independent of Kubernetes or any Pod that consumes it.

Originally, PersistentVolumes required administrators to manually pre-provision the requested storage prior to its use in Kubernetes. However, starting with version 1.6, the notion of Dynamic Provisioning of volumes was introduced as a stable feature. Dynamic Provisioning allows Kubernetes to automatically provision volumes, as needed, without any administrative interventions eliminating the need to pre-provision storage, thus providing a powerful abstraction to build new types of stateful workloads on Kubernetes.

StorageClass and Provisioner

StorageClass objects are created and managed by cluster administrators who can define available storage based on qualitative attributes such as “fast”, “slow”, “gold”, or “silver”, etc. These attributes can, in turn, reflect actual storage settings such as quality of service, availability, replication, backup policies, speed of disk, etc, as defined in the storage system.

Consumers of Kubernetes storage classes (cluster users) use another object called  PersistentVolumeClaim (PVC) to claim portion or all available storage described by the storage class (see PersistentVolumeClaim below). This level of indirection decouples storage consumption from storage implementation. This allows applications to be portable from environment to another (e.g. from an on-premise deployment to a cloud provider) as long as there are storage classes defined that can satisfy the claim used by a pod.

The following example defines a StorageClass object that uses the Dell EMC ScaleIO storage platform. The parameters section is what is passed to ScaleIO for use, and in this case, its credentials, the system name, protection domain, and file system type (ScaleIO Parameters). If the ScaleIO system had multiple types of Protection Domains segregated by performance, network constraints, or multi-tenancy, a StorageClass object could be made for each.

kind: StorageClass
 name: sio-standard
 gateway: https://localhost:443/api 
 system: scaleio
 protectionDomain: default
 secretRef: sio-secret
 fsType: xfs



As mentioned earlier, the PersistentVolumeClaim (PVC) acts as a “claim check” that defines persistent volumes to be consumed by applications. The creation of a PVC causes Kubernetes to dynamically provision a new volume that satisfies the claim from a referenced StorageClass. The volume created is persistent, meaning its life cycle is independent of its use by applications deployed in the cluster.

The following shows an example of a PVC that defines a claim against StorageClass sio-standard that was defined earlier.

kind: PersistentVolumeClaim
apiVersion: v1
  name: pvc-sio-standard
  annotations: sio-standard
    - ReadWriteOnce
      storage: 32Gi

Once this PVC is created, Kubernetes automatically instructs the storage platform (via the ScaleIO volume plugin) to create the volume requested by the claim without any administrative interventions. The volume is then formatted so it can be ready for consumption by pods. This greatly enhances storage automation and orchestration by Kubernetes allowing storage to be made available as it is needed.

For instance, the previous PVC causes ScaleIO to create a new volume with 32gb in size with access mode set to be Read/Write. The following figure shows a volume, with an auto-generated name, that was dynamically provisioned by Kubernetes for the defined PVC.

Using the PVC in a Deployment

A Deployment is a mechanism to orchestrate the creation and management of pods in a cluster. It defines the number of pod replicas that must be running at all time to satisfy the stable state of an application. A Deployment can define a pod which references a deployed PVC to provide storage for applications in the pod. For instance, the following Deployment defines a pod for Postgres which references the PVC, that was created earlier, under the volumes: attribute. The pod then uses attribute volumeMounts: to reference to the volume provided by the PVC.

apiVersion: extensions/v1beta1
kind: Deployment
 name: postgresdynamic
  app: postgresdynamic
  role: master
 replicas: 1
     app: postgresdynamic
     role: master
   - name: postgresdynamic
     image: postgres
     - containerPort: 5432
       value: "Password123!"
     - name: postgresdynamic
       mountPath: /var/lib/postgresql/data
  - name: postgresdynamic
    claimName: pvc-sio-standard

Once the Deployment is created, Kubernetes will bind-mount the specified path unto the Postgres container, giving access to the volume created by the PVC. This can be verified in the following figure which shows the volume being mapped in the ScaleIO GUI.

You can further verify the volume consumed by the Postgres pod using information from command kubectl describe pods <pod-name> as shown in the following figure.

For the sake of completeness, let us create a Postgres database by starting a terminal session inside the container. First, get the name of the pod running Postgres with command kubectl get pods. Next, use command kubectl exec -it <pod-name> -- /bin/bash to start the terminal session and use the psql command-line tool to create the database as shown figure below.

To demonstrate the persistent nature of Kubernetes volumes at work, we can simulate a pod failure (and restart) by deleting the pod completely from the cluster. The Replica Controller will immediately cause a new pod to be rescheduled, for Postgres, to maintain the replica state as shown in the figure below.

If the pod happens to be rescheduled on a new node, Kubernetes will automatically remap the volume to the new node (removing the old one) making the data available for the pod seamlessly. The following figure (generated with command kubectl describe pods <pod-name>) shows the volume information for the Postgres pod after it has been scheduled on a new node in the cluster. Notice that the volume is the same as before.

Let us take this further by verifying that the database is intact. As before, get the pod name for the Postgres container. Then start a session into the pod, as was done earlier. Then validate that the database is still intact as shown in the following figure. As expected, we see that the database is still there!

Recycle and Cleanup

It is important to understand that the life cycle of the PVC is independent of that of the pods that use them. Therefore, it is possible to reuse persisted volumes by different pods with different application needs. As these pods are no longer needed, they can be deleted without affecting the volumes.

To remove a volume backed by a PVC, simply delete the PVC. Kubernetes will do all the required orchestration to talk to ScaleIO and remove the dynamic volume that was associated with the deleted PVC. This completes the entire volume lifecycle and demonstrates how Kubernetes can manage storage automatically without any outside interventions.

Test it with Vagrant!

So you’ve gone through and learned what it takes to use dynamic provisioning with Kubernetes and it’s time for you to try it on your own. Go to the {code} Labs and follow ScaleIO with Docker Swarm, Mesos with Marathon, and Kubernetes. Test out persistent applications with Kubernetes in a clean and isolated environment on your laptop. If you need a jumpstart, look on virtual machine MDMI within the Vagrant environment for a folder named k8examples that has all of the files you need to run your own Postgres Deployment!