Tanzu default storage class

I was recently playing with Helm charts on Tanzu Kubernetes clusters and ran into a few problems. Here is a quick tip if you hit something similar.

While deploying a Helm chart I ran into pods going into a crash loop. Digging into it, I found that the persistent volumes had not been created. The chart created some PVCs but could not create the matching PVs. Describing the PVC gave me this error:

no persistent volumes available for this claim and no storage class is set

Although TKG clusters have the vSphere CSI as their PV provisioner, the PVC needs a storage class assigned so it knows where to create the volumes. With Helm charts you cannot specify which storage class to use when installing a chart, the way you would in a deployment.

Setting a default storage class

Setting a default storage class means every PVC created without one uses that class. And in theory setting a storage class as the default in Kubernetes is straightforward:

Find out which storage classes you have with kubectl get sc:

kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
sp-tanzu-global csi.vsphere.vmware.com Delete Immediate true 9d
sp-tanzu-global-latebinding csi.vsphere.vmware.com Delete WaitForFirstConsumer true 9d

As you can see there are two storage classes, but since the second is the same as the first with a different binding mode, neither is set as the default, which would show up here as (default).

Now you can make one the default by patching the class:

kubectl patch storageclass sp-tanzu-global -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Replace sp-tanzu-global with the name of your storage class. Run kubectl get sc again and you should see:

kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
sp-tanzu-global (default) csi.vsphere.vmware.com Delete Immediate true 9d
sp-tanzu-global-latebinding csi.vsphere.vmware.com Delete WaitForFirstConsumer true 9d

So far so good. Run your Helm chart straight away now and it should work. But I saw another problem: after about a minute the storage class is reset to not-default, which means Helm charts installed afterwards throw the same error about no storage class being set for the PVC.

That is specific to Tanzu clusters. Because the cluster is provisioned from a YAML file, changing the default storage class inside the cluster itself does not change the cluster’s configuration. Every so often Tanzu compares the actual cluster configuration against the YAML cluster configuration and reconciles the differences, with the YAML winning. Which means we have to set the default storage class there:

Change your context to the vSphere namespace and list your clusters (kubectl config use-context <namespace-name>). Depending on which API version was used to provision the cluster, this differs:

If you see:

kubectl get tkc
No resources found in cl-lab-core-ns-mma-tkg2 namespace.

then you provisioned your cluster with v1beta1. If you do see your cluster here, it was provisioned with v1alpha3 or older. Take the matching section below.

The default storage class in v1beta1

Run kubectl edit cluster <cluster name>. The configuration is probably fairly long. What you are looking for is the spec.topology.variables block. With a lot elided it should look roughly like this:

Look for storageClass and defaultStorageClass. That is most likely missing, so add it somewhere at that level. After that you should have a default storage class, so check in your cluster with kubectl get sc.

The default storage class in v1alpha3

If your cluster is of type TKC, as established above, you cannot edit it with kubectl edit cluster and have to use kubectl edit tkc instead.

What you want here is the spec.settings.storage level, and in the end it should look roughly like this (shortened):

You were probably missing the defaultClass value, so add it with your storage class. After that you should see your storage class when you go back to the cluster and run kubectl get sc again.

Volumes still are not created

Even after all that, some of my PVCs had volumes assigned and some did not. If you still hit the problem of no PVs being created for your deployment, look at the age of your PVCs with kubectl get PVC -A. If the problematic PVCs are still there even after deleting and redeploying the Helm chart, they may not have been created by a manifest-based persistentVolumeClaim in the chart but by a job. Which is why uninstalling the Helm chart does not delete them. Delete them manually and try again. That solved it for me.