IaaS Control Plane – API inaccessible

We recently upgraded the Supervisor cluster of our VMware IaaS Control Plane platform, formerly vSphere with Tanzu, to vSphere 8.0 Update 3 in our lab. The upgrade itself worked as expected, and both the Supervisor cluster and the Kubernetes guest clusters were working afterwards.

The problem

After a while, kubectl access to our Kubernetes clusters stopped working. The config status of the Supervisor cluster reported this:

Initialized vSphere resources
Deployed Control Plane VMs
Configured Control Plane VMs
Configured Load Balancer fronting the kubernetes API Server
Configured Core Supervisor Services
Service: velero.vsphere.vmware.com. Status: Configuring
Service: tkg.vsphere.vmware.com. Reason: Reconciling. Message: Reconciling.

Every service running on the Kubernetes guest clusters was still reachable, and the kube-api VIPs of the Kubernetes clusters still answered.

Looking closer, I found that authentication against our Supervisor cluster had stopped working. So I wanted to check with kubectl on the Supervisor control plane VMs (cpVMs) whether all the services were still active. I could not log in through the Supervisor VIP, so I connected over SSH to the management IP address of one of the three cpVMs directly.

To my surprise, neither the kube-api nor the etcd service was running:

 
crictl ps | grep -iE 'etcd|kube-api'

Because kube-api only runs when etcd is running, I checked the etcd logs:

cat /var/log/pods/kube-system_etcd-4233ab1d5ddccf36bc5bba316d1972b0_654c066164da8fbdd6d33ad93af301dc/etcd/10364.log

stderr F {"level":"warn","ts":"2024-10-08T18:22:39.077642Z","caller":"wal/repair.go:81","msg":"failed to copy","from":"/var/lib/etcd/member/wal/0000000000000181-000000001c149b6d.wal.broken","to":"/var/lib/etcd/member/wal/0000000000000181-000000001c149b6d.wal","error":"write /var/lib/etcd/member/wal/0000000000000181-000000001c149b6d.wal.broken: no space left on device"}

The error […]no space left on device[…] is the decisive one.

So I checked whether there was enough space on the cpVM’s vDisk:

 
df -h | grep /dev/root
/dev/root 32G 32G 0 100% /

Which made it reasonable to assume the services were not starting because there was no disk space left. I checked all three cpVMs, and every one of them was at 100% or 99%.

The solution

To get the services running again I had to reclaim space. A quick search for the ten largest files gave this:

 
find / -path /proc -prune -o -type f -exec du -Sh {} + | sort -rh | head -n 10
1.1G /var/log/vmware/upgrade-ctl-cli.log.1
884M /var/log/vmware/svchost/stderr.log
730M /var/log/vmware/upgrade-ctl-cli.log
385M /var/log/vmware/audit/kube-apiserver.log
307M /var/log/vmware/fluentbit/consolidated.log
250M /storage/container-registry/docker/registry/v2/blobs/sha256/a0/a0dd531132ecd058d0b0249cf2f32cecccdfbe8d13cfb93b75101aafcd2a50a6/data
248M /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/4364e490859d064c9434f9e480d74bc62402a0d812480201d644a4ea9d7ff6c3
248M /storage/container-registry/docker/registry/v2/blobs/sha256/43/4364e490859d064c9434f9e480d74bc62402a0d812480201d644a4ea9d7ff6c3/data
234M /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/fcb275778b51abf28182241bffffc6f9a25861f29ed844d71364f59e4485fb1e
234M /storage/container-registry/docker/registry/v2/blobs/sha256/fc/fcb275778b51abf28182241bffffc6f9a25861f29ed844d71364f59e4485fb1e/data

Then I truncated the log files:

 
echo > /var/log/vmware/upgrade-ctl-cli.log.1
echo > /var/log/vmware/svchost/stderr.log
echo > /var/log/vmware/upgrade-ctl-cli.log
echo > /var/log/vmware/audit/kube-apiserver.log
echo > /var/log/vmware/fluentbit/consolidated.log

Deleting the log files reclaimed enough space on all three cpVMs for etcd and kube-api to start. The free space kept shrinking though, and after a while the problem was back. So I repeated the same procedure and waited for the cluster to reach a healthy state.

Then I verified the etcd status and worked out which member was the leader:

etcdctl member list -w table
+------------------+---------+----------------------------------+----------------------------+----------------------------+------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
+------------------+---------+----------------------------------+----------------------------+----------------------------+------------+
| 10a00138f3a1ed4f | started | 421dd5792ebae985affbca516bb385c7 | https://172.16.100.11:2380 | https://172.16.100.11:2379 | false |
| 7546e437eef94d66 | started | 421d85be9b7ab8b9dad06f0c6487d976 | https://172.16.100.12:2380 | https://172.16.100.12:2379 | false |
| dec23b7a3b3cc58a | started | 421d411b9b2dca2e5176b3ff2dd4b66f | https://172.16.100.13:2380 | https://172.16.100.13:2379 | false |
+------------------+---------+----------------------------------+----------------------------+----------------------------+------------+

# Check the endpoint status of all members and get the leader.
etcdctl --endpoints=https://172.16.100.11:2379,https://172.16.100.12:2379,https://172.16.100.13:2379 -w table endpoint status
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://172.16.100.11:2379 | 10a00138f3a1ed4f | 3.5.11 | 164 MB | false | false | 318 | 668808659 | 668808659 | |
| https://172.16.100.12:2379 | 7546e437eef94d66 | 3.5.11 | 164 MB | true | false | 318 | 668808659 | 668808659 | |
| https://172.16.100.13:2379 | dec23b7a3b3cc58a | 3.5.11 | 165 MB | false | false | 318 | 668808659 | 668808659 | |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

# Check the endpoint health of all members.
etcdctl --endpoints=https://172.16.100.11:2379,https://172.16.100.12:2379,https://172.16.100.13:2379 -w table endpoint health
+----------------------------+--------+-------------+-------+
| ENDPOINT | HEALTH | TOOK | ERROR |
+----------------------------+--------+-------------+-------+
| https://172.16.100.11:2379 | true | 15.673792ms | |
| https://172.16.100.11:2379 | true | 17.78887ms | |
| https://172.16.100.11:2379 | true | 15.571392ms | |
+----------------------------+--------+-------------+-------+

To solve the problem for good, I then had all the cpVMs recreated by removing the EAM agency one at a time.

Removing the EAM agency deletes the Supervisor control plane VM it refers to. In a production environment this should only be done under the direction of, and on the explicit instruction of, VMware support.

First I gathered the information about the cpVMs:

VM	Role	Management IP	VIP	etcd member
vmware-vsc-apiserver-hv4ntr	SupervisorControlPlaneVM (1)	172.16.100.11		421dd5792ebae985affbca516bb385c7
vmware-vsc-apiserver-dhpm69	SupervisorControlPlaneVM (2)	172.16.100.12	172.16.100.10	421d85be9b7ab8b9dad06f0c6487d976 (etcd leader)
vmware-vsc-apiserver-dfll6t	SupervisorControlPlaneVM (3)	172.16.100.13		421d411b9b2dca2e5176b3ff2dd4b66f

Then I removed one cpVM after another, like this:

1. Remove the first non-leader cpVM through the vCenter UI: Administration > vCenter Server Extensions > vSphere ESX Agent Manager > Configure > vmware-vcs-apiserver-hv4ntr > Delete Agency
2. Wait until the new cpVM SupervisorControlPlaneVM (4) is provisioned and the cluster is healthy.
3. Repeat step 1 for the second non-leader cpVM: Administration > vCenter Server Extensions > vSphere ESX Agent Manager > Configure > vmware-vcs-apiserver-dfll6t > Delete Agency
4. Wait until the new cpVM SupervisorControlPlaneVM (5) is provisioned and the cluster is healthy.
5. Repeat step 1 for the leader cpVM: Administration > vCenter Server Extensions > vSphere ESX Agent Manager > Configure > vmware-vcs-apiserver-dhpm69 > Delete Agency
6. Wait until the new cpVM SupervisorControlPlaneVM (6) is provisioned and the cluster is healthy.

Once all three cpVMs had been recreated, their disk usage normalised too, and the problem was solved.