vSphere with Tanzu and NSX Advanced Load Balancer – avi-secret not found

I recently ran into an interesting behaviour during a functional test of a VMware vSphere with Tanzu container runtime platform combined with the VMware NSX Advanced Load Balancer. I find functional tests interesting: you get to see first hand how your own systems behave under difficult conditions, and at what point they give up.

The platform was built on vSphere with Tanzu (vSphere 8.0 Update 2), NSX Data Center (4.1.2.1) and NSX (Avi) Advanced Load Balancer (22.1.5). Every component was deployed in an integrated setup, meaning every interface is driven automatically, which gives developers and platform administrators a genuinely cloud-like experience.

NSX Data Center is there primarily for network and security automation, the Antrea CNI integration in NSX for instance, and the NSX Advanced Load Balancer for the automated provisioning of L4 load balancer services, which can be extended to L7 ingress. You get that degree of automation literally out of the box when you first deploy the vSphere with Tanzu container platform.

The problem

When I powered off all three NSX ALB controllers to simulate a disaster, everything behaved as expected. The surprise came after the NSX ALB control plane had been restored successfully.

Deploying a service of type LoadBalancer, I found I was getting no external IP address from the NSX ALB controller:

kubectl get svc -n ft-nsx-alb

NAMESPACE       NAME               TYPE         CLUSTER-IP  EXTERNAL-IP  PORT(S)  AGE
ft-nsx-alb      service/kubernetes LoadBalancer 10.96.0.10  <pending>    443/TCP  1d
ft-nsx-alb      service/supervisor LoadBalancer 10.96.0.134 <pending>    80/TCP   1d

The NSX ALB control plane gave me nothing to go on, because nothing was being logged at all. Which was itself a clue, so I checked the responsible service on the Supervisor cluster.

The Avi Kubernetes Operator (AKO) service handles communication between the Supervisor cluster and the NSX ALB controller cluster, and that service was not in a proper Running state:

kubectl get pods -A | grep -v Running

NAMESPACE          NAME                                                      READY  STATUS           RESTARTS         AGE
vmware-system-ako  vmware-system-ako-ako-controller-manager-65d78d698d-c944k 1/2    CrashLoopBackOff 1465 (2m24s ago) 49d

So I checked every resource in the vmware-system-ako namespace:

kubectl -n vmware-system-ako get all

NAME                                                          READY STATUS           RESTARTS         AGE
pod/vmware-system-ako-ako-controller-manager-65d78d698d-c944k 1/2   CrashLoopBackOff 1465 (4m17s ago) 49d

NAME                                                     READY UP-TO-DATE AVAILABLE AGE
deployment.apps/vmware-system-ako-ako-controller-manager 0/1   1          0         49d

NAME                                                                DESIRED CURRENT READY AGE
replicaset.apps/vmware-system-ako-ako-controller-manager-65d78d698d 1       1       0     49d

A kubectl describe on the failing pod gave this:

kubectl -n vmware-system-ako describe pod vmware-system-ako-ako-controller-manager-65d78d698d-c944k
[...]
Events:
Type    Reason  Age                       From    Message
----    ------  ----                      ----    -------
Warning BackOff 3m14s (x33529 over 5d10h) kubelet Back-off restarting failed container manager in pod vmware-system-ako-ako-controller-manager-65d78d698d-c944k_vmware-system-ako(e629cb54-f4fe-409f-8ac4-f2b0ac58b506)

And the logs showed the actual problem:

kubectl -n vmware-system-ako logs vmware-system-ako-ako-controller-manager-65d78d698d-c944k infra | more

2024-02-02T09:16:01.591Z INFO infra-main/main.go:49 AKO-Infra is running with version: ob-21883866-460f000-7e6ff10
2024-02-02T09:16:01.591Z INFO infra-main/main.go:55 We are running inside kubernetes cluster. Won't use kubeconfig files.
2024-02-02T09:16:01.594Z INFO infra-main/main.go:76 Successfully created kube client for ako-infra
2024-02-02T09:16:01.594Z INFO utils/utils.go:173 Initializing configmap informer in vmware-system-ako
2024-02-02T09:16:01.675Z INFO lib/dynamic_client.go:134 Skipped initializing dynamic informers for cniPlugin
2024-02-02T09:16:01.682Z INFO ingestion/vcf_k8s_controller.go:346 Got data from ConfigMap {"advancedL4":"true","cloudName":"/infra/sites/default/enforcement-points/default/transport-zones/overlay-tz","clusterID":"domain-c[...]","controllerIP":"[...]","credentialsSecretName":"avi-secret","credentialsSecretNamespace":"vmware-system-ako","logLevel":"WARN","serverURL":"https://[...]"}
2024-02-02T09:16:01.682Z INFO ingestion/vcf_k8s_controller.go:427 TransportZone to use for AKO is set to /infra/sites/default/enforcement-points/default/transport-zones/overlay-tz
E0202 09:16:20.192221 1 avisession.go:668] Client error for URI: login. Error: Post "https://[...]/login": dial tcp [...]:443: connect: no route to host
E0202 09:16:20.193030 1 avisession.go:714] CheckControllerStatus is disabled for this session, not going to retry.
E0202 09:16:20.193046 1 avisession.go:716] Failed to invoke API. Error: Post "https://[...]/login": dial tcp [...]:443: connect: no route to host
E0202 09:16:20.193123 1 avisession.go:383] response error: Rest request error, returning to caller: Post " https://[...]/login": dial tcp [...]:443: connect: no route to host
2024-02-02T09:16:20.193Z ERROR ingestion/vcf_k8s_controller.go:381 Failed to connect to AVI controller using secret provided by NCP, the secret would be deleted, err: Rest request error, returning to caller: Post "https://[...]/login": dial tcp [...]:443: connect: no route to host
2024-02-02T09:16:20.201Z INFO ingestion/vcf_k8s_controller.go:210 ConfigMap Add
2024-02-02T09:16:20.204Z INFO ingestion/vcf_k8s_controller.go:346 Got data from ConfigMap {"advancedL4":"true","cloudName":"/infra/sites/default/enforcement-points/default/transport-zones/overlay-tz","clusterID":"domain-c[...]","controllerIP":"[...]","credentialsSecretName":"avi-secret","credentialsSecretNamespace":"vmware-system-ako","logLevel":"WARN","serverURL":"https://[...]"}
2024-02-02T09:16:20.206Z WARN ingestion/vcf_k8s_controller.go:361 Failed to get Secret, got err: secrets "avi-secret" not found
2024-02-02T09:16:20.206Z INFO ingestion/vcf_k8s_controller.go:210 ConfigMap Add
2024-02-02T09:16:20.208Z INFO ingestion/vcf_k8s_controller.go:346 Got data from ConfigMap
[...]

The line […]Failed to connect to AVI controller using secret provided by NCP, the secret would be deleted[…] is the decisive one.

The solution

So the NSX Container Plugin (NCP) service appeared to be responsible for generating the missing secret. I restarted it to trigger the secret regeneration:

kubectl -n vmware-system-nsx get pods

NAME                       READY   STATUS    RESTARTS   AGE
nsx-ncp-5f4f7d6597-7rstp   2/2     Running   0          49d
nsx-ncp-5f4f7d6597-ppl7w   2/2     Running   0          49d

kubectl -n vmware-system-nsx delete pod nsx-ncp-5f4f7d6597-7rstp
kubectl -n vmware-system-nsx delete pod nsx-ncp-5f4f7d6597-ppl7w

Once the NCP pods were recreated, an avi-init-secret appeared, which in turn triggered a reboot of the AKO service. Shortly afterwards the avi-secret object I was waiting for appeared too:

kubectl -n vmware-system-ako get secrets

NAME            TYPE   DATA AGE
avi-init-secret Opaque 3    26h
avi-secret      Opaque 3    26h

A quick check of my deployment showed that the NSX ALB controller was assigning load balancer VIP addresses again:

kubectl get svc -n ft-nsx-alb

NAMESPACE  NAME               TYPE         CLUSTER-IP  EXTERNAL-IP  PORT(S) AGE
ft-nsx-alb service/kubernetes LoadBalancer 10.96.0.10  172.16.22.31 443/TCP 1d
ft-nsx-alb service/supervisor LoadBalancer 10.96.0.134 172.16.22.32 80/TCP  1d

TL;DR

After a total outage of the NSX ALB controller cluster, restarting the NCP pods can work wonders.