vSphere with Tanzu and NSX Advanced Load Balancer with a private CA

From VMware vSphere 8.0 Update 2 onwards you can deploy vSphere with Tanzu using VMware NSX as the networking stack and VMware NSX Advanced Load Balancer (ALB) as the integrated L4 load balancer. Beyond that, the NSX Advanced Load Balancer can also act as the L7 ingress (AKO) controller, which brings additional enterprise features: container and app security with WAF, real-time application performance monitoring, network and end-user analytics and more.

That means you can deliver classic L4/L7 load balancing services and cover microservice requirements for Kubernetes platforms at the same time. Every NSX ALB Service Engine instance, the data plane component that does the actual load balancing, is managed from one platform, whether it balances virtual machines or Kubernetes-based workloads.

The problem

On a recent Tanzu project built on vSphere 8.0 Update 2, NSX 4.1.2.1 and NSX ALB 22.1.5, we hit a problem that stopped us initialising the Supervisor cluster. Enabling the Supervisor cluster ran into a timeout during the workload network configuration phase.

Checking the NSX ALB configuration, we found that not a single object had been created and no corresponding events had been logged.

So we looked at the NCP logs and found these entries:

[ncp GreenThread-125 W] nsx_ujo.ncp.nsx.policy.ako_bootstrap_service Unexpected exception from NSX manager when acquiring AVI auth token: Unexpected error from backend manager (['nsx-mgr-01:443', 'nsx-mgr-01a:443', 'nsx-mgr-01b:443', 'nsx-mgr-01c:443']) for PUT policy/api/v1/infra/alb-auth-token: Error: I/O error on GET request for https://192.168.100.33/api/user: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain.; nested exception is javax.net.ssl.SSLHandshakeException: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain.

[ncp GreenThread-125 W] nsx_ujo.ncp.nsx.policy.ako_bootstrap_service get_avi_auth_token failed, cause: Failed to get Avi auth token: Unexpected error from backend manager (['nsx-mgr-01:443', 'nsx-mgr-01a:443', 'nsx-mgr-01b:443', 'nsx-mgr-01c:443']) for PUT policy/api/v1/infra/alb-auth-token: Error: I/O error on GET request for https://192.168.100.33/api/user: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain.; nested exception is javax.net.ssl.SSLHandshakeException: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain., args: (), kwargs: {}

[ncp GreenThread-125 I] nsx_ujo.common.controller AviSecretController worker 1 failed to sync Bootstrap due to retryable exception: Failed to get Avi auth token: Unexpected error from backend manager (['nsx-mgr-01:443', 'nsx-mgr-01a:443', 'nsx-mgr-01b:443', 'nsx-mgr-01c:443']) for PUT policy/api/v1/infra/alb-auth-token: Error: I/O error on GET request for https://192.168.100.33/api/user: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain.; nested exception is javax.net.ssl.SSLHandshakeException: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain.

It looked related to the NSX Advanced Load Balancer registration process, where the NSX ALB controller cluster registers with the NSX Manager cluster (link).

The line […]PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain[…] in particular pointed at a certificate problem, which did not follow directly, because the private PKI/CA certificates were in the NSX certificate trust store.

The solution

As it turns out, the JVM (Java Virtual Machine) process on the NSX Manager nodes has its own certificate authority trust store, which holds only public CAs. So if you issue the NSX ALB a certificate from a private PKI/CA, no trust is established, because that private CA is not in the JVM trust store.

VMware then supplied a concrete fix: import the private CAs into the JVM trust store. Run these commands in sequence on every NSX Manager:

keytool -importcert -alias <private-Root-CA> -keystore /usr/java/jre/lib/security/cacerts -storepass changeit -file <path-to-root-ca-cert>
keytool -importcert -alias <private-Intermediate-CA> -keystore /usr/java/jre/lib/security/cacerts -storepass changeit -file <path-to-intermediate-ca-cert>
sudo cp <path-to-root-ca-cert> /usr/local/share/ca-certificates/
sudo cp <path-to-intermediate-ca-cert> /usr/local/share/ca-certificates/
sudo update-ca-certificates
service proton restart

After that we could initialise the Supervisor cluster successfully. We had to cancel the current deployment (disable Supervisor) and start again.

Notes

[Update] 05.02.2024: the documentation has since been updated and now mentions this behaviour with the JVM trust store and certificates issued by a private CA.