IaaS Control Plane – wrong NSX edge cluster for T1 placement

I recently upgraded an IaaS Control Plane environment, formerly vSphere with Tanzu, to vSphere 8.0.3. Everything ran cleanly and operations carried on as usual afterwards. Then, while building a few new dashboards in VMware Aria Operations for Logs for the NSX T1 gateways of the vSphere namespaces, I noticed that the T1 SR instances had not been instantiated on the NSX edge cluster they were meant to be on. The NSX edge cluster setup for the Supervisor cluster looks like this:

The Supervisor cluster’s T0 gateway is hosted on edge cluster 1, while every T1 gateway of the vSphere namespaces is hosted on edge cluster 2.

The problem

The fact that the vSphere namespaces’ T1 SR instances were not instantiated on the intended NSX edge cluster pointed at something in the creation workflow of the vSphere namespaces. So the first thing I tried to establish was whether the behaviour is consistent.

I created one vSphere namespace with network overrides and one without. The namespace without network overrides was provisioned on the dedicated T1 edge cluster as intended, while the one with network overrides was provisioned on the edge cluster that also holds the T0 gateway of the Supervisor system namespace.

After finding no indication of a problem in the logs, I opened a VMware SR, and the cause was clear shortly afterwards. The update to vSphere 8.0.3 changed the behaviour of T1 provisioning.

From vSphere 8.0.3 onwards this is what happens:

• When a vSphere namespace is created with network overrides, the T1 SRs are placed on the edge cluster that hosts the T0 SRs of the new vSphere namespace.
• When no network overrides are given, the T1 SRs are instantiated on the default edge cluster from the Supervisor network configuration.

So how do I get my new vSphere namespaces with network overrides onto the edge cluster I intended?

The solution

If new vSphere namespaces have already been created after the vSphere 8.0.3 update, or are planned, this workaround is what there is for now:

# [STEP-1]
# Login to a NSX manager via root user
# Get the respective T1 gateway ID of the newly created vSphere Namespace
curl -k -u 'admin:PASSWORD' --request GET 'https://localhost/policy/api/v1/infra/tier-1s' | grep VSPHERE_NAMESPACE_T1_NAME-rtr -B 1
"id" : "t1_778759f1-e306-3322-8fe4-6ddkjh658b09_rtr",
"display_name" : "t1-domain-c5008:23e875a6-44gf-4814-94ff-e840ghm7bc03-VSPHERE_NAMESPACE_T1_NAME-rtr",

# [STEP-2]
# Get the locale-services ID of the respective T1 gateway
# locale-servces ID = t1_778759f1-e306-3322-8fe4-6ddkjh658b09_rtr-0

curl -k -u 'admin:PASSWORD' --request GET 'https://localhost/policy/api/v1/infra/tier-1s/t1_778759f1-e306-3322-8fe4-6ddkjh658b09_rtr/locale-servces'
{
"results" : [ {
"edge_cluster_path" : "/infra/sites/default/enforcement-points/default/edge-clusters/294ss994-8128-4217-aa42-a182b954ak8b",
"resource_type" : "LocaleServices",
"id" : "t1_778759f1-e306-3322-8fe4-6ddkjh658b09_rtr-0",
"display_name" : "t1_778759f1-e306-3322-8fe4-6ddkjh658b09_rtr-0",
"path" : "/infra/tier-1s/t1_778759f1-e306-3322-8fe4-6ddkjh658b09_rtr/locale-services/t1_778759f1-e306-3322-8fe4-6ddkjh658b09_rtr-0",
"relative_path" : "t1_778759f1-e306-3322-8fe4-6ddkjh658b09_rtr-0",
"parent_path" : "/infra/tier-1s/t1_778759f1-e306-3322-8fe4-6ddkjh658b09_rtr",
"remote_path" : "",
"unique_id" : "84ca333a-8619-d3f31-a108-13d222c4c349",
"realization_id" : "84ca333a-8619-d3f31-a108-13d222c4c349",
"owner_id" : "48cb77bb-9t43-4285-a245-b2117f8a8b87",
"marked_for_delete" : false,
"overridden" : false,
"_create_time" : 1730469777339,
"_create_user" : "wcp-cluster-user-380fdd7d-6f2c-4cdw-82df-fb7dfddd1b9b-dbd7f126-633c-4a13-badf-f5wwd8cfd11e",
"_last_modified_time" : 1732889723175,
"_last_modified_user" : "admin",
"_system_owned" : false,
"_protection" : "REQUIRE_OVERRIDE",
"_revision" : 1
} ],
"result_count" : 1,
"sort_by" : "display_name",
"sort_ascending" : true

# Verify the associated edge cluster in the 'edge_cluster_path' parameter

curl -k -u 'admin:PASSWORD' --request GET 'https://localhost/policy/api/v1/infra/sites/default/enforcement-points/default/edge-clusters' | grep '\"id\" : \"294ss994-8128-4217-aa42-a182b954ak8b\"' -A 1
"id" : "294ss994-8128-4217-aa42-a182b954ak8b",
"display_name" : "k8s-t0-ec",

# [STEP-3]
# Get the 'path' parameter of the desired T1 edge cluster
# Current edge cluster = 294ss994-8128-4217-aa42-a182b954ak8b
# Desired edge cluster = 486hc7c3-2d29-ad51-b6c3-ff52alo86f4b

curl -k -u 'admin:PASSWORD' --request GET 'https://localhost/policy/api/v1/infra/sites/default/enforcement-points/default/edge-clusters'
{
"results" : [ {
"nsx_id" : "294ss994-8128-4217-aa42-a182b954ak8b",
"inter_site_forwarding_enabled" : false,
"member_node_type" : "EDGE_NODE",
"resource_type" : "PolicyEdgeCluster",
"id" : "294ss994-8128-4217-aa42-a182b954ak8b",
"display_name" : "k8s-t0-ec",
"tags" : [ ],
"path" : "/infra/sites/default/enforcement-points/default/edge-clusters/294ss994-8128-4217-aa42-a182b954ak8b",
"relative_path" : "294ss994-8128-4217-aa42-a182b954ak8b",
"parent_path" : "/infra/sites/default/enforcement-points/default",
"remote_path" : "",
"unique_id" : "294ss994-8128-4217-aa42-a182b954ak8b",
"realization_id" : "294ss994-8128-4217-aa42-a182b954ak8b",
"owner_id" : "48cb77bb-9t43-4285-a245-b2117f8a8b87",
"marked_for_delete" : false,
"overridden" : false,
"_create_time" : 1702468345681,
"_create_user" : "admin",
"_last_modified_time" : 1702471331793,
"_last_modified_user" : "admin",
"_system_owned" : false,
"_protection" : "NOT_PROTECTED",
"_revision" : 1
}, {
"nsx_id" : "486hc7c3-2d29-ad51-b6c3-ff52alo86f4b",
"inter_site_forwarding_enabled" : false,
"member_node_type" : "EDGE_NODE",
"resource_type" : "PolicyEdgeCluster",
"id" : "486hc7c3-2d29-ad51-b6c3-ff52alo86f4b",
"display_name" : "k8s-t1-ec",
"tags" : [ ],
"path" : "/infra/sites/default/enforcement-points/default/edge-clusters/486hc7c3-2d29-ad51-b6c3-ff52alo86f4b",
"relative_path" : "486hc7c3-2d29-ad51-b6c3-ff52alo86f4b",
"parent_path" : "/infra/sites/default/enforcement-points/default",
"remote_path" : "",
"unique_id" : "486hc7c3-2d29-ad51-b6c3-ff52alo86f4b",
"realization_id" : "486hc7c3-2d29-ad51-b6c3-ff52alo86f4b",
"owner_id" : "48cb77bb-9t43-4285-a245-b2117f8a8b87",
"marked_for_delete" : false,
"overridden" : false,
"_create_time" : 1702468349973,
"_create_user" : "admin",
"_last_modified_time" : 1702471337162,
"_last_modified_user" : "admin",
"_system_owned" : false,
"_protection" : "NOT_PROTECTED",
"_revision" : 1
} ],
"result_count" : 2,
"sort_by" : "display_name",
"sort_ascending" : true

# [STEP-4]
# Update the 'edge_cluster_path' parameter in the T1 locale-services from step 2 to the desired T1 edge cluster path from step 3

curl -k -u 'admin:PASSWORD' \
--request PUT 'https://localhost/policy/api/v1/infra/tier-1s/t1_778759f1-e306-3322-8fe4-6ddkjh658b09_rtr/locale-servces/t1_778759f1-e306-3322-8fe4-6ddkjh658b09_rtr-0' \
--header 'X-Allow-Overwrite: True' \
--header 'Content-Type: application/json' \
--data-raw '{
"edge_cluster_path": "/infra/sites/default/enforcement-points/default/edge-clusters/486hc7c3-2d29-ad51-b6c3-ff52alo86f4b",
"_revision": 0
}'

Careful: an SR relocation is a destructive operation, and you have to expect short interruptions in connectivity to the workloads behind the affected T1 gateway.

The procedure itself is short:

1. Find the NSX object ID of the affected T1 gateway
2. Find the locale-services ID of the affected T1 gateway
3. List every edge cluster in the NSX domain
   1. compare their path parameter with the one in the affected T1 gateway’s edge_cluster_path parameter
   2. find the path parameter of the correct edge cluster
4. Replace the affected T1 gateway’s edge_cluster_path parameter with the path parameter of the correct edge cluster

Once the PUT request goes through, NSX starts placing the T1 SR instances on the edge cluster you wanted.