Deploying a VCF 9.1 Supervisor on a One-Node Nested Lab: VLAN Trunking, Large VNA, and the Tiny-to-Small Fix

At the end of my previous VCF 9.1 homelab article, one physical MINISFORUM MS-A2 was running a complete nested management domain, including VCF Automation. That deployment proved the full stack could run. It also showed me that proving something once and keeping it as the best platform for the next experiment are two different goals.

The next milestone was a healthy vSphere Supervisor backed by NSX VPC networking. Automation and six nested hosts consumed headroom that I now needed for a Virtual Network Appliance (VNA), its load-balancer services, and a Kubernetes control plane. I powered off and deleted the old nested ESXi VMs and Installer, retained the physical ESXi host, outer vCenter, datastore, router, DNS design, and addressing plan, then rebuilt only the three-host management domain. This time the generated JSON omitted VCF Automation completely.

The rebuild was only the beginning. I reconstructed the nested VLAN path across both virtualization layers, deployed a Large VNA, worked around an unsupported AMD Ryzen model check inside that VNA, and followed Supervisor failures through content libraries, vLCM, Spherelet, registry timeouts, and Kubernetes leader-election failures. The decisive change was surprisingly small: resizing the Supervisor control plane from Tiny to Small. After that one-way resize, both Config Status and Host Config Status reached Running.

Close  ×

Why I Rebuilt the Management Domain

#

The earlier six-host design was useful because it gave me a complete management domain plus three future workload hosts. VCF Automation also worked after several recovery steps. For the Supervisor phase, however, that breadth had become a resource tax. There was no value in continuing to repair and rebalance a topology that no longer matched the immediate goal.

I kept the physical foundation intact:

• the physical ESXi 9.1 host on the MS-A2
• the outer vCenter at vc01.lab.devynharrington.com
• the nested-vcf backing datastore
• the 192.168.88.0/24 management network
• the MikroTik gateway, DNS, and established VCF addresses

I recreated inst01 and only nested-esx01 through nested-esx03. Each nested host ended at 24 vCPUs, 80 GB of RAM, a 32 GB vSAN cache disk, and a 1,000 GB thin-provisioned capacity disk. My first 16-vCPU and 64-GB sizing was not enough for the Installer’s capacity checks. A separate attempt to set 24 vCPUs while retaining 16 cores per socket failed because configSpec.numCoresPerSocket must divide the total vCPU count. I corrected the topology to 24 cores per socket.

$DeploymentPhase = "Infrastructure"
$DeployVCFAutomation = $false
$NestedESXiMGMTvCPU = "24"
$NestedESXiMGMTvMEM = "80"
$NestedESXiMGMTCachevDisk = "32"
$NestedESXiMGMTCapacityvDisk = "1000"

Close  ×

I used a distinct configuration, devyn-vcf-9.1-no-automation-1tb.ps1, and added an explicit GenerateJson phase to the driver. That let me generate and inspect a fresh management-domain specification without redeploying infrastructure that was already correct or automatically beginning bring-up.

./vcf-automated-fleet-deployment-no-automation-v2.ps1 `
  -EnvConfigFile ./devyn-vcf-9.1-no-automation-1tb.ps1 `
  -Phase GenerateJson

The important JSON check was structural. Setting a fictional enabled: false value inside vcfAutomationSpec would still leave an object the schema could try to validate. For a no-Automation deployment, the generated specification omitted vcfAutomationSpec entirely.

Fresh ESXi deployments also meant fresh TLS certificates. I recollected every SHA-256 fingerprint with openssl and inserted it into a newly named JSON file. One character mattered: the fingerprint for nested-esx01 ended in D8, not DB. Installer validation accepted hosts 2 and 3 but continued rejecting host 1 until I corrected that transcription.

Close  ×

Close  ×

The Installer then hit a separate lab-specific failure: /tmp/vcf-installer-vsan-hcl.json was missing even though the source vSAN HCL database existed. I validated the source JSON and copied it to the exact temporary path expected by the failed workflow before retrying.

jq empty /nfs/vmware/vcf/nfs-mount/vsan-hcl/all.json
cp --preserve=mode,ownership \
  /nfs/vmware/vcf/nfs-mount/vsan-hcl/all.json \
  /tmp/vcf-installer-vsan-hcl.json
ls -lh /tmp/vcf-installer-vsan-hcl.json

Close  ×

I also let the long Monitor VCF Management Services Deployment Task continue. It eventually advanced into Software Depot deployment. As in the previous build, slow progress was not proof of a dead workflow.

The Addressing Plan I Preserved

#

Reusing the earlier addressing made the rebuild easier to reason about, but I still verified every layer instead of assuming a familiar address meant the same function.

The resulting path looked like this:

flowchart LR
  R["MikroTik
192.168.88.1"] --> O["Physical ESXi + outer vCenter"]
  O -->|"VM Network
adapters 1/2"| M["Nested management
vmnic0/vmnic1"]
  O -->|"VCF-Trunk VLAN 4095
adapters 3/4"| U["vmnic2 → uplink1
vmnic3 → uplink2"]
  U --> T["Host TEPs
VLAN 2005 / 10.1.34.0/24"]
  T --> D["Distributed TGW"]
  D --> E["VCF-DTGW-External
VLAN 2006 / 10.1.35.1/26"]
  D --> V["Large VNA
192.168.88.90"]
  V --> S["Supervisor API VIP
192.168.88.91:6443"]
  S --> P["VPC 172.30.0.0/16
Services 172.29.0.0/16"]

Reconstructing Networking Across Two Virtualization Layers

#

Nested networking becomes confusing when a virtual NIC, an ESXi vmnic, a VDS uplink, and a VMkernel adapter are treated as interchangeable. They are four different objects along one path.

Outer vCenter: Preserve Management and Add the Trunk

#

On the physical vCenter, every nested ESXi VM had four virtual NICs. I left adapters 1 and 2 on VM Network, moved adapters 3 and 4 to VCF-Trunk, and kept all four connected. The outer VCF-Trunk port group used VLAN ID 4095, VMware’s virtual-guest-tagging mode for passing guest-generated 802.1Q tags.

There was no reason to move the management adapters onto the trunk merely to make the settings symmetrical. Their existing untagged management path worked. The trunk pair had a different job: carry the tagged networks produced inside the nested environment.

Close  ×

Inner vCenter: Map vmnics to Uplinks

#

Inside each nested ESXi host, those four virtual devices enumerated as vmnic0 through vmnic3. I used the VDS workflow to replace the original vmnic0/vmnic1 assignments with the trunk-backed transport pair recorded in my deployment notes: vmnic2 to uplink1 and vmnic3 to uplink2 across all three hosts. The final host view below confirms that vmnic2 and vmnic3 are attached to sddc1-cl01-vds01, while vmnic0 and vmnic1 are no longer attached to a switch. This particular view proves the selected transport devices, but it does not display the logical uplink1 and uplink2 slot names.

A vmnic is the ESXi device. An uplink is the VDS’s logical slot for that device. A VMkernel adapter such as vmk1 is an IP endpoint using a port group carried by those uplinks.

I did not blindly migrate vmk0. Losing the working management path while editing the transport path would have turned a controlled network change into a recovery exercise.

Close  ×

The VMkernel map was equally deliberate:

Close  ×

Proving the Underlay Without Overclaiming

#

I verified the physical-adapter and VMkernel state, then ran TEP, vMotion, and vSAN tests from the correct stacks and interfaces.

esxcli network nic list
esxcli network ip interface ipv4 get

vmkping -S vxlan -I vmk10 -d -s 1572 10.1.34.103
vmkping -S vxlan -I vmk11 -d -s 1572 10.1.34.104
vmkping -S vmotion -I vmk1 10.1.32.102
vmkping -I vmk2 10.1.33.102

All TEP pairs returned zero packet loss. The 1,572-byte payload with do-not-fragment demonstrated that the TEP underlay could carry more than a 1,600-byte frame once headers were included. The vMotion and vSAN pings proved reachability on those VMkernel paths.

They did not prove every later Kubernetes service, registry, NAT, or VNA path. That limitation mattered when registry timeouts later tempted me to blame the trunk.

Close  ×

Close  ×

Building the Distributed External Path

#

Before activating Supervisor, the distributed Transit Gateway needed a real exit and the VPC connectivity profile needed a service gateway, external addresses, and NAT.

Under Configure > Networking > External Connections, I created a Distributed VLAN Connection, not a centralized Tier-0/Edge connection:

The completed object showed Success, one attached Transit Gateway, and the expected VLAN/gateway pair.

Close  ×

The initial Default VPC Connectivity Profile still reported four incompatibilities: Service Gateway disabled, automatic SNAT disabled, no external IP block, and no Edge/VNA resource path. I created VCF-External-IP-Block, retained Day0 Private Tgw Ip Block (172.31.0.0/16), selected vna-cluster01, enabled north-south services, enabled Default Outbound NAT, and selected the external block for that NAT.

Close  ×

Close  ×

Broadcom’s Supervisor with VPC troubleshooting guidance documents how incomplete workload networking, routing, and MTU can surface as image-registry and core-service failures. That is why I completed these objects and validated the trunk before asking Supervisor to deploy.

Why I Deployed a Large VNA

#

The form factors shown by the VNA wizard were Small at 2 vCPUs/4 GB, Medium at 4 vCPUs/8 GB, Large at 8 vCPUs/32 GB, and Extra Large at 16 vCPUs/64 GB. A previous Small VNA reached 100 percent logical load-balancer capacity even with a Tiny Supervisor. A Medium attempt still did not produce a successful result in this lab. I therefore used Large for the repeatable final attempt.

Logical load-balancer capacity is not the same as guest CPU utilization. It is a capacity and reservation model. Broadcom’s LB Edge Capacity alarm explanation describes the alarm as capacity usage against the node’s load-balancer capacity, with a default warning threshold of 80 percent. The Large profile prevented the earlier capacity blocker here; it does not establish Large as a universal Supervisor requirement.

Close  ×

A one-node VNA cluster was acceptable for this one-physical-host proof of concept. It does not provide physical high availability. Adding a second virtual node on the same physical MS-A2 would not protect the service from failure of that host.

Close  ×

The Unsupported Ryzen CPU Check

#

The appliance booted on the AMD Ryzen 9 9955HX, but NSX Edge configuration stopped with:

NSX Edge configuration has failed. Unsupported CPU: AMD Ryzen 9 9955HX 16-Core Processor

The check in /opt/vmware/nsx-edge/bin/config.py accepted AMD EPYC model strings but rejected Ryzen. I backed up the script before changing anything.

cp -p /opt/vmware/nsx-edge/bin/config.py \
  /opt/vmware/nsx-edge/bin/config.py.pre-ryzen-workaround
ls -lh /opt/vmware/nsx-edge/bin/config.py*
grep -n -A2 -B2 'Unsupported CPU|AMD EPYC' \
  /opt/vmware/nsx-edge/bin/config.py

Close  ×

The lab-only edit commented the conditional model-name check and its self.error_exit call around lines 221 and 222. I did not remove the backup. I verified the edit and rebooted the VNA because retrying the service without a reboot did not re-run the full initialization path.

Close  ×

After reboot, /config/vmware/edge/config.json existed. The service_datapath, service_dispatcher, service_rcpm, service_rsd, and service_nsxa containers were Up, and systemctl --failed returned no failed units. A temporary L2 configuration failure appeared during convergence, and the node briefly showed Degraded/Down while HA was Admin Down on the single-node cluster. I watched it rather than rebuilding again. Manager, controller, tunnel, proxy, and storage connectivity recovered, and the VNA converged to 1 Up / Success.

Close  ×

Close  ×

Preparing the Supervisor

#

I created a subscribed Supervisor content library on vsanDatastore with immediate download from:

https://wp-content.broadcom.com/supervisor/v1/latest/lib.json

The vSphere Client initially appeared empty. Creating supervisor-library-v2 while troubleshooting did not solve that display delay, so I used PowerCLI to verify the state behind the UI. It showed the original supervisor-library, the duplicate, a separate Kubernetes Service Content Library, Supervisor images supervisor-9.0.0.0100-24845085 and supervisor-9.0.2.0100-25262241, and Spherelet items for Kubernetes versions 1.28 through 1.32.

Get-ContentLibrary |
  Format-Table Name, Type, SubscriptionUrl -AutoSize

$library = Get-ContentLibrary -Name supervisor-library-v2
Get-ContentLibraryItem -ContentLibrary $library |
  Format-Table Name, Type, Size -AutoSize

Once the client caught up, I assigned the original 11.8-GB supervisor-library under Supervisor Management > Content Distribution. The duplicate was not part of the final deployment.

Close  ×

The Supervisor wizard used these values:

I left the optional API Server DNS Name blank because the evidence did not include a deliberately created DNS record for it. I also exported the configuration before selecting Finish so I would have the exact inputs for comparison and a future deployment.

Close  ×

Following the Supervisor Failures

#

The deployment created its control-plane VM, configured management and workload networking, and configured the Supervisor API load balancer. Then core services began failing. tkg.vsphere.vmware.com and velero.vsphere.vmware.com reported timeouts reaching the internal registry:

docker-registry.kube-system.svc:5000 ... i/o timeout

Those symptoms resemble the network failure described in Broadcom KB 452173, so it was reasonable to re-check the network. It was not reasonable to declare that the root cause without reconciling the successful TEP tests, working API VIP, and later in-cluster observations.

Close  ×

What HTTP 401 Proved

#

I repeatedly called the API VIP at 192.168.88.91:6443:

for i in $(seq 1 10); do
  curl -k -sS --connect-timeout 5 --max-time 15 -o /dev/null \
    -w 'HTTP=%{http_code} TOTAL=%{time_total}s\n' \
    https://192.168.88.91:6443/readyz
  sleep 2
done

The endpoint quickly returned HTTP 401. That is an authentication rejection, not a connection failure. It proved that my client could reach an HTTPS server at the API VIP and receive an application response. It did not prove every internal service request was healthy.

PowerCLI discovered SupervisorControlPlaneVM (1) and its management address, 192.168.88.180. That was separate from the floating API address .91. The UI also showed the control-plane node at 10.1.35.6, while the VM carried 172.30.0.2 on the VPC/pod side. Treating those values as one interchangeable “Supervisor IP” would have obscured the route being tested.

Close  ×

Host Configuration Stopped at 0 of 12

#

All three hosts stalled at Installed and Started Kubernetes Node Agent on the ESXi Host, with Host Configuration at 0 of 12 conditions. Spherelet was not running, /etc/vmware/spherelet contained zero-byte configuration files, and its log was empty.

Close  ×

Close  ×

The desired cluster image was missing the Spherelet VIB required by VMware vSphere with Kubernetes - Supervisor Cluster 9.0.1.32.5.0-25510706. vLCM reported remediation was required. Its pre-check also warned that the custom nested-esxi-customization VIB could be removed and that the virtual NVMe device failed hardware compatibility. Broadcom’s vLCM VIB behavior guidance reinforces why image remediation and manually added packages need to be evaluated together. A blind remediation could remove the VIB that makes this nested appliance usable.

Close  ×

I found the exact Spherelet VIB in the vCenter patch store, copied it to nested-esx01, and installed it as a diagnostic step:

/storage/updatemgr/patch-store/hostupdate/vmw/vib20/spherelet/
VMware_bootbank_spherelet_9.0.1.32.5.0-25510706.vib

esxcli software vib install \
  -v /tmp/VMware_bootbank_spherelet_9.0.1.32.5.0-25510706.vib
esxcli software vib list | grep -i spherelet

Close  ×

The installation succeeded, yet the services stayed stopped and the configuration files remained empty. That result was useful: the VIB was only one piece. The supported desired-state workflow still had to create credentials and configuration and start the agents. Manual package installation was not the solution.

The Nested NVMe Finding

#

The vSAN health check NVMe device is VMware certified flagged the VMware virtual NVMe controller, VID 15ad and DID 07f0, on all three hosts. This was intentionally emulated hardware in a nested lab. I silenced that specific finding so it would not continue gating the lab workflow.

Close  ×

Silencing an alarm changes its reporting behavior. It does not add a controller to the hardware compatibility list or make the design suitable for production. After clearing that nested-only gate, I let the image-solution and compliance workflows continue without blindly applying a full remediation that could remove nested-esxi-customization.

The Decisive Evidence Was Inside the Control Plane

#

Once host configuration progressed, TKG still oscillated among Configuring, Reconciling, and ReconcileFailed. I moved the investigation into the control-plane VM.

The in-cluster kubernetes service was 172.29.0.1:443, with endpoint 10.1.35.6:6443. A 30-iteration request loop to https://172.29.0.1:443/readyz mostly returned rapid 401 responses. All four Kubernetes nodes were Ready, none reported MemoryPressure, DiskPressure, or PIDPressure, and kube-proxy was Running.

Close  ×

Close  ×

The controller evidence told a different story. Calls to 172.29.0.1:443 sometimes exceeded their deadlines, controllers lost leader election, probes timed out or received connection refused, and capi-kubeadm-bootstrap had restarted 168 times.

Failed to update lock optimistically ... context deadline exceeded
failed to renew lease ... leader election lost
Restart Count: 168
Back-off restarting failed container manager

Close  ×

Close  ×

The Tiny control-plane VM had only 2 vCPUs and 8 GB of RAM. During reconciliation, its load average ranged roughly from 19 to 35. CPU0 reached 100 percent, CPU1 approached 90 percent, and approximately 6.8 of 7.7 GiB was in use. This pattern explained the contradiction: networking could work, but an overloaded API server could still answer too slowly for leader leases, probes, registry pulls, and reconciliation deadlines.

I treat that as the root cause strongly supported by this lab’s evidence, not as a universal product statement. The important correlation was sustained pressure, repeated controller failure, intermittent success on the same endpoints, and convergence immediately after adding control-plane resources.

The Tiny-to-Small Fix

#

Under vcf-mgmt-supervisor01 > Configure > General > Control Plane, I changed the control-plane size from Tiny to Small:

Close  ×

The UI warned that I would not be able to scale down to a smaller size after saving. I accepted that one-way decision and let desired state reconfigure the VM. I did not reboot or delete the control-plane VM manually.

Close  ×

Close  ×

The status did not become perfect instantly. During reconciliation, completed conditions temporarily regressed from 8 of 9 to 7 of 9 while the control-plane VM and workload network were reconfigured. Then the core services turned green, the control-plane condition completed, and host configuration converged across all three nested ESXi hosts.

That progression is why resizing is the primary resolution narrative. A Large VNA had removed the earlier logical load-balancer-capacity blocker. The distributed external connection, VPC profile, NAT, trunk, and TEP paths were already in place. Clearing the nested hardware gate allowed desired state to move forward. The Tiny-to-Small change finally gave the Kubernetes control plane enough headroom to keep its API and controllers responsive through reconciliation.

Final Validation

#

Before calling the deployment complete, I verified the outcome at each layer:

• vna-cluster01 showed one node Up and deployment state Success.
• VCF-DTGW-External showed Success on VLAN 2006 with gateway 10.1.35.1/26.
• the Default VPC Connectivity Profile had its VNA, north-south services, external IP block, and outbound NAT.
• the Supervisor API load balancer was configured and 192.168.88.91:6443 responded.
• core Supervisor services reached a healthy state.
• all three nested hosts completed their host-configuration conditions.
• Config Status became Running.
• Host Config Status became Running.

The Large VNA prevented the load-balancer capacity failure I observed with smaller profiles in this lab. The Small Supervisor control plane resolved the severe CPU pressure and API instability that remained. Neither observation means every VCF 9.1 Supervisor needs those sizes; they are the known-good choices for this one-node nested design.

What I Learned

#

• Rebuild when the old topology no longer serves the next objective. Removing Automation created the headroom I needed for VNA and Supervisor work.
• Keep the outer virtual NIC, inner vmnic, VDS uplink, and VMkernel layers conceptually separate. Similar names do not make them the same object.
• A passing vmkping proves the tested path. It does not certify every application path above it.
• An HTTP 401 can be positive connectivity evidence when authentication is intentionally absent.
• Treat manual VIB installation as a diagnostic unless the complete desired-state workflow also writes configuration and starts the service.
• Silencing a nested hardware alarm does not make virtual hardware certified.
• Logical VNA load-balancer capacity and observed guest CPU are different measurements.
• Intermittent endpoint success plus missed leases, high restart counts, and extreme load can point to resource starvation rather than a total network outage.
• Preserve the state transitions. The brief regression after resizing was part of healthy reconciliation, not a reason to interrupt it.

The lab now has what I was working toward: a functioning VCF 9.1 Supervisor on a single physical host, with VPC networking, north-south services, a healthy VNA, and all nested hosts configured. The next article will use that foundation to deploy and validate vSphere Kubernetes Service rather than spending another round rebuilding the platform underneath it.