VCF 9.1.1: A TEP-less VLAN-backed VPC Homelab with VKS and NGINX

I deployed VMware Cloud Foundation (VCF) 9.1.1 on three nested ESXi hosts running on a single Minisforum MS-A2, then used vSphere Kubernetes Service (VKS) to run NGINX and access it from my Mac. This walkthrough covers the deployment, networking, storage validation, and fixes along the way.

William Lam’s work provided the foundation. I followed his TEP-less VLAN-backed VPC walkthrough and nested VCF deployment guide, adapting his upstream automation scripts for my Ryzen hardware and network.

1. Understand the design and prepare the lab

#

The lab uses a VLAN-backed VPC (Virtual Private Cloud) for workload networking. A DTGW (Distributed Transit Gateway) connects it to the external network, and a VNA (Virtual Network Appliance) provides network services. Both remain part of the TEP-less design.

The Supervisor provides the vSphere-integrated Kubernetes control plane. VKS provisions the guest Kubernetes cluster that runs the applications. Its nodes use subnets allocated from a Public SubnetSet in the namespace.

TEP-less means the ESXi hosts do not need Tunnel Endpoint IPs; their TEP assignment was NO_IP. The guest cluster uses Antrea for pod networking. Antrea supports multiple traffic modes, and I did not verify which was active, so TEP-less does not imply overlay-free pod networking.

The diagram shows logical relationships; application traffic does not pass through the Supervisor. The outer vCenter manages the nested infrastructure; the nested vCenter manages the VCF domain.

The router interface vlan2006-external uses 10.1.35.1/24. Use the same gateway CIDR for the transit gateway’s external connection and 10.1.35.0/24 for the External IP Block.

The outer inventory reported approximately 628 GB of tiered capacity, not physical DRAM. All three nested hosts share the physical host’s CPU, memory, storage, and failure domain. The cache/capacity labels above come from the deployment script, not general vSAN ESA sizing guidance.

My earlier physical-host and VCF article covers the existing foundation. Its six-host topology, Automation deployment, and JSON bring-up are a different build. Here I retained the physical host, outer vCenter, datastore, router, and local assets, then rebuilt the three-host nested domain.

Prerequisites to check before starting

#

The upstream repository calls for an outer vCenter-managed environment, PowerShell Core/PowerCLI, the nested ESXi and Installer OVAs, an entitled software source, and networking that permits nested guests.

The Broadcom PowerCLI guide recommends PowerShell 7.4 or later and the VCF.PowerCLI module. Review the VCF 9.1 depot activation article and Cloud Proxy DNS prerequisite article before bring-up.

Use your own domain, IPs, datastore and cluster names, port groups, and local paths. Generated names and allocated addresses will vary by deployment.

2. Deploy the nested hosts and Installer from PowerShell

#

I worked in ~/VCF91-Lab on the Mac with the following scripts and OVAs:

Close  ×

Start with William’s repository and adapt its sample configuration to your environment. The filenames and -Phase Infrastructure parameter below belong to my customized scripts.

Mac terminal — zsh, then enter PowerShell:

cd ~/VCF91-Lab
pwsh

PowerShell on the Mac — recorded local Infrastructure phase:

./vcf-automated-fleet-deployment-9.1.1-tepless.ps1 `
  -EnvConfigFile ./devyn-vcf-9.1.1-tepless-no-automation-3tb.ps1 `
  -Phase Infrastructure

The PowerShell backtick continues a line; do not put spaces after it. The Infrastructure phase prepared the nested hosts and Installer. The subsequent management-domain deployment was performed through the Installer UI.

Close  ×

Before accepting the script’s Y/N prompt, I checked VCF 9.1.1.0, three hosts, 24 vCPU/128 GB each, 32 GB cache and 3000 GB capacity VMDKs, Day-0 Automation DISABLED, and TEP assignment NONE with overlayVtepSpec.vtepType = NO_IP. The summary also enabled the nested-lab vSAN ESA HCL auto-disk-claim exception. The Installer review later calls this Allow auto claim of HCL incompatible disks. It is a lab accommodation, not a statement of supported physical hardware.

Close  ×

I recreated the nested lab and reinstalled the 9.1.1 binaries before the successful rebuild.

Checkpoint: in the outer vCenter, verify the three nested hosts and inst01, including their networks and configured resources. Confirm access to https://inst01.vcf.lab.devynharrington.com and prepare the entitled 9.1.1 binaries before starting the fleet wizard.

3. Complete the successful VCF Installer UI workflow

#

I opened the Installer, chose Deploy a new VCF fleet, and used the new-management-domain path without existing components.

Close  ×

In Plan, I selected the Simple deployment model and Small deployment size. The selection below shows the settings used for the successful rebuild.

Close  ×

Select VLAN-backed VPC in Network Options

#

In Plan → Network Options, click Customize. Under VPC Network Configuration, change Full Stack VPC to VLAN backed VPC.

Close  ×

The Distributed and Centralized gateway choices disappear when VLAN backed VPC is selected. Confirm this selection before clicking Next. It sets the host TEP configuration to overlayVtepSpec.vtepType = NO_IP; the external VLAN, gateway, IP block, and VNA are configured after deployment in section 5.

General information and host entry

#

In Prepare, I entered the following values. This table follows the final review, so it supersedes earlier default screens.

General Information form before final settings

Close  ×

For each host, enter its current root password and verify the presented certificate fingerprint against the host. Rebuilding a host creates a new certificate, so do not reuse a previous deployment’s thumbprint.

Close  ×

Networks and the management-services address pool

#

Enter the ESX, VM, and VCF management networks, then the vMotion and vSAN networks. Keep the management-services pool separate from the external workload block.

I selected Single IPv4 range for the 30-address management-services pool listed above; the form required at least 12. The DHCP pool ended at .191. The VNA and Supervisor floating addresses, .90 and .91, were outside both pools.

Close  ×

VLAN 0 applies to these nested networks. The external workload network uses VLAN 2006.

Operations, Cloud Proxy, and management services

#

Enter the corresponding root, administrator, and system-user passwords in the wizard.

Nested vCenter, storage, and distributed switch

#

Close  ×

Close  ×

I kept the Installer mapping on vmnic0/vmnic1. The trunk migration comes after deployment, as section 4 explains.

NSX, SDDC Manager, validation, and completion

#

The VLAN backed VPC selection made in Plan → Network Options is retained in the saved configuration as vtepType: NO_IP.

The overlay transport-zone name remains in the review even though host TEP IPs are not assigned. Enter the NSX root/admin/audit and SDDC Manager root/VCF/admin credentials, review the specification, and run validation. Resolve any errors before starting deployment. Keep exported specifications private because they can contain credentials.

Close  ×

The Installer reported a successful deployment:

Close  ×

About 5 hours 43 minutes elapsed between the deployment-start and completion screenshots in my nested lab. Confirm the green stage results and success banner before continuing to postdeployment networking.

4. Move the completed VDS onto the outer trunk adapters

#

The Installer used vmnic0/vmnic1. After deployment, I moved the VDS uplinks to the outer-trunk-backed vmnic2/vmnic3 adapters for the VLAN-backed workload network.

My earlier attempt to specify vmnic2/vmnic3 too soon failed ESX Host Configuration validation: vmnic0 was attached to vSwitch0, but was missing from the input specification. That is why I retained vmnic0/vmnic1 during the successful Installer run.

Close  ×

In outer vCenter → VM → Edit Settings, I checked each nested host’s extra adapters were connected to VCF-Trunk, configured with VLAN 4095. Match adapter MAC addresses to the nested host’s physical-adapter inventory to confirm the mapping.

Close  ×

Close  ×

The Broadcom nested-VLAN reference distinguishes outer switch types: VLAN 4095 is the standard-port-group virtual guest tagging setting; a distributed port group instead uses VLAN Trunking with an allowed range. Permit VLAN 2006 across the actual path and retain the management/storage connectivity required by the nested hosts.

In the nested vCenter → Networking → vcf-mgmt-cl01-vds01 → Add and Manage Hosts, I selected the nested hosts and managed their physical adapters.

Close  ×

Apply changes deliberately, preserving a verified working path while migrating a host. Review the wizard’s impact and keep console access available. Do not detach all working adapters at once or assign the same host’s uplink to two adapters. After each host, verify management reachability and storage networking before proceeding.

Close  ×

I completed the trunk correction during the rebuild. Figure S24 shows the starting state; Figure H181 illustrates the final mapping from a previous deployment.

5. Configure the transit gateway, external block, and Large VNA

#

With VCF up and the trunk path corrected, I rechecked the router’s /24.

Close  ×

In nested vCenter → Network → Networks → Transit Gateways, click Set Up Network Connectivity to begin.

Close  ×

The prerequisite screen calls for the external VLAN path and disabling ICMP redirects on its gateway. I verified the intended VLAN connectivity, then applied the recorded RouterOS setting.

Close  ×

MikroTik RouterOS terminal — disable and verify ICMP redirects:

/ip/settings/set send-redirects=no
/ip/settings/print

Close  ×

In External Network Connectivity, enter VLAN 2006 and gateway CIDR 10.1.35.1/24. Add the block in the same workflow:

External-connection and IP-block entry forms

Close  ×

Close  ×

Close  ×

The gateway field uses the host address, 10.1.35.1/24; the block uses the network, 10.1.35.0/24. Exclude the router address and any other reserved addresses from allocation.

The transit gateway provides the external connection; the VPC connectivity profile links the VPC to the external IP block and services. My Supervisor used the Default NSX project and Default VPC Connectivity Profile. Verify that profile references this /24 block before enabling Supervisor.

Submit the VNA configuration

#

On VPC Services, configure the VNA cluster and add its node.

Intermediate VNA forms: Medium and DHCP are initial defaults

Close  ×

Close  ×

The initial service form showed Medium, and the unfilled node form showed DHCP. Neither establishes the final submission. The recorded management plan used .90; the healthy result later confirms that address. The submitted review conclusively shows Large, one node, and the correct external connection:

Close  ×

Close  ×

Monitor deployment under Configure → Networking → VNA Clusters. Figure S34 shows vna01 at 10% Deploying VM.

6. Recover VNA root access and apply the Ryzen workaround

#

The VNA did not deploy successfully on my Ryzen lab without intervention. Deployment stalled at around 51%, the same symptom I had encountered in the earlier attempt. The VNA hit an AMD CPU-model validation guard: the installed Python code rejected an AMD processor unless its model string contained AMD EPYC.

To get deployment moving again, I opened vna01’s VM web console in the nested vCenter, restarted the appliance, and used GRUB password recovery to reset its root password. Once I could log in as root, I applied the sed workaround below, verified the change, and rebooted. The recovery steps and commands are shown in order below; the stall near 51% was an observation from my lab.

William’s original Ryzen/NSX Edge article explains the guard and workaround. This is an unsupported homelab modification. His upgrade follow-up covers changes to file paths and line numbers; upgrades or redeployments can replace the modified file.

Use the VNA console and GRUB password recovery

#

1. In the nested vCenter, select vna01, launch its VM web console, and restart the appliance.
2. Hold left Shift or press Esc during boot to interrupt GRUB. If you miss the menu, restart and try again. Select the boot entry and press e to edit.
3. If GRUB authenticates the edit, use its credentials. The documented unchanged default is username root, password NSX@VM!WaR10. A changed GRUB password takes precedence. This is not the appliance’s new root login password.
4. Locate the line beginning with linux and move to the end of the logical line, even if it wraps across the display. On the Mac, the guidance was Ctrl+E or Fn+Right; confirm the actual cursor position.
5. Append a space and the recovery parameter below.
6. Press Ctrl+X to boot. Follow the recovery prompt to set and confirm the appliance root password, then complete boot and log in as root.

VNA VM console — append to the GRUB linux line, not a shell command:

systemd.wants=PasswordRecovery.service

Close  ×

The parameter and boot workflow are documented in Broadcom KB 416526 and the Edge/Manager recovery procedure in KB 316043. I reached the VNA root shell through this process. No recovered appliance password is included in this article.

Console message about the separate GRUB password

Close  ×

Inspect the installed file before editing

#

VNA root shell — advised backup and recorded inspection command:

cp -pn /opt/vmware/nsx-edge/bin/config.py \
  /opt/vmware/nsx-edge/bin/config.py.pre-ryzen-workaround

grep -n -A5 -B5 'Unsupported CPU\|AMD EPYC' \
  /opt/vmware/nsx-edge/bin/config.py

The backup was advised in the conversation; a separate successful backup result was not captured. Verify that your own backup exists before modifying the file. The inspection was captured and located this guard at lines 224–225 in my installed build.

VNA configuration file — inspected Python excerpt, not a command to execute:

if "AMD" in vendor_info and "AMD EPYC" not in model_name:
    self.error_exit("Unsupported CPU: %s" % model_name)

Close  ×

VNA root shell — exact recorded edit, verification, and reboot:

sed -i '224,225s/^/# /' /opt/vmware/nsx-edge/bin/config.py
sed -n '224,225p' /opt/vmware/nsx-edge/bin/config.py
reboot

Close  ×

Close  ×

Do not apply those line numbers blindly to another version. Find the guard in that installed file and confirm that only the intended two lines will be commented. The screenshot shows both lines prefixed with #. The first sed command changes the file; the second prints the edited lines so you can verify them.

Wait for the VNA deployment to recover

#

After applying the workaround and rebooting, I returned to Configure → Networking → VNA Clusters. The node briefly showed L2 Config Failed, then In Progress. I left the deployment running; it reached Up / Success after roughly five minutes.

Close  ×

With the cluster and node Up / Success, I continued. BFD Sessions: Not Available remained visible. This lab used a single VNA node.

7. Create the two subscribed content libraries

#

The Supervisor image source and the guest Kubernetes release source serve different consumers. I used separate names and subscriptions:

In nested vCenter → Content Libraries → Create, name the library, select the nested vCenter, choose Subscribed content library, enter its URL, and select vsanDatastore. Use automatic synchronization, no authentication for these public subscriptions, and download content immediately. Wait for the needed images to finish downloading before using them.

Downloading immediately uses storage and transfer time up front so the images are available locally.

The local Fleet-gateway subscription I tried could not fetch its items in this build:

Content Library subscription field — local URL that failed in my lab:

https://vcf-flt01.vcf.lab.devynharrington.com/depot-service/content-gateway/PROD/COMP/VKR/lib.json

I used the public VKR subscription instead, which worked in my lab.

Broadcom’s library-association guidance places the Supervisor image-source association under Supervisor Management → Content Distribution → Supervisor Images Library. The 9.1 Supervisor library article confirms the public Supervisor endpoint.

After Supervisor is running, associate VKR-Subscribed-Library under Supervisor → Configure → General → Kubernetes Service, then make it available through the namespace’s VM Service / Content Libraries configuration, as shown in section 10.

8. Activate a Medium Supervisor

#

With the VNA healthy and image sources prepared, I opened Supervisor Management → Add Supervisor, chose VCF Networking with VPC, and selected the zone containing vcf-mgmt-cl01.

Close  ×

Close  ×

DHCP for the control-plane management configuration and the explicit .91 floating address are separate fields. Ensure DHCP and routing are available on the selected management network; copying the floating IP alone does not supply them.

Close  ×

Initial Supervisor sizing before my Medium selection

Close  ×

Close  ×

I chose Medium for the planned Kubernetes use. The UI warned that it could not later scale down. Keep these sizes separate: Simple/Small fleet, Large VNA, Medium Supervisor, and later best-effort-small VKS node VMs. They size different components.

Review the network, storage, sizing, and endpoint choices and activate the Supervisor. Entering the API DNS name does not itself prove that a DNS A record was created. The resulting API address was 10.1.35.7; its current DNS answer remains an uncaptured check.

Close  ×

9. Follow Supervisor reconciliation through to both Running states

#

The guest, management-network, and Kubernetes control-plane conditions completed while workload networking and service configuration were still in progress.

Close  ×

During reconciliation, I observed the following errors and status changes:

NSX, Pinniped, and CPU-pressure evidence during reconciliation

Close  ×

Close  ×

Close  ×

Close  ×

Close  ×

Delayed host conditions and Apply Solution health checks

Close  ×

Close  ×

For a similar delay, inspect Supervisor Management → status details, the cluster’s recent tasks, and cluster → Monitor → vSAN → Skyline Health. Resolve failures; silence only understood warnings related to nested hardware.

Close  ×

I did not isolate a single cause of the recovery.

Both Config Status and Host Config Status reached Running, while the Supervisor remained Medium:

Close  ×

The UI showed three hosts, three system namespaces, API address 10.1.35.7, and Supervisor version v1.34.9+vmware.1-vsc.9.1.1.0-25712839. Both Running columns—not just the existence of a control-plane VM—were the checkpoint for continuing.

Mac terminal — suggested DNS verification, not a captured completed test:

nslookup vcf-mgmt-supervisor01.vcf.lab.devynharrington.com 192.168.88.1

The expected target for this lab is 10.1.35.7. My later CLI used the API IP directly.

10. Configure the namespace and its Public SubnetSet

#

At Supervisor → Configure → General → Kubernetes Service, I associated VKR-Subscribed-Library.

Close  ×

In Supervisor Management / Namespaces → Create Namespace, I selected vcf-mgmt-supervisor01, named the namespace homelab-vks-ns, and selected the zone containing the cluster. I left Override default network settings unchecked, then reviewed and created the namespace.

Close  ×

Then I configured the namespace resources:

If the VKS wizard cannot find the VM class, release, or storage, check these namespace associations before recreating the library or cluster.

Create public before creating VKS

#

In homelab-vks-ns → Resources → Network service → SubnetSets → New SubnetSet, I used these settings:

Close  ×

Close  ×

The 32 IPs setting requests a /27; fewer addresses are available for guest nodes. The screenshot does not show the allocated CIDR.

Public is a VPC access mode, not public internet exposure; these remain private lab addresses. The default private network was rejected without SNAT (source network address translation), so I used public.

11. Create VKS with Custom Configuration and verify it from Supervisor

#

In Resources → Kubernetes service → Create, I selected Custom Configuration to choose public as the primary node network.

Close  ×

General settings and storage

#

Close  ×

Close  ×

Cluster API manages the cluster using a ClusterClass topology. A PVC (PersistentVolumeClaim) requests application storage; its default storage class is separate from the node VM storage class.

I kept the generated name kubernetes-cluster-dhby in all commands. homelab-vks01 refers to the earlier failed deployment.

Primary network and replicas

#

Close  ×

Intermediate primary-network form before the MTU correction

Close  ×

Close  ×

If you do not change the network to public and leave the default private network selected, you will get this error at step 5, Review and Confirm, in this VPC namespace without SNAT:

Close  ×

I set the guest primary-interface MTU to 1500 because jumbo frames were unverified across the external VLAN path. The separate VDS, vMotion, and vSAN settings remained 9000.

The default YAML showed Service CIDR 10.96.0.0/12 and domain cluster.local; I did not verify these against the full runtime configuration.

Close  ×

Close  ×

With public selected, I reviewed and created the cluster, then confirmed its status was Available:

Close  ×

Authenticate to Supervisor and select its namespace

#

The VMware Supervisor CLI reference distinguishes three context scopes:

Mac terminal — create the Supervisor context:

vcf context create supervisor-redeploy \
  --endpoint https://10.1.35.7 \
  --username [email protected] \
  --auth-type basic \
  --type k8s \
  --insecure-skip-tls-verify

Enter the SSO password at the prompt. This lab command skips TLS certificate verification; omit --insecure-skip-tls-verify when your client trusts the endpoint certificate.

Close  ×

Login succeeded and the contexts were saved despite a Darwin ARM64 telemetry:v9.0.2 lookup returning MANIFEST_UNKNOWN.

Mac terminal — Supervisor namespace context:

vcf context use supervisor-redeploy:homelab-vks-ns
kubectl config current-context
kubectl get clusters -n homelab-vks-ns

Close  ×

A Harbor plugin-source discovery warning also appeared, but the cluster query succeeded. The warning remained unresolved; I did not need to install Harbor for this query.

Close  ×

Both guest-node VMs were powered on, though the nested ESXi hosts still showed warnings.

If the cluster is still reconciling, use these optional diagnostics; not all were run during this deployment.

Mac terminal — optional diagnostics in the Supervisor namespace context:

kubectl get clusters -n homelab-vks-ns -w
kubectl get machines -n homelab-vks-ns -o wide
kubectl get virtualmachines -n homelab-vks-ns -o wide
kubectl get events -n homelab-vks-ns --sort-by=.lastTimestamp
kubectl describe subnetset public -n homelab-vks-ns
kubectl get subnets -n homelab-vks-ns -o wide
kubectl get cluster kubernetes-cluster-dhby -n homelab-vks-ns -o yaml

Stop the watch with Ctrl+C, then run the remaining commands. Cluster creation continues.

12. Enter the guest cluster with its downloaded kubeconfig

#

With VKS Available, I connected to the guest Kubernetes API to verify its nodes and run a workload. The Supervisor context manages the VKS cluster object; the guest context accesses its nodes, pods, and Services. The vSphere namespace homelab-vks-ns and guest namespace lab-validation belong to different clusters.

From supervisor-redeploy:homelab-vks-ns, I tried switching to the expected guest context:

Mac terminal — switch to the guest context:

vcf context use supervisor-redeploy:homelab-vks-ns:kubernetes-cluster-dhby
kubectl config current-context

Close  ×

The guest context was not found. A telemetry:v9.0.2 lookup also returned MANIFEST_UNKNOWN, but that does not establish why the context was missing. I checked the available contexts:

Mac terminal — inspect VCF contexts:

vcf context list

Close  ×

Instead, I downloaded the kubeconfig from nested vCenter → homelab-vks-ns → Resources → Kubernetes Service → kubernetes-cluster-dhby → Download Kubeconfig File. Keep this credential file private and use your own cluster name and file path.

Mac terminal — locate the downloaded file:

ls -lt ~/Downloads/*kube*

I first used --kubeconfig to query the guest cluster explicitly:

Mac terminal — guest-cluster access with explicit kubeconfig:

kubectl --kubeconfig="$HOME/Downloads/kubernetes-cluster-dhby-kubeconfig.yaml" config get-contexts
kubectl --kubeconfig="$HOME/Downloads/kubernetes-cluster-dhby-kubeconfig.yaml" get nodes -o wide
kubectl --kubeconfig="$HOME/Downloads/kubernetes-cluster-dhby-kubeconfig.yaml" get pods -A -o wide

Close  ×

The context was kubernetes-cluster-dhby-admin@kubernetes-cluster-dhby. After the queries succeeded, I selected this kubeconfig for the session:

Mac terminal — select the guest cluster for subsequent commands:

export KUBECONFIG="$HOME/Downloads/kubernetes-cluster-dhby-kubeconfig.yaml"

All kubectl commands below use this export; repeat it in a new terminal. It leaves the default kubeconfig unchanged and bypasses the missing VCF guest context.

13. Check nodes, system pods, storage classes, and resource use

#

I checked guest-cluster readiness before deploying the test workload.

Mac terminal — guest-cluster context:

kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl get storageclass
kubectl top nodes

Close  ×

Both nodes ran Photon OS on amd64. The worker’s role column showed <none>; its node-pool configuration identified it as the worker. Generated node names change when the cluster is recreated.

The system-pod query showed 20 pods Running, with all listed containers ready, summarized below. Figure A05 shows the other checks.

Some pods had earlier restarts, but none showed an active crash loop. Antrea provides networking, CoreDNS handles DNS, metrics-server reports resource use, and vSphere CSI integrates storage. The workload below tests these functions beyond pod readiness.

I had two AutoRAID storage classes:

Both use csi.vsphere.vmware.com, support volume expansion, and reclaim volumes with Delete. Immediate binds or provisions storage when the PVC is created; WaitForFirstConsumer waits for a pod’s scheduling requirements. The manifest below uses the default class, so check yours before applying it. See the storage-class documentation.

Checkpoint: nodes and system containers are ready, storage is available, and metrics work. Resource figures are snapshots, not capacity benchmarks.

14. Create the BusyBox and PVC validation workload

#

I used a BusyBox pod and PVC in the guest namespace lab-validation to test image retrieval, networking, and storage.

Paste the entire block into the guest-cluster terminal, including the closing EOF. Scroll to view the manifest; Copy copies the entire block.

Mac terminal — guest-cluster context:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: lab-validation
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-storage
  namespace: lab-validation
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: lab-test
  namespace: lab-validation
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 1000
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: test
      image: busybox:1.37
      command: ["sh", "-c", "sleep 86400"]
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
      resources:
        requests:
          cpu: 10m
          memory: 16Mi
        limits:
          cpu: 100m
          memory: 64Mi
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: test-storage
EOF

The pod mounts a 1 GiB, ReadWriteOnce volume at /data, using the default latebinding AutoRAID class. It runs as nonroot UID/GID 1000 with fsGroup: 1000, RuntimeDefault seccomp, no privilege escalation, and all Linux capabilities dropped. See the security-context documentation.

I waited for readiness, then checked the pod and PVC:

Mac terminal — guest-cluster context:

kubectl wait -n lab-validation --for=condition=Ready pod/lab-test --timeout=180s
kubectl get pods,pvc -n lab-validation -o wide

Close  ×

The wait timed out with lab-test in ImagePullBackOff, while test-storage was Bound. I inspected the pod to isolate the image-pull failure:

Mac terminal — inspect the failing pod:

kubectl describe pod lab-test -n lab-validation

Events showed SuccessfulAttachVolume: scheduling and volume attachment had succeeded. The image had not been pulled, so I investigated DNS before testing writes to the volume.

15. Diagnose the image-pull timeout and permit lab DNS on MikroTik

#

In the Events section at the bottom of the kubectl describe pod lab-test -n lab-validation output from the previous step, I found this Docker Hub DNS timeout:

Failed to pull image "busybox:1.37":
failed to pull and unpack image "docker.io/library/busybox:1.37":
failed to resolve image: failed to do request:
Head "https://registry-1.docker.io/v2/library/busybox/manifests/1.37":
dial tcp: lookup registry-1.docker.io on 127.0.0.53:53:
read udp 127.0.0.1:44928->127.0.0.53:53: i/o timeout

The node’s local resolver stub, 127.0.0.53, timed out resolving Docker Hub before connecting to the registry. The error does not identify the upstream resolver or indicate an authentication or rate-limit failure.

I inspected the MikroTik configuration. Use your router’s address and login:

Mac terminal — connect to MikroTik:

ssh [email protected]

MikroTik RouterOS terminal — inspect DNS, firewall, interfaces, and NAT:

/ip/dns/print
/ip/firewall/filter/print detail
/interface/list/member/print
/ip/firewall/nat/print detail

DNS was already enabled with allow-remote-requests=yes and dynamic upstream servers 206.225.75.226 and 206.225.75.225. WAN masquerade was also configured; its presence alone did not verify internet access.

The input rule defconf: drop all not coming from LAN matched in-interface-list=!LAN, and vlan2006-external was missing from the LAN list:

Close  ×

Queries to the router use its input chain; forwarding and NAT rules do not override it. I allowed DNS from 10.1.35.0/24 on vlan2006-external before the input drop.

Run each line once, after checking the interface, subnet, and drop-rule comment against your router. The place-before selector must match the intended drop rule. DNS needs both UDP and TCP port 53; see MikroTik’s DNS documentation.

MikroTik RouterOS terminal — allow lab DNS:

/ip/firewall/filter/add chain=input action=accept in-interface=vlan2006-external src-address=10.1.35.0/24 protocol=udp dst-port=53 place-before=[find where comment="defconf: drop all not coming from LAN"] comment="VKS lab DNS UDP"

/ip/firewall/filter/add chain=input action=accept in-interface=vlan2006-external src-address=10.1.35.0/24 protocol=tcp dst-port=53 place-before=[find where comment="defconf: drop all not coming from LAN"] comment="VKS lab DNS TCP"

With DNS allowed only from the lab interface and subnet, I watched the existing pod retry its image pull.

Mac terminal — guest-cluster context:

kubectl get pod lab-test -n lab-validation -w

Close  ×

The pod reached 1/1 Running with zero restarts. Press Ctrl+C to stop the watch. If your pod remains in backoff, run kubectl describe pod lab-test -n lab-validation again and check the latest error.

Recovery after the firewall change supports the DNS diagnosis. The BusyBox pull and later NGINX rollout were functional checks; I did not capture firewall counters or a packet trace.

16. Prove cluster DNS and a write/read on the mounted volume

#

With BusyBox running, I tested cluster DNS:

Mac terminal — run inside the guest test pod:

kubectl exec -n lab-validation lab-test -- nslookup kubernetes.default.svc.cluster.local

Close  ×

DNS server 10.96.0.10 resolved kubernetes.default.svc.cluster.local to 10.96.0.1. BusyBox ran on the worker and CoreDNS on the control plane, exercising DNS across nodes. This tested an internal name, not an external-domain lookup.

These guest addresses do not verify the full Service CIDR or resolve the earlier Supervisor CIDR discrepancy.

Next I wrote a file to /data and read it back.

Mac terminal — run inside the guest test pod:

kubectl exec -n lab-validation lab-test -- sh -c 'echo "VCF 9.1.1 VKS storage test passed" > /data/test.txt && cat /data/test.txt'

Close  ×

The bound PVC, successful attachment, and write/read confirmed provisioning and basic storage I/O. Persistence across pod replacement, failure recovery, and performance were not tested.

17. Deploy NGINX and open it through a LoadBalancer

#

I deployed one NGINX replica in lab-validation with a LoadBalancer Service for access from my Mac. It does not use the BusyBox PVC.

The unprivileged NGINX image listens on 8080 and supports the nonroot settings below. I used the moving tag stable-alpine; its digest was not captured.

Paste the entire block, including EOF, into the guest-cluster terminal. The lab-validation namespace must already exist. Scroll to view the full manifest; Copy copies the entire block.

Mac terminal — guest-cluster context:

kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: lab-web
  namespace: lab-validation
spec:
  replicas: 1
  selector:
    matchLabels:
      app: lab-web
  template:
    metadata:
      labels:
        app: lab-web
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 101
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: nginx
          image: nginxinc/nginx-unprivileged:stable-alpine
          ports:
            - containerPort: 8080
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          resources:
            requests:
              cpu: 25m
              memory: 32Mi
            limits:
              cpu: 250m
              memory: 128Mi
          readinessProbe:
            httpGet:
              path: /
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: lab-web
  namespace: lab-validation
spec:
  type: LoadBalancer
  selector:
    app: lab-web
  ports:
    - name: http
      port: 80
      targetPort: 8080
EOF

The Service selects app: lab-web and forwards 80/TCP → 8080. The pod runs as nonroot UID 101 with restricted privileges and an HTTP readiness probe.

I checked the rollout:

Mac terminal — guest-cluster context:

kubectl rollout status deployment/lab-web -n lab-validation --timeout=180s

Close  ×

After the rollout succeeded, I checked the Service address:

Mac terminal — guest-cluster context:

kubectl get service lab-web -n lab-validation -w

Close  ×

Press Ctrl+C once the address appears, then open your Service’s assigned IP. Check it again if you recreate the Service.

Mac browser — open the assigned address:

http://10.1.35.34

The application path was Mac browser → 10.1.35.34:80 → lab-web Service → NGINX pod on 8080. Browser access uses Service port 80, not NodePort 30703.

Close  ×

NGINX was reachable from my home network through the VKS LoadBalancer. This tested private lab HTTP access; public hosting and HTTPS were outside the scope.

18. Results, practical lessons, and the next application

#

The rebuild reached these milestones:

This was a single-physical-host nested lab with one Supervisor control-plane VM and one VKS control plane and worker. It demonstrated basic functionality, not production availability, performance, or recovery.

Future work: a custom application

#

Next, I plan to deploy a custom application on this VKS cluster using an AMD64-compatible container image.

19. References

#

Sources for the deployment workflow and supporting concepts: