Upgrading VCF 9.0.x to 9.1: What Actually Matters

Table of Contents

When I started preparing for VCF 9.1, I expected a familiar maintenance window: download the binaries from the depot, run the prechecks, and work through the component upgrades. Easy day—basically the same rhythm we followed getting to 9.0.2.

That was not quite how this one went. The component upgrades were only half the work. VCF 9.1 introduces a new Management Services architecture, additional IP pools and DNS records, a License Server appliance, a new lifecycle inventory model, and a blue-green Automation migration. Most of the problems we encountered came from those dependencies rather than the upgrade buttons themselves.

This guidance applies to supported VCF 9.0.x-to-9.1 paths. Our starting build was 9.0.2, and the environment included VCF Operations, Automation, vCenter, NSX, ESX, and workload domains. The sequence below follows our path, including a stale SDDC Manager lock and failed Automation import that changed how I would prepare next time. Generate the Upgrade Planner path for your exact topology, but use these lessons to make its prompts and stopping points easier to understand.

Before the Window: Generate the Path and Finish the Network Design
#

Start by identifying the Identity Broker deployment
#

Before generating the plan, check Fleet Management → Identity & Access → VCF SSO Overview and identify the Identity Broker mode, network, and datastore. If a healthy 9.0 Identity Broker is already on the planned Management Services network, do not rebuild it or request another FQDN. Select its mode in the Upgrade Planner; the 9.1 workflow migrates an external appliance into Management Services and powers down the old VMs. Validate SSO before deleting them.

An appliance on a different network or unsupported datastore needs a separate, potentially disruptive transition. KB 444734 covers that exception, and KB 441285 collects known failures.

My starting point was the VCF Upgrade Planner path for VCF 9.0 to 9.1 with Automation. Thank you to William Lam for turning the product dependencies into an order that was easy to follow. Generate a path using the versions and optional products actually deployed, then stage everything the current workflow makes available—especially the Operations PAK. After Operations reaches 9.1, its new licensing interface provides the License Server OVA download; the remaining 9.1 binaries may not appear until SDDC Manager is also upgraded and the new Software Depot is configured. The first VCF instance needs the full fleet-level set; later instances need fewer components, as listed in KB 425089.

Use the VCF 9.1 Planning and Preparation Workbook to tie each FQDN to an address, VLAN, certificate, and owner. This caught more problems than confirming that the management /24 had free space. Download a clean workbook and adapt it to your environment.

The execution order that follows is deliberate—and Broadcom treats the management-component sequence as mandatory: upgrade VCF Operations with the product PAK, deploy and connect the License Server, precheck and upgrade SDDC Manager, configure the Software Depot, deploy Management Services, verify Fleet inventory, upgrade the management domain, and then migrate Automation and the remaining products. Workload domains can follow later as a Day‑N activity. The Management Services network detail appears with the phase that consumes it, but it should be completed and validated before the maintenance window.

Phase 1: Upgrade VCF Operations with the Product PAK
#

Sign in to the Broadcom Support Portal and download the VCF Operations product-upgrade PAK for the intended 9.1 build. Choose the Virtual Appliance package for the Operations cluster, or the combined Virtual Appliance and Cloud Proxy package when the environment also needs its Cloud Proxies updated. Do not use the Cloud Proxy-only PAK for the core cluster. Verify the downloaded file’s build and checksum before the window; KB 431008 explains the package variants and how the wrong one can appear to install without upgrading the product.

Open the VCF Operations administration interface directly at https://<vcf-operations-fqdn>/admin and sign in with the local Operations admin account. This is separate from the normal product interface at /ui. From the admin interface, select Software Update → Install a Software Update, upload the VCF Operations product-upgrade PAK—not a management-pack or adapter package—and monitor the installation. KB 428018 documents the same admin URL and upload path.

This step unlocks the remaining binary workflow. While Operations is on 9.0, SDDC Manager may offer 9.1 in Binary Management but show no Upgrade, Patch, or Install binaries. KB 440282 explains that after applying the PAK, you upgrade SDDC Manager, configure the new Software Depot, and retrieve the other component binaries.

Although the warning recommends Fleet Management, this transition still uses the direct PAK and later moves the 9.0 Fleet Management data into 9.1. Confirm that the Operations cluster is online, the PAK matches the target build, and required backups exist.

Next, deploy the License Server appliance
#

VCF Operations 9.1 introduces a new licensing interface. Go to Manage → Licensing → Licenses & Registration. The page provides the Broadcom portal link for downloading the License Server OVA and displays the unique registration key required during deployment.

Deploy the OVA to the management-domain vCenter using the reserved License Server FQDN and IP, and enter that registration key when prompted. The OVA deployment is a manual step. In connected mode, what happens after deployment is automated: the appliance connects to VCF Operations, associates with the VCF Business Services Console, registers, and retrieves the allocated licenses. Confirm that sequence completes before starting the vCenter and ESX phases.

Create both an A record and PTR record before deployment. If the environment uses 172.17.0.0/16 for DNS or another reachable network, use License Server 9.1.0.0200 or later; earlier builds can conflict with the appliance’s Docker network and boot to a black console. KB 441526 covers that condition.

Entitlements originate in the Business Services Console; Operations manages them, and the License Server connects them to vCenter, where components and hosts obtain their licenses. VCF 9.1 also automates the ongoing connected-mode exchange: usage is sent to Broadcom and an updated license file is downloaded and applied every 24 hours. In VCF 9.0 connected mode, an administrator still had to acknowledge the refreshed license file at least once every 180 days. Disconnected mode continues to use manually transferred usage and license files. See Broadcom’s VCF 9.1 licensing architecture article, its VCF 9.0 licensing comparison, and KB 437242.

Verify Manage → Licensing → Licenses & Registration, confirm the server is connected, and compare allocated capacity with managed cores. An underlicensed vCenter can block the ESX precheck; see KB 447737.

Phase 2: Precheck and Upgrade SDDC Manager, Then Deploy Management Services
#

Our SDDC Manager health check reported a resource lock with no active workflow in the UI. KB 439473 covers stale locks left by earlier tasks. Verify that no legitimate workflow owns the lock, follow the KB’s backup safeguards, and rerun the full health check after remediation.

Once the precheck passes, start the SDDC Manager 9.1 upgrade—do not stop at the green check. Monitor the task through completion, sign back in, and verify the reported version and overall health. Then configure the new Software Depot connection and retrieve the remaining binaries. That upgraded SDDC Manager and depot workflow is what enables the Management Services deployment and the component upgrades that follow.

What VCF Management Services actually is
#

VCF Management Services is a set of containerized services running on VCF Services Runtime, a Kubernetes-based platform deployed as VMs in the management domain. It is management infrastructure for VCF itself—not a general-purpose Kubernetes cluster for application workloads. Broadcom’s VCF Management Services documentation describes the architecture and its role in a VCF 9.1 deployment.

The main consolidation is lifecycle management. Responsibilities from the standalone 9.0 Fleet Management appliance move into Fleet Lifecycle for fleet-wide products and SDDC Lifecycle for each VCF instance. The platform also hosts Software Depot, Salt RaaS, Salt Master, and Telemetry. An existing 9.0 Identity Broker is migrated into it, while Real-Time Metrics and Log Management can be added as Management Services components.

VCF Operations, vCenter, NSX, Automation, and Operations for Networks remain separate products managed through this lifecycle model; they do not run inside Management Services. The License Server is also a separate appliance, deployed from the OVA downloaded through the VCF Operations licensing workflow described in Phase 1.

The first VCF instance carries the fleet-level services and has the largest binary and network footprint. Additional instances deploy a smaller set of instance-level services and use the fleet services established by the first instance. This architecture is why the upgrade needs a dedicated runtime IP pool plus separate fleet, instance, runtime, licensing, and optional-service FQDNs.

Enter the Management Services addresses
#

For the initial Management Services deployment, Broadcom documents a minimum of 12 addresses and allows another 18 to be added later for new services, scale-out, and future upgrades—30 addresses in total. All of these ranges must be on the management network, and later ranges do not have to be contiguous. For targets from 9.1.0.0 through 9.1.0.300, the UI expects an aligned /28 for the initial allocation or /27 for the full allocation. Starting with 9.1.0.400, the UI can exclude addresses from the CIDR or accept a comma-separated list of contiguous or non-contiguous addresses. If the management network cannot hold the range, an API-driven deployment can use a custom VLAN-backed network. KB 440223 explains the supported choices.

Enter the CIDR as an aligned network address/prefix, not the first usable host. For 192.0.2.160/28, .160 is the network address, .161–.174 are the 14 traditional host addresses, and .175 is broadcast; the workflow uses this block size for its 12-address minimum. A /28 has 16 total addresses, mask 255.255.255.240, and boundaries every 16 addresses.

For 192.0.2.160/27, .160 is the network address, .161–.190 are the 30 usable addresses, and .191 is broadcast. A /27 has 32 total addresses, mask 255.255.255.224, and boundaries every 32 addresses. Reserve the complete block in IPAM. Named endpoints remain outside this runtime CIDR.

The runtime range is only part of the request. The installer also needs unique, DNS-backed endpoints outside that range:

Installer endpoint	Services using the endpoint	Example FQDN	Example IP
Fleet components FQDN	Fleet Lifecycle, Salt RaaS, and Software Depot	`vcf-fleet-01.example.com`	`192.0.2.40`
Instance components FQDN	SDDC Lifecycle, Salt Master, metrics, and telemetry	`vcf-instance-01.example.com`	`192.0.2.41`
VCF services runtime FQDN	VCF Services Runtime	`vcf-runtime-01.example.com`	`192.0.2.42`
VCF Identity Broker FQDN—conditional	Identity Broker when one is not already deployed	`vcf-idb-01.example.com`	`192.0.2.43`
License Server FQDN	Local VCF License Server appliance	`vcf-license-01.example.com`	`192.0.2.44`
VCF Automation services runtime FQDN	Internal Automation services runtime	`vcfa-runtime-01.example.com`	`192.0.2.72`

Create forward and reverse records, keep the names lowercase, and verify that every endpoint resolves to its own address. The Identity Broker row applies only when one is not already deployed or its documented transition requires a new address.

This validation failed because the runtime and fleet fields were given the same FQDN. DNS therefore resolved both fields to 192.0.2.42, producing both a duplicate-name and duplicate-address error. Give the runtime, fleet, and instance endpoints separate FQDNs and separate IPs.

Also verify that every named endpoint and every address in the runtime block is actually unused before starting the deployment. Check IPAM and DNS, then use the network team’s normal duplicate-address checks—such as ARP or neighbor-table inspection from the local segment—rather than relying on ping alone. The installer can catch duplicates within its input and some DNS conflicts, but it should not be treated as the authoritative test for an address already assigned to another device.

VCF Services Runtime also uses an internal network, 198.18.0.0/15 by default. That is not the Management Services address pool, and it must not overlap anything routed in the environment. If it does, Broadcom documents alternative internal ranges and a JSON deployment method in KB 440541.

Validate placement against the latency diagram
#

The workbook tells you where components will live; the latency diagram tells you whether they should live there.

Source: Broadcom’s VMware Ports and Protocols network-diagram portal.

The diagram shows up to 300 ms for many fleet-level paths, but tighter limits of 50 ms around collectors and management components, 100 ms between vCenter and SDDC Manager or Supervisor, and 150 ms for several NSX Edge, workload-domain, Enhanced Linked Mode, and long-distance vMotion paths. It also calls for at least 10 Mbps on the illustrated NSX-to-ESX Edge path. These values should drive placement and inter-site testing. KB 412252 notes that installation may not enforce every limit, but exceeding them can still affect stability and performance.

Once prechecks were clean, we deployed the Management Services platform planned above. During this transition, the standalone 9.0 Fleet Management appliance data and responsibilities move into the new lifecycle model; the old appliance has no direct upgrade path. Do not use the 9.0 Fleet Management appliance to perform 9.1 upgrades. After validating Fleet Lifecycle and SDDC Lifecycle, the old appliance can be decommissioned.

One practical warning from the product guide: do not move the deployed Management Services VMs into another folder or resource pool. Later patch, scale-out, and deployment workflows depend on their expected placement.

Phase 3: Verify the Fleet Inventory Before Continuing
#

Management Services completed, but Automation did not import into Fleet Lifecycle. The application was running; lifecycle discovery was not.

The task reported that Automation could not be discovered through SSH.

A healthy application is not automatically ready for lifecycle management. Fleet Lifecycle must reach it, validate credentials and identity, associate the correct vCenter, and add it to inventory. KB 441858 distinguishes remediation for a failed deployment from one completed with warnings.

Account for every component before continuing and validate its certificate. VCF Operations for Networks, for example, requires every platform and collector FQDN and IP in the certificate SAN; see KB 424807. It cannot be upgraded from its own UI in this path; use the VCF Operations Fleet Lifecycle workflow. KB 440459 documents the related Fleet import failure when the certificate SAN is incomplete.

Phase 4: Upgrade the Management Domain and Validate Its Consumers
#

Follow the planner’s component order. In our path, every core phase had a precheck and validation gate before the next dependency.

A mixed-version state is normal during the plan, but do not leave it there unnecessarily. After vCenter and NSX, I checked DNS, certificates, provider connections, north-south and east-west traffic, and an Automation provisioning operation. The phase was complete only when dependent services still worked.

Our workloads remained online while ESX hosts were evacuated and upgraded one at a time; the main interruption was brief management-plane availability around vCenter. This is not a guarantee—cluster capacity, pinned workloads, storage, NSX, and patch method change the impact.

The ESX prechecks surfaced the same Intel volume-management warning seen previously. Acknowledging that known warning cleared the precheck; no separate driver, firmware, or VIB fix was applied. Record and understand a warning before accepting it: acknowledgment releases the workflow but does not prove the underlying condition is fixed.

Phase 5: Upgrade Automation and Finish the Remaining Products
#

Automation uses a blue-green migration: the new 9.1 nodes are built while the 9.0.x nodes remain available, then the old nodes are shut down after the migration completes. Their addresses cannot be reused at the start.

For targets through 9.1.0.300, reserve an eight-address /29. Three addresses are assigned to the new Automation nodes, two are retained for failed-node replacement or rolling maintenance, and three remain unused. The Automation services-runtime FQDN needs another address outside the /29, making the planning requirement nine addresses in total.

Starting with 9.1.0.400, the unused three addresses are no longer required. Provide five unique addresses—three for the Automation nodes and two for replacement or rolling-maintenance capacity. Those five can be contiguous or non-contiguous and can be supplied using the newer address-selection options. The separate services-runtime FQDN still needs its own address, making the 9.1.0.400-and-later planning requirement six addresses in total.

The existing user-facing Automation FQDN and load-balancer VIP stay the same and transfer to the 9.1 deployment; the new runtime FQDN is an internal endpoint. Because users, integrations, and saved Automation workflows continue to use the same front-end address, they do not need to be rebuilt or repointed after the migration. The blue-green design minimizes disruption, although I would still avoid scheduling critical in-flight workflow executions during the final cutover and validation period.

After migration, validate cloud accounts, regions, projects, quotas, namespaces, identity, and one representative provisioning workflow.

VCF Operations for Logs is not the new Log Management deployment
#

An existing VCF Operations for Logs deployment does not become 9.1 Log Management in place. Log Management is a separate Management Services deployment with a new DNS-backed VIP/FQDN. In our case, it was one of the last steps because we were waiting for that record.

Reserve a record such as vcf-logs-01.example.com with forward and reverse resolution and include it in the certificate SAN. Do not reuse the Operations FQDN or assume the old Operations for Logs name satisfies the prompt. Additional Log Management VIPs also need SAN entries. Finish other optional products in the planner’s order and confirm each one joins Fleet inventory.

Existing logs do not have to be abandoned. After deploying Log Management 9.1, use the supported Transfer Log Data workflow to move data from the existing VCF Operations for Logs instance into the new Log Management deployment. Broadcom includes this transition in its VCF 9.1 upgrade guidance. Plan the transfer separately from deploying the new service, and validate retention, available capacity, and access to historical searches afterward.

What I Would Carry into the Next Upgrade
#

My most useful artifact was a control sheet containing the planner order, FQDNs and addresses, certificate SANs, Management Services and Automation ranges, latency results, prerequisite owners, and a validation gate for each phase. I would not carry an unexplained lock, missing Fleet component, certificate mismatch, or failed provider connection into the next phase.

VCF 9.1 brings useful capabilities after the lifecycle work is finished. VCF Networking Automation adds services for distributed VPCs, including distributed Transit Gateway connectivity for Supervisor and Automation, a VNA-backed Layer 4 load balancer, and VPC VLAN extension. VKS and VM Fast-Deploy can use linked clones, and namespace consumption is more streamlined for self-service Kubernetes. Eligible TPM-backed hosts can also use live ESX patching for supported patches. Broadcom summarizes those additions in VCF 9.1 What’s New.

Accurate DNS, sufficient addresses, measured latency, clean inventory, valid certificates, and a check after each dependency changed kept the upgrade predictable.

References Used During the Upgrade
#

Always confirm that a KB and its remediation apply to the exact VCF build and task state before using it.

Before the Window: Generate the Path and Finish the Network Design #

Start by identifying the Identity Broker deployment #

Phase 1: Upgrade VCF Operations with the Product PAK #

Next, deploy the License Server appliance #

Phase 2: Precheck and Upgrade SDDC Manager, Then Deploy Management Services #

What VCF Management Services actually is #

Enter the Management Services addresses #

Validate placement against the latency diagram #

Phase 3: Verify the Fleet Inventory Before Continuing #

Phase 4: Upgrade the Management Domain and Validate Its Consumers #

Phase 5: Upgrade Automation and Finish the Remaining Products #

VCF Operations for Logs is not the new Log Management deployment #

What I Would Carry into the Next Upgrade #

References Used During the Upgrade #