#Upgrading Clusters on Huawei DCS

This guide explains how to complete Phase 2 of the upgrade workflow for clusters on Huawei DCS. Before you upgrade Kubernetes, complete the Distribution Version upgrade described in Upgrading Clusters.

#Upgrade Sequence

Upgrade DCS clusters in the following order:

1. (Prerequisite) Upgrade the ACP platform on the management cluster first. This brings the cluster-api-provider-dcs controller and the related CAPI components (core, KubeadmControlPlane provider, bootstrap provider) to versions that understand the new schema. Trigger workload-cluster upgrades only after the management-side controllers have rolled out and become Ready.
2. Upgrade the Distribution Version (Aligned Extensions) on the workload cluster. See Upgrading Distribution Version.
3. Upgrade Kube-OVN to the chart version required by the target ACP release and wait for the AppRelease to reach Success.
4. Upgrade the control plane Kubernetes version.
5. Upgrade worker nodes to the target Kubernetes version.

Cluster API orchestrates rolling updates with built-in safety mechanisms to reduce service disruption.

#Prerequisites

Before you start, ensure all of the following prerequisites are met:

• The Distribution Version upgrade is complete
• The control plane is reachable
• All nodes are healthy and in Ready state
• A current etcd backup has been taken and verified by using the supported ACP backup procedure
• The IP Pool has sufficient capacity for rolling updates
• The VM template supports the target Kubernetes version. See OS Support Matrix for version mapping
• For cross-version upgrades that span more than one Kubernetes minor, the intermediate-version Core images and VM templates are pre-staged. See Cross-Version Upgrade Preparation
• The target Kubernetes version is compatible with your workloads and add-ons
• DCS VM templates are 4.2.1 or later if you use pool-managed persistent disks, because safe shutdown and disk detach depend on guest tools
• If you rely on pool-managed persistent disks, keep KubeadmControlPlane.spec.rolloutStrategy.rollingUpdate.maxSurge = 0 and each MachineDeployment.spec.strategy.rollingUpdate.maxSurge = 0
• If the cluster uses additional NICs, keep the required DCSIpHostnamePool.spec.pool[].additionNic[] entries in place and confirm that the DCS DVS and Port Groups are reachable from the hosts that may receive replacement VMs

#Using YAML

YAML-based upgrades do not depend on Fleet Essentials.

#Required Values From the OS Support Matrix

The authoritative mapping between an ACP release, its Alauda OS image, the Kubernetes version, the matching CoreDNS, etcd, and Kube-OVN versions lives in OS Support Matrix. Locate the row that corresponds to the target ACP version before you start; the row supplies every value the YAML steps below need.

The cells you read from that row map to the upgrade manifests as follows:

The CoreDNS and etcd image tags are control-plane-only because clusterConfiguration is a KubeadmControlPlane field. Worker nodes inherit container image versions from the new VM template; the MachineDeployment does not carry its own dns/etcd tags. The Kube-OVN annotation lives on the Cluster resource, not on KubeadmControlPlane, because the DCS provider watches it independently of the Kubernetes control plane rollout.

Confirm with the cluster's platform owner that the target Alauda OS image has already been uploaded to the DCS platform under the same name as the Alauda OS Image Version value in the matrix row. The upgrade fails if that VM template is not present on DCS when the DCSMachineTemplate is applied.

#Upgrade Kube-OVN Before the Control Plane

Follow Upgrade Kube-OVN Before the Control Plane and select the Huawei DCS procedure that matches the installed DCS provider version. That shared procedure owns the provider version boundary, the Kube-OVN chart-name migration at v4.4, legacy behavior, and the required AppRelease health checks.

Do not use the legacy direct targetRevision patch for a Kube-OVN v4.4+ target. Upgrade the DCS provider to v1.0.22 or later first so it can reconcile the complete source and the related component repository changes.

#Upgrade Control Plane Infrastructure

Upgrading the control plane machine template lets you roll out updated VM specifications, system patches, and infrastructure settings.

#Procedure

1. Create an updated machine template

   Copy the existing DCSMachineTemplate referenced by KubeadmControlPlane and save it as a new file:

   kubectl get dcsmachinetemplate <current-template-name> -n cpaas-system -o yaml > new-cp-template.yaml

2. Modify the template specifications

   Update the new template as needed:

   • Set metadata.name to <new-template-name>
   • Update spec.template.spec.vmTemplateName
   • Update spec.template.spec.vmConfig.dcsMachineCpuSpec.quantity
   • Update spec.template.spec.vmConfig.dcsMachineMemorySpec.quantity
   • Update spec.template.spec.vmConfig.dcsMachineDiskSpec for system and template-local disks only
   • Strip server-generated metadata (resourceVersion, uid, generation, creationTimestamp, managedFields, kubectl.kubernetes.io/last-applied-configuration annotation) and the entire status field from the copied manifest.
   • Leave spec.template.spec.providerID unset. The DCS provider sets providerID to dcs://<machine-name> once the VM is created; pre-filling it in the template breaks the controller's identity binding.

   Keep pool-managed persistent disks, including /var/cpaas, in DCSIpHostnamePool.spec.pool[].persistentDisk. Keep additional NICs in DCSIpHostnamePool.spec.pool[].additionNic[].

3. Apply the updated template

   kubectl apply -f new-cp-template.yaml -n cpaas-system

4. Update the control plane reference

   Modify the KubeadmControlPlane resource to reference the new template:

   kubectl patch kubeadmcontrolplane <kcp-name> -n cpaas-system --type='merge' -p='{"spec":{"machineTemplate":{"infrastructureRef":{"name":"<new-template-name>"}}}}'

5. Monitor the rolling update

   kubectl get kubeadmcontrolplane <kcp-name> -n cpaas-system -w
   kubectl get machines -n cpaas-system -l cluster.x-k8s.io/control-plane

#Upgrade Control Plane Kubernetes Version

Upgrading the control plane Kubernetes version on immutable OS is a delete-recreate workflow. The control plane VMs are replaced one by one from a new DCSMachineTemplate that points at the target Alauda OS VM template, and the KubeadmControlPlane resource is patched to carry the matching Kubernetes version, CoreDNS image tag, and etcd image tag.

Before you start, complete the shared Upgrade Kube-OVN Before the Control Plane procedure and verify that the cni-kube-ovn AppRelease is at the target revision with phase=Success. Then collect every required control-plane value from the target ACP row in the OS Support Matrix as described in Required Values From the OS Support Matrix.

#Procedure

1. Create a new DCSMachineTemplate for the target Kubernetes version

   Copy the existing control-plane template and update metadata.name to a new name and spec.template.spec.vmTemplateName to the Alauda OS Image Version value read from the target row in the OS Support Matrix. Keep pool-managed persistent disks in DCSIpHostnamePool.spec.pool[].persistentDisk and additional NICs in DCSIpHostnamePool.spec.pool[].additionNic[] rather than reintroducing them as template fields.

   kubectl get dcsmachinetemplate <current-cp-template-name> -n cpaas-system -o yaml > new-cp-template.yaml
   # edit metadata.name and spec.template.spec.vmTemplateName
   kubectl apply -f new-cp-template.yaml -n cpaas-system

2. Patch the KubeadmControlPlane with the target Kubernetes values

   Update the KubeadmControlPlane resource in a single edit to keep spec.version, the CoreDNS image tag, the etcd image tag, and the infrastructure template reference consistent with the same Alauda OS release:

   • spec.version ← Kubernetes Version from the OS Support Matrix row

   • spec.kubeadmConfigSpec.clusterConfiguration.dns.imageTag ← coredns column from the same row

   • spec.kubeadmConfigSpec.clusterConfiguration.etcd.local.imageTag ← etcd column from the same row

   • spec.machineTemplate.infrastructureRef.name ← the new DCSMachineTemplate name created in step 1

   • When the target is Kubernetes 1.35 or later, update /etc/kubernetes/patches/kubeletconfiguration0+strategic.json in spec.kubeadmConfigSpec.files in this same edit, as described in Required kubelet patch for Kubernetes 1.35

     kubectl edit kubeadmcontrolplane <kcp-name> -n cpaas-system

   Updating only spec.version is not sufficient. The CoreDNS and etcd image tags must move together with the Kubernetes version because they are built from the same Alauda OS release; leaving them at the previous values can result in CoreDNS and etcd pods that do not match the new Kubernetes minor version.

3. Monitor the rolling update

   kubectl get kubeadmcontrolplane <kcp-name> -n cpaas-system -w
   kubectl get machines -n cpaas-system -l cluster.x-k8s.io/control-plane
   kubectl get nodes

   KubeadmControlPlane.spec.rolloutStrategy.rollingUpdate.maxSurge must remain 0 when the cluster relies on pool-managed persistent disks, so the control plane VMs are replaced one at a time.

#Upgrade Worker Nodes

Worker node Kubernetes upgrades are managed through MachineDeployment resources. Worker upgrades carry fewer fields than the control plane: the CoreDNS and etcd image tags are part of KubeadmControlPlane.spec.kubeadmConfigSpec.clusterConfiguration, which MachineDeployment does not have. Worker nodes inherit Kubernetes component versions from the new VM template; the MachineDeployment only needs the target Kubernetes version and the new template reference.

Before you start, read the Alauda OS Image Version and Kubernetes Version cells from the target ACP row in the OS Support Matrix as described in Required Values From the OS Support Matrix.

#Procedure

1. Create a new DCSMachineTemplate for worker nodes

   • Create a new DCSMachineTemplate with a vmTemplateName set to the Alauda OS Image Version value from the target row in the OS Support Matrix
   • Keep /var/cpaas and any other upgrade-preserved disks in DCSIpHostnamePool.spec.pool[].persistentDisk rather than reintroducing them as template disks
   • Keep additional NICs in DCSIpHostnamePool.spec.pool[].additionNic[]

2. Update the MachineDeployment

   • Set spec.template.spec.version to the Kubernetes Version value from the same OS Support Matrix row
   • Set spec.template.spec.infrastructureRef.name to the new DCSMachineTemplate name created in step 1
   • When the target is Kubernetes 1.35 or later, create a new KubeadmConfigTemplate containing the required kubelet patch and set spec.template.spec.bootstrap.configRef.name to that template in the same edit; for earlier versions, update the bootstrap reference only when other bootstrap changes are required

3. Monitor the rolling update

   • Verify that the rolling update completes successfully
   • Verify that the new worker nodes join the cluster with the target Kubernetes version

   MachineDeployment.spec.strategy.rollingUpdate.maxSurge must remain 0 when the cluster relies on pool-managed persistent disks, so the worker nodes are replaced one at a time.

#Recovering From a Failed Phase 2 Upgrade

Do not treat a Kubernetes minor downgrade as an ordinary rollback. Choose the recovery path from the rollout stage:

1. No target-version control-plane Machine has been created: restore the previous Kube-OVN state and the previous KubeadmControlPlane and MachineDeployment manifest values. This cancels the target rollout before a new control-plane data format is introduced.
2. Only the machine template or OS image changed, and the Kubernetes minor did not change: point the controlling resource back to the previous template. Cluster API performs another replacement rollout. Keep the Kubernetes minor unchanged.
3. A control-plane Machine on the target Kubernetes minor has joined the cluster: do not patch Kubernetes, CoreDNS, or etcd back to the previous minor. Stop further rollout, repair forward on the target minor, or restore the cluster from the verified pre-upgrade backup by using the supported ACP recovery procedure.

If a target-minor control-plane Machine was created but never joined, first restore healthy etcd quorum and determine whether the failed replacement can be removed safely. Do not assume that changing the version fields alone is sufficient.

Keep these infrastructure facts in mind during any recovery:

• The old VMs are gone. They were destroyed during the upgrade. Template recovery builds a fresh set of replacement machines; it does not restore the original VMs.
• The old DCSMachineTemplate resource must still exist. Do not delete the previous template until the new rollout is healthy. If you already deleted it, recreate it from version control or backup before attempting same-minor template recovery.
• Pool-managed disk identity is preserved, but data state is not. Disks declared in DCSIpHostnamePool.spec.pool[].persistentDisk reattach to the replacement machines at the same IP slot, but data written during the upgrade window remains on those disks.

For stage 1, use the DCS rule in Restore Kube-OVN During Stage-1 Recovery. The recovery action depends on the installed provider version: current providers restore the complete source from the annotation, while legacy providers require the earlier-chart revision patch. Wait until the restored AppRelease passes the shared verification before changing control-plane manifests.

The KubeadmControlPlane controller can block replacement while etcd is unhealthy. Recover quorum before retrying any safe replacement action.

---

#Fleet Essentials Boundary

Fleet Essentials 1.0.4 and later can request the ACP 4.3-and-later Distribution Version upgrade through CVO. That is Phase 1 of this guide. Fleet Essentials does not perform the DCS Kubernetes and Alauda OS replacement in Phase 2, and it does not manage DCSIpHostnamePool.spec.pool[].persistentDisk. Use the YAML procedure on this page for the complete Phase 2 rollout.

Cluster creation and routine node-pool management through Fleet Essentials are outside this upgrade limitation.

#Additional Resources

• Migrate Existing Huawei DCS Clusters to Pool-Managed Persistent Disks
• Cluster API Upgrade Documentation
• Kubernetes Version Skew Policy