#Upgrading Clusters on Bare Metal

This guide explains how to complete Phase 2 of the upgrade workflow for clusters on bare metal. Before you upgrade Kubernetes, complete the Distribution Version upgrade described in Upgrading Clusters.

#Key Considerations

Bare-metal upgrades replace every node — they do not run kubeadm upgrade on the existing OS. The mechanism is:

1. CAPI deletes one Machine according to the rollout strategy.
2. The provider writes a clean plan to the inventory's plan secret.
3. The host stops kubelet + CRI workload + containerd, then returns to Available in the same pool.
4. CAPI creates a replacement Machine.
5. The provider picks an Available inventory from the same pool, resolves the new Kubernetes version against elemental-image-catalog, and writes a reprovision plan.
6. The host runs cloud-init clean → elemental upgrade --system <new-image> → reboot. initramfs clears Kubernetes persistent state; cloud-init re-executes and runs kubeadm init / kubeadm join against the new control plane.

Two structural consequences operators must internalize before starting:

• Bare-metal does not preserve Kubernetes-managed disk state. /var/lib/kubelet, /var/lib/containerd, /var/lib/etcd, /etc/kubernetes are cleared by the initramfs cleanup step of every reprovision. There is no equivalent of the DCS provider's pool-managed persistent disks today.
• The same MachineInventory may not be re-picked. The provider does not guarantee that the inventory released by a clean plan is the same one re-allocated to the replacement Machine. Replacements are performed as delete-then-add when maxSurge=0 (no overlap between old and new node), so pool capacity sizing need only accommodate the desired replica count — old and new nodes do not coexist during the rollout.

#Upgrade Sequence

Upgrade bare-metal clusters in the following order:

1. (Prerequisite) Upgrade the ACP platform on the global cluster. This brings the bare-metal provider, elemental-operator, and the related CAPI components to versions that understand the new schema. Trigger workload-cluster upgrades only after the management-side controllers have rolled out and become Ready.
2. Upgrade the Distribution Version (Aligned Extensions) on the workload cluster. See Upgrading Distribution Version.
3. Ensure the target Kubernetes version is present in elemental-image-catalog (and the matching -iso repository is available for any future host re-registration).
4. Upgrade Kube-OVN to the chart version required by the target ACP release and wait for the AppRelease to reach Success.
5. Upgrade the control plane Kubernetes version (replaces all control-plane nodes one at a time).
6. Upgrade worker nodes to the target Kubernetes version (replaces all worker nodes within the maxUnavailable budget).

Cluster API orchestrates the rolling replacement.

#Prerequisites

Before you start, ensure all of the following prerequisites are met:

• The Distribution Version upgrade is complete.
• The installed Bare Metal provider is v0.0.1 or later and its AppRelease reports the same requested and installed revision with phase=Success.
• The control plane is reachable through the existing <control-plane-vip>:<control-plane-port>.
• All current nodes are healthy and Ready.
• The target Kubernetes version is a key in elemental-image-catalog. If it is not, add it before starting (see Update the Image Catalog).
• The target kube-ovn (chart) version has been read from the matching ACP row in the OS Support Matrix.
• BaremetalCluster.spec.networkType is kube-ovn; the provider skips Kube-OVN reconciliation for any other value.
• The platform registry is reachable from every host in the cluster.
• Both KubeadmControlPlane.spec.rolloutStrategy.rollingUpdate.maxSurge = 0 and MachineDeployment.spec.strategy.rollingUpdate.maxSurge = 0 are set — bare-metal does not over-provision physical hosts.
• The relevant pools have enough capacity to replace one node at a time without falling below the desired replica count.

#Update the Image Catalog

elemental-image-catalog is the resource that introduces a Kubernetes version to the bare-metal provider.

Option A — Add a chart override. Append the new version under provider.imageCatalog.images (uses global.registry.address as the registry):

provider:
  imageCatalog:
    images:
      <new-kubernetes-version>:
        repository: tkestack/baremetal-base-image
        tag: <official-image-tag>

Reapply the bare-metal provider plugin. The chart re-renders the ConfigMap; the provider's in-process watch hot-reloads the cache without restarting.

Option B — Patch the ConfigMap directly. Use this only for an approved image reference that must be pinned directly in the catalog:

kubectl -n cpaas-system patch configmap elemental-image-catalog \
  --type='json' \
  -p='[{"op":"add","path":"/data/<new-kubernetes-version>","value":"<registry-address>/tkestack/baremetal-base-image:<new-tag>"}]'

In either case, verify before continuing:

kubectl -n cpaas-system get configmap elemental-image-catalog -o yaml

The key must include the leading v (for example v1.34.5). If the key is missing when CAPI creates the replacement Machine, the resulting BaremetalMachine enters Failed with ImageResolved=False / Reason=ImageCatalogMiss and no reprovision plan is written — adding the key later restarts the reconciliation automatically.

When the new version corresponds to a new Alauda OS / base-image release, the ISO variant for that release should also be available before the upgrade. The bare-metal provider does not rebuild SeedImages automatically, but you may want to refresh MachineRegistration / SeedImage for future host onboarding so the new hosts come up on the new release. The ISO repository is the same as the base-image repository with -iso appended.

#Upgrade Kube-OVN Before the Control Plane

Follow the Bare Metal v0.0.1+ flow in Upgrade Kube-OVN Before the Control Plane. Bare Metal has no earlier supported provider flow in this documentation.

The shared procedure verifies BaremetalCluster.spec.networkType, updates the CAPI Cluster annotation, lets the provider reconcile the complete chart source and specification, and checks the chart name, requested revision, installed revision, phase, and conditions. Do not patch the AppRelease source manually.

#Upgrade the Control Plane

Before continuing, complete the shared Upgrade Kube-OVN Before the Control Plane procedure and verify that cni-kube-ovn is at the target installed revision with phase=Success.

Patch KubeadmControlPlane.spec.version to the new Kubernetes version. Where component image tags are pinned (DNS, etcd), update them in the same edit:

kubectl -n cpaas-system edit kubeadmcontrolplane <cluster-name>-control-plane

• spec.version ← target Kubernetes version (must match an elemental-image-catalog key).
• spec.kubeadmConfigSpec.clusterConfiguration.dns.imageTag ← matching CoreDNS image tag for the new release.
• spec.kubeadmConfigSpec.clusterConfiguration.etcd.local.imageTag ← matching etcd image tag for the new release.
• When the target is Kubernetes 1.35 or later, update /etc/kubernetes/patches/kubeletconfiguration0+strategic.json in spec.kubeadmConfigSpec.files in this same edit, as described in Required kubelet patch for Kubernetes 1.35.

The bare-metal provider does not require a new BaremetalMachineTemplate for a Kubernetes-only upgrade: the template only carries the pool reference, and the upgrade image is resolved from Machine.spec.version through the catalog. A new template is only required when you want to move the control plane to a different pool.

Cluster API rolls control-plane nodes one at a time (because maxSurge=0):

kubectl -n cpaas-system get kubeadmcontrolplane <cluster-name>-control-plane -w
kubectl -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io
kubectl -n cpaas-system get machineinventorypools.infrastructure.cluster.x-k8s.io <cluster-name>-control-plane-pool
kubectl get nodes -o wide                                                    # workload cluster

The expected sequence on each node:

1. CAPI marks one old Machine for deletion; corresponding BaremetalMachine moves to Preparing.
2. The provider writes a clean plan; the inventory ends up Available with cleared owner annotations.
3. CAPI creates a new Machine (with the target version); a new BaremetalMachine is created.
4. The provider allocates an Available inventory from the control-plane pool, resolves the new image, and writes a reprovision plan.
5. The host runs elemental upgrade --system <new-image>, reboots, and kubeadm joins the surviving control plane. BaremetalMachine.status.phase becomes Running.

Repeat steps 1–5 until every control-plane node has been replaced.

Throughout the rollout, alive continues to manage the VIP. As control-plane membership changes, the bare-metal provider re-renders the alive chart values (the peer list and ipvs.ips) and rolls out the static-pod manifests.

#Upgrade Workers

Patch each MachineDeployment.spec.template.spec.version to the new Kubernetes version. CAPI replaces workers within the maxUnavailable budget — the bare-metal provider re-uses the same worker pool and the same image-catalog resolution logic as the control plane.

For Kubernetes 1.35 or later, first create a new worker KubeadmConfigTemplate containing the required kubelet patch, then update spec.template.spec.bootstrap.configRef.name in the same MachineDeployment edit as the version bump. The command below includes that reference. For an earlier target with no bootstrap change, omit the bootstrap object. Follow Updating Bootstrap Templates to create the immutable template.

kubectl -n cpaas-system patch machinedeployment <cluster-name>-workers \
  --type='merge' \
  -p='{"spec":{"template":{"spec":{"version":"<new-kubernetes-version>","bootstrap":{"configRef":{"name":"<new-bootstrap-template>"}}}}}}'

Watch:

kubectl -n cpaas-system get machinedeployments.cluster.x-k8s.io -w
kubectl -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io
kubectl get nodes -o wide

Worker upgrades carry fewer knobs than the control plane: there is no clusterConfiguration block on MachineDeployment, so no DNS / etcd tags to update. Kubernetes component versions on worker nodes follow the new base image.

For earlier versions, a bootstrap-template change is needed only when the release changes settings such as cloud-init or kubeletExtraArgs.

#Cross-Version Upgrades

Kubernetes control planes must move through supported minor-version hops. Use Preparing for Cross-Version Upgrades to identify and pre-stage every intermediate ACP row.

For each intermediate row, and finally for the target row:

1. Ensure that row's Kubernetes version exists in elemental-image-catalog.
2. Upgrade Kube-OVN to that row's kube-ovn (chart) version and wait for phase=Success.
3. Upgrade the control plane to that row's Kubernetes version and wait for every control-plane node to become Ready.
4. Upgrade every worker MachineDeployment to the same Kubernetes version and wait for the rollout to complete before starting the next hop.

The version values always come from the OS Support Matrix; they are not fixed examples in this guide. The bare-metal rollout strategy already serializes node replacement, so cross-version upgrades do not require additional pool capacity beyond what is needed for a same-minor upgrade.

#Recovering From a Failed Phase 2 Upgrade

Do not treat a Kubernetes minor downgrade as an ordinary rollback:

1. If no target-version control-plane Machine has been created, restore Kube-OVN by using Restore Kube-OVN During Stage-1 Recovery, then restore the previous KubeadmControlPlane and MachineDeployment manifest values.
2. If a target-minor control-plane machine has joined, stop further rollout and repair forward on the target minor, or restore the cluster from the verified pre-upgrade backup by using the supported ACP recovery procedure. Do not patch Kubernetes, CoreDNS, or etcd back to the previous minor.

Every host that completed a reprovision plan has already replaced its OS image and cleared Kubernetes-managed state. Changing version fields does not reconstruct that earlier node state.

#Verification

After the rollout completes:

kubectl -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io
kubectl -n cpaas-system get machines.cluster.x-k8s.io
kubectl -n cpaas-system get apprelease cni-kube-ovn
kubectl get nodes -o wide

The cni-kube-ovn AppRelease should report the target installed revision with phase=Success. All BaremetalMachine objects should be Running. Every CAPI Machine should report the new spec.version. Every workload Node should be Ready with the new kubelet version, and every MachineInventory plan secret should carry baremetal.alauda.io/plan.type=reprovision (the most recent plan applied).

MachineInventoryPool.status should satisfy available + allocated + preparing + reprovisioning + unavailable = total with preparing = reprovisioning = 0.

#Troubleshooting

For the full operator-side state machine reference (every condition reason and recovery action), see Provider Overview.

#Additional Resources

• Cluster API Upgrade Documentation
• Kubernetes Version Skew Policy