-
Notifications
You must be signed in to change notification settings - Fork 6.8k
Description
What happened?
kubeadm-config configmap is not updated as per changes from kubeadm config static file before upgrading which caused kubeadm upgrade apply to fail.
Found this when updating image path in a new update but can also be due to other kubeadm configmap related changes just like image path updates (see ansible o/p below)
Running :
kubeadm init phase upload-config kubeadm
--config={{ kube_config_dir }}/kubeadm-config.yaml
before upragde fixes the issue in our case, so contributing for everyone's benefit
What did you expect to happen?
Upgrade to succeed
How can we reproduce it (as minimally and precisely as possible)?
change image path and upgrade
OS
RHEL 8
Version of Ansible
9.13.0
Version of Python
3
Version of Kubespray (commit)
v2.29.0
Network plugin used
cilium
Full inventory with variables
<>
Command used to invoke ansible
<>
Output of ansible run
TASK [kubernetes/control-plane : Kubeadm | Upgrade first control plane node to 1.33.7-<>] ***
fatal: [k8s-<>]: FAILED! => {"changed": true, "cmd": ["timeout", "-k", "600s", "600s", "/usr/local/bin/kubeadm", "upgrade", "apply", "-y", "v1.33.7-<>", "--config=/etc/kubernetes/kubeadm-config.yaml", "--skip-phases=addon/coredns"], "delta": "0:00:24.558125", "end": "2026-01-11 21:59:10.661781", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2026-01-11 21:58:46.103656", "stderr": "W0111 21:58:46.136025 2056849 upgradeconfiguration.go:43] [config] WARNING: Ignored configuration document with GroupVersionKind kubeproxy.config.k8s.io/v1alpha1, Kind=KubeProxyConfiguration\nW0111 21:58:46.136099 2056849 upgradeconfiguration.go:43] [config] WARNING: Ignored configuration document with GroupVersionKind kubelet.config.k8s.io/v1beta1, Kind=KubeletConfiguration\nW0111 21:58:46.136105 2056849 upgradeconfiguration.go:43] [config] WARNING: Ignored configuration document with GroupVersionKind kubeadm.k8s.io/v1beta4, Kind=InitConfiguration\nW0111 21:58:46.136110 2056849 upgradeconfiguration.go:43] [config] WARNING: Ignored configuration document with GroupVersionKind kubeadm.k8s.io/v1beta4, Kind=ClusterConfiguration\nW0111 21:58:46.181153 2056849 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.9.240.10]; the provided value is: [10.9.240.3]\n\t[WARNING SystemVerification]: cgroups v1 support is in maintenance mode, please migrate to cgroups v2\n\t[WARNING SystemVerification]: kernel release 4.18.0-553.89.1.el8_10.x86_64 is unsupported. Recommended LTS version from the 4.x series is 4.19. Any 5.x or 6.x versions are also supported. For cgroups v2 support, the minimal version is 4.15 and the recommended version is 5.8+\nW0111 21:58:49.232714 2056849 checks.go:843] detected that the sandbox image "<>pause:3.10" of the container runtime is inconsistent with that used by kubeadm.It is recommended to use "<>pause:3.10" as the CRI sandbox image.\nerror execution phase preflight: [preflight] Some fatal errors occurred:\n\t[ERROR ImagePull]: failed to pull image <>/kube-apiserver:v1.33.7-<>: failed to pull image <>/kube-apiserver:v1.33.7-<>: unable to pull image or OCI artifact: pull image err: initializing source docker://<>/kube-apiserver:v1.33.7-<>: reading manifest v1.33.7-<> in <>/kube-apiserver: manifest unknown: The named manifest is not known to the registry.; artifact err: get manifest: build image source: reading manifest v1.33.7-<> in <>/kube-apiserver: manifest unknown: The named manifest is not known to the registry.\n\t[ERROR ImagePull]: failed to pull image <>/kube-controller-manager:v1.33.7-<>: failed to pull image <>/kube-controller-manager:v1.33.7-<>: unable to pull image or OCI artifact: pull image err: initializing source docker://<>/kube-controller-manager:v1.33.7-<>: reading manifest v1.33.7-<> in <>/kube-controller-manager: manifest unknown: The named manifest is not known to the registry.;,
"[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...", "To see the stack trace of this error execute with --v=5 or higher"], "stdout": "[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...\n[upgrade] Use 'kubeadm init phase upload-config --config your-config-file' to re-upload it.[upgrade/preflight] You have chosen to upgrade the cluster version to "v1.33.7-<>"\n[upgrade/versions] Cluster version: v1.32.10-<>\n[upgrade/versions] kubeadm version: v1.33.7-<>\n[upgrade/preflight] Pulling images required for setting up a Kubernetes cluster\n[upgrade/preflight] This might take a minute or two, depending on the speed of your internet connection\n[upgrade/preflight] You can also perform this action beforehand using 'kubeadm config images pull'", "stdout_lines": ["[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...", "[upgrade] Use 'kubeadm init phase upload-config --config your-config-file' to re-upload it.", "[upgrade/preflight] Running preflight checks",
Anything else we need to know
No response