Restart Kubernetes Control Planes Safely

🛡️ Safe Control-Plane Restart Runbook

This is the operator runbook for the three-node cicd-ac-k8s stacked-etcd control plane. It is intentionally explicit about where every command runs, so a node is never restarted from itself without a healthy administrative control plane remaining online.

One node at a timeStacked etcd protectedCordon · Drain · Reboot · Validate · Uncordon

PURPOSE

Safely rotate the three CICD AC Kubernetes stacked control-plane nodes through
planned VM reboots without losing etcd quorum or API availability.

This runbook deliberately restarts only one control plane at a time:

  1. Cordon and drain the selected control plane from a separate healthy control plane.
  2. Reboot the selected VM locally or through Proxmox.
  3. Validate the rebooted node locally: containerd, kubelet, API readiness, and static pods.
  4. Validate the recovered node from the designated healthy control plane.
  5. Uncordon the recovered node only after all checks pass.

Never restart or power off two control planes at the same time.

DOCUMENT REVISION

Version: 1.0
Updated: 2026-07-04

Initial release
  Added a complete, host-specific restart sequence for CP-01, CP-02, and CP-03.
  Added explicit execution locations for every command.
  Added control-plane safety gates, local post-reboot checks, cluster re-entry checks,
  final uncordon checks, and memory-health verification.

Hard stop: do not reboot, power off, drain, or otherwise take down two control-plane nodes at once. Complete the full validation and uncordon of the current node before beginning the next section.

1. 🧭 Cluster Topology and Command-Execution Map

CLUSTER CONTROL-PLANE TOPOLOGY

Node                     Address          Role
  cicd-ac-k8s-cp-01      192.168.8.202    Kubernetes control plane + local etcd member
  cicd-ac-k8s-cp-02      192.168.8.203    Kubernetes control plane + local etcd member
  cicd-ac-k8s-cp-03      192.168.8.204    Kubernetes control plane + local etcd member

Architecture
  kubeadm highly available stacked-etcd control plane.
  Each control-plane VM hosts kubelet, containerd, and the local static pods:
    etcd
    kube-apiserver
    kube-controller-manager
    kube-scheduler

Safety rule
  Keep two healthy control planes online while one control plane is restarted.

Restart target	Run cluster preflight, cordon, drain, verify, and uncordon on	Run reboot and local post-reboot validation on
`cicd-ac-k8s-cp-01`	`cicd-ac-k8s-cp-02`	`cicd-ac-k8s-cp-01`
`cicd-ac-k8s-cp-02`	`cicd-ac-k8s-cp-01`	`cicd-ac-k8s-cp-02`
`cicd-ac-k8s-cp-03`	`cicd-ac-k8s-cp-01`	`cicd-ac-k8s-cp-03`

CP-01 Restart

Use CP-02 as the healthy administrative control plane. CP-01 is the node being rebooted and therefore cannot be the host that validates or uncordons itself.

CP-02 Restart

Use CP-01 as the healthy administrative control plane. Reboot and validate the local services only from CP-02 after it returns.

CP-03 Restart

Use CP-01 as the healthy administrative control plane. Reboot and validate the local services only from CP-03 after it returns.

Preconditions

You have SSH access as acllc and can use sudo on all three control planes.
The cluster is healthy before beginning: all control planes are Ready and schedulable.
The current section's preflight is the authority for whether it is safe to proceed.
Do not add --force to any drain command. A failed drain must be investigated before rebooting the node.
Seeing DaemonSet pods and static control-plane pods after a drain is expected. They are not ordinary evictable workloads.

I confirmed the cluster is healthy, I will restart one control plane at a time, and I will not add --force to drain.

2. 🔁 Restart CP-01 — `cicd-ac-k8s-cp-01` (`192.168.8.202`)

Execution map for this section: run the cluster-side commands on cicd-ac-k8s-cp-02 ; run the reboot and local post-reboot validation on cicd-ac-k8s-cp-01 . CP-03 remains online as the third healthy etcd/control-plane member.

Run on CP-02

2.1 Preflight, cordon, and drain CP-01

SSH to cicd-ac-k8s-cp-02. This block refuses to proceed unless CP-02 and CP-03 are both Ready and schedulable.

sudo bash -ceu '
export KUBECONFIG=/etc/kubernetes/admin.conf

ANCHOR="$(hostname -s)"
EXPECTED_ANCHOR="cicd-ac-k8s-cp-02"
TARGET="cicd-ac-k8s-cp-01"
HEALTHY_CONTROL_PLANES=(
  "cicd-ac-k8s-cp-02"
  "cicd-ac-k8s-cp-03"
)

if [ "${ANCHOR}" != "${EXPECTED_ANCHOR}" ]; then
  echo "ERROR: Run this CP-01 preflight and drain block on ${EXPECTED_ANCHOR}, not ${ANCHOR}."
  exit 1
fi

echo "=== Confirm Kubernetes API readiness ==="
kubectl get --raw="/readyz?verbose"

echo
echo "=== Confirm CP-02 and CP-03 are Ready and schedulable ==="
for node in "${HEALTHY_CONTROL_PLANES[@]}"; do
  ready="$(kubectl get node "${node}" -o jsonpath="{.status.conditions[?(@.type==\"Ready\")].status}")"
  unschedulable="$(kubectl get node "${node}" -o jsonpath="{.spec.unschedulable}")"

  if [ "${ready}" != "True" ]; then
    echo "ERROR: ${node} is not Ready. Do not restart CP-01."
    exit 1
  fi

  if [ "${unschedulable}" = "true" ]; then
    echo "ERROR: ${node} is cordoned. Restore and validate it before restarting CP-01."
    exit 1
  fi

  echo "${node}: Ready and schedulable"
done

echo
echo "=== Confirm CP-01 is healthy before planned maintenance ==="
kubectl get node "${TARGET}" -o wide

target_ready="$(kubectl get node "${TARGET}" -o jsonpath="{.status.conditions[?(@.type==\"Ready\")].status}")"
target_unschedulable="$(kubectl get node "${TARGET}" -o jsonpath="{.spec.unschedulable}")"

if [ "${target_ready}" != "True" ]; then
  echo "ERROR: ${TARGET} is not Ready. Investigate before starting a planned reboot."
  exit 1
fi

if [ "${target_unschedulable}" = "true" ]; then
  echo "ERROR: ${TARGET} is already cordoned. Resolve the prior maintenance state first."
  exit 1
fi

echo
echo "=== Workloads currently assigned to CP-01 ==="
kubectl get pods -A -o wide \
  --field-selector "spec.nodeName=${TARGET}"

echo
echo "=== Cordon CP-01 ==="
kubectl cordon "${TARGET}"

echo
echo "=== Drain CP-01 ==="
kubectl drain "${TARGET}" \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --timeout=10m

echo
echo "=== CP-01 status after drain ==="
kubectl get node "${TARGET}"
kubectl get pods -A -o wide \
  --field-selector "spec.nodeName=${TARGET}"

echo
echo "SUCCESS: CP-01 is cordoned and drained for reboot."
'

Run on CP-01

2.2 Reboot CP-01

SSH to cicd-ac-k8s-cp-01 and run the command below. Using the Proxmox Reboot action for the same VM is also acceptable. Do not manually restart etcd, kubelet, or any individual control-plane static pod.

sudo bash -ceu '
EXPECTED_NODE="cicd-ac-k8s-cp-01"
NODE="$(hostname -s)"

if [ "${NODE}" != "${EXPECTED_NODE}" ]; then
  echo "ERROR: Run this reboot command on ${EXPECTED_NODE}, not ${NODE}."
  exit 1
fi

systemctl reboot
'

Run on CP-01 after SSH reconnects

2.3 Validate the recovered CP-01 locally

Wait for CP-01 to boot and accept SSH connections, then run this block on CP-01. It checks the local services, API readiness, node state, and the four local static containers.

sudo bash -ceu '
export KUBECONFIG=/etc/kubernetes/admin.conf

NODE="$(hostname -s)"
EXPECTED_NODE="cicd-ac-k8s-cp-01"

if [ "${NODE}" != "${EXPECTED_NODE}" ]; then
  echo "ERROR: Run this local post-reboot validation on ${EXPECTED_NODE}, not ${NODE}."
  exit 1
fi

echo "=== Host identity ==="
hostnamectl --static
ip -brief address show

echo
echo "=== Required local services ==="
for service in containerd kubelet; do
  systemctl is-active --quiet "${service}"
  printf "%s: active\n" "${service}"
done

echo
echo "=== Wait for Kubernetes API readiness: maximum 5 minutes ==="
deadline=$((SECONDS + 300))
ready_file="$(mktemp)"
trap "rm -f \"${ready_file}\"" EXIT

until kubectl get --raw="/readyz?verbose" >"${ready_file}" 2>&1; do
  if (( SECONDS >= deadline )); then
    echo "ERROR: Kubernetes API did not become ready within 5 minutes."
    cat "${ready_file}"
    exit 1
  fi

  sleep 5
done

cat "${ready_file}"

echo
echo "=== Rejoined Kubernetes node ==="
kubectl get node "${NODE}" -o wide

node_ready="$(kubectl get node "${NODE}" -o jsonpath="{.status.conditions[?(@.type==\"Ready\")].status}")"

if [ "${node_ready}" != "True" ]; then
  echo "ERROR: ${NODE} is not Ready. Current Ready status: ${node_ready}"
  exit 1
fi

echo
echo "=== Required local control-plane static containers ==="
for component in etcd kube-apiserver kube-controller-manager kube-scheduler; do
  if ! crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps \
    | grep -Fq "${component}-${NODE}"
  then
    echo "ERROR: Running static container not found: ${component}-${NODE}"
    crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps
    exit 1
  fi

done

crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps \
  | egrep "etcd|kube-apiserver|kube-controller-manager|kube-scheduler"

echo
echo "=== Cluster view of CP-01 control-plane pods ==="
kubectl -n kube-system get pods -o wide \
  | egrep "etcd-${NODE}|kube-apiserver-${NODE}|kube-controller-manager-${NODE}|kube-scheduler-${NODE}"

echo
echo "=== Kubelet warnings from this boot ==="
journalctl -u kubelet -b --priority=warning --no-pager || true

echo
echo "SUCCESS: ${NODE} is Ready, local services are active, and all local control-plane static containers are running."
'

Run on CP-02

2.4 Verify CP-01 from the healthy control plane and uncordon it

Return to CP-02 only after the local CP-01 validation succeeds. This block waits for CP-01 and its control-plane pods to be Ready, then restores scheduling.

sudo bash -ceu '
export KUBECONFIG=/etc/kubernetes/admin.conf

ANCHOR="$(hostname -s)"
EXPECTED_ANCHOR="cicd-ac-k8s-cp-02"
TARGET="cicd-ac-k8s-cp-01"

if [ "${ANCHOR}" != "${EXPECTED_ANCHOR}" ]; then
  echo "ERROR: Run this CP-01 recovery and uncordon block on ${EXPECTED_ANCHOR}, not ${ANCHOR}."
  exit 1
fi

echo "=== Wait for CP-01 node readiness ==="
kubectl wait \
  --for=condition=Ready \
  "node/${TARGET}" \
  --timeout=5m

echo
echo "=== Confirm API readiness and all cluster nodes ==="
kubectl get --raw="/readyz?verbose"
kubectl get nodes -o wide

echo
echo "=== Confirm CP-01 control-plane pods are Ready ==="
for component in etcd kube-apiserver kube-controller-manager kube-scheduler; do
  kubectl -n kube-system wait \
    --for=condition=Ready \
    "pod/${component}-${TARGET}" \
    --timeout=5m
done

kubectl -n kube-system get pods -o wide \
  | egrep "etcd-${TARGET}|kube-apiserver-${TARGET}|kube-controller-manager-${TARGET}|kube-scheduler-${TARGET}"

echo
echo "=== Restore CP-01 scheduling ==="
kubectl uncordon "${TARGET}"

echo
echo "=== Final CP-01 status ==="
kubectl get node "${TARGET}"

if [ "$(kubectl get node "${TARGET}" -o jsonpath="{.spec.unschedulable}")" = "true" ]; then
  echo "ERROR: ${TARGET} remains cordoned after uncordon."
  exit 1
fi

echo
echo "SUCCESS: CP-01 is Ready and returned to normal scheduling."
'

CP-01 is Ready, its four control-plane components are Ready, and CP-01 is no longer SchedulingDisabled.

3. 🔁 Restart CP-02 — `cicd-ac-k8s-cp-02` (`192.168.8.203`)

Execution map for this section: run the cluster-side commands on cicd-ac-k8s-cp-01 ; run the reboot and local post-reboot validation on cicd-ac-k8s-cp-02 . Do not start this section until CP-01 has completed Section 2 and is Ready and schedulable.

Run on CP-01

3.1 Preflight, cordon, and drain CP-02

SSH to CP-01. This block refuses to proceed unless CP-01 and CP-03 are both Ready and schedulable.

sudo bash -ceu '
export KUBECONFIG=/etc/kubernetes/admin.conf

ANCHOR="$(hostname -s)"
EXPECTED_ANCHOR="cicd-ac-k8s-cp-01"
TARGET="cicd-ac-k8s-cp-02"
HEALTHY_CONTROL_PLANES=(
  "cicd-ac-k8s-cp-01"
  "cicd-ac-k8s-cp-03"
)

if [ "${ANCHOR}" != "${EXPECTED_ANCHOR}" ]; then
  echo "ERROR: Run this CP-02 preflight and drain block on ${EXPECTED_ANCHOR}, not ${ANCHOR}."
  exit 1
fi

echo "=== Confirm Kubernetes API readiness ==="
kubectl get --raw="/readyz?verbose"

echo
echo "=== Confirm CP-01 and CP-03 are Ready and schedulable ==="
for node in "${HEALTHY_CONTROL_PLANES[@]}"; do
  ready="$(kubectl get node "${node}" -o jsonpath="{.status.conditions[?(@.type==\"Ready\")].status}")"
  unschedulable="$(kubectl get node "${node}" -o jsonpath="{.spec.unschedulable}")"

  if [ "${ready}" != "True" ]; then
    echo "ERROR: ${node} is not Ready. Do not restart CP-02."
    exit 1
  fi

  if [ "${unschedulable}" = "true" ]; then
    echo "ERROR: ${node} is cordoned. Restore and validate it before restarting CP-02."
    exit 1
  fi

  echo "${node}: Ready and schedulable"
done

echo
echo "=== Confirm CP-02 is healthy before planned maintenance ==="
kubectl get node "${TARGET}" -o wide

target_ready="$(kubectl get node "${TARGET}" -o jsonpath="{.status.conditions[?(@.type==\"Ready\")].status}")"
target_unschedulable="$(kubectl get node "${TARGET}" -o jsonpath="{.spec.unschedulable}")"

if [ "${target_ready}" != "True" ]; then
  echo "ERROR: ${TARGET} is not Ready. Investigate before starting a planned reboot."
  exit 1
fi

if [ "${target_unschedulable}" = "true" ]; then
  echo "ERROR: ${TARGET} is already cordoned. Resolve the prior maintenance state first."
  exit 1
fi

echo
echo "=== Workloads currently assigned to CP-02 ==="
kubectl get pods -A -o wide \
  --field-selector "spec.nodeName=${TARGET}"

echo
echo "=== Cordon CP-02 ==="
kubectl cordon "${TARGET}"

echo
echo "=== Drain CP-02 ==="
kubectl drain "${TARGET}" \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --timeout=10m

echo
echo "=== CP-02 status after drain ==="
kubectl get node "${TARGET}"
kubectl get pods -A -o wide \
  --field-selector "spec.nodeName=${TARGET}"

echo
echo "SUCCESS: CP-02 is cordoned and drained for reboot."
'

Run on CP-02

3.2 Reboot CP-02

sudo bash -ceu '
EXPECTED_NODE="cicd-ac-k8s-cp-02"
NODE="$(hostname -s)"

if [ "${NODE}" != "${EXPECTED_NODE}" ]; then
  echo "ERROR: Run this reboot command on ${EXPECTED_NODE}, not ${NODE}."
  exit 1
fi

systemctl reboot
'

Run on CP-02 after SSH reconnects

3.3 Validate the recovered CP-02 locally

Wait for CP-02 to boot and accept SSH connections, then run this exact local validation.

sudo bash -ceu '
export KUBECONFIG=/etc/kubernetes/admin.conf

NODE="$(hostname -s)"
EXPECTED_NODE="cicd-ac-k8s-cp-02"

if [ "${NODE}" != "${EXPECTED_NODE}" ]; then
  echo "ERROR: Run this local post-reboot validation on ${EXPECTED_NODE}, not ${NODE}."
  exit 1
fi

echo "=== Host identity ==="
hostnamectl --static
ip -brief address show

echo
echo "=== Required local services ==="
for service in containerd kubelet; do
  systemctl is-active --quiet "${service}"
  printf "%s: active\n" "${service}"
done

echo
echo "=== Wait for Kubernetes API readiness: maximum 5 minutes ==="
deadline=$((SECONDS + 300))
ready_file="$(mktemp)"
trap "rm -f \"${ready_file}\"" EXIT

until kubectl get --raw="/readyz?verbose" >"${ready_file}" 2>&1; do
  if (( SECONDS >= deadline )); then
    echo "ERROR: Kubernetes API did not become ready within 5 minutes."
    cat "${ready_file}"
    exit 1
  fi

  sleep 5
done

cat "${ready_file}"

echo
echo "=== Rejoined Kubernetes node ==="
kubectl get node "${NODE}" -o wide

node_ready="$(kubectl get node "${NODE}" -o jsonpath="{.status.conditions[?(@.type==\"Ready\")].status}")"

if [ "${node_ready}" != "True" ]; then
  echo "ERROR: ${NODE} is not Ready. Current Ready status: ${node_ready}"
  exit 1
fi

echo
echo "=== Required local control-plane static containers ==="
for component in etcd kube-apiserver kube-controller-manager kube-scheduler; do
  if ! crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps \
    | grep -Fq "${component}-${NODE}"
  then
    echo "ERROR: Running static container not found: ${component}-${NODE}"
    crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps
    exit 1
  fi

done

crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps \
  | egrep "etcd|kube-apiserver|kube-controller-manager|kube-scheduler"

echo
echo "=== Cluster view of CP-02 control-plane pods ==="
kubectl -n kube-system get pods -o wide \
  | egrep "etcd-${NODE}|kube-apiserver-${NODE}|kube-controller-manager-${NODE}|kube-scheduler-${NODE}"

echo
echo "=== Kubelet warnings from this boot ==="
journalctl -u kubelet -b --priority=warning --no-pager || true

echo
echo "SUCCESS: ${NODE} is Ready, local services are active, and all local control-plane static containers are running."
'

Run on CP-01

3.4 Verify CP-02 from the healthy control plane and uncordon it

sudo bash -ceu '
export KUBECONFIG=/etc/kubernetes/admin.conf

ANCHOR="$(hostname -s)"
EXPECTED_ANCHOR="cicd-ac-k8s-cp-01"
TARGET="cicd-ac-k8s-cp-02"

if [ "${ANCHOR}" != "${EXPECTED_ANCHOR}" ]; then
  echo "ERROR: Run this CP-02 recovery and uncordon block on ${EXPECTED_ANCHOR}, not ${ANCHOR}."
  exit 1
fi

echo "=== Wait for CP-02 node readiness ==="
kubectl wait \
  --for=condition=Ready \
  "node/${TARGET}" \
  --timeout=5m

echo
echo "=== Confirm API readiness and all cluster nodes ==="
kubectl get --raw="/readyz?verbose"
kubectl get nodes -o wide

echo
echo "=== Confirm CP-02 control-plane pods are Ready ==="
for component in etcd kube-apiserver kube-controller-manager kube-scheduler; do
  kubectl -n kube-system wait \
    --for=condition=Ready \
    "pod/${component}-${TARGET}" \
    --timeout=5m
done

kubectl -n kube-system get pods -o wide \
  | egrep "etcd-${TARGET}|kube-apiserver-${TARGET}|kube-controller-manager-${TARGET}|kube-scheduler-${TARGET}"

echo
echo "=== Restore CP-02 scheduling ==="
kubectl uncordon "${TARGET}"

echo
echo "=== Final CP-02 status ==="
kubectl get node "${TARGET}"

if [ "$(kubectl get node "${TARGET}" -o jsonpath="{.spec.unschedulable}")" = "true" ]; then
  echo "ERROR: ${TARGET} remains cordoned after uncordon."
  exit 1
fi

echo
echo "SUCCESS: CP-02 is Ready and returned to normal scheduling."
'

CP-02 is Ready, its four control-plane components are Ready, and CP-02 is no longer SchedulingDisabled.

4. 🔁 Restart CP-03 — `cicd-ac-k8s-cp-03` (`192.168.8.204`)

Execution map for this section: run the cluster-side commands on cicd-ac-k8s-cp-01 ; run the reboot and local post-reboot validation on cicd-ac-k8s-cp-03 . Do not start this section until CP-02 has completed Section 3 and is Ready and schedulable.

Run on CP-01

4.1 Preflight, cordon, and drain CP-03

SSH to CP-01. This block refuses to proceed unless CP-01 and CP-02 are both Ready and schedulable.

sudo bash -ceu '
export KUBECONFIG=/etc/kubernetes/admin.conf

ANCHOR="$(hostname -s)"
EXPECTED_ANCHOR="cicd-ac-k8s-cp-01"
TARGET="cicd-ac-k8s-cp-03"
HEALTHY_CONTROL_PLANES=(
  "cicd-ac-k8s-cp-01"
  "cicd-ac-k8s-cp-02"
)

if [ "${ANCHOR}" != "${EXPECTED_ANCHOR}" ]; then
  echo "ERROR: Run this CP-03 preflight and drain block on ${EXPECTED_ANCHOR}, not ${ANCHOR}."
  exit 1
fi

echo "=== Confirm Kubernetes API readiness ==="
kubectl get --raw="/readyz?verbose"

echo
echo "=== Confirm CP-01 and CP-02 are Ready and schedulable ==="
for node in "${HEALTHY_CONTROL_PLANES[@]}"; do
  ready="$(kubectl get node "${node}" -o jsonpath="{.status.conditions[?(@.type==\"Ready\")].status}")"
  unschedulable="$(kubectl get node "${node}" -o jsonpath="{.spec.unschedulable}")"

  if [ "${ready}" != "True" ]; then
    echo "ERROR: ${node} is not Ready. Do not restart CP-03."
    exit 1
  fi

  if [ "${unschedulable}" = "true" ]; then
    echo "ERROR: ${node} is cordoned. Restore and validate it before restarting CP-03."
    exit 1
  fi

  echo "${node}: Ready and schedulable"
done

echo
echo "=== Confirm CP-03 is healthy before planned maintenance ==="
kubectl get node "${TARGET}" -o wide

target_ready="$(kubectl get node "${TARGET}" -o jsonpath="{.status.conditions[?(@.type==\"Ready\")].status}")"
target_unschedulable="$(kubectl get node "${TARGET}" -o jsonpath="{.spec.unschedulable}")"

if [ "${target_ready}" != "True" ]; then
  echo "ERROR: ${TARGET} is not Ready. Investigate before starting a planned reboot."
  exit 1
fi

if [ "${target_unschedulable}" = "true" ]; then
  echo "ERROR: ${TARGET} is already cordoned. Resolve the prior maintenance state first."
  exit 1
fi

echo
echo "=== Workloads currently assigned to CP-03 ==="
kubectl get pods -A -o wide \
  --field-selector "spec.nodeName=${TARGET}"

echo
echo "=== Cordon CP-03 ==="
kubectl cordon "${TARGET}"

echo
echo "=== Drain CP-03 ==="
kubectl drain "${TARGET}" \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --timeout=10m

echo
echo "=== CP-03 status after drain ==="
kubectl get node "${TARGET}"
kubectl get pods -A -o wide \
  --field-selector "spec.nodeName=${TARGET}"

echo
echo "SUCCESS: CP-03 is cordoned and drained for reboot."
'

Run on CP-03

4.2 Reboot CP-03

sudo bash -ceu '
EXPECTED_NODE="cicd-ac-k8s-cp-03"
NODE="$(hostname -s)"

if [ "${NODE}" != "${EXPECTED_NODE}" ]; then
  echo "ERROR: Run this reboot command on ${EXPECTED_NODE}, not ${NODE}."
  exit 1
fi

systemctl reboot
'

Run on CP-03 after SSH reconnects

4.3 Validate the recovered CP-03 locally

Wait for CP-03 to boot and accept SSH connections, then run this exact local validation.

sudo bash -ceu '
export KUBECONFIG=/etc/kubernetes/admin.conf

NODE="$(hostname -s)"
EXPECTED_NODE="cicd-ac-k8s-cp-03"

if [ "${NODE}" != "${EXPECTED_NODE}" ]; then
  echo "ERROR: Run this local post-reboot validation on ${EXPECTED_NODE}, not ${NODE}."
  exit 1
fi

echo "=== Host identity ==="
hostnamectl --static
ip -brief address show

echo
echo "=== Required local services ==="
for service in containerd kubelet; do
  systemctl is-active --quiet "${service}"
  printf "%s: active\n" "${service}"
done

echo
echo "=== Wait for Kubernetes API readiness: maximum 5 minutes ==="
deadline=$((SECONDS + 300))
ready_file="$(mktemp)"
trap "rm -f \"${ready_file}\"" EXIT

until kubectl get --raw="/readyz?verbose" >"${ready_file}" 2>&1; do
  if (( SECONDS >= deadline )); then
    echo "ERROR: Kubernetes API did not become ready within 5 minutes."
    cat "${ready_file}"
    exit 1
  fi

  sleep 5
done

cat "${ready_file}"

echo
echo "=== Rejoined Kubernetes node ==="
kubectl get node "${NODE}" -o wide

node_ready="$(kubectl get node "${NODE}" -o jsonpath="{.status.conditions[?(@.type==\"Ready\")].status}")"

if [ "${node_ready}" != "True" ]; then
  echo "ERROR: ${NODE} is not Ready. Current Ready status: ${node_ready}"
  exit 1
fi

echo
echo "=== Required local control-plane static containers ==="
for component in etcd kube-apiserver kube-controller-manager kube-scheduler; do
  if ! crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps \
    | grep -Fq "${component}-${NODE}"
  then
    echo "ERROR: Running static container not found: ${component}-${NODE}"
    crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps
    exit 1
  fi

done

crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps \
  | egrep "etcd|kube-apiserver|kube-controller-manager|kube-scheduler"

echo
echo "=== Cluster view of CP-03 control-plane pods ==="
kubectl -n kube-system get pods -o wide \
  | egrep "etcd-${NODE}|kube-apiserver-${NODE}|kube-controller-manager-${NODE}|kube-scheduler-${NODE}"

echo
echo "=== Kubelet warnings from this boot ==="
journalctl -u kubelet -b --priority=warning --no-pager || true

echo
echo "SUCCESS: ${NODE} is Ready, local services are active, and all local control-plane static containers are running."
'

Run on CP-01

4.4 Verify CP-03 from the healthy control plane and uncordon it

sudo bash -ceu '
export KUBECONFIG=/etc/kubernetes/admin.conf

ANCHOR="$(hostname -s)"
EXPECTED_ANCHOR="cicd-ac-k8s-cp-01"
TARGET="cicd-ac-k8s-cp-03"

if [ "${ANCHOR}" != "${EXPECTED_ANCHOR}" ]; then
  echo "ERROR: Run this CP-03 recovery and uncordon block on ${EXPECTED_ANCHOR}, not ${ANCHOR}."
  exit 1
fi

echo "=== Wait for CP-03 node readiness ==="
kubectl wait \
  --for=condition=Ready \
  "node/${TARGET}" \
  --timeout=5m

echo
echo "=== Confirm API readiness and all cluster nodes ==="
kubectl get --raw="/readyz?verbose"
kubectl get nodes -o wide

echo
echo "=== Confirm CP-03 control-plane pods are Ready ==="
for component in etcd kube-apiserver kube-controller-manager kube-scheduler; do
  kubectl -n kube-system wait \
    --for=condition=Ready \
    "pod/${component}-${TARGET}" \
    --timeout=5m
done

kubectl -n kube-system get pods -o wide \
  | egrep "etcd-${TARGET}|kube-apiserver-${TARGET}|kube-controller-manager-${TARGET}|kube-scheduler-${TARGET}"

echo
echo "=== Restore CP-03 scheduling ==="
kubectl uncordon "${TARGET}"

echo
echo "=== Final CP-03 status ==="
kubectl get node "${TARGET}"

if [ "$(kubectl get node "${TARGET}" -o jsonpath="{.spec.unschedulable}")" = "true" ]; then
  echo "ERROR: ${TARGET} remains cordoned after uncordon."
  exit 1
fi

echo
echo "SUCCESS: CP-03 is Ready and returned to normal scheduling."
'

CP-03 is Ready, its four control-plane components are Ready, and CP-03 is no longer SchedulingDisabled.

5. 🧠 Check Memory Health After Each Control Plane Settles

Run this on the control plane that was just restarted, after it has remained stable for about 10 minutes. For example, run it on CP-01 after Section 2, CP-02 after Section 3, and CP-03 after Section 4.

sudo bash -ceu '
echo "=== free -h ==="
free -h

echo
echo "=== Key memory counters ==="
grep -E "MemTotal|MemAvailable|SwapTotal|SwapFree|Cached|SReclaimable" /proc/meminfo

echo
echo "=== Paging activity ==="
vmstat 1 5

echo
echo "=== Largest resident-memory processes ==="
ps -eo pid,user,comm,rss,%mem --sort=-rss | head -n 20

echo
echo "=== OOM events since this boot ==="
journalctl -k -b --no-pager \
  | grep -Ei "out of memory|oom-killer|killed process" || true
'

Current CP-01 result: after the completed CP-01 reboot, the node had approximately 5.9 GiB of available memory, no swap usage, no paging, and no OOM events. Keep all three control planes at 8 GiB unless repeated checks show sustained low available memory, active paging, swap consumption, or OOM events. If a memory increase becomes necessary, resize all three control-plane VMs consistently through the approved infrastructure-as-code workflow.

6. 🏁 Final Whole-Cluster Validation

After CP-03 is restored to normal scheduling, run this final validation on CP-01.

Run on CP-01

sudo bash -ceu '
export KUBECONFIG=/etc/kubernetes/admin.conf

ANCHOR="$(hostname -s)"
EXPECTED_ANCHOR="cicd-ac-k8s-cp-01"

if [ "${ANCHOR}" != "${EXPECTED_ANCHOR}" ]; then
  echo "ERROR: Run final cluster validation on ${EXPECTED_ANCHOR}, not ${ANCHOR}."
  exit 1
fi

echo "=== Kubernetes API readiness ==="
kubectl get --raw="/readyz?verbose"

echo
echo "=== Cluster node readiness ==="
kubectl get nodes -o wide

for node in cicd-ac-k8s-cp-01 cicd-ac-k8s-cp-02 cicd-ac-k8s-cp-03; do
  ready="$(kubectl get node "${node}" -o jsonpath="{.status.conditions[?(@.type==\"Ready\")].status}")"
  unschedulable="$(kubectl get node "${node}" -o jsonpath="{.spec.unschedulable}")"

  if [ "${ready}" != "True" ]; then
    echo "ERROR: ${node} is not Ready."
    exit 1
  fi

  if [ "${unschedulable}" = "true" ]; then
    echo "ERROR: ${node} remains cordoned."
    exit 1
  fi

done

echo
echo "=== Control-plane static pods ==="
kubectl -n kube-system get pods -o wide \
  | egrep "etcd-cicd-ac-k8s-cp-0[123]|kube-apiserver-cicd-ac-k8s-cp-0[123]|kube-controller-manager-cicd-ac-k8s-cp-0[123]|kube-scheduler-cicd-ac-k8s-cp-0[123]"

echo
echo "SUCCESS: all three control planes are Ready and schedulable."
'

Completion Standard

The maintenance rotation is complete only when all of the following are true:

The API readiness endpoint passes.
CP-01, CP-02, and CP-03 each show Ready.
No control plane shows SchedulingDisabled.
All twelve static control-plane pods are 1/1 Running.
The post-reboot memory checks show no paging, swap use, or OOM events.

All three control planes are Ready and schedulable, API readiness passes, and final cluster validation completed.

7. 🩺 Stop Conditions and Recovery Boundaries

Stop and investigate before continuing when a preflight fails, a drain fails, a node does not become Ready within five minutes, a required static container is missing, a control-plane pod does not become Ready, or an uncordon leaves the node SchedulingDisabled.

Do not use --force to bypass a failed drain.
Do not restart individual etcd, kube-apiserver, kube-controller-manager, or kube-scheduler containers as a substitute for the documented VM reboot.
Do not begin CP-02 until CP-01 has passed all recovery checks and has been uncordoned.
Do not begin CP-03 until CP-02 has passed all recovery checks and has been uncordoned.
Do not alter control-plane memory only in Proxmox. Any approved size change must be represented in Terraform and deployed through the established manual workflow.

🛡️ Safe Control-Plane Restart Runbook

1. 🧭 Cluster Topology and Command-Execution Map​

CP-01 Restart

CP-02 Restart

CP-03 Restart

Preconditions​

2. 🔁 Restart CP-01 — cicd-ac-k8s-cp-01 (192.168.8.202)​

2.1 Preflight, cordon, and drain CP-01

2.2 Reboot CP-01

2.3 Validate the recovered CP-01 locally

2.4 Verify CP-01 from the healthy control plane and uncordon it

3. 🔁 Restart CP-02 — cicd-ac-k8s-cp-02 (192.168.8.203)​

3.1 Preflight, cordon, and drain CP-02

3.2 Reboot CP-02

3.3 Validate the recovered CP-02 locally

3.4 Verify CP-02 from the healthy control plane and uncordon it

4. 🔁 Restart CP-03 — cicd-ac-k8s-cp-03 (192.168.8.204)​

4.1 Preflight, cordon, and drain CP-03

4.2 Reboot CP-03

4.3 Validate the recovered CP-03 locally

4.4 Verify CP-03 from the healthy control plane and uncordon it

5. 🧠 Check Memory Health After Each Control Plane Settles​

6. 🏁 Final Whole-Cluster Validation​

Completion Standard​

7. 🩺 Stop Conditions and Recovery Boundaries​

1. 🧭 Cluster Topology and Command-Execution Map

Preconditions

2. 🔁 Restart CP-01 — `cicd-ac-k8s-cp-01` (`192.168.8.202`)

3. 🔁 Restart CP-02 — `cicd-ac-k8s-cp-02` (`192.168.8.203`)

4. 🔁 Restart CP-03 — `cicd-ac-k8s-cp-03` (`192.168.8.204`)

5. 🧠 Check Memory Health After Each Control Plane Settles

6. 🏁 Final Whole-Cluster Validation

Completion Standard

7. 🩺 Stop Conditions and Recovery Boundaries