Configure Kubernetes Worker Nodes

1. 🧭 Scope Legend

Use these scope markers throughout this page and the wider Kubernetes CI/CD documentation.

1.COMMON · NO CHANGE NEEDEDConfigure once and reuse across all GitHub organizations and private repositories.

2.CHANGE PER GITHUB ORGRepeat or recreate for every GitHub organization or product.

3.CHANGE PER REPOSITORYRepeat for every private repository inside the GitHub organization.

Shared-worker-pool rule: the DEV, QA, and PROD worker pools are shared by Shelvera, FP, and future GitHub organizations. Adding another organization or private repository does not require another worker pool or app-specific node labels by default. Plan the combined maxRunners for every organization against the total CPU, RAM, disk, and container capacity available in each environment pool.

2. 🧾 Worker Node Inputs

COMMON · NO CHANGE NEEDEDThese are shared cluster and environment-worker values. They are not GitHub organization or repository inputs and may persist using the versioned common storage keys on this page. The worker join command remains memory-only and must never persist.

Template Hostname

CP-01 Hostname

Control Plane Endpoint

K8s Repo Stream

Pause Image Tag

PROD Workers

COMMON · NO CHANGE NEEDEDConfigure the shared production worker pool once; all product runners targeting PROD consume this common capacity.

Prod Worker 1 Hostname

Prod Worker 1 IP

Prod Worker 2 Hostname

Prod Worker 2 IP

Prod Worker 3 Hostname

Prod Worker 3 IP

Prod Worker 4 Hostname

Prod Worker 4 IP

QA Workers

COMMON · NO CHANGE NEEDEDConfigure the shared QA worker pool once; all product runners targeting QA consume this common capacity.

QA Worker 1 Hostname

QA Worker 1 IP

QA Worker 2 Hostname

QA Worker 2 IP

QA Worker 3 Hostname

QA Worker 3 IP

QA Worker 4 Hostname

QA Worker 4 IP

DEV Workers

COMMON · NO CHANGE NEEDEDConfigure the shared development worker pool once; all product runners targeting DEV consume this common capacity.

Dev Worker 1 Hostname

Dev Worker 1 IP

Dev Worker 2 Hostname

Dev Worker 2 IP

Dev Worker 3 Hostname

Dev Worker 3 IP

Dev Worker 4 Hostname

Dev Worker 4 IP

Paste the worker join command here after generating it from the control plane. Do not persist this value.

Worker Join Command

Template Name:

tmpl-ac-cicd-env-wk-00

Control Plane Endpoint:

ac-cicd-api.aspireclan.com:443

Kubernetes Repo URL:

https://pkgs.k8s.io/core:/stable:/v1.35/deb/

3. 📦 VM Specs

COMMON · NO CHANGE NEEDEDUse this worker sizing as the shared baseline and adjust capacity based on the combined runner demand across all GitHub organizations.

CPU: 8 vCPU
RAM: 32 GB
Disk: 200 GB
Clone: tmpl-ub-24-min

Worker VM specs reviewed

4. 🧱 Prepare the Base Worker Template VM

COMMON · NO CHANGE NEEDEDPrepare one reusable base template for worker nodes in the shared Kubernetes cluster.

lsblk

sudo growpart /dev/sda 3
sudo pvresize /dev/sda3
sudo lvextend -l +100%FREE /dev/ubuntu-vg/ubuntu-lv
sudo resize2fs /dev/ubuntu-vg/ubuntu-lv
df -h

HDD resize reference: Resize Ubuntu VM HDD

HDD resized successfully

swapon --show

sudo modprobe overlay
sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF

sudo sysctl --system

sudo apt install -y containerd

sudo mkdir -p /etc/containerd

containerd config default | sudo tee /etc/containerd/config.toml >/dev/null

sudo nano /etc/containerd/config.toml

Nano tip: press Ctrl + W to search.

Find:

SystemdCgroup = false

Change To:

SystemdCgroup = true

Find:

sandbox_image = "registry.k8s.io/pause:3.8"

Change To:

sandbox_image = "registry.k8s.io/pause:3.10.1"

sudo systemctl daemon-reload
sudo systemctl enable containerd
sudo systemctl restart containerd

sudo ufw disable
sudo systemctl disable ufw

sudo mkdir -p /etc/apt/keyrings

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.35/deb/Release.key | \
sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.35/deb/ /" | \
sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt update
sudo apt install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
sudo systemctl enable kubelet

sudo kubeadm config images pull

sudo systemctl status containerd --no-pager -l

sudo systemctl status kubelet --no-pager -l

swapon --show

cat <<EOF | sudo tee /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF

sudo crictl info

sudo swapoff -a
sudo sed -i.bak '/ swap / s/^/#/' /etc/fstab

swapon --show

kubeadm version

kubectl version --client

sudo apt update && sudo apt upgrade -y

5. 🧹 Clean the VM Before Templating

COMMON · NO CHANGE NEEDEDClean the shared base image before converting it into the reusable worker template.

sudo apt autoremove -y
sudo apt clean
sudo journalctl --vacuum-time=3d
sudo truncate -s 0 /etc/machine-id
sudo rm -f /var/lib/dbus/machine-id
sudo ln -s /etc/machine-id /var/lib/dbus/machine-id

Optional cleanup:

sudo rm -rf /tmp/*
sudo rm -rf /var/tmp/*

Then shut it down:

sudo shutdown -h now

Worker template cleaned and shut down

6. 📀 Create Template

COMMON · NO CHANGE NEEDEDCreate one reusable worker template for the shared cluster rather than one template per product or repository.

Template Name:

tmpl-ac-cicd-env-wk-00

Create the template only after all base configuration and cleanup steps are complete.

Worker template created successfully

7. 🧬 Clone Template for Worker VMs

COMMON · NO CHANGE NEEDEDClone the shared template into environment-based worker pools. Do not create product-specific workers unless capacity, compliance, tooling, or failure-boundary requirements justify dedicated nodes.

Clone the above template and configure each worker VM.

PROD Worker Nodes

COMMON · NO CHANGE NEEDEDThese worker nodes belong to the shared environment pool and are reused by every GitHub organization and repository.

ac-cicd-prod-wk-01:

192.168.8.71 ac-cicd-prod-wk-01

ac-cicd-prod-wk-02:

192.168.8.72 ac-cicd-prod-wk-02

ac-cicd-prod-wk-03:

192.168.8.73 ac-cicd-prod-wk-03

ac-cicd-prod-wk-04:

192.168.8.74 ac-cicd-prod-wk-04

QA Worker Nodes

COMMON · NO CHANGE NEEDEDThese worker nodes belong to the shared environment pool and are reused by every GitHub organization and repository.

ac-cicd-qa-wk-01:

192.168.8.81 ac-cicd-qa-wk-01

ac-cicd-qa-wk-02:

192.168.8.82 ac-cicd-qa-wk-02

ac-cicd-qa-wk-03:

192.168.8.83 ac-cicd-qa-wk-03

ac-cicd-qa-wk-04:

192.168.8.84 ac-cicd-qa-wk-04

DEV Worker Nodes

COMMON · NO CHANGE NEEDEDThese worker nodes belong to the shared environment pool and are reused by every GitHub organization and repository.

ac-cicd-dev-wk-01:

192.168.8.91 ac-cicd-dev-wk-01

ac-cicd-dev-wk-02:

192.168.8.92 ac-cicd-dev-wk-02

ac-cicd-dev-wk-03:

192.168.8.93 ac-cicd-dev-wk-03

ac-cicd-dev-wk-04:

192.168.8.94 ac-cicd-dev-wk-04

Complete configuration of the VM using: Configure VMs

Worker VMs cloned and basic VM configuration completed

8. 🚀 Join Worker Nodes to the Cluster

COMMON · NO CHANGE NEEDEDJoin each environment worker to the shared Kubernetes cluster once; this is not repeated per GitHub organization or repository.

Run the following on each worker after hostname and IP configuration is complete.

kubeadm version

kubectl version --client

Set the hostname of the target worker first.

sudo hostnamectl set-hostname <<TARGET_WORKER_HOSTNAME>>
hostnamectl

Available worker hostnames:

ac-cicd-prod-wk-01
ac-cicd-prod-wk-02
ac-cicd-prod-wk-03
ac-cicd-prod-wk-04
ac-cicd-qa-wk-01
ac-cicd-qa-wk-02
ac-cicd-qa-wk-03
ac-cicd-qa-wk-04
ac-cicd-dev-wk-01
ac-cicd-dev-wk-02
ac-cicd-dev-wk-03
ac-cicd-dev-wk-04

sudo reboot

swapon --show

If swap still shows anything, run:

sudo swapoff -a
sudo sed -i.bak '/ swap / s/^/#/' /etc/fstab

swapon --show

Generate the join command on ac-cicd-cp-01:

kubeadm token create --print-join-command

Join the worker to the cluster using the exact one-line command.

sudo kubeadm join ac-cicd-api.aspireclan.com:443 --token xxxxxx.xxxxxxxxxxxxxxxx --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Worker node joined to the cluster

9. 🔥 Enable Firewall (Post-Bootstrap)

COMMON · NO CHANGE NEEDEDApply the common worker firewall baseline after each node joins the shared cluster.

# kubelet API
sudo ufw allow 10250/tcp

# NodePort services (optional, but keep if you use NodePort)
sudo ufw allow 30000:32767/tcp

sudo ufw allow from 192.168.8.0/22

# Enable firewall
sudo ufw enable

sudo ufw status

Firewall enabled on worker node

10. ✅ Verify from Control Plane

COMMON · NO CHANGE NEEDEDVerify the shared environment worker pools centrally from the control plane.

Run these on ac-cicd-cp-01.

kubectl get nodes -o wide

kubectl get pods -n kube-system -o wide

Worker nodes verified from control plane

11. 🏷️ Apply Labels and Taints to Newly Added Workers

COMMON · NO CHANGE NEEDEDKeep node targeting environment-based with env and workload labels. Do not add app-specific labels by default; organization isolation belongs in namespaces, secrets, runner scale sets, and repository configuration.

Run these on ac-cicd-cp-01 while building the full worker pool so all worker nodes match the same ARC runner targeting pattern.

kubectl label node ac-cicd-dev-wk-01 workload=arc-runner --overwrite
kubectl label node ac-cicd-qa-wk-01 workload=arc-runner --overwrite
kubectl label node ac-cicd-prod-wk-01 workload=arc-runner --overwrite
kubectl label node ac-cicd-prod-wk-02 workload=arc-runner --overwrite
kubectl label node ac-cicd-dev-wk-02 env=dev workload=arc-runner --overwrite
kubectl label node ac-cicd-dev-wk-03 env=dev workload=arc-runner --overwrite
kubectl label node ac-cicd-dev-wk-04 env=dev workload=arc-runner --overwrite
kubectl label node ac-cicd-qa-wk-02 env=qa workload=arc-runner --overwrite
kubectl label node ac-cicd-qa-wk-03 env=qa workload=arc-runner --overwrite
kubectl label node ac-cicd-qa-wk-04 env=qa workload=arc-runner --overwrite
kubectl label node ac-cicd-prod-wk-03 env=prod workload=arc-runner --overwrite
kubectl label node ac-cicd-prod-wk-04 env=prod workload=arc-runner --overwrite

kubectl taint node ac-cicd-dev-wk-02 env=dev:NoSchedule --overwrite
kubectl taint node ac-cicd-dev-wk-03 env=dev:NoSchedule --overwrite
kubectl taint node ac-cicd-dev-wk-04 env=dev:NoSchedule --overwrite
kubectl taint node ac-cicd-qa-wk-02 env=qa:NoSchedule --overwrite
kubectl taint node ac-cicd-qa-wk-03 env=qa:NoSchedule --overwrite
kubectl taint node ac-cicd-qa-wk-04 env=qa:NoSchedule --overwrite
kubectl taint node ac-cicd-prod-wk-03 env=prod:NoSchedule --overwrite
kubectl taint node ac-cicd-prod-wk-04 env=prod:NoSchedule --overwrite

Worker labels and taints applied

12. 🔎 Verify Labels and Taints

COMMON · NO CHANGE NEEDEDConfirm the common environment and ARC-runner labels and taints across the shared worker pools.

Run these on ac-cicd-cp-01.

kubectl get nodes --show-labels

kubectl describe nodes | egrep "Name:|Roles:|Taints:|env=|workload="

kubectl get nodes -L env,workload

Worker labels and taints verified

13. 🔁 After Server Restart (Worker)

COMMON · NO CHANGE NEEDEDUse the same post-restart validation for every shared worker node.

Run these on a worker node after a server restart.

sudo systemctl status kubelet --no-pager -l

sudo systemctl status containerd --no-pager -l

swapon --show

sudo swapoff -a
sudo sed -i.bak '/ swap / s/^/#/' /etc/fstab

swapon --show

sudo systemctl daemon-reload
sudo systemctl restart containerd
sudo systemctl restart kubelet

sudo systemctl status containerd --no-pager -l

sudo systemctl status kubelet --no-pager -l

Run this on ac-cicd-cp-01 to confirm the node is back.

kubectl get nodes -o wide

Post-restart worker checks completed

14. 📋 Worker Inventory

COMMON · NO CHANGE NEEDEDMaintain one shared inventory organized by PROD, QA, and DEV worker pools.

PROD Worker Nodes

COMMON · NO CHANGE NEEDEDThese worker nodes belong to the shared environment pool and are reused by every GitHub organization and repository.

ac-cicd-prod-wk-01:

192.168.8.71 ac-cicd-prod-wk-01

ac-cicd-prod-wk-02:

192.168.8.72 ac-cicd-prod-wk-02

ac-cicd-prod-wk-03:

192.168.8.73 ac-cicd-prod-wk-03

ac-cicd-prod-wk-04:

192.168.8.74 ac-cicd-prod-wk-04

QA Worker Nodes

COMMON · NO CHANGE NEEDEDThese worker nodes belong to the shared environment pool and are reused by every GitHub organization and repository.

ac-cicd-qa-wk-01:

192.168.8.81 ac-cicd-qa-wk-01

ac-cicd-qa-wk-02:

192.168.8.82 ac-cicd-qa-wk-02

ac-cicd-qa-wk-03:

192.168.8.83 ac-cicd-qa-wk-03

ac-cicd-qa-wk-04:

192.168.8.84 ac-cicd-qa-wk-04

DEV Worker Nodes

COMMON · NO CHANGE NEEDEDThese worker nodes belong to the shared environment pool and are reused by every GitHub organization and repository.

ac-cicd-dev-wk-01:

192.168.8.91 ac-cicd-dev-wk-01

ac-cicd-dev-wk-02:

192.168.8.92 ac-cicd-dev-wk-02

ac-cicd-dev-wk-03:

192.168.8.93 ac-cicd-dev-wk-03

ac-cicd-dev-wk-04:

192.168.8.94 ac-cicd-dev-wk-04

15. 🧾 Flat Worker Inventory

COMMON · NO CHANGE NEEDEDUse this consolidated list as the common worker inventory for cluster operations and capacity planning.

192.168.8.71 ac-cicd-prod-wk-01
192.168.8.72 ac-cicd-prod-wk-02
192.168.8.73 ac-cicd-prod-wk-03
192.168.8.74 ac-cicd-prod-wk-04
192.168.8.81 ac-cicd-qa-wk-01
192.168.8.82 ac-cicd-qa-wk-02
192.168.8.83 ac-cicd-qa-wk-03
192.168.8.84 ac-cicd-qa-wk-04
192.168.8.91 ac-cicd-dev-wk-01
192.168.8.92 ac-cicd-dev-wk-02
192.168.8.93 ac-cicd-dev-wk-03
192.168.8.94 ac-cicd-dev-wk-04