Install Shared ARC Controller

1. 🧭 Scope Legend

These markers separate the cluster-shared controller from the organization, repository, and deployment-branch resources created on later pages.

1.COMMON · NO CHANGE NEEDEDConfigure once and reuse across all GitHub organizations and private repositories.

2.CHANGE PER GITHUB ORGRepeat or recreate for every GitHub organization or product.

3.CHANGE PER REPOSITORYRepeat for every private repository inside the GitHub organization.

4.CHANGE PER DEPLOYMENT BRANCHRepeat once for each deployment branch/environment mapping, such as dev, qa, or prod.

2. ✅ Why This Classification Makes Sense

COMMON · NO CHANGE NEEDEDCHANGE PER GITHUB ORGCHANGE PER REPOSITORYCHANGE PER DEPLOYMENT BRANCHThe cluster and ARC controller are shared infrastructure. GitHub identity, runner registration, credentials, and environment placement are introduced later at the organization, repository, and deployment-branch scopes.

Your request is the correct architecture. Install exactly one ARC controller in this shared Kubernetes cluster. Do not install separate controllers for FP, Shelvera, or future products. The controller watches the cluster and reconciles runner scale sets created later for different GitHub organizations, repositories, and deployment environments.

ONE SHARED CLUSTER
  cicd-ac-k8s-api.aspireclan.com / 192.168.8.200
  3 control planes
  4 development workers
  4 QA workers
  4 production workers

ONE SHARED ARC CONTROLLER
  Namespace: arc-systems
  Release:   arc
  Chart:     gha-runner-scale-set-controller 0.14.2

REPEATED LATER
  Per GitHub organization/product:
    GitHub App
    organization/environment namespaces
    Harbor project and organization credentials

  Per private repository:
    runner scale set
    repository GitHub URL
    repository-level Harbor settings

  Per deployment branch/environment:
    dev, qa, or prod namespace mapping
    node selector
    taint toleration
    min/max runner capacity

COMMON · NO CHANGE NEEDED
  Shared Kubernetes cluster
  Helm CLI on cicd-ac-k8s-cp-01
  ARC controller namespace and release
  ARC controller chart version
  ARC controller replicas, resources, logging, and placement
  ARC CRDs and controller RBAC

CHANGE PER GITHUB ORG
  GitHub App and installation
  GitHub App Kubernetes Secret
  Harbor project
  Harbor runtime and CI robot accounts
  organization working folder

CHANGE PER REPOSITORY
  GitHub repository URL
  runner scale-set name and Helm release
  repository runner values file
  repository-level GitHub Actions secrets and variables

CHANGE PER DEPLOYMENT BRANCH
  dev, qa, or prod runner namespace
  environment node selector
  environment taint toleration
  runner capacity and deployment target credentials

Page boundary: this eighth page installs only the shared ARC controller and its CRDs. It does not create a GitHub App, Harbor credentials, runner namespace, listener, or runner scale set. Those are intentionally deferred to the tenant-onboarding page.

3. 🧱 Shared ARC Controller Inputs

COMMON · NO CHANGE NEEDEDThese values are configured once for the shared cluster. The defaults match the approved Aspireclan Kubernetes topology and pin the controller and Helm versions for a clean rebuild.

Administration Hostname

Administration IP

SSH User

Kubeconfig

Helm Version

Helm Linux AMD64 SHA-256

Controller Namespace

Controller Release

Controller Chart Version

Controller Replicas

Request CPU

Request Memory

Limit CPU

Limit Memory

Log Level

4. 🏢 Future Tenant Identity Preview

CHANGE PER GITHUB ORGCHANGE PER REPOSITORYCHANGE PER DEPLOYMENT BRANCHThese blank fields demonstrate how FP, Shelvera, and future products will be isolated later. They do not alter the common controller installation on this page.

Service / App Name

App Short Form

GitHub Organization

Target Environment

Organization and environment namespace:

arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>>

Organization GitHub App secret:

<<APP_SHORT_FORM>>-arc-ghapp-secret

Organization Harbor project:

<<APP_SHORT_FORM>>-ci-cd

Repository and branch runner scale set:

<<SERVICE_NAME>>-<<ENVIRONMENT>>-arc

Repository GitHub configuration URL:

https://github.com/<<GITHUB_ORGANIZATION>>/<<SERVICE_NAME>>

Branch node selector:

environment=<<ENVIRONMENT>>, workload=github-runner

Branch taint toleration:

environment=<<ENVIRONMENT>>:NoSchedule

5. 📊 From-Scratch Sequence Checkpoint

COMMON · NO CHANGE NEEDEDThis checkpoint identifies what the earlier rebuild pages must produce, what this page installs, and what remains for the later onboarding pages.

FROM-SCRATCH SEQUENCE CHECKPOINT

Required before this page
  Load-balancer VM, HAProxy, Keepalived, and API VIP configured
  Three control-plane nodes Ready
  Four development workers Ready and isolated
  Four QA workers Ready and isolated
  Four production workers Ready and isolated
  15 Kubernetes nodes total
  Kubernetes API /readyz returns ok
  GitHub and GHCR outbound access available

Implemented by this page
  Pinned Helm installation on the first control plane
  Shared ARC controller values and Ansible role
  Shared ARC controller Helm release in arc-systems
  Two controller replicas on control-plane nodes
  ARC actions.github.com CRDs

Implemented by later pages
  GitHub organization onboarding
  Repository runner scale sets
  Deployment-branch listener and runner mappings

6. 📁 Files Created by This Page

COMMON · NO CHANGE NEEDEDCreate only the controller values, controller Ansible role, controller playbook, and controller workflow. Preserve every Terraform, control-plane, and worker file created by the preceding rebuild pages.

helm/common/arc-controller/values.yaml
ansible/roles/arc-controller/defaults/main.yml
ansible/roles/arc-controller/tasks/main.yml
ansible/playbooks/shared-k8s/09-install-arc-controller.yml
.github/workflows/ansible-install-arc-controller.yml

Preserve without replacement
  terraform/**
  ansible/inventories/shared-k8s/**
  ansible/roles/common/**
  ansible/roles/containerd/**
  ansible/roles/kubernetes-common/**
  ansible/roles/kubernetes-control-plane/**
  ansible/roles/kubernetes-worker/**
  ansible/playbooks/shared-k8s/01-common-baseline.yml
  ansible/playbooks/shared-k8s/02-configure-load-balancer.yml
  ansible/playbooks/shared-k8s/03-prepare-kubernetes-nodes.yml
  ansible/playbooks/shared-k8s/04-bootstrap-first-control-plane.yml
  ansible/playbooks/shared-k8s/05-join-control-planes.yml
  ansible/playbooks/shared-k8s/06-install-cni.yml
  ansible/playbooks/shared-k8s/07-join-dev-workers.yml
  ansible/playbooks/shared-k8s/07-join-qa-workers.yml
  ansible/playbooks/shared-k8s/07-join-prod-workers.yml
  ansible/playbooks/shared-k8s/08-label-and-taint-workers.yml
  ansible/playbooks/shared-k8s/08-label-and-taint-qa-workers.yml
  ansible/playbooks/shared-k8s/08-label-and-taint-prod-workers.yml
  .github/workflows/ansible-configure-control-planes.yml
  .github/workflows/ansible-configure-dev-workers.yml
  .github/workflows/ansible-configure-qa-workers.yml
  .github/workflows/ansible-configure-prod-workers.yml
  .github/workflows/ansible-configure-shared-k8s.yml
  .github/workflows/ansible-configure-load-balancer.yml

7. 🌿 Create the Feature Branch

COMMON · NO CHANGE NEEDEDUse the approved dev-validation and prod-execution flow for this shared cluster change.

feature/install-shared-arc-controller
    ↓ merge or push
   dev
    ↓ a dev push runs validation only
 dev → prod promotion
    ↓ the merge creates a prod push
   prod
    ↓ validation, installation, and reconciliation

Pull requests may still be used for code review, but no pull_request workflow trigger is configured.

Not used by this infrastructure execution flow:
local, qa, main

cd D:\code\ASPIRECLAN-LLC-Org\ac-cicd-infra

git switch dev
git pull --ff-only origin dev

git switch -c feature/install-shared-arc-controller

8. 🔎 Verify the Prerequisite Cluster and Internet Egress

COMMON · NO CHANGE NEEDEDARC requires the cluster produced by the preceding pages and outbound access to GitHub and GHCR. Verify all 15 Kubernetes nodes before adding the controller.

ssh \
  -i ~/.ssh/id_ed25519_ansible \
  -o IdentitiesOnly=yes \
  acllc@192.168.8.202 \
  'sudo bash -s' <<'REMOTE'
set -euo pipefail
export KUBECONFIG=/etc/kubernetes/admin.conf

echo "=== API READINESS ==="
kubectl get --raw=/readyz

echo
echo "=== ALL NODES ==="
kubectl get nodes -o wide

echo
echo "=== ENVIRONMENT WORKERS ==="
kubectl get nodes -L environment,workload

echo
echo "=== EXACT NODE COUNT ==="
node_count="$(kubectl get nodes -o name | wc -l)"
echo "Kubernetes node count: ${node_count}"
test "${node_count}" -eq 15

echo
echo "=== WAIT FOR EVERY NODE TO BE READY ==="
kubectl wait \
  --for=condition=Ready \
  node \
  --all \
  --timeout=2m

echo
echo "=== EXISTING ARC RESOURCES ==="
if [ -x /usr/local/bin/helm ]; then
  /usr/local/bin/helm list -A
else
  echo "Helm is not installed yet. This is expected before the prod installation."
fi

kubectl get crd | grep actions.github.com || true
kubectl get pods -A | grep -Ei 'arc|runner|listener' || true
REMOTE

ssh \
  -i ~/.ssh/id_ed25519_ansible \
  -o IdentitiesOnly=yes \
  acllc@192.168.8.202 \
  'set -e
   getent hosts ghcr.io
   getent hosts api.github.com
   curl -fsSI --connect-timeout 10 https://ghcr.io/v2/ >/dev/null
   curl -fsSI --connect-timeout 10 https://api.github.com/ >/dev/null
   echo "GitHub and GHCR egress checks passed."'

Cluster readiness and GitHub/GHCR egress verified

9. 📝 Create the Shared Controller Values File

COMMON · NO CHANGE NEEDEDThe controller runs on the control-plane pool rather than consuming a dev, QA, or production worker. Two replicas enable chart-managed leader election and improve controller availability.

Create helm/common/arc-controller/values.yaml:

replicaCount: 2

resources:
  requests:
    cpu: "100m"
    memory: "128Mi"
  limits:
    cpu: "500m"
    memory: "512Mi"

nodeSelector:
  node-role.kubernetes.io/control-plane: ""

tolerations:
  - key: node-role.kubernetes.io/control-plane
    operator: Exists
    effect: NoSchedule

# Prefer placing the two controller replicas on different control-plane VMs.
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          topologyKey: kubernetes.io/hostname
          labelSelector:
            matchLabels:
              app.kubernetes.io/part-of: gha-rs-controller

priorityClassName: system-cluster-critical

flags:
  logLevel: "info"
  logFormat: "json"

Shared ARC controller values file created

10. ⚙️ Create the ARC Controller Role Defaults

COMMON · NO CHANGE NEEDEDPin the Helm binary checksum, controller chart version, namespace, release, kubeconfig, and Git-managed values file.

Create ansible/roles/arc-controller/defaults/main.yml:

---
arc_admin_hostname: "cicd-ac-k8s-cp-01"
arc_admin_ip: "192.168.8.202"
arc_kubeconfig: "/etc/kubernetes/admin.conf"

arc_helm_version: "v3.21.0"
arc_helm_archive_sha256: "0093eb572e3d2380f094df162ddb525e219249de88957afe24cfbb19632acd36"
arc_helm_archive_url: >-
  https://get.helm.sh/helm-{{ arc_helm_version }}-linux-amd64.tar.gz
arc_helm_archive_path: "/tmp/helm-{{ arc_helm_version }}-linux-amd64.tar.gz"
arc_helm_extract_directory: "/tmp/helm-{{ arc_helm_version }}-linux-amd64"
arc_helm_binary_path: "/usr/local/bin/helm"

arc_controller_namespace: "arc-systems"
arc_controller_release_name: "arc"
arc_controller_chart_version: "0.14.2"
arc_controller_chart: >-
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller
arc_controller_replica_count: 2
arc_controller_values_source: >-
  {{ playbook_dir }}/../../../helm/common/arc-controller/values.yaml
arc_controller_work_directory: "/etc/ac-cicd-infra/arc-controller"
arc_controller_values_remote: >-
  {{ arc_controller_work_directory }}/values.yaml

ARC controller role defaults created

11. 🛠️ Create the ARC Controller Role Tasks

COMMON · NO CHANGE NEEDEDInstall the pinned Helm binary, render the pinned chart, perform an atomic Helm reconciliation, wait for the controller, and verify the ARC CRDs.

Create ansible/roles/arc-controller/tasks/main.yml:

---
- name: Confirm the ARC controller is being managed from the first control plane
  ansible.builtin.assert:
    that:
      - inventory_hostname == groups['first_control_plane'][0]
      - inventory_hostname == arc_admin_hostname
      - ansible_facts["default_ipv4"]["address"] == arc_admin_ip
    fail_msg: >-
      The ARC controller playbook must run from the approved first control plane.

- name: Force refresh the APT package cache before installing ARC prerequisites
  ansible.builtin.apt:
    update_cache: true
  register: arc_apt_cache_refresh
  retries: 5
  delay: 15
  until: arc_apt_cache_refresh is succeeded

- name: Install ARC controller administration prerequisites
  ansible.builtin.apt:
    name:
      - ca-certificates
      - curl
      - gzip
      - tar
    state: present
  register: arc_prerequisite_install
  retries: 3
  delay: 10
  until: arc_prerequisite_install is succeeded

- name: Check the currently installed Helm version
  ansible.builtin.command:
    cmd: "{{ arc_helm_binary_path }} version --short"
  register: arc_existing_helm_version
  changed_when: false
  failed_when: false

- name: Download the approved Helm archive
  ansible.builtin.get_url:
    url: "{{ arc_helm_archive_url }}"
    dest: "{{ arc_helm_archive_path }}"
    checksum: "sha256:{{ arc_helm_archive_sha256 }}"
    owner: root
    group: root
    mode: "0644"
  when: arc_helm_version not in (arc_existing_helm_version.stdout | default(''))

- name: Recreate the Helm extraction directory
  ansible.builtin.file:
    path: "{{ arc_helm_extract_directory }}"
    state: absent
  when: arc_helm_version not in (arc_existing_helm_version.stdout | default(''))

- name: Create the Helm extraction directory
  ansible.builtin.file:
    path: "{{ arc_helm_extract_directory }}"
    state: directory
    owner: root
    group: root
    mode: "0755"
  when: arc_helm_version not in (arc_existing_helm_version.stdout | default(''))

- name: Extract the approved Helm archive
  ansible.builtin.unarchive:
    src: "{{ arc_helm_archive_path }}"
    dest: "{{ arc_helm_extract_directory }}"
    remote_src: true
  when: arc_helm_version not in (arc_existing_helm_version.stdout | default(''))

- name: Install the approved Helm binary
  ansible.builtin.copy:
    src: "{{ arc_helm_extract_directory }}/linux-amd64/helm"
    dest: "{{ arc_helm_binary_path }}"
    remote_src: true
    owner: root
    group: root
    mode: "0755"
  when: arc_helm_version not in (arc_existing_helm_version.stdout | default(''))

- name: Verify the installed Helm version
  ansible.builtin.command:
    cmd: "{{ arc_helm_binary_path }} version --short"
  register: arc_installed_helm_version
  changed_when: false
  failed_when: arc_helm_version not in arc_installed_helm_version.stdout

- name: Create the ARC controller working directory
  ansible.builtin.file:
    path: "{{ arc_controller_work_directory }}"
    state: directory
    owner: root
    group: root
    mode: "0755"

- name: Copy the Git-managed ARC controller values file
  ansible.builtin.copy:
    src: "{{ arc_controller_values_source }}"
    dest: "{{ arc_controller_values_remote }}"
    owner: root
    group: root
    mode: "0644"

- name: Confirm the Kubernetes API is ready before installing ARC
  ansible.builtin.command:
    cmd: >-
      kubectl
      --kubeconfig={{ arc_kubeconfig }}
      get --raw=/readyz
  register: arc_api_ready
  changed_when: false
  retries: 12
  delay: 10
  until: arc_api_ready.stdout | trim == "ok"

- name: Render the pinned ARC controller chart before installation
  ansible.builtin.shell:
    executable: /bin/bash
    cmd: |
      set -euo pipefail

      {{ arc_helm_binary_path }} template \
        {{ arc_controller_release_name }} \
        {{ arc_controller_chart }} \
        --namespace {{ arc_controller_namespace }} \
        --version {{ arc_controller_chart_version }} \
        --values {{ arc_controller_values_remote }} \
        --kubeconfig {{ arc_kubeconfig }} \
        > {{ arc_controller_work_directory }}/rendered.yaml
  changed_when: false

- name: Install or reconcile the shared ARC controller
  ansible.builtin.command:
    argv:
      - "{{ arc_helm_binary_path }}"
      - upgrade
      - --install
      - "{{ arc_controller_release_name }}"
      - "{{ arc_controller_chart }}"
      - --namespace
      - "{{ arc_controller_namespace }}"
      - --create-namespace
      - --version
      - "{{ arc_controller_chart_version }}"
      - --values
      - "{{ arc_controller_values_remote }}"
      - --kubeconfig
      - "{{ arc_kubeconfig }}"
      - --atomic
      - --wait
      - --timeout
      - 10m
      - --history-max
      - "10"
  register: arc_controller_helm_apply
  changed_when: >-
    'has been upgraded' in arc_controller_helm_apply.stdout or
    'has been installed' in arc_controller_helm_apply.stdout

- name: Wait for the ARC controller deployment to become available
  ansible.builtin.command:
    cmd: >-
      kubectl
      --kubeconfig={{ arc_kubeconfig }}
      wait
      --namespace {{ arc_controller_namespace }}
      --for=condition=Available
      deployment
      --selector=app.kubernetes.io/part-of=gha-rs-controller
      --timeout=10m
  changed_when: false

- name: Confirm the approved ARC controller replica count is available
  ansible.builtin.shell:
    executable: /bin/bash
    cmd: |
      set -euo pipefail

      available="$({{ arc_helm_binary_path }} status \
        {{ arc_controller_release_name }} \
        --namespace {{ arc_controller_namespace }} \
        --kubeconfig {{ arc_kubeconfig }} >/dev/null && \
        kubectl \
          --kubeconfig={{ arc_kubeconfig }} \
          get deployment \
          --namespace {{ arc_controller_namespace }} \
          --selector=app.kubernetes.io/part-of=gha-rs-controller \
          --output=jsonpath='{.items[0].status.availableReplicas}')"

      test "${available:-0}" = "{{ arc_controller_replica_count }}"
  register: arc_controller_replica_check
  changed_when: false
  retries: 30
  delay: 10
  until: arc_controller_replica_check.rc == 0

- name: Verify ARC custom resource definitions
  ansible.builtin.shell:
    executable: /bin/bash
    cmd: |
      set -euo pipefail

      kubectl \
        --kubeconfig={{ arc_kubeconfig }} \
        get crd \
        --output=name |
        grep -E 'actions\.github\.com$'
  register: arc_crd_check
  changed_when: false

- name: Display the installed ARC controller release
  ansible.builtin.command:
    cmd: >-
      {{ arc_helm_binary_path }} status
      {{ arc_controller_release_name }}
      --namespace {{ arc_controller_namespace }}
      --kubeconfig {{ arc_kubeconfig }}
  register: arc_controller_status
  changed_when: false

- name: Print the ARC controller release status
  ansible.builtin.debug:
    var: arc_controller_status.stdout_lines

ARC controller role tasks created

12. 📘 Create the Shared ARC Controller Playbook

COMMON · NO CHANGE NEEDEDThe playbook targets only the first control plane. The controller pods themselves may run on any control plane allowed by the values file.

Create ansible/playbooks/shared-k8s/09-install-arc-controller.yml:

---
- name: Install the shared Actions Runner Controller
  hosts: first_control_plane
  become: true
  gather_facts: true

  roles:
    - role: arc-controller

Shared ARC controller playbook created

13. 🔄 Create the ARC Controller GitHub Actions Workflow

COMMON · NO CHANGE NEEDEDA push to dev validates the values and Ansible targeting only. A push to prod validates and then installs or reconciles the controller after shared-k8s environment approval. There is no pull_request trigger, narrow path filters isolate ARC changes, and the shared Ansible concurrency group prevents simultaneous cluster mutations.

Create .github/workflows/ansible-install-arc-controller.yml:

name: Ansible Install - Shared ARC Controller

on:
  push:
    branches:
      - dev
      - prod
    paths:
      - "helm/common/arc-controller/**"
      - "ansible/roles/arc-controller/**"
      - "ansible/playbooks/shared-k8s/09-install-arc-controller.yml"
      - ".github/workflows/ansible-install-arc-controller.yml"

  workflow_dispatch:

permissions:
  contents: read

concurrency:
  group: shared-k8s-ansible
  cancel-in-progress: false

env:
  ANSIBLE_CONFIG: ${{ github.workspace }}/ansible/ansible.cfg

jobs:
  validate:
    name: Validate shared ARC controller configuration

    runs-on:
      - self-hosted
      - Linux
      - X64
      - prod
      - terraform
      - deploy
      - ac-cicd-infra

    timeout-minutes: 30

    steps:
      - name: Checkout repository
        uses: actions/checkout@v5

      - name: Verify required files
        shell: bash
        run: |
          set -euo pipefail

          required_files=(
            "helm/common/arc-controller/values.yaml"
            "ansible/roles/arc-controller/defaults/main.yml"
            "ansible/roles/arc-controller/tasks/main.yml"
            "ansible/playbooks/shared-k8s/09-install-arc-controller.yml"
          )

          for file in "${required_files[@]}"; do
            if [ ! -f "${file}" ]; then
              echo "ERROR: Missing ${file}"
              exit 1
            fi
          done

      - name: Parse the ARC controller values YAML
        shell: bash
        run: |
          set -euo pipefail

          python3 - <<'PY'
          from pathlib import Path
          import yaml

          path = Path("helm/common/arc-controller/values.yaml")
          values = yaml.safe_load(path.read_text(encoding="utf-8"))

          assert values["replicaCount"] >= 2
          assert values["nodeSelector"]["node-role.kubernetes.io/control-plane"] == ""
          assert values["priorityClassName"] == "system-cluster-critical"
          assert values["flags"]["logFormat"] == "json"

          print("ARC controller values YAML is valid.")
          PY

      - name: Validate the ARC playbook target
        working-directory: ansible
        shell: bash
        run: |
          set -euo pipefail

          output="$(
            ansible-playbook \
              -i inventories/shared-k8s/hosts.ini \
              playbooks/shared-k8s/09-install-arc-controller.yml \
              --list-hosts
          )"

          printf '%s\n' "${output}"

          grep -Fq "cicd-ac-k8s-cp-01" <<< "${output}"

          if grep -Eq 'cicd-ac-k8s-(dev|qa|prod)-wk-' <<< "${output}"; then
            echo "ERROR: ARC controller playbook resolves to a worker node."
            exit 1
          fi

      - name: Syntax-check the ARC controller playbook
        working-directory: ansible
        shell: bash
        run: |
          set -euo pipefail

          ansible-playbook \
            -i inventories/shared-k8s/hosts.ini \
            playbooks/shared-k8s/09-install-arc-controller.yml \
            --syntax-check

  configure:
    name: Install or reconcile the shared ARC controller

    needs:
      - validate

    if: >-
      (github.event_name == 'push' && github.ref_name == 'prod') ||
      (github.event_name == 'workflow_dispatch' && github.ref_name == 'prod')

    environment:
      name: shared-k8s

    runs-on:
      - self-hosted
      - Linux
      - X64
      - prod
      - terraform
      - deploy
      - ac-cicd-infra

    timeout-minutes: 90

    steps:
      - name: Checkout repository
        uses: actions/checkout@v5

      - name: Verify the production branch
        shell: bash
        run: |
          set -euo pipefail

          if [ "${GITHUB_REF_NAME}" != "prod" ]; then
            echo "ERROR: Shared ARC controller changes are permitted only from prod."
            exit 1
          fi

      - name: Prepare the existing Ansible SSH key
        shell: bash
        run: |
          set -euo pipefail

          KEY_PATH="${HOME}/.ssh/id_ed25519_ansible"

          if [ ! -f "${KEY_PATH}" ]; then
            echo "ERROR: Missing Ansible key: ${KEY_PATH}"
            exit 1
          fi

          chmod 600 "${KEY_PATH}"
          echo "ANSIBLE_PRIVATE_KEY_FILE=${KEY_PATH}" >> "${GITHUB_ENV}"

      - name: Refresh the first-control-plane SSH host key
        shell: bash
        run: |
          set -euo pipefail

          mkdir -p "${HOME}/.ssh"
          chmod 700 "${HOME}/.ssh"
          touch "${HOME}/.ssh/known_hosts"
          chmod 600 "${HOME}/.ssh/known_hosts"

          ssh-keygen \
            -f "${HOME}/.ssh/known_hosts" \
            -R "192.168.8.202" || true

          captured=false

          for attempt in $(seq 1 30); do
            if ssh-keyscan \
              -T 5 \
              -H "192.168.8.202" \
              >> "${HOME}/.ssh/known_hosts" 2>/dev/null
            then
              echo "SSH host key captured for 192.168.8.202."
              captured=true
              break
            fi

            echo "Waiting for SSH on 192.168.8.202 (attempt ${attempt}/30)..."
            sleep 10
          done

          if [ "${captured}" != "true" ]; then
            echo "ERROR: Unable to capture the SSH host key for 192.168.8.202."
            exit 1
          fi

      - name: Prepare the Ansible remote temporary directory
        shell: bash
        run: |
          set -euo pipefail

          ssh \
            -i "${ANSIBLE_PRIVATE_KEY_FILE}" \
            -o IdentitiesOnly=yes \
            -o BatchMode=yes \
            "acllc@192.168.8.202" \
            'sudo install -d -m 0700 -o acllc -g acllc /var/tmp/ansible-acllc'

      - name: Verify Ansible connectivity
        working-directory: ansible
        shell: bash
        run: |
          set -euo pipefail

          ansible \
            -i inventories/shared-k8s/hosts.ini \
            first_control_plane \
            --private-key "${ANSIBLE_PRIVATE_KEY_FILE}" \
            -m ansible.builtin.ping

      - name: Confirm the complete Kubernetes cluster is healthy
        working-directory: ansible
        shell: bash
        run: |
          set -euo pipefail

          ansible \
            -i inventories/shared-k8s/hosts.ini \
            first_control_plane \
            --private-key "${ANSIBLE_PRIVATE_KEY_FILE}" \
            --become \
            --extra-vars ansible_shell_executable=/bin/bash \
            -m ansible.builtin.shell \
            -a '
              set -euo pipefail
              export KUBECONFIG=/etc/kubernetes/admin.conf

              node_count="$(kubectl get nodes -o name | wc -l)"
              echo "Kubernetes node count: ${node_count}"
              test "${node_count}" -eq 15

              kubectl wait \
                --for=condition=Ready \
                node \
                --all \
                --timeout=2m

              kubectl get --raw=/readyz | grep -Fx ok
            '

      - name: Install or reconcile the shared ARC controller
        working-directory: ansible
        shell: bash
        run: |
          set -euo pipefail

          ansible-playbook \
            -i inventories/shared-k8s/hosts.ini \
            --private-key "${ANSIBLE_PRIVATE_KEY_FILE}" \
            playbooks/shared-k8s/09-install-arc-controller.yml

      - name: Verify the shared ARC controller
        working-directory: ansible
        shell: bash
        run: |
          set -euo pipefail

          ansible \
            -i inventories/shared-k8s/hosts.ini \
            first_control_plane \
            --private-key "${ANSIBLE_PRIVATE_KEY_FILE}" \
            --become \
            --extra-vars ansible_shell_executable=/bin/bash \
            -m ansible.builtin.shell \
            -a '
              set -euo pipefail
              export KUBECONFIG=/etc/kubernetes/admin.conf

              test -x /usr/local/bin/helm
              /usr/local/bin/helm status arc -n arc-systems

              kubectl get namespace arc-systems

              controller_deployment="$(
                kubectl get deployment \
                  -n arc-systems \
                  -l app.kubernetes.io/part-of=gha-rs-controller \
                  -o jsonpath="{.items[0].metadata.name}"
              )"

              if [ -z "${controller_deployment}" ]; then
                echo "ERROR: ARC controller deployment was not found."
                exit 1
              fi

              echo "ARC controller deployment: ${controller_deployment}"

              kubectl rollout status \
                "deployment/${controller_deployment}" \
                -n arc-systems \
                --timeout=2m

              kubectl get deployment,pods \
                -n arc-systems \
                -o wide

              kubectl get crd | grep actions.github.com

              available="$(
                kubectl get deployment \
                  "${controller_deployment}" \
                  -n arc-systems \
                  -o jsonpath="{.status.availableReplicas}"
              )"

              echo "Available ARC controller replicas: ${available:-0}"
              test "${available:-0}" -eq 2
            '

Shared ARC controller workflow created

14. 🧪 Review and Commit Only the Common Controller Files

COMMON · NO CHANGE NEEDEDProve that the Terraform, inventory, node roles, and environment worker workflows created by the preceding rebuild pages are unchanged.

git status
git diff --check
git diff --stat

git diff -- \
  helm/common/arc-controller/values.yaml \
  ansible/roles/arc-controller/defaults/main.yml \
  ansible/roles/arc-controller/tasks/main.yml \
  ansible/playbooks/shared-k8s/09-install-arc-controller.yml \
  .github/workflows/ansible-install-arc-controller.yml

git diff --exit-code -- \
  terraform \
  ansible/inventories/shared-k8s \
  ansible/roles/common \
  ansible/roles/containerd \
  ansible/roles/kubernetes-common \
  ansible/roles/kubernetes-worker \
  .github/workflows/ansible-configure-control-planes.yml \
  .github/workflows/ansible-configure-dev-workers.yml \
  .github/workflows/ansible-configure-qa-workers.yml \
  .github/workflows/ansible-configure-prod-workers.yml

The final git diff --exit-code command must return exit code 0.

git add \
  helm/common/arc-controller/values.yaml \
  ansible/roles/arc-controller/defaults/main.yml \
  ansible/roles/arc-controller/tasks/main.yml \
  ansible/playbooks/shared-k8s/09-install-arc-controller.yml \
  .github/workflows/ansible-install-arc-controller.yml

git commit -m "Install shared Actions Runner Controller"

git push -u origin feature/install-shared-arc-controller

15. ✅ Validate Through dev

COMMON · NO CHANGE NEEDEDUse a pull request only for review if desired. The workflow does not run for pull requests; validation starts only after the merge or direct update creates a push to dev.

gh pr create \
  --base dev \
  --head feature/install-shared-arc-controller \
  --title "Install shared Actions Runner Controller" \
  --body "Adds the cluster-shared ARC controller using a pinned Helm chart and an idempotent Ansible workflow. The pull request is for review only; validation starts after the merge creates a dev push."

dev validation completed successfully

16. 🚀 Promote and Install from prod

COMMON · NO CHANGE NEEDEDAfter dev validation, promote the same commit to prod. The resulting prod push runs validation and installation; the shared-k8s environment approval gates the configure job.

gh pr create \
  --base prod \
  --head dev \
  --title "Install shared Actions Runner Controller" \
  --body "Promotes the validated cluster-shared ARC controller installation to prod. The pull request itself does not run the workflow; installation starts after merge creates the prod push."

Expected production sequence:

Validate values and Ansible target
Verify all 15 Kubernetes nodes are Ready
Install or verify Helm v3.21.0
Render ARC controller chart 0.14.2
Install or reconcile arc-systems/arc
Wait for 2 controller replicas
Verify actions.github.com CRDs

Shared ARC controller installed from prod

17. 🔬 Verify the Shared ARC Controller

COMMON · NO CHANGE NEEDEDVerify Helm, the release, controller replicas, control-plane placement, logs, CRDs, and cluster readiness against the clean-rebuild acceptance checkpoint.

ssh \
  -i ~/.ssh/id_ed25519_ansible \
  -o IdentitiesOnly=yes \
  acllc@192.168.8.202 \
  'sudo bash -s' <<'REMOTE'
set -euo pipefail
export KUBECONFIG=/etc/kubernetes/admin.conf
HELM=/usr/local/bin/helm

echo "=== HELM VERSION ==="
test -x "${HELM}"
"${HELM}" version --short

echo
echo "=== ARC RELEASE ==="
"${HELM}" list -n arc-systems
"${HELM}" status arc -n arc-systems

echo
echo "=== ARC NAMESPACE ==="
kubectl get namespace arc-systems

echo
echo "=== ARC DEPLOYMENT AND PODS ==="
kubectl get deployment,pods -n arc-systems -o wide

controller_deployment="$(
  kubectl get deployment \
    -n arc-systems \
    -l app.kubernetes.io/part-of=gha-rs-controller \
    -o jsonpath='{.items[0].metadata.name}'
)"

test -n "${controller_deployment}"
echo "ARC controller deployment: ${controller_deployment}"

kubectl rollout status \
  "deployment/${controller_deployment}" \
  -n arc-systems \
  --timeout=2m

available="$(
  kubectl get deployment \
    "${controller_deployment}" \
    -n arc-systems \
    -o jsonpath='{.status.availableReplicas}'
)"

echo "Available ARC controller replicas: ${available:-0}"
test "${available:-0}" -eq 2

echo
echo "=== ARC CRDS ==="
kubectl get crd | grep actions.github.com

echo
echo "=== ARC CONTROLLER LOGS ==="
kubectl logs \
  -n arc-systems \
  "deployment/${controller_deployment}" \
  --all-containers=true \
  --tail=100

echo
echo "=== CLUSTER READINESS ==="
kubectl get nodes -L environment,workload
kubectl get --raw=/readyz
REMOTE

Shared ARC controller:
  Namespace:       arc-systems
  Helm release:    arc
  Chart version:   0.14.2
  Deployment:      arc-gha-rs-controller
  Controller pods: 2 available
  Placement:       Kubernetes control-plane nodes
  Leader election: enabled because replicaCount is greater than 1
  CRDs:            actions.github.com resources present

Not created by this page:
  GitHub Apps
  organization runner namespaces
  Harbor organization credentials
  repository runner scale sets
  dev, QA, or production listener/runner pods

Next:
  onboard the first GitHub organization/product
  configure organization and branch isolation
  create the first repository runner scale set

FROM-SCRATCH ACCEPTANCE CHECKPOINT

Required Helm state
  Helm binary:        /usr/local/bin/helm
  Helm version:       v3.21.0
  Release:            arc
  Namespace:          arc-systems / Active
  Release status:     deployed
  Controller chart:   0.14.2
  Deployment:         arc-gha-rs-controller
  Desired replicas:   2
  Ready replicas:     2
  Available replicas: 2
  Placement:          two Kubernetes control-plane nodes when scheduling permits

Required ARC CRDs
  autoscalinglisteners.actions.github.com
  autoscalingrunnersets.actions.github.com
  ephemeralrunners.actions.github.com
  ephemeralrunnersets.actions.github.com

Success rule
  Do not continue to organization onboarding until the Helm release is deployed,
  two controller replicas are available, all four CRDs exist, and /readyz returns ok.

Shared ARC controller and CRDs verified

18. 🧩 What Happens on the Next Page

CHANGE PER GITHUB ORGCHANGE PER REPOSITORYCHANGE PER DEPLOYMENT BRANCHThe next page repeats only the tenant-specific pieces. It reuses the controller installed here without reinstalling or modifying it.

COMMON · NO CHANGE NEEDED
  Shared Kubernetes cluster
  Helm CLI on cicd-ac-k8s-cp-01
  ARC controller namespace and release
  ARC controller chart version
  ARC controller replicas, resources, logging, and placement
  ARC CRDs and controller RBAC

CHANGE PER GITHUB ORG
  GitHub App and installation
  GitHub App Kubernetes Secret
  Harbor project
  Harbor runtime and CI robot accounts
  organization working folder

CHANGE PER REPOSITORY
  GitHub repository URL
  runner scale-set name and Helm release
  repository runner values file
  repository-level GitHub Actions secrets and variables

CHANGE PER DEPLOYMENT BRANCH
  dev, qa, or prod runner namespace
  environment node selector
  environment taint toleration
  runner capacity and deployment target credentials

Important: installing the controller does not create a listener or runner pod. Those appear only after a repository or organization runner scale set is installed and successfully authenticates to GitHub.

19. 🩺 Failure Handling

COMMON · NO CHANGE NEEDEDUse these checks for controller installation failures. Do not reset Kubernetes or rebuild any VM for a Helm or ARC chart problem.

Helm checksum or download fails

curl -fL -o /tmp/helm.tar.gz \
  https://get.helm.sh/helm-v3.21.0-linux-amd64.tar.gz

sha256sum /tmp/helm.tar.gz

echo "Expected: 0093eb572e3d2380f094df162ddb525e219249de88957afe24cfbb19632acd36"

dev validation succeeds but Helm and arc-systems are absent

This is expected when only the dev branch has been pushed. The dev workflow validates configuration only. Promote the same commit to prod so the prod push runs the configure job, installs Helm at /usr/local/bin/helm, creates arc-systems, and installs ARC.

test -x /usr/local/bin/helm   && /usr/local/bin/helm version --short   || echo "Helm is not installed yet"

sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf get namespace arc-systems   || echo "ARC namespace is not installed yet"

The workflow fails with $2: unbound variable

Do not embed an awk '$2 == "Ready"' expression inside the single-quoted Ansible ad-hoc shell argument. The outer GitHub runner shell can consume $2. Usekubectl wait --for=condition=Ready node --all and explicitly select/bin/bash with ansible_shell_executable, as implemented in the workflow above.

An ARC-only push starts unrelated Ansible workflows

Remove broad ansible/** path filters from component workflows. Keep this ARC workflow limited to the controller values, role, playbook, and workflow file, and use the sharedshared-k8s-ansible concurrency group across cluster-changing Ansible workflows.

Controller pods remain Pending

sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf get pods -n arc-systems -o wide
sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf describe pods -n arc-systems
sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf get nodes --show-labels
sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf describe node cicd-ac-k8s-cp-01 | grep -A4 -E 'Taints:|Labels:'

Helm release fails atomically

sudo /usr/local/bin/helm history arc -n arc-systems
sudo /usr/local/bin/helm status arc -n arc-systems || true
sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf get events -n arc-systems --sort-by=.lastTimestamp
sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf get pods -n arc-systems -o wide

Controller is Running but CRDs are missing

sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf get crd | grep actions.github.com || true
sudo /usr/local/bin/helm get manifest arc -n arc-systems | grep -n 'CustomResourceDefinition' || true
sudo /usr/local/bin/helm upgrade --install arc \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller \
  --namespace arc-systems \
  --version 0.14.2 \
  --values /etc/ac-cicd-infra/arc-controller/values.yaml \
  --atomic --wait --timeout 10m

Rollback a bad controller upgrade

sudo /usr/local/bin/helm history arc -n arc-systems
sudo /usr/local/bin/helm rollback arc <PREVIOUS_REVISION> \
  -n arc-systems \
  --wait \
  --timeout 10m

20. 🏁 Rebuild Status After Successful Completion

COMMON · NO CHANGE NEEDEDCHANGE PER GITHUB ORGCHANGE PER REPOSITORYCHANGE PER DEPLOYMENT BRANCHAfter this page succeeds during a clean rebuild, the shared cluster and controller meet the required acceptance checkpoint. The remaining work is tenant onboarding and runner scale-set isolation.

FROM-SCRATCH REBUILD CHECKPOINT AFTER THIS PAGE

Expected infrastructure
  Load balancer, API VIP, and three control planes operational
  Development, QA, and production worker pools Ready
  15 Kubernetes nodes Ready

Expected shared ARC controller
  Values file and Ansible role committed
  Helm v3.21.0 installed on cicd-ac-k8s-cp-01
  ARC chart 0.14.2 deployed as arc in arc-systems
  Two controller replicas available
  Four actions.github.com CRDs installed

Next documentation steps
  GitHub organization onboarding
  Repository runner scale sets
  Deployment-branch runner mappings

Consistency rule
  This checkpoint records the required result of executing this page during a clean rebuild;
  it does not assume any previously retained VM, Helm release, namespace, or ARC resource.