Configure ARC Runners

1. 🧭 Scope Legend

Use these scope markers throughout this page and the wider Kubernetes CI/CD documentation.

1.COMMON · NO CHANGE NEEDEDConfigure once and reuse across all GitHub organizations and private repositories.

2.CHANGE PER GITHUB ORGRepeat or recreate for every GitHub organization or product.

3.CHANGE PER REPOSITORYRepeat for every private repository inside the GitHub organization.

4.CHANGE PER DEPLOYMENT BRANCHRepeat once for each deployment branch/environment mapping, such as dev, qa, or prod.

2. 🧱 Shared ARC Infrastructure Inputs

COMMON · NO CHANGE NEEDEDThese values describe the existing shared control plane, Harbor endpoint, ARC controller, runner image, worker labels, and resource defaults. Configure them once and reuse them across organizations.

CP-01 Hostname

CP-01 IP

CP-01 SSH User

Windows PEM Directory

Harbor Hostname

Runner Image Name

Runner Image Tag

Workload Label

Listener Request CPU

Listener Request Memory

Listener Limit CPU

Listener Limit Memory

Runner Request CPU

Runner Request Memory

Runner Limit CPU

Runner Limit Memory

ARC Controller Namespace

ARC Controller Release

ARC Controller Chart Version

Runner Scale Set Chart Version

3. 🏢 Generic GitHub Organization and Repository Inputs

CHANGE PER GITHUB ORGCHANGE PER REPOSITORYCHANGE PER DEPLOYMENT BRANCHUse one blank reusable input set. The app short form, GitHub organization, and environment define organization isolation; the service/app name identifies the private repository and its runner scale set.

Service / App Name

App Short Form

GitHub Organization

Target Environment

CHANGE PER REPOSITORYCHANGE PER DEPLOYMENT BRANCHSet repository-specific runner capacity. Plan the combined maximum across every organization using the same environment worker pool.

Min Runners

Max Runners

Enter the four generic identity fields. Placeholder examples are never treated as entered values. The app short form must be unique inside the shared cluster because it becomes part of namespaces, secrets, Harbor projects, and working folders.

4. 🧮 Derived Organization and Repository Values

CHANGE PER GITHUB ORGCHANGE PER REPOSITORYCHANGE PER DEPLOYMENT BRANCHReview the generated names before creating Kubernetes, Harbor, GitHub App, or Helm resources. Organization-scoped resources are reused by repositories in the same organization and environment; scale sets and values files remain repository-specific.

Shared ARC Controller:

arc-systems/arc (chart 0.14.0)

Organization Working Folder:

~/arc/<<APP_SHORT_FORM>>

ARC Runner Namespace:

arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>>

GitHub App Secret:

<<APP_SHORT_FORM>>-arc-ghapp-secret

Harbor Project:

<<APP_SHORT_FORM>>-ci-cd

Harbor Pull Secret:

<<APP_SHORT_FORM>>-harbor-regcred

Harbor CI Credentials Secret:

<<APP_SHORT_FORM>>-harbor-credentials

Runner Scale Set / Helm Release:

<<SERVICE_NAME>>-<<ENVIRONMENT>>-arc

GitHub Repository URL:

https://github.com/<<GITHUB_ORGANIZATION>>/<<SERVICE_NAME>>

Runner Image:

harbor.aspireclan.com/<<APP_SHORT_FORM>>-ci-cd/actions-runner-azcli:v1.0.0

Runner Values File:

~/arc/<<APP_SHORT_FORM>>/<<SERVICE_NAME>>-<<ENVIRONMENT>>-values.yaml

5. 🔎 Verify Current Cluster State

COMMON · NO CHANGE NEEDEDInspect the existing shared cluster and ARC controller before creating any organization or repository resources.

Run these on ac-cicd-cp-01.

kubectl config current-context

kubectl get nodes -o wide

kubectl get ns

kubectl get crd | grep actions.github.com || true

kubectl get pods -A | egrep 'arc|runner|listener' || true

helm version || true

helm list -A || true

Existing shared cluster state verified

6. 🔒 Verify Organization Isolation Before Changes

CHANGE PER GITHUB ORGCHANGE PER DEPLOYMENT BRANCHConfirm that the derived namespace, GitHub App secret, Harbor project, pull secret, CI credentials secret, and working folder are unique to this app short form and environment.

echo "Target GitHub organization: <<GITHUB_ORGANIZATION>>"
echo "Target app short form: <<APP_SHORT_FORM>>"
echo "Target environment: <<ENVIRONMENT>>"
echo "Target namespace: arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>>"
echo "GitHub App secret: <<APP_SHORT_FORM>>-arc-ghapp-secret"
echo "Harbor project: <<APP_SHORT_FORM>>-ci-cd"
echo "Harbor pull secret: <<APP_SHORT_FORM>>-harbor-regcred"
echo "Harbor CI credentials secret: <<APP_SHORT_FORM>>-harbor-credentials"

kubectl get namespace arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>> 2>/dev/null || true
kubectl get secrets -n arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>> 2>/dev/null || true
helm list -n arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>> 2>/dev/null || true

Organization namespace and secret isolation verified

7. 🧹 Optional One-Time Cleanup of a Misplaced Repository Runner

CHANGE PER REPOSITORYCHANGE PER DEPLOYMENT BRANCHUse this only when the same repository scale set was previously installed in the wrong namespace. The release name is repository-specific, so the commands do not remove other repositories.

Previous Runner Namespace

helm list -n <<PREVIOUS_RUNNER_NAMESPACE>> | grep -F "<<SERVICE_NAME>>-<<ENVIRONMENT>>-arc" || true
kubectl get autoscalingrunnersets -n <<PREVIOUS_RUNNER_NAMESPACE>> | grep -F "<<SERVICE_NAME>>-<<ENVIRONMENT>>-arc" || true

helm uninstall <<SERVICE_NAME>>-<<ENVIRONMENT>>-arc   --namespace <<PREVIOUS_RUNNER_NAMESPACE>> || true

kubectl get autoscalingrunnersets -n <<PREVIOUS_RUNNER_NAMESPACE>> | grep -F "<<SERVICE_NAME>>-<<ENVIRONMENT>>-arc" || true

Misplaced repository runner release removed, if applicable

8. 🏷️ Verify Shared Worker Labels and Taints

COMMON · NO CHANGE NEEDEDAll organizations use the shared environment worker pools by default. Keep environment and workload labels; do not introduce app-specific node labels unless hard compute isolation is required.

Run these on ac-cicd-cp-01.

kubectl get nodes --show-labels

kubectl describe nodes | egrep "Name:|Roles:|Taints:|env=|workload="

kubectl get nodes -L env,workload

Shared worker environment and workload labels verified

Capacity rule: namespaces provide logical isolation but do not reserve physical resources. Plan the combined maxRunners across all organizations against the CPU, RAM, disk, and Docker capacity of the selected environment worker pool.

9. 📁 Prepare Organization ARC Working Folder

CHANGE PER GITHUB ORGCreate one organization-scoped folder under the shared ARC root and reuse it for every repository belonging to that organization.

mkdir -p ~/arc
mkdir -p ~/arc/<<APP_SHORT_FORM>>
cd ~/arc/<<APP_SHORT_FORM>>

Organization ARC working folder created

10. 📦 Install Helm

COMMON · NO CHANGE NEEDEDInstall Helm once on the shared control-plane administration host. Do not reinstall it for each organization.

sudo apt-get install curl gpg apt-transport-https --yes

curl -fsSL https://packages.buildkite.com/helm-linux/helm-debian/gpgkey | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null

echo "deb [signed-by=/usr/share/keyrings/helm.gpg] https://packages.buildkite.com/helm-linux/helm-debian/any/ any main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list

sudo apt-get update
sudo apt-get install helm
helm version

Helm installed successfully

11. ⚙️ Create Shared ARC Controller Values File

COMMON · NO CHANGE NEEDEDThe controller is cluster-shared. Create and maintain one controller values file for all current and future GitHub organizations.

cat > ~/arc/arc-controller-values.yaml <<'EOF'
replicaCount: 1

nodeSelector:
  node-role.kubernetes.io/control-plane: ""

tolerations:
  - key: node-role.kubernetes.io/control-plane
    operator: Exists
    effect: NoSchedule

resources:
  requests:
    cpu: "100m"
    memory: "128Mi"
  limits:
    cpu: "500m"
    memory: "512Mi"
EOF

Shared ARC controller values file created

12. 🚀 Install Shared ARC Controller Only When Missing

COMMON · NO CHANGE NEEDEDDo not install a separate controller per GitHub organization. Treat controller upgrades as planned shared-cluster maintenance.

if helm status arc -n arc-systems >/dev/null 2>&1; then
  echo "Shared ARC controller arc-systems/arc already exists. Skipping installation."
  helm status arc -n arc-systems
else
  helm upgrade --install arc     --namespace arc-systems     --create-namespace     --version 0.14.0     -f ~/arc/arc-controller-values.yaml     oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller
fi

Shared ARC controller present

13. ✅ Verify Shared ARC Controller

COMMON · NO CHANGE NEEDEDVerify the single controller release, deployment, pods, and ARC custom resource definitions.

helm list -n arc-systems

helm status arc -n arc-systems

kubectl get pods -n arc-systems -o wide

kubectl get deploy -n arc-systems

kubectl get crd | grep actions.github.com

Shared ARC controller verified

14. 🔐 Copy GitHub App PEM to the Organization Working Folder

CHANGE PER GITHUB ORGUse one GitHub App per GitHub organization/product. Keep its private key in the organization-scoped ARC folder and never persist the key or its contents in the browser.

GitHub App PEM Filename

Run this from the Windows 11 machine PowerShell.

ssh acllc@192.168.8.62 "mkdir -p ~/arc/<<APP_SHORT_FORM>>"

scp "C:\Users\Manoj\Downloads\pem\<<GITHUB_APP_PEM_FILE>>" acllc@192.168.8.62:~/arc/<<APP_SHORT_FORM>>/

Back on ac-cicd-cp-01:

ls -l ~/arc/<<APP_SHORT_FORM>>/*.pem

chmod 600 ~/arc/<<APP_SHORT_FORM>>/*.pem

GitHub App requirements for repository-scoped ARC:

Repository permissions: Administration — Read and write and Metadata — Read-only. Organization permissions: Self-hosted runners — Read and write. Install the GitHub App for the repositories that will use ARC runners, and approve permission changes in the organization.

Installation ID tip: GitHub organization → Settings → GitHub Apps → Installed GitHub Apps → Configure. The browser URL ends with the Installation ID.

https://github.com/organizations/<<GITHUB_ORGANIZATION>>/settings/installations/INSTALLATION_ID

GitHub App permissions, installation access, PEM, and Installation ID verified

15. 🧬 Configure the Organization Runner Namespace

CHANGE PER GITHUB ORGCHANGE PER DEPLOYMENT BRANCHCreate one isolated ARC runner namespace per app short form and environment. Multiple private repositories in the same organization and environment share this namespace while keeping separate runner scale sets.

mkdir -p ~/arc/<<APP_SHORT_FORM>>
cd ~/arc/<<APP_SHORT_FORM>>

kubectl create namespace arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>>   --dry-run=client   -o yaml | kubectl apply -f -

kubectl get ns arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>>

Organization runner namespace created

🐳 Create the Organization Harbor Pull Secret

CHANGE PER GITHUB ORGCHANGE PER DEPLOYMENT BRANCHCreate this shared pull secret once in each organization/environment namespace. Every repository runner scale set in that namespace references it; do not recreate it during each repository workflow run.

Runtime Robot Username

Runtime Robot Password

read -p "Harbor runtime robot username: " HARBOR_RUNTIME_USERNAME
read -s -p "Harbor runtime robot password: " HARBOR_RUNTIME_PASSWORD
echo

Copy-pastable Harbor runtime credentials:

Runtime Robot Username:

<<ENTER_RUNTIME_ROBOT_USERNAME>>

Runtime Robot Password:

<<ENTER_RUNTIME_ROBOT_PASSWORD>>

kubectl create secret docker-registry <<APP_SHORT_FORM>>-harbor-regcred   --namespace arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>>   --docker-server=harbor.aspireclan.com   --docker-username="$HARBOR_RUNTIME_USERNAME"   --docker-password="$HARBOR_RUNTIME_PASSWORD"   --docker-email='noreply@aspireclan.com'   --dry-run=client -o yaml | kubectl apply -f -

kubectl get secret <<APP_SHORT_FORM>>-harbor-regcred -n arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>>

unset HARBOR_RUNTIME_USERNAME HARBOR_RUNTIME_PASSWORD

echo "USER=[$HARBOR_RUNTIME_USERNAME]"
echo "PASS=[$HARBOR_RUNTIME_PASSWORD]"

Organization Harbor pull secret created with Pull-only runtime credentials

16. 🔑 Create the Organization GitHub App Secret

CHANGE PER GITHUB ORGCHANGE PER DEPLOYMENT BRANCHCreate one GitHub App secret in each organization/environment runner namespace. The App ID, Installation ID, and private key are sensitive and never persist in localStorage.

GitHub App ID

GitHub App Installation ID

kubectl create secret generic <<APP_SHORT_FORM>>-arc-ghapp-secret   --namespace arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>>   --from-literal=github_app_id='<<ENTER_GITHUB_APP_ID>>'   --from-literal=github_app_installation_id='<<ENTER_GITHUB_APP_INSTALLATION_ID>>'   --from-file=github_app_private_key=$HOME/arc/<<APP_SHORT_FORM>>/<<GITHUB_APP_PEM_FILE>>   --dry-run=client -o yaml | kubectl apply -f -

kubectl get secret <<APP_SHORT_FORM>>-arc-ghapp-secret -n arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>>

echo -n "App ID: "
kubectl get secret <<APP_SHORT_FORM>>-arc-ghapp-secret   -n arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>>   -o jsonpath='{.data.github_app_id}' | base64 -d
echo

echo -n "Installation ID: "
kubectl get secret <<APP_SHORT_FORM>>-arc-ghapp-secret   -n arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>>   -o jsonpath='{.data.github_app_installation_id}' | base64 -d
echo

Organization GitHub App secret created

17. 🤖 Create the Organization Harbor CI Credentials Secret

CHANGE PER GITHUB ORGCHANGE PER DEPLOYMENT BRANCHUse the Harbor CI robot with Pull and Push permissions. ARC runner jobs consume this secret when they authenticate to build and push images.

CI Robot Username

CI Robot Password

read -p "Harbor CI robot username: " HARBOR_USERNAME
read -s -p "Harbor CI robot password: " HARBOR_PASSWORD
echo

Copy-pastable Harbor CI credentials:

CI Robot Username:

<<ENTER_CI_ROBOT_USERNAME>>

CI Robot Password:

<<ENTER_CI_ROBOT_PASSWORD>>

kubectl create secret generic <<APP_SHORT_FORM>>-harbor-credentials   --namespace arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>>   --from-literal=HARBOR_USERNAME="$HARBOR_USERNAME"   --from-literal=HARBOR_PASSWORD="$HARBOR_PASSWORD"   --dry-run=client -o yaml | kubectl apply -f -

kubectl get secret <<APP_SHORT_FORM>>-harbor-credentials -n arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>>

kubectl get secret <<APP_SHORT_FORM>>-harbor-credentials   -n arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>>   -o jsonpath='{.type}{"\n"}{.data.HARBOR_USERNAME}{"\n"}{.data.HARBOR_PASSWORD}{"\n"}'

unset HARBOR_USERNAME HARBOR_PASSWORD

echo "USER=[$HARBOR_USERNAME]"
echo "PASS=[$HARBOR_PASSWORD]"

Organization Harbor CI credentials secret created

18. 🔏 Configure GitHub Free Repository-Level Harbor Settings

CHANGE PER REPOSITORYPrivate repositories on GitHub Free receive Harbor credentials and configuration at repository scope. Do not depend on organization-level or environment-level Actions secrets or variables.

read -p "Harbor CI robot username: " HARBOR_USERNAME
read -s -p "Harbor CI robot password: " HARBOR_PASSWORD
echo

REPOSITORIES=(
  "<<SERVICE_NAME>>"
  # "another-private-repository"
)

for REPOSITORY in "${REPOSITORIES[@]}"; do
  gh secret set HARBOR_USERNAME     --repo "<<GITHUB_ORGANIZATION>>/${REPOSITORY}"     --body "$HARBOR_USERNAME"

  gh secret set HARBOR_PASSWORD     --repo "<<GITHUB_ORGANIZATION>>/${REPOSITORY}"     --body "$HARBOR_PASSWORD"

  gh variable set HARBOR_REGISTRY     --repo "<<GITHUB_ORGANIZATION>>/${REPOSITORY}"     --body "harbor.aspireclan.com"

  gh variable set HARBOR_PROJECT     --repo "<<GITHUB_ORGANIZATION>>/${REPOSITORY}"     --body "<<APP_SHORT_FORM>>-ci-cd"
done

unset HARBOR_USERNAME HARBOR_PASSWORD

REPOSITORIES=(
  "<<SERVICE_NAME>>"
  # "another-private-repository"
)

for REPOSITORY in "${REPOSITORIES[@]}"; do
  echo "===== <<GITHUB_ORGANIZATION>>/${REPOSITORY} ====="
  gh secret list --repo "<<GITHUB_ORGANIZATION>>/${REPOSITORY}"
  gh variable list --repo "<<GITHUB_ORGANIZATION>>/${REPOSITORY}"
done

Repository-level Harbor secrets and variables configured

19. 📝 Create the Repository Runner Scale Set Values File

CHANGE PER REPOSITORYCHANGE PER DEPLOYMENT BRANCHCreate one values file per private repository and environment. It references the organization-scoped GitHub App and Harbor secrets while retaining a repository-specific scale-set name and GitHub URL.

Values File:

~/arc/<<APP_SHORT_FORM>>/<<SERVICE_NAME>>-<<ENVIRONMENT>>-values.yaml

cat > ~/arc/<<APP_SHORT_FORM>>/<<SERVICE_NAME>>-<<ENVIRONMENT>>-values.yaml <<'EOF'
githubConfigUrl: "https://github.com/<<GITHUB_ORGANIZATION>>/<<SERVICE_NAME>>"
githubConfigSecret: <<APP_SHORT_FORM>>-arc-ghapp-secret

runnerScaleSetName: "<<SERVICE_NAME>>-<<ENVIRONMENT>>-arc"

minRunners: 0
maxRunners: 4

listenerTemplate:
  spec:
    nodeSelector:
      env: <<ENVIRONMENT>>
      workload: arc-runner
    tolerations:
      - key: env
        operator: Equal
        value: <<ENVIRONMENT>>
        effect: NoSchedule
    containers:
      - name: listener
        resources:
          requests:
            cpu: "250m"
            memory: "256Mi"
          limits:
            cpu: "500m"
            memory: "512Mi"

template:
  spec:
    nodeSelector:
      env: <<ENVIRONMENT>>
      workload: arc-runner
    tolerations:
      - key: env
        operator: Equal
        value: <<ENVIRONMENT>>
        effect: NoSchedule

    imagePullSecrets:
      - name: <<APP_SHORT_FORM>>-harbor-regcred

    initContainers:
      - name: init-dind-externals
        image: harbor.aspireclan.com/<<APP_SHORT_FORM>>-ci-cd/actions-runner-azcli:v1.0.0
        imagePullPolicy: IfNotPresent
        command: ["cp", "-r", "/home/runner/externals/.", "/home/runner/tmpDir/"]
        volumeMounts:
          - name: dind-externals
            mountPath: /home/runner/tmpDir

      - name: dind
        image: docker:dind
        args:
          - dockerd
          - --host=unix:///var/run/docker.sock
          - --group=$(DOCKER_GROUP_GID)
          - --feature=cdi=false
        env:
          - name: DOCKER_GROUP_GID
            value: "123"
        securityContext:
          privileged: true
        restartPolicy: Always
        startupProbe:
          exec:
            command: ["docker", "info"]
          initialDelaySeconds: 0
          failureThreshold: 24
          periodSeconds: 5
        volumeMounts:
          - name: work
            mountPath: /home/runner/_work
          - name: dind-sock
            mountPath: /var/run
          - name: dind-externals
            mountPath: /home/runner/externals

    containers:
      - name: runner
        image: harbor.aspireclan.com/<<APP_SHORT_FORM>>-ci-cd/actions-runner-azcli:v1.0.0
        imagePullPolicy: IfNotPresent
        command: ["/home/runner/run.sh"]
        env:
          - name: DOCKER_HOST
            value: unix:///var/run/docker.sock
          - name: RUNNER_WAIT_FOR_DOCKER_IN_SECONDS
            value: "120"
          - name: HARBOR_USERNAME
            valueFrom:
              secretKeyRef:
                name: <<APP_SHORT_FORM>>-harbor-credentials
                key: HARBOR_USERNAME
          - name: HARBOR_PASSWORD
            valueFrom:
              secretKeyRef:
                name: <<APP_SHORT_FORM>>-harbor-credentials
                key: HARBOR_PASSWORD
        resources:
          requests:
            cpu: "1000m"
            memory: "2Gi"
          limits:
            cpu: "2000m"
            memory: "4Gi"
        volumeMounts:
          - name: work
            mountPath: /home/runner/_work
          - name: dind-sock
            mountPath: /var/run

    volumes:
      - name: work
        emptyDir: {}
      - name: dind-sock
        emptyDir: {}
      - name: dind-externals
        emptyDir: {}
EOF

Repository runner scale set values file created

20. 🔍 Review Repository Values and Organization Secret References

CHANGE PER REPOSITORYCHANGE PER DEPLOYMENT BRANCHConfirm that the repository values file points to the correct repository URL, scale-set name, organization secrets, shared worker labels, and resource limits.

grep -n -E "githubConfigUrl:|githubConfigSecret:|runnerScaleSetName:|imagePullSecrets:|name: <<APP_SHORT_FORM>>-harbor-regcred|name: <<APP_SHORT_FORM>>-harbor-credentials" ~/arc/<<APP_SHORT_FORM>>/<<SERVICE_NAME>>-<<ENVIRONMENT>>-values.yaml

grep -n -A12 -B2 "listenerTemplate" ~/arc/<<APP_SHORT_FORM>>/<<SERVICE_NAME>>-<<ENVIRONMENT>>-values.yaml

grep -n -A28 -B5 "HARBOR_USERNAME" ~/arc/<<APP_SHORT_FORM>>/<<SERVICE_NAME>>-<<ENVIRONMENT>>-values.yaml

grep -n -A14 -B5 "nodeSelector" ~/arc/<<APP_SHORT_FORM>>/<<SERVICE_NAME>>-<<ENVIRONMENT>>-values.yaml

grep -n -A12 -B2 "resources:" ~/arc/<<APP_SHORT_FORM>>/<<SERVICE_NAME>>-<<ENVIRONMENT>>-values.yaml

Repository values and organization secret references reviewed

21. 🚀 Install the Repository Runner Scale Set

CHANGE PER REPOSITORYCHANGE PER DEPLOYMENT BRANCHInstall one Helm release per private repository and environment into the organization/environment namespace.

helm upgrade --install <<SERVICE_NAME>>-<<ENVIRONMENT>>-arc   --namespace arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>>   --create-namespace   --version 0.14.0   -f ~/arc/<<APP_SHORT_FORM>>/<<SERVICE_NAME>>-<<ENVIRONMENT>>-values.yaml   --wait   --timeout 5m   oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set

Repository runner scale set installed

22. ✅ Verify Repository Runner Registration and Placement

CHANGE PER REPOSITORYCHANGE PER DEPLOYMENT BRANCHVerify the repository-specific Helm release and ARC resources while confirming that listener and runner pods use the shared environment worker pool.

helm status <<SERVICE_NAME>>-<<ENVIRONMENT>>-arc -n arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>>

kubectl get autoscalingrunnersets -n arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>> -o wide

kubectl get pods -n arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>> -o wide

kubectl get pods -n arc-systems -o wide | grep -F "<<SERVICE_NAME>>-<<ENVIRONMENT>>-arc" || true

kubectl get pods -A -o wide | egrep "arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>>|<<SERVICE_NAME>>-<<ENVIRONMENT>>-arc|listener|runner" || true

kubectl get nodes -L env,workload

kubectl logs   -n arc-systems   deployment/arc-gha-rs-controller   --all-containers=true   --since=5m | grep -Ei '<<SERVICE_NAME>>|<<GITHUB_ORGANIZATION>>|error|forbidden|listener|scale set' || true

Repository runner registered and placed on the correct shared worker pool

23. 📡 Watch Repository Runner Deployment

CHANGE PER REPOSITORYCHANGE PER DEPLOYMENT BRANCHUse the shared-cluster watch for broad troubleshooting and the namespace/release filters for the selected private repository.

kubectl get pods -A -o wide -w

Repository-specific commands:

kubectl get pods -n arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>> -w

kubectl get pods -n arc-systems -w | grep --line-buffered -F "<<SERVICE_NAME>>-<<ENVIRONMENT>>-arc"

Repository ARC runners verified and watch commands completed

24. 🩺 Troubleshoot a Missing ARC Listener

CHANGE PER GITHUB ORGCHANGE PER REPOSITORYCHANGE PER DEPLOYMENT BRANCHUse this when the Helm release exists but the expected listener pod is missing. The probe validates the exact GitHub App identity, installation, repository visibility, granted permissions, and registration endpoints used by ARC.

Failure boundary: ARC creates the listener only after it successfully registers the runner scale set with GitHub. A Helm release can show as deployed while GitHub registration is still failing. With minRunners: 0, no runner pod is normal until a workflow is queued, but the listener should still exist after successful registration.

24.1 Confirm the failure occurs before listener creation

kubectl get autoscalingrunnersets -n arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>> -o wide
kubectl get pods -n arc-systems -o wide | grep -F "<<SERVICE_NAME>>-<<ENVIRONMENT>>-arc" || true
kubectl logs   -n arc-systems   deployment/arc-gha-rs-controller   --all-containers=true   --since=10m | grep -Ei '<<SERVICE_NAME>>|<<GITHUB_ORGANIZATION>>|registration-token|403|404|forbidden|error|listener|scale set' || true

If the controller log shows a failed call to a repository or organization runner registration-token endpoint, continue with the probe below instead of troubleshooting node scheduling or image pulls.

24.2 Run the definitive GitHub App probe

Run this on ac-cicd-cp-01. The script reads the same Kubernetes Secret used by ARC and tests the same GitHub endpoints. It does not print the private key, installation token, or runner registration tokens.

cat > /tmp/arc-github-app-probe.sh <<'EOF'
#!/usr/bin/env bash
set -Eeuo pipefail

NAMESPACE="arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>>"
SECRET_NAME="<<APP_SHORT_FORM>>-arc-ghapp-secret"
GITHUB_ORG="<<GITHUB_ORGANIZATION>>"
GITHUB_REPO="<<SERVICE_NAME>>"
API_VERSION="2022-11-28"

for command_name in kubectl jq curl openssl base64; do
  if ! command -v "${command_name}" >/dev/null 2>&1; then
    echo "ERROR: Required command not found: ${command_name}"
    exit 1
  fi
done

TMP_DIR="$(mktemp -d)"
PRIVATE_KEY_FILE="${TMP_DIR}/github-app.pem"

cleanup() {
  rm -rf "${TMP_DIR}"
}
trap cleanup EXIT

decode_secret_value() {
  local key="$1"

  kubectl get secret "${SECRET_NAME}"     --namespace "${NAMESPACE}"     --output "jsonpath={.data.${key}}" |
    base64 --decode
}

echo
echo "===== 1. Reading GitHub App identity from Kubernetes ====="

APP_ID="$(decode_secret_value github_app_id)"
INSTALLATION_ID="$(decode_secret_value github_app_installation_id)"

decode_secret_value github_app_private_key > "${PRIVATE_KEY_FILE}"
chmod 600 "${PRIVATE_KEY_FILE}"

echo "Namespace:       ${NAMESPACE}"
echo "Secret:          ${SECRET_NAME}"
echo "App ID:          ${APP_ID}"
echo "Installation ID: ${INSTALLATION_ID}"
echo "Target:          ${GITHUB_ORG}/${GITHUB_REPO}"

if ! openssl pkey   -in "${PRIVATE_KEY_FILE}"   -check   -noout >/dev/null 2>&1; then
  echo "ERROR: The private key stored in Kubernetes is not a valid private key."
  exit 1
fi

echo "Private key:     structurally valid"

base64_url_encode() {
  openssl base64 -A |
    tr '+/' '-_' |
    tr -d '='
}

NOW="$(date +%s)"
ISSUED_AT="$((NOW - 60))"
EXPIRES_AT="$((NOW + 540))"

JWT_HEADER="$(
  printf '%s' '{"alg":"RS256","typ":"JWT"}' |
    base64_url_encode
)"

JWT_PAYLOAD="$(
  printf '{"iat":%s,"exp":%s,"iss":"%s"}'     "${ISSUED_AT}"     "${EXPIRES_AT}"     "${APP_ID}" |
    base64_url_encode
)"

JWT_UNSIGNED="${JWT_HEADER}.${JWT_PAYLOAD}"

JWT_SIGNATURE="$(
  printf '%s' "${JWT_UNSIGNED}" |
    openssl dgst       -sha256       -sign "${PRIVATE_KEY_FILE}" |
    base64_url_encode
)"

APP_JWT="${JWT_UNSIGNED}.${JWT_SIGNATURE}"

echo
echo "===== 2. Listing installations belonging to this GitHub App ====="

INSTALLATIONS_STATUS="$(
  curl --silent --show-error     --output "${TMP_DIR}/installations.json"     --write-out '%{http_code}'     --header "Accept: application/vnd.github+json"     --header "Authorization: Bearer ${APP_JWT}"     --header "X-GitHub-Api-Version: ${API_VERSION}"     "https://api.github.com/app/installations?per_page=100"
)"

echo "HTTP status: ${INSTALLATIONS_STATUS}"

if [[ "${INSTALLATIONS_STATUS}" == "200" ]]; then
  jq --arg expected_id "${INSTALLATION_ID}" '
    map({
      id: (.id | tostring),
      account: .account.login,
      account_type: .account.type,
      repository_selection: .repository_selection,
      matches_kubernetes_installation_id:
        ((.id | tostring) == $expected_id)
    })
  ' "${TMP_DIR}/installations.json"
else
  jq '{message, documentation_url, status}'     "${TMP_DIR}/installations.json" 2>/dev/null ||
    cat "${TMP_DIR}/installations.json"
  exit 1
fi

echo
echo "===== 3. Creating a fresh installation access token ====="

ACCESS_STATUS="$(
  curl --silent --show-error     --request POST     --dump-header "${TMP_DIR}/access-token.headers"     --output "${TMP_DIR}/access-token.json"     --write-out '%{http_code}'     --header "Accept: application/vnd.github+json"     --header "Authorization: Bearer ${APP_JWT}"     --header "X-GitHub-Api-Version: ${API_VERSION}"     "https://api.github.com/app/installations/${INSTALLATION_ID}/access_tokens"
)"

echo "HTTP status: ${ACCESS_STATUS}"

if [[ "${ACCESS_STATUS}" != "201" ]]; then
  jq 'del(.token)' "${TMP_DIR}/access-token.json" 2>/dev/null ||
    cat "${TMP_DIR}/access-token.json"
  exit 1
fi

jq '{
  expires_at,
  repository_selection,
  permissions
} | del(.token)' "${TMP_DIR}/access-token.json"

INSTALLATION_TOKEN="$(
  jq --raw-output '.token // empty'     "${TMP_DIR}/access-token.json"
)"

if [[ -z "${INSTALLATION_TOKEN}" ]]; then
  echo "ERROR: GitHub returned no installation token."
  exit 1
fi

echo
echo "===== 4. Repositories visible to the installation token ====="

VISIBLE_REPOSITORIES_STATUS="$(
  curl --silent --show-error     --output "${TMP_DIR}/repositories.json"     --write-out '%{http_code}'     --header "Accept: application/vnd.github+json"     --header "Authorization: Bearer ${INSTALLATION_TOKEN}"     --header "X-GitHub-Api-Version: ${API_VERSION}"     "https://api.github.com/installation/repositories?per_page=100"
)"

echo "HTTP status: ${VISIBLE_REPOSITORIES_STATUS}"

if [[ "${VISIBLE_REPOSITORIES_STATUS}" == "200" ]]; then
  jq --raw-output     '.repositories[]?.full_name'     "${TMP_DIR}/repositories.json"

  if jq --exit-status     --arg target "${GITHUB_ORG}/${GITHUB_REPO}" '
      any(
        .repositories[]?;
        ((.full_name | ascii_downcase) ==
         ($target | ascii_downcase))
      )
    ' "${TMP_DIR}/repositories.json" >/dev/null; then
    echo
    echo "TARGET_REPOSITORY_VISIBLE=yes"
  else
    echo
    echo "TARGET_REPOSITORY_VISIBLE=no"
  fi
else
  jq '{message, documentation_url, status}'     "${TMP_DIR}/repositories.json" 2>/dev/null ||
    cat "${TMP_DIR}/repositories.json"
fi

echo
echo "===== 5. Testing normal repository visibility ====="

REPOSITORY_GET_STATUS="$(
  curl --silent --show-error     --output "${TMP_DIR}/repository-get.json"     --write-out '%{http_code}'     --header "Accept: application/vnd.github+json"     --header "Authorization: Bearer ${INSTALLATION_TOKEN}"     --header "X-GitHub-Api-Version: ${API_VERSION}"     "https://api.github.com/repos/${GITHUB_ORG}/${GITHUB_REPO}"
)"

echo "REPOSITORY_GET_HTTP=${REPOSITORY_GET_STATUS}"

if [[ "${REPOSITORY_GET_STATUS}" == "200" ]]; then
  jq '{
    full_name,
    private,
    archived,
    disabled,
    permissions
  }' "${TMP_DIR}/repository-get.json"
else
  jq '{message, documentation_url, status}'     "${TMP_DIR}/repository-get.json" 2>/dev/null ||
    cat "${TMP_DIR}/repository-get.json"
fi

echo
echo "===== 6. Testing the exact repository registration call used by ARC ====="

REPOSITORY_REGISTRATION_STATUS="$(
  curl --silent --show-error     --request POST     --dump-header "${TMP_DIR}/repository-registration.headers"     --output "${TMP_DIR}/repository-registration.json"     --write-out '%{http_code}'     --header "Accept: application/vnd.github+json"     --header "Authorization: Bearer ${INSTALLATION_TOKEN}"     --header "X-GitHub-Api-Version: ${API_VERSION}"     "https://api.github.com/repos/${GITHUB_ORG}/${GITHUB_REPO}/actions/runners/registration-token"
)"

echo "REPOSITORY_REGISTRATION_HTTP=${REPOSITORY_REGISTRATION_STATUS}"

grep --ignore-case   '^x-accepted-github-permissions:'   "${TMP_DIR}/repository-registration.headers" || true

if [[ "${REPOSITORY_REGISTRATION_STATUS}" == "201" ]]; then
  echo "Repository registration token call succeeded."
  echo "The returned temporary token is intentionally not displayed."
else
  jq '{message, documentation_url, status}'     "${TMP_DIR}/repository-registration.json" 2>/dev/null ||
    cat "${TMP_DIR}/repository-registration.json"
fi

echo
echo "===== 7. Testing organization-level registration ====="

ORGANIZATION_REGISTRATION_STATUS="$(
  curl --silent --show-error     --request POST     --dump-header "${TMP_DIR}/organization-registration.headers"     --output "${TMP_DIR}/organization-registration.json"     --write-out '%{http_code}'     --header "Accept: application/vnd.github+json"     --header "Authorization: Bearer ${INSTALLATION_TOKEN}"     --header "X-GitHub-Api-Version: ${API_VERSION}"     "https://api.github.com/orgs/${GITHUB_ORG}/actions/runners/registration-token"
)"

echo "ORGANIZATION_REGISTRATION_HTTP=${ORGANIZATION_REGISTRATION_STATUS}"

grep --ignore-case   '^x-accepted-github-permissions:'   "${TMP_DIR}/organization-registration.headers" || true

if [[ "${ORGANIZATION_REGISTRATION_STATUS}" == "201" ]]; then
  echo "Organization registration token call succeeded."
  echo "The returned temporary token is intentionally not displayed."
else
  jq '{message, documentation_url, status}'     "${TMP_DIR}/organization-registration.json" 2>/dev/null ||
    cat "${TMP_DIR}/organization-registration.json"
fi

echo
echo "===== RESULT SUMMARY ====="
echo "Installation token:       ${ACCESS_STATUS}"
echo "Repository visible:       ${REPOSITORY_GET_STATUS}"
echo "Repository registration:  ${REPOSITORY_REGISTRATION_STATUS}"
echo "Organization registration:${ORGANIZATION_REGISTRATION_STATUS}"
EOF

chmod 700 /tmp/arc-github-app-probe.sh
/tmp/arc-github-app-probe.sh

24.3 Interpret the four summary results

Installation token:
Repository visible:
Repository registration:
Organization registration:

Result A — Target repository is not visible

TARGET_REPOSITORY_VISIBLE=no
REPOSITORY_GET_HTTP=404

The GitHub App installation represented by github_app_installation_id cannot see the configured repository. Verify the exact organization login and repository name, confirm that the matching Installation ID belongs to that organization, and add the repository under the installed app’s selected repository access.

kubectl get secret <<APP_SHORT_FORM>>-arc-ghapp-secret   -n arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>>   -o jsonpath='{.data.github_app_installation_id}' | base64 -d
echo

echo "Expected target: <<GITHUB_ORGANIZATION>>/<<SERVICE_NAME>>"

Result B — Repository is visible, but registration returns 403

REPOSITORY_GET_HTTP=200
REPOSITORY_REGISTRATION_HTTP=403
Resource not accessible by integration

The fresh installation token must include these permissions:

• Repository Administration — Read and write
• Repository Metadata — Read-only
• Repository Contents — Read-only
• Organization Self-hosted runners — Read and write

When permissions were added after the app was installed, an organization owner must approve them under:

<<GITHUB_ORGANIZATION>> → Settings → GitHub Apps → Installed GitHub Apps → Configure
→ Review request / Approve new permissions / Accept new permissions

Re-run the probe after approval. It creates a fresh installation token, so missing permissions in that result cannot be explained by an old cached token.

Result C — Repository GET succeeds, repository registration is 404, organization registration is 201

REPOSITORY_GET_HTTP=200
REPOSITORY_REGISTRATION_HTTP=404
ORGANIZATION_REGISTRATION_HTTP=201

This can indicate a repository-level GitHub/ARC limitation involving an enterprise-created GitHub App installed into an organization. The durable options are to create the GitHub App directly under the target organization or use organization-level ARC registration with appropriate runner-group access controls.

# Repository-level registration
githubConfigUrl: "https://github.com/<<GITHUB_ORGANIZATION>>/<<SERVICE_NAME>>"

# Organization-level alternative
githubConfigUrl: "https://github.com/<<GITHUB_ORGANIZATION>>"

Result D — Repository registration succeeds with 201

REPOSITORY_REGISTRATION_HTTP=201

GitHub authentication is correct. Force ARC to obtain fresh credentials and reconcile the existing scale set:

kubectl rollout restart   deployment/arc-gha-rs-controller   --namespace arc-systems

kubectl rollout status   deployment/arc-gha-rs-controller   --namespace arc-systems   --timeout=2m

kubectl annotate autoscalingrunnerset   <<SERVICE_NAME>>-<<ENVIRONMENT>>-arc   --namespace arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>>   troubleshooting.arc/reconcile-at="$(date +%s)"   --overwrite

24.4 Watch the fresh reconciliation without filtering the first error

Start with the unfiltered controller log because a narrow grep can hide the first meaningful error after successful GitHub authentication.

kubectl logs   --namespace arc-systems   deployment/arc-gha-rs-controller   --all-containers=true   --since=3m   --follow

In another terminal:

kubectl get   autoscalingrunnersets,autoscalinglisteners,ephemeralrunnersets,ephemeralrunners,pods   --all-namespaces   --watch

24.5 Inspect the live resource rather than only the local values file

helm get values <<SERVICE_NAME>>-<<ENVIRONMENT>>-arc -n arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>>

kubectl get autoscalingrunnerset   <<SERVICE_NAME>>-<<ENVIRONMENT>>-arc   --namespace arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>>   --output yaml

Success criteria: repository registration returns 201, the controller assigns a runner scale-set ID, and a listener named from <<SERVICE_NAME>>-<<ENVIRONMENT>>-arc appears in arc-systems on the worker selected for <<ENVIRONMENT>>.

Missing-listener GitHub App and ARC troubleshooting completed