Onboard GitHub Organization for ARC

1. 🧭 Scope Legend

These markers show exactly which values are cluster-shared, organization-specific, repository-specific, or deployment-branch-specific.

1.COMMON · NO CHANGE NEEDEDConfigure once and reuse across every GitHub organization, product, repository, and deployment branch.

2.CHANGE PER GITHUB ORGRepeat for each GitHub organization or separately isolated product onboarding unit.

3.CHANGE PER REPOSITORYRepeat for every private repository selected for an ARC runner scale set.

4.CHANGE PER DEPLOYMENT BRANCHRepeat for each dev, qa, or prod runner and deployment mapping.

2. 🎯 Page Purpose and Boundary

COMMON · NO CHANGE NEEDEDCHANGE PER GITHUB ORGCHANGE PER REPOSITORYCHANGE PER DEPLOYMENT BRANCHThis page onboards one GitHub organization or product identity. It validates the GitHub App installation without creating any runner scale set or deployment-branch resource.

THIS PAGE CREATES
  One GitHub App for one organization/product onboarding unit
  One GitHub App installation in the selected GitHub organization
  One protected GitHub Environment in ac-cicd-infra
  One non-secret organization configuration file
  One organization-specific validation workflow

THIS PAGE DOES NOT CREATE
  Kubernetes runner namespaces
  Kubernetes GitHub App secrets
  Repository runner scale sets
  ARC listener pods
  ARC runner pods
  dev, QA, or production branch mappings

The next page consumes the organization credentials created here and creates the first repository runner scale set.

COMMON · NO CHANGE NEEDED
  Shared Kubernetes cluster
  Shared ARC controller in arc-systems
  ARC CRDs and controller RBAC
  Existing Proxmox, Terraform, Ansible, CNI, and worker pools

CHANGE PER GITHUB ORG / PRODUCT
  Product or tenant short form
  GitHub organization owner
  GitHub App registration and installation
  GitHub App Client ID, App ID, and installation ID
  GitHub App private key
  Protected GitHub Environment in ac-cicd-infra
  Organization onboarding metadata file and validation workflow

CHANGE PER REPOSITORY
  Repository selected in the GitHub App installation
  Repository URL
  Runner scale-set name and Helm release
  Repository-specific values and access policies

CHANGE PER DEPLOYMENT BRANCH
  dev, qa, or prod runner namespace
  Environment worker node selector and taint toleration
  Runner minimum and maximum capacity
  Deployment credentials and target servers

Security and rebuild boundary: this page never accepts or stores the private-key content in browser storage. Enter only the local PEM file path. The key is piped directly into a protected GitHub Environment secret and must never be committed to Git. During a clean rebuild, recreate and verify the environment even when an earlier environment with the same name once existed.

3. 🧾 Required Inputs

COMMON · NO CHANGE NEEDEDCHANGE PER GITHUB ORGCHANGE PER REPOSITORYCHANGE PER DEPLOYMENT BRANCHCommon inputs are prefilled. Organization inputs are required for this page. Repository and branch inputs are previews used to demonstrate the next phase.

Common shared-cluster inputs

Administration Hostname

Administration IP

SSH User

Kubeconfig

ARC Controller Namespace

ARC Controller Release

Infrastructure Repository

GitHub REST API Version

GitHub App Token Action Version

Organization or product onboarding inputs

Tenant / Product Name

App Short Form

GitHub Organization

GitHub App Name

GitHub App Client ID

GitHub App ID

GitHub App Installation ID

Local Private-Key File Path

GitHub App Repository AccessOnly selected repositories — recommendedAll repositories

Repository and deployment-branch preview inputs

Service / Repository Name

Target Environment

4. 🧮 Derived Organization, Repository, and Branch Names

CHANGE PER GITHUB ORGCHANGE PER REPOSITORYCHANGE PER DEPLOYMENT BRANCHReview the derived names before creating the GitHub App or files. The same naming rules will be reused by later runner scale-set pages.

GitHub App name:

<<APP_SHORT_FORM>>-arc-runners

Protected credential environment:

arc-org-<<APP_SHORT_FORM>>

Organization configuration path:

kubernetes/tenants/<<APP_SHORT_FORM>>/organization/config.yaml

Organization workflow path:

.github/workflows/onboard-<<APP_SHORT_FORM>>-github-org.yml

Future Kubernetes GitHub App secret:

<<APP_SHORT_FORM>>-arc-ghapp-secret

Future Harbor project:

<<APP_SHORT_FORM>>-ci-cd

Future branch namespace:

arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>>

Future repository runner scale set:

<<SERVICE_NAME>>-<<ENVIRONMENT>>-arc

Future repository GitHub URL:

https://github.com/<<GITHUB_ORGANIZATION>>/<<SERVICE_NAME>>

5. 📊 From-Scratch Sequence Status

COMMON · NO CHANGE NEEDEDCHANGE PER GITHUB ORGUse this checkpoint after rebuilding the shared cluster and ARC controller. This page then creates and verifies one isolated GitHub organization identity.

FROM-SCRATCH SEQUENCE CHECKPOINT

Required before this page
  Load balancer, API VIP, and three control planes operational
  Development, QA, and production worker pools Ready
  15 Kubernetes nodes Ready
  Shared ARC controller deployed as arc in arc-systems
  Two ARC controller replicas available
  Four actions.github.com CRDs installed

Implemented by this page
  One GitHub App and installation for one organization or product
  One protected credential environment in ac-cicd-infra
  One non-secret organization configuration file
  One organization-specific validation workflow
  dev configuration validation
  prod GitHub App installation verification

Not created by this page
  Kubernetes runner namespaces or GitHub App Secrets
  Repository runner scale sets
  ARC listener or runner pods
  Deployment-branch mappings

Source consistency
  The generated FP configuration and workflow must match the cleaned repository
  when the FP input values are entered.

6. 🔬 Verify the Shared ARC Controller Prerequisite

COMMON · NO CHANGE NEEDEDOrganization onboarding requires the controller installed by the preceding page. Verify the clean-rebuild result before adding GitHub identity.

ssh \
  -i ~/.ssh/id_ed25519_ansible \
  -o IdentitiesOnly=yes \
  acllc@192.168.8.202 \
  'sudo bash -s' <<'REMOTE'
set -euo pipefail
export KUBECONFIG=/etc/kubernetes/admin.conf
HELM=/usr/local/bin/helm

echo "=== SHARED ARC RELEASE ==="
"${HELM}" status arc -n arc-systems

echo
echo "=== SHARED ARC CONTROLLER ==="
kubectl get deployment,pods -n arc-systems -o wide

controller_deployment="$(
  kubectl get deployment \
    -n arc-systems \
    -l app.kubernetes.io/part-of=gha-rs-controller \
    -o jsonpath='{.items[0].metadata.name}'
)"

test -n "${controller_deployment}"
kubectl rollout status \
  "deployment/${controller_deployment}" \
  -n arc-systems \
  --timeout=2m

echo
echo "=== ARC CRDS ==="
kubectl get crd | grep actions.github.com
REMOTE

Shared ARC controller verified

7. 🌿 Create the Organization-Onboarding Feature Branch

CHANGE PER GITHUB ORGThe dev push validates non-secret configuration. The prod push authenticates with the protected environment and verifies the GitHub App installation.

feature/onboard-<<APP_SHORT_FORM>>-github-org
    ↓ merge or push
   dev
    ↓ configuration validation only
 dev → prod promotion
    ↓ the merge creates a prod push
   prod
    ↓ validation plus GitHub App installation verification

There is no pull_request workflow trigger.
Pull requests may still be used for code review.

Not used by this infrastructure execution flow:
local, qa, main

cd D:\code\ASPIRECLAN-LLC-Org\ac-cicd-infra

git switch dev
git pull --ff-only origin dev

git switch -c feature/onboard-<<APP_SHORT_FORM>>-github-org

8. 🤖 Create the GitHub App

CHANGE PER GITHUB ORGCHANGE PER REPOSITORYCreate one organization-owned GitHub App for this product or isolation boundary. Include repository Administration permission because the next phase will register repository-scoped runner scale sets.

GitHub App owner
  <<GITHUB_ORGANIZATION>>

GitHub App name
  <<APP_SHORT_FORM>>-arc-runners

Homepage URL
  https://github.com/actions/actions-runner-controller

Webhook
  Inactive — no webhook URL is required for this ARC authentication app

Repository permissions
  Administration: Read and write
  Metadata: Read-only

Organization permissions
  Self-hosted runners: Read and write

Installation scope
  Only on this account

Repository access
  Only selected repositories for this product or onboarding unit

1. Open the GitHub organization settings and create a new GitHub App owned by the organization.
2. Apply the exact name, homepage, webhook, and permissions shown above.
3. For product isolation, choose Only selected repositories unless all organization repositories should use this App.
4. Create the App and record both the current Client ID and numeric App ID.

GitHub App created with approved permissions

9. 🔑 Generate and Protect the GitHub App Private Key

CHANGE PER GITHUB ORGGenerate one PEM private key from the GitHub App settings. Store it outside the repository and never paste its content into the MDX inputs or a Git-managed file.

$TenantShortForm = "<<APP_SHORT_FORM>>"
$DownloadedPrivateKey = "C:\Users\Manoj\Downloads\<github-app-private-key>.pem"
$SecureDirectory = Join-Path $env:USERPROFILE ".aspireclan\arc\$TenantShortForm"
$SecurePrivateKey = Join-Path $SecureDirectory "github-app-private-key.pem"

New-Item -ItemType Directory -Path $SecureDirectory -Force | Out-Null
Copy-Item -LiteralPath $DownloadedPrivateKey -Destination $SecurePrivateKey -Force

Write-Host "Private key copied outside the Git repository:"
Write-Host $SecurePrivateKey

# Do not add this file to Git.
# Keep the original download only until the secure copy and GitHub secret are verified.

Add or confirm these repository ignore rules:

# GitHub App and private cryptographic keys
*.pem
*.key
*.p8

# Local ARC credential working folders
.arc-secrets/
arc-secrets/

# Never ignore committed public certificates by broad directory rules.
# Add a specific exception only when a public certificate is intentionally versioned.

Private key stored outside Git and ignored

10. 📦 Install the GitHub App in the Organization

CHANGE PER GITHUB ORGCHANGE PER REPOSITORYInstall the App in the approved organization and grant access only to the repositories that belong to this product or onboarding unit.

1. Open the GitHub App page and choose Install App.
2. Install it into <<GITHUB_ORGANIZATION>>.
3. Choose the approved repository-access mode: Only selected repositories for this product or onboarding unit.
4. After installation, copy the numeric installation ID from the installation page URL.
5. Return to the input section and enter the Client ID, App ID, installation ID, and local PEM path.

Expected installation URL format:

https://github.com/organizations/<<GITHUB_ORGANIZATION>>/settings/installations/<<GITHUB_APP_INSTALLATION_ID>>

GitHub App installed and installation ID recorded

11. 🛡️ Create the Protected GitHub Environment and Store Credentials

CHANGE PER GITHUB ORGUse one GitHub Environment per organization or product onboarding unit. These commands are Windows PowerShell-safe and intentionally use one-line GitHub CLI commands instead of Bash backslash continuations.

$InfraRepository = "ASPIRECLAN-LLC-Org/ac-cicd-infra"
$EnvironmentName = "arc-org-<<APP_SHORT_FORM>>"
$GitHubOrganization = "<<GITHUB_ORGANIZATION>>"
$AppClientId = "<<GITHUB_APP_CLIENT_ID>>"
$AppId = "<<GITHUB_APP_ID>>"
$InstallationId = "<<GITHUB_APP_INSTALLATION_ID>>"
$PrivateKeyPath = "C:\Users\Manoj\Downloads\<github-app-private-key>.pem"

if (-not (Test-Path -LiteralPath $PrivateKeyPath -PathType Leaf)) {
  throw "GitHub App private key not found: $PrivateKeyPath"
}

Write-Host "Checking GitHub authentication..."
gh auth status

Write-Host "Checking repository access..."
gh repo view $InfraRepository

Write-Host "Creating or reconciling GitHub Environment..."
gh api --method PUT "repos/$InfraRepository/environments/$EnvironmentName"

Write-Host "Setting environment variables..."
gh variable set ARC_GITHUB_ORGANIZATION --body $GitHubOrganization --env $EnvironmentName --repo $InfraRepository
gh variable set ARC_GITHUB_APP_CLIENT_ID --body $AppClientId --env $EnvironmentName --repo $InfraRepository
gh variable set ARC_GITHUB_APP_ID --body $AppId --env $EnvironmentName --repo $InfraRepository
gh variable set ARC_GITHUB_APP_INSTALLATION_ID --body $InstallationId --env $EnvironmentName --repo $InfraRepository

Write-Host "Setting GitHub App private-key environment secret..."
Get-Content -Raw -LiteralPath $PrivateKeyPath | gh secret set ARC_GITHUB_APP_PRIVATE_KEY --env $EnvironmentName --repo $InfraRepository

Write-Host "Environment variables:"
gh variable list --env $EnvironmentName --repo $InfraRepository

Write-Host "Environment secret names:"
gh secret list --env $EnvironmentName --repo $InfraRepository

Write-Host "Environment details:"
gh api "repos/$InfraRepository/environments/$EnvironmentName"

In the repository UI, open the arc-org-<<APP_SHORT_FORM>> environment, restrict deployment branches to prod, and enable a required reviewer when that protection is available.

Protected organization credential environment created

12. 📝 Create the Non-Secret Organization Configuration

CHANGE PER GITHUB ORGCHANGE PER REPOSITORYCHANGE PER DEPLOYMENT BRANCHCommit only identifiers, permissions, naming, and future resource names. Do not place private-key content in this file.

Create kubernetes/tenants/<<APP_SHORT_FORM>>/organization/config.yaml:

schemaVersion: 1

tenant:
  displayName: "<<TENANT_OR_PRODUCT_NAME>>"
  shortForm: "<<APP_SHORT_FORM>>"

github:
  organization: "<<GITHUB_ORGANIZATION>>"

  app:
    name: "<<APP_SHORT_FORM>>-arc-runners"
    clientId: "<<GITHUB_APP_CLIENT_ID>>"
    appId: "<<GITHUB_APP_ID>>"
    installationId: "<<GITHUB_APP_INSTALLATION_ID>>"
    credentialEnvironment: "arc-org-<<APP_SHORT_FORM>>"
    repositoryAccess: "selected"

    permissions:
      repository:
        administration: read-write
        metadata: read-only
      organization:
        selfHostedRunners: read-write

security:
  privateKeyCommittedToGit: false
  privateKeyStoredAsEnvironmentSecret: true
  kubernetesSecretCreated: false

futureResources:
  harborProject: "<<APP_SHORT_FORM>>-ci-cd"
  githubAppSecretName: "<<APP_SHORT_FORM>>-arc-ghapp-secret"
  developmentNamespace: "arc-runners-<<APP_SHORT_FORM>>-dev"
  qaNamespace: "arc-runners-<<APP_SHORT_FORM>>-qa"
  productionNamespace: "arc-runners-<<APP_SHORT_FORM>>-prod"

Organization configuration file created

13. 🔄 Create the Organization Validation Workflow

CHANGE PER GITHUB ORGA dev push validates structure and secret hygiene only. A prod push uses the protected environment to create a short-lived installation token and verify the exact GitHub App installation.

Create .github/workflows/onboard-<<APP_SHORT_FORM>>-github-org.yml:

name: Validate ARC GitHub Organization - <<APP_SHORT_FORM>>

on:
  push:
    branches:
      - dev
      - prod
    paths:
      - "kubernetes/tenants/<<APP_SHORT_FORM>>/organization/**"
      - ".github/workflows/onboard-<<APP_SHORT_FORM>>-github-org.yml"

  workflow_dispatch:

permissions:
  contents: read

concurrency:
  group: arc-org-<<APP_SHORT_FORM>>-onboarding
  cancel-in-progress: false

jobs:
  validate:
    name: Validate <<APP_SHORT_FORM>> GitHub organization configuration

    runs-on:
      - self-hosted
      - Linux
      - X64
      - prod
      - terraform
      - deploy
      - ac-cicd-infra

    timeout-minutes: 20

    steps:
      - name: Checkout repository
        uses: actions/checkout@v5

      - name: Verify required organization files
        shell: bash
        run: |
          set -euo pipefail

          test -f "kubernetes/tenants/<<APP_SHORT_FORM>>/organization/config.yaml"
          test -f ".github/workflows/onboard-<<APP_SHORT_FORM>>-github-org.yml"

      - name: Validate organization configuration
        shell: bash
        run: |
          set -euo pipefail

          python3 - <<'PY'
          from pathlib import Path
          import yaml

          path = Path("kubernetes/tenants/<<APP_SHORT_FORM>>/organization/config.yaml")
          config = yaml.safe_load(path.read_text(encoding="utf-8"))

          assert config["schemaVersion"] == 1
          assert config["tenant"]["shortForm"] == "<<APP_SHORT_FORM>>"
          assert config["github"]["organization"] == "<<GITHUB_ORGANIZATION>>"
          assert config["github"]["app"]["name"] == "<<APP_SHORT_FORM>>-arc-runners"
          assert config["github"]["app"]["clientId"] == "<<GITHUB_APP_CLIENT_ID>>"
          assert config["github"]["app"]["appId"] == "<<GITHUB_APP_ID>>"
          assert config["github"]["app"]["installationId"] == "<<GITHUB_APP_INSTALLATION_ID>>"
          assert config["github"]["app"]["credentialEnvironment"] == "arc-org-<<APP_SHORT_FORM>>"
          assert config["security"]["privateKeyCommittedToGit"] is False
          assert config["security"]["privateKeyStoredAsEnvironmentSecret"] is True
          assert config["security"]["kubernetesSecretCreated"] is False

          print("Organization configuration is valid.")
          PY

      - name: Reject committed GitHub App private keys
        shell: bash
        run: |
          set -euo pipefail
        
          echo "Checking for tracked private-key files..."
        
          if git ls-files |
            grep -E '\.(pem|key|p8)$'
          then
            echo "ERROR: A private-key file is tracked by Git."
            exit 1
          fi
        
          echo "Checking tracked files for PEM private-key boundaries..."
        
          if git grep \
            -n \
            -E \
            -- '-----BEGIN ([A-Z0-9]+ )?PRIVATE KEY-----' \
            -- . \
            ':!docs/**'
          then
            echo "ERROR: Private-key contents were found in a tracked file."
            exit 1
          fi
        
          echo "No private GitHub App key is tracked."
        
  verify-installation:
    name: Verify <<APP_SHORT_FORM>> GitHub App installation

    needs:
      - validate

    if: >-
      (github.event_name == 'push' && github.ref_name == 'prod') ||
      (github.event_name == 'workflow_dispatch' && github.ref_name == 'prod')

    environment:
      name: arc-org-<<APP_SHORT_FORM>>

    runs-on:
      - self-hosted
      - Linux
      - X64
      - prod
      - terraform
      - deploy
      - ac-cicd-infra

    timeout-minutes: 30

    steps:
      - name: Checkout repository
        uses: actions/checkout@v5

      - name: Verify required environment credentials
        shell: bash
        env:
          ARC_GITHUB_ORGANIZATION: ${{ vars.ARC_GITHUB_ORGANIZATION }}
          ARC_GITHUB_APP_CLIENT_ID: ${{ vars.ARC_GITHUB_APP_CLIENT_ID }}
          ARC_GITHUB_APP_ID: ${{ vars.ARC_GITHUB_APP_ID }}
          ARC_GITHUB_APP_INSTALLATION_ID: ${{ vars.ARC_GITHUB_APP_INSTALLATION_ID }}
          ARC_GITHUB_APP_PRIVATE_KEY: ${{ secrets.ARC_GITHUB_APP_PRIVATE_KEY }}
        run: |
          set -euo pipefail

          test "${ARC_GITHUB_ORGANIZATION}" = "<<GITHUB_ORGANIZATION>>"
          test -n "${ARC_GITHUB_APP_CLIENT_ID}"
          test -n "${ARC_GITHUB_APP_ID}"
          test -n "${ARC_GITHUB_APP_INSTALLATION_ID}"
          test -n "${ARC_GITHUB_APP_PRIVATE_KEY}"

          case "${ARC_GITHUB_APP_PRIVATE_KEY}" in
            *"BEGIN "*"PRIVATE KEY"*) ;;
            *)
              echo "ERROR: ARC_GITHUB_APP_PRIVATE_KEY is not a PEM private key."
              exit 1
              ;;
          esac

      - name: Create a short-lived GitHub App installation token
        id: app-token
        uses: actions/create-github-app-token@v3.2.0
        with:
          client-id: ${{ vars.ARC_GITHUB_APP_CLIENT_ID }}
          private-key: ${{ secrets.ARC_GITHUB_APP_PRIVATE_KEY }}
          owner: "<<GITHUB_ORGANIZATION>>"

      - name: Verify the expected installation identity
        shell: bash
        env:
          EXPECTED_INSTALLATION_ID: ${{ vars.ARC_GITHUB_APP_INSTALLATION_ID }}
          ACTUAL_INSTALLATION_ID: ${{ steps.app-token.outputs.installation-id }}
          APP_SLUG: ${{ steps.app-token.outputs.app-slug }}
        run: |
          set -euo pipefail

          echo "GitHub App slug: ${APP_SLUG}"
          echo "Installation ID: ${ACTUAL_INSTALLATION_ID}"

          if [ "${ACTUAL_INSTALLATION_ID}" != "${EXPECTED_INSTALLATION_ID}" ]; then
            echo "ERROR: The GitHub App installation ID does not match the approved value."
            exit 1
          fi

      - name: Verify the organization and installed repositories
        shell: bash
        env:
          GH_TOKEN: ${{ steps.app-token.outputs.token }}
          GH_API_VERSION: "2026-03-10"
        run: |
          set -euo pipefail

          actual_org="$(
            gh api               --header "X-GitHub-Api-Version: ${GH_API_VERSION}"               "/orgs/<<GITHUB_ORGANIZATION>>"               --jq .login
          )"

          test "${actual_org,,}" = "<<github_organization>>"

          echo "Verified GitHub organization: ${actual_org}"
          echo "Repositories currently accessible to this installation:"

          gh api             --paginate             --header "X-GitHub-Api-Version: ${GH_API_VERSION}"             /installation/repositories             --jq '.repositories[].full_name' |
            sort -u

      - name: Confirm organization onboarding completed
        shell: bash
        run: |
          set -euo pipefail

          echo "GitHub organization/product onboarding verified."
          echo "No Kubernetes runner namespace, listener, or runner scale set was created by this workflow."

Organization onboarding workflow created

14. 🧪 Review and Commit Only Organization-Onboarding Files

CHANGE PER GITHUB ORGThe shared cluster, ARC controller, worker definitions, and existing tenant environments must remain unchanged.

kubernetes/tenants/<<APP_SHORT_FORM>>/organization/config.yaml
.github/workflows/onboard-<<APP_SHORT_FORM>>-github-org.yml

Preserve without replacement
  terraform/**
  ansible/**
  kubernetes/common/**
  helm/common/**
  helm/tenants/**
  kubernetes/tenants/*/dev/**
  kubernetes/tenants/*/qa/**
  kubernetes/tenants/*/prod/**
  .github/workflows/terraform-*.yml
  .github/workflows/ansible-*.yml

Never commit
  *.pem
  *.key
  GitHub App private-key contents
  GitHub App installation access tokens
  Kubernetes Secret manifests containing credentials

git status
git diff --check
git diff --stat

git diff -- kubernetes/tenants/<<APP_SHORT_FORM>>/organization/config.yaml .github/workflows/onboard-<<APP_SHORT_FORM>>-github-org.yml

git diff --exit-code -- terraform ansible kubernetes/common helm/common helm/tenants .github/workflows/terraform-plan-shared-k8s.yml .github/workflows/terraform-apply-shared-k8s.yml .github/workflows/ansible-install-arc-controller.yml

git add kubernetes/tenants/<<APP_SHORT_FORM>>/organization/config.yaml .github/workflows/onboard-<<APP_SHORT_FORM>>-github-org.yml

git commit -m "Onboard <<TENANT_OR_PRODUCT_NAME>> GitHub organization for ARC"

git push -u origin feature/onboard-<<APP_SHORT_FORM>>-github-org

15. ✅ Validate Through dev

CHANGE PER GITHUB ORGA pull request may be used for review, but no workflow runs for the pull request. Validation starts only after the merge or direct update creates a dev push.

gh pr create --base dev --head feature/onboard-<<APP_SHORT_FORM>>-github-org --title "Onboard <<TENANT_OR_PRODUCT_NAME>> GitHub organization for ARC" --body "Adds non-secret organization metadata and a GitHub App installation validation workflow. The pull request is for review only; validation starts after the merge creates a dev push."

Expected dev result:

Validate <<APP_SHORT_FORM>> GitHub organization configuration — success
Verify <<APP_SHORT_FORM>> GitHub App installation — skipped

dev push validation completed

16. 🚀 Promote and Verify Through prod

CHANGE PER GITHUB ORGThe prod push validates the same files, requests approval for the protected environment, creates a short-lived App installation token, and confirms the exact installation ID.

gh pr create --base prod --head dev --title "Verify <<TENANT_OR_PRODUCT_NAME>> GitHub organization onboarding" --body "Promotes the validated organization onboarding metadata to prod. The prod push authenticates with the protected GitHub App environment and verifies the installation."

Expected prod result:

Validate <<APP_SHORT_FORM>> GitHub organization configuration — success
Verify <<APP_SHORT_FORM>> GitHub App installation — success
Installation ID — <<GITHUB_APP_INSTALLATION_ID>>
No Kubernetes resources created

prod GitHub App installation verification completed

17. 🔎 Manual Verification

CHANGE PER GITHUB ORGConfirm the clean-rebuild acceptance state: protected environment metadata, secret name, workflow results, installation identity, and committed non-secret configuration. Secret values remain hidden.

$InfraRepository = "ASPIRECLAN-LLC-Org/ac-cicd-infra"
$EnvironmentName = "arc-org-<<APP_SHORT_FORM>>"
$WorkflowName = "Validate ARC GitHub Organization - <<APP_SHORT_FORM>>"
$ConfigPath = ".\kubernetes\tenants\<<APP_SHORT_FORM>>\organization\config.yaml"

Write-Host "=== ENVIRONMENT VARIABLES ==="
gh variable list --env $EnvironmentName --repo $InfraRepository

Write-Host "=== ENVIRONMENT SECRET NAMES ==="
gh secret list --env $EnvironmentName --repo $InfraRepository

Write-Host "=== ENVIRONMENT DETAILS ==="
gh api "repos/$InfraRepository/environments/$EnvironmentName"

Write-Host "=== RECENT WORKFLOW RUNS ==="
gh run list --workflow $WorkflowName --repo $InfraRepository --limit 5

Write-Host "=== ORGANIZATION CONFIGURATION ==="
Get-Content -Raw -LiteralPath $ConfigPath

FROM-SCRATCH ORGANIZATION ONBOARDING ACCEPTANCE CHECKPOINT

GitHub organization/product onboarding
  Tenant:                    <<TENANT_OR_PRODUCT_NAME>>
  Short form:                <<APP_SHORT_FORM>>
  GitHub organization:       <<GITHUB_ORGANIZATION>>
  GitHub App:                <<APP_SHORT_FORM>>-arc-runners
  GitHub App Client ID:      recorded
  GitHub App ID:             recorded
  Installation ID:           <<GITHUB_APP_INSTALLATION_ID>>
  Repository access:         Only selected repositories for this product or onboarding unit
  Credential environment:    arc-org-<<APP_SHORT_FORM>>
  Private key in Git:        no
  Private key in environment secret: yes
  dev validation:            successful
  prod installation check:   successful

Shared infrastructure changed
  Kubernetes cluster:        no
  ARC controller:            no
  Worker nodes:              no
  Terraform state:           no

Success rule
  Do not continue until the protected environment contains all four variables and
  the private-key secret, the dev validation succeeds, and the prod workflow verifies
  the expected GitHub App installation ID.

Next
  Select and onboard the first private repository
  Create the environment-specific Kubernetes namespace and GitHub App secret
  Install the first repository runner scale set

Organization onboarding manually verified

18. 🧭 Repository and Deployment-Branch Hand-Off

CHANGE PER REPOSITORYCHANGE PER DEPLOYMENT BRANCHThe next page consumes this organization identity. It creates a repository and environment namespace, reproduces the GitHub App secret in that namespace, and installs the first runner scale set.

Repository GitHub URL:

https://github.com/<<GITHUB_ORGANIZATION>>/<<SERVICE_NAME>>

Runner namespace:

arc-runners-<<APP_SHORT_FORM>>-<<ENVIRONMENT>>

Kubernetes GitHub App secret:

<<APP_SHORT_FORM>>-arc-ghapp-secret

Runner scale-set name / runs-on label:

<<SERVICE_NAME>>-<<ENVIRONMENT>>-arc

Worker node selector:

environment=<<ENVIRONMENT>>, workload=github-runner

Worker toleration:

environment=<<ENVIRONMENT>>:NoSchedule

19. 🩺 Failure Handling

CHANGE PER GITHUB ORGCHANGE PER REPOSITORYCorrect GitHub identity, installation, permission, or environment-secret problems here. Do not rebuild Kubernetes VMs or reinstall the shared ARC controller.

The App token action cannot find an installation

Confirm all four values:
  Organization: <<GITHUB_ORGANIZATION>>
  GitHub App:   <<APP_SHORT_FORM>>-arc-runners
  Client ID:   <<GITHUB_APP_CLIENT_ID>>
  Environment: arc-org-<<APP_SHORT_FORM>>

Then verify the App is installed at:
  https://github.com/organizations/<<GITHUB_ORGANIZATION>>/settings/installations

The installation ID does not match

Update ARC_GITHUB_APP_INSTALLATION_ID in the protected environment and the non-secret organization configuration so both contain the numeric ID from the installation URL.

The private key is rejected

Get-Content -Raw -LiteralPath "C:\Users\Manoj\Downloads\<github-app-private-key>.pem" | gh secret set ARC_GITHUB_APP_PRIVATE_KEY --env "arc-org-<<APP_SHORT_FORM>>" --repo "ASPIRECLAN-LLC-Org/ac-cicd-infra"

Do not Base64-encode the PEM for this workflow. Store the original multiline PEM content.

The App cannot access an intended repository

Open the App installation settings and add the repository to the selected-repository list. Permission or repository-selection changes may require an organization owner to approve the update.

A PEM file was accidentally staged

git restore --staged <PRIVATE-KEY-FILE>
git status
git check-ignore -v <PRIVATE-KEY-FILE>

An organization-only runner design does not need repository Administration permission

Repository Administration read/write is required for repository-scoped runner registration. This architecture keeps it because the next phase creates private repository runner scale sets.

20. 🏁 Rebuild Checkpoint After Successful Completion

COMMON · NO CHANGE NEEDEDCHANGE PER GITHUB ORGCHANGE PER REPOSITORYCHANGE PER DEPLOYMENT BRANCHRecord the required organization-onboarding result for a clean rebuild. Repository runner scale sets become the next phase only after this checkpoint passes.

FROM-SCRATCH REBUILD CHECKPOINT AFTER THIS PAGE

Expected shared platform
  Load balancer, API VIP, control planes, and all worker pools operational
  Shared ARC controller deployed and healthy

Expected organization identity
  GitHub App created and installed in the selected organization
  Protected arc-org-<short-form> environment created in ac-cicd-infra
  ARC_GITHUB_ORGANIZATION variable present
  ARC_GITHUB_APP_CLIENT_ID variable present
  ARC_GITHUB_APP_ID variable present
  ARC_GITHUB_APP_INSTALLATION_ID variable present
  ARC_GITHUB_APP_PRIVATE_KEY secret present
  Organization config and validation workflow committed
  dev validation successful
  prod installation verification successful

Not expected yet
  Kubernetes runner namespace
  Kubernetes GitHub App Secret
  Repository runner scale set
  Listener or ephemeral runner pods

Next documentation step
  Configure the first repository and deployment-environment runner scale set

Consistency rule
  This checkpoint records the required result of executing this page during a clean rebuild;
  it does not assume any previously retained GitHub Environment, App installation, or workflow run.