Self-hosted CI runners on AKS

The fleet's CI queues behind an org-wide ceiling of 60 concurrent GitHub-hosted jobs. One MeshWeaver.Plugins pull request is 105 jobs / ~350 job-minutes, so a handful of open PRs saturates the org and everything else waits — measured 2026-09-07: 61 hosted jobs running, 18 Plugins runs queued behind them, and $1,987 of a $3,000 monthly Actions budget consumed on day 7 with prevent_further_usage: true armed.

Self-hosted minutes do not count against that budget, and the AKS cluster that hosts the portals has real spare capacity. This page is how runners live there without ever becoming a risk to the portals, which is the only hard constraint on the whole design.

The portals and the runners share nodes. Every decision below exists to make one sentence true: under pressure the scheduler removes a runner, never a portal. If you change a cap, a request, or the priority class, re-derive the arithmetic in "The arithmetic" first — the numbers are measurements, not preferences.

What is deployed

Install (private cluster, so through the API-server-side runner):

az aks command invoke -g <aks-resource-group> -n <aks-cluster> \
  --file controller-values.yaml \
  --command "helm upgrade --install arc \
    oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller \
    --version 0.14.2 --namespace arc-systems -f controller-values.yaml --wait --timeout 5m"

The guardrails (PriorityClass, both namespaces, the ResourceQuota and the LimitRange) are a plain manifest applied before the controller — they are the safety mechanism, so they must exist before anything that could consume capacity does. Both files live in the private deploy repo beside the other environment material; nothing here is environment-specific.

Why the shared pool and not a dedicated one

Chart Ownership and the Runner Pool recorded the intended shape in 2026-09 as "ARC on the existing cluster, on a dedicated node pool so CI cannot starve the portal", and named the honest trade: a dedicated pool converts a variable per-job cost into a standing one.

What actually shipped uses the existing silos pool, for one reason that page could not yet weigh: the headroom was measured. The pool carries 20,703m of unrequested CPU and 120 GiB of unrequested memory, and its real utilisation is 2–14%. A dedicated pool would spend money to buy isolation that the priority class, the quota and the reserve already provide arithmetically — and it would spend it whether or not CI is running.

That page's reasoning is not wrong, it is the next step, and it is written up as such under "How to scale it". The trigger to take it is steady rather than bursty CI load, or a reserve calculation that stops closing. Everything else that page records for the lane — a persistent git mirror, a warm layer store, opt-in runner labels with hosted runners as the fallback — is unchanged by this and still outstanding.

Three independent brakes, not one

maxRunners alone is a setting — an upgrade, a merge or a typo can move it and nothing notices. The design therefore stacks three mechanisms that fail in different ways:

1. maxRunners: 8 on the scale set — the normal operating limit. Application-level.
2. A ResourceQuota on arc-runners — pods: 9, requests.cpu: 4500m, requests.memory: 36Gi. Enforced by the API server: no ARC version, value override or hand-applied pod can exceed it. It sits one runner above maxRunners so normal operation never touches it and a runaway still stops.
3. A LimitRange giving every container a default request and limit, and a hard max of 4 CPU / 8Gi. An unbounded pod is the one shape that could starve a portal regardless of its priority, and this makes one unschedulable.

The priority class, and why -10 exactly

Every production pod on this cluster runs at the default priority 0 with no priorityClassName — measured, both portal Deployments plus portal-next, the website and whisper. So any negative value ranks runners below all of them.

value: -10
preemptionPolicy: Never

• Negative ⇒ when a portal pod cannot schedule, the scheduler preempts a runner to make room. That is the required ordering, and it is what makes a KEDA scale-out or a rolling update safe even if runners have taken the slack.
• preemptionPolicy: Never ⇒ a runner can never evict anything itself. The ordering holds in both directions rather than depending on which pod the scheduler happens to consider first.
• -10 and not lower is deliberate. The cluster autoscaler treats pods below --expendable-pods-priority-cutoff (default -10) as expendable: it will not grow the pool for them and it will delete a node out from under them. At exactly -10 a runner is not expendable, so a pending runner may still grow the silos pool within its existing max of 4 nodes, and a running job is not killed by a scale-down. Going to -11 would make runners free but would trade away both of those.

Cost consequence, stated plainly: because runners are not expendable they can pull the silos pool from 3 nodes to its configured maximum of 4. That is capacity the pool is already allowed to use, and one node-hour is far cheaper than the hosted minutes it replaces — but it is not free, and it is the one way this design spends money on Azure rather than saving it on Actions.

The arithmetic

All figures are requests, because the scheduler places pods on requests, not on usage. Measured on the silos pool (3 × Standard_D16s_v5) on 2026-09-07.

Capacity. Allocatable per node 15,740m CPU / 59,144 Mi ⇒ pool 47,220m CPU / 177,432 Mi. Already requested 26,517m CPU (56.2%) / 54,466 Mi (30.7%) ⇒ free 20,703m CPU / 122,966 Mi.

🚨 Do not size this from kubectl top. Actual usage on the same pool read 2% / 14% / 3% CPU, which invites a cap five times too large. The portals request 4 CPU + 8Gi per replica and use a fraction of it; that reservation is what a new pod has to fit around.

What must stay free for the portals — the two ways production grows without anyone deploying:

What is left for runners: 20,703 − 16,000 = 4,703m CPU, and 122,966 − 32,768 = 90,198 Mi. CPU is the binding constraint, so it sets the cap.

Per-runner request: 500m CPU / 4Gi. Limit: 2 CPU / 6Gi.

• Memory is requested honestly (4Gi) because it is incompressible — under-requesting it is how a node reaches memory pressure and the kubelet starts killing things.
• CPU is requested modestly (500m) because a CI job is bursty and the pool's CPU is genuinely idle. Under contention the kernel shares CPU in proportion to requests, so a portal at 4,000m gets 8× the share of a runner at 500m — CPU contention degrades the runner first, by construction.

Cap: 4,703m ÷ 500m = 9.4 ⇒ maxRunners: 8, with the quota one above at 9.

Resulting pool state at full runner load, on 3 nodes:

Both reserves are satisfied without relying on preemption or on a 4th node. Preemption and the autoscaler are the second and third lines of defence, not the plan.

Worth doing? Yes. The Module bundles matrix is 34 jobs / ~152 job-minutes. At 8 concurrent runners that is ~19 minutes of wall clock starting immediately, against up to an hour spent queued today — and it removes those 152 minutes per run from a budget that was 66% consumed on day 7.

How to scale it

In order of preference. Re-derive the table above before any of them.

1. Raise maxRunners — only if the reserve arithmetic still holds. The ResourceQuota must be raised in the same change or it becomes the real cap and runners sit Pending for no visible reason.
2. Raise the silos pool maximum (az aks nodepool update --max-count). Each node adds 15,740m / 59,144 Mi of headroom, so the reserve is satisfied with room to spare and the cap moves with it. Costs a node only while the autoscaler holds it.
3. A dedicated node pool — az aks nodepool add with --enable-cluster-autoscaler --min-count 0, a taint, and a matching toleration + nodeSelector on the scale set. This is the right answer once CI load is steady rather than bursty: runners then contend with nothing, the priority class stops being load-bearing, and cost is strictly proportional to CI usage. It is deliberately not the starting point, because it spends money to solve a problem the measured headroom does not yet have.

Never raise a cap to make a queue shorter without re-reading the reserve. The reserve is what keeps a portal scale-out from having to preempt anything.

Where jobs run — GitHub Actions cost is zero

Maintainer, 2026-09-17: "we still incur cost for github actions. wtf? please see that it goes to 0" · "disable for any private repo" · "and when free capacity gone => defer to our infra".

The switch is two ORG variables of visibility private:

Why core's reusable lanes carry it, although core is public. GitHub bills a called workflow's hosted minutes to the CALLER ("Billing for GitHub-hosted runners is always associated with the caller"). Core's own runs are free, but every node-repo-*.yml lane and auto-arm.yml is called by MeshWeaver.Plugins and the private satellites, so a runs-on: ubuntu-latest inside one is paid by whichever private repository called it. Measured 2026-09-17: $4,174 of Actions cost for September 1–17, the residual being thousands of short jobs on ubuntu-latest, each billed a whole minute.

The fallback semantics — one expression, two answers. A called workflow's vars context is the CALLER's, and a private org variable is invisible to a public repository. So core writes:

• a job: runs-on: ${{ vars.MW_RUNNER || 'ubuntu-latest' }} (or MW_RUNNER_DOCKER). A private caller resolves the self-hosted set; core's own calls (main-cd.yml, dotnet-test.yml, auto-arm.yml on core's pull requests) resolve ubuntu-latest, which is free there.
• a runner / workspace-runner input: the SAME expression as its default: — a workflow_call input default may read github, inputs and vars — so a caller that passes nothing is covered too, and every inputs.runner consumer (runs-on, DOTNET_INSTALL_DIR, the browser step's if:) sees the resolved label. node-repo-gate.yml's shards default to MW_RUNNER_DOCKER; module-pack's pack/tests to MW_RUNNER, its build-workspace to MW_RUNNER_DOCKER.
• 🚨 Never a selected-visibility variable inside a core lane (MW_RUNNER_GATE, MW_RUNNER_HEAVY): adding core to its selection would move core's own runs onto self-hosted runners. A caller may pass those as with: runner:.

A PRIVATE repository's own workflows use the same variables with the opposite fallback — runs-on: ${{ vars.MW_RUNNER || 'aks-silos' }} — because there an unset variable must not cost money; the same holds for a with: runner: it passes to a core lane.

🚨 A REPOSITORY variable of the same name overrides the org one. Measured 2026-09-17: MeshWeaver.Plugins carries a repo-level MW_RUNNER_GATE=ubuntu-latest, which is why its gate shards and global workspace build still ran hosted after the org value said aks-silos-dind. Read gh api repos/Systemorph/<repo>/actions/variables before concluding the org switch applies.

Rolling back or "free minutes first" is a variable flip, not a workflow edit — set MW_RUNNER / MW_RUNNER_DOCKER to ubuntu-latest at org level. That only helps while the Actions budget allows hosted minutes at all: with the budget at $0, a hosted job in a private repository does not start.

The gate. check-reusable-workflow-runners.py runs in core's CI's own shell job and refuses, in any workflow_call workflow, a literal label (ubuntu-latest, aks-silos, a list), a selected-visibility variable, a fallback other than 'ubuntu-latest', and an input whose default does not reach MW_RUNNER/MW_RUNNER_DOCKER. Its self-test takes --root and mutates every real runner expression back to ubuntu-latest, demanding a fire for each.

What a job needs when it moves. The runner image (Systemorph/Memex deployments/aks/ci-runners/runner-image/Dockerfile) is not GitHub's hosted image. It carries git, gh, az, jq, python3 (no pip — actions/setup-python brings one), curl, unzip, sqlite3, .NET 10 at /usr/share/dotnet (writable by runner, DOTNET_INSTALL_DIR=/home/runner/.dotnet in the pod env), kubectl/helm/kubelogin and the docker CLI with buildx. It carries no node/npm, pwsh, zip, yq, zstd, make, rsync, nor the hosted toolcache. A lane that needs one installs it idempotently and names the image as the real fix (module-pack's prepare and build-workspace do this for zstd); a "free disk space" step that removes /usr/share/dotnet must be gated on runner.environment == 'github-hosted'. runner.temp is /home/runner/_work/_temp under ARC and /home/runner/work/_temp on hosted, so an actions/cache entry whose path embeds it misses across runner kinds — harmless, but expected.

Proof is the jobs API, not the colour of the wall — runner_name is aks-silos-… / aks-silos-dind-… on ARC and GitHub Actions <n> on hosted:

gh api --paginate "repos/Systemorph/<repo>/actions/runs/<id>/jobs?per_page=100" \
  --jq '.jobs[] | [.name, .conclusion, .runner_name] | @tsv'

Measured before the switch merged, on two private callers pinned to the change's commit (both probe pull requests closed unmerged). Every reusable-lane job that started ran on ARC, zero on GitHub Actions <n>, with the conclusions of the same jobs on each repository's green main:

The gate-shard row is the decisive one: MW_RUNNER_DOCKER is private, so core — where the default expression is written — cannot see it; only the caller's vars context can have resolved it. The push/schedule-only jobs (supersede, publish-bake, tag-modules, ci-failure, resolve-locks) do not run on a pull request and are proven on the first main run after merge.

Moving a job family onto it

One family at a time, behind a variable, hosted as the fallback. The first candidate is the plugins repo's Module bundles matrix: 34 jobs / ~152 of the ~350 job-minutes per run, all of them independent, none of them holding a credential that a runner would newly see.

The mechanism is cross-repo, which is the part worth knowing before starting:

• modules-floor and modules-rest in the plugins repo do not carry a runs-on. They are uses: calls into core's reusable node-repo-module-pack.yml, pinned by full sha.
• That reusable hard-coded runs-on: ubuntu-latest in six jobs — select, prepare, build-workspace, pack, tests, verify — when this section was written. Since 2026-09-17 none does; see "Where jobs run — GitHub Actions cost is zero" above.

So the change is: add a runs-on input with a default of ubuntu-latest to the reusable, use it on the matrix jobs, then pass it from the caller as runs-on: ${{ vars.MW_RUNNER_LABEL || 'ubuntu-latest' }}. Reverting is then one variable, with no workflow edit at all.

🚨 The default is load-bearing. A required input added to a reusable workflow is a silent startup failure in every caller that has not been updated in the same breath — no checks appear at all, which reads exactly like CI being slow rather than like a break.

🚨 Do not point a workflow at the label before a job has provably run on a runner. A runs-on label nothing serves does not fail — it queues, until the job hits its 45-minute cap. That is strictly worse than the hosted queue it was meant to escape.

Bumping the caller's pinned sha is a deliberate act in its own PR, per that workflow's own rule about a reusable that receives secrets.

Telling a self-hosted job from a hosted one

• In the run log, the first line of the job's Set up job group names the runner: Runner name: '<scale-set>-...' and Runner group: Default for self-hosted, against Runner Image: ubuntu-... / Runner ImageOS on a hosted one. A hosted job always prints a Runner Image block; a self-hosted job never does.
• In the API, GET /repos/{o}/{r}/actions/runs/{id}/jobs gives each job a labels array and a runner_name. A job that ran on ARC carries the scale-set name as its label.
• From the cluster, kubectl -n arc-runners get pods shows one ephemeral pod per running job, named after the scale set. Zero pods and a queued job means the listener is not connected — check kubectl -n arc-systems logs deploy/arc-gha-rs-controller.
• From GitHub, gh api /orgs/<org>/actions/runners lists every registered runner with its status (online / offline) and busy flag.

🚨 A registered runner is not a working runner. The failure that looks most like success here is a scale set that registers, shows online, and never picks up a job — because its labels do not match any runs-on. Verify with a run whose job log names the runner, never with the runner list alone.

Tearing it down

Removing it is two commands and leaves nothing behind:

az aks command invoke -g <aks-resource-group> -n <aks-cluster> --command "\
  helm uninstall <scale-set-release> -n arc-runners; \
  helm uninstall arc -n arc-systems; \
  kubectl delete ns arc-runners arc-systems; \
  kubectl delete priorityclass arc-runner-low"

Uninstall the scale set before the controller — the controller is what finalizes the scale-set CRs, and removing it first leaves them stuck deleting. Nothing in the portal namespaces references any of these objects, so a teardown cannot affect a portal.

Any workflow pointing at the self-hosted label must go back to ubuntu-latest first, or its jobs queue forever against a label nothing serves — for the fleet that is the org variables MW_RUNNER / MW_RUNNER_DOCKER (above), plus every private repository's own aks-silos fallback.

The credential

ARC authenticates as a GitHub App — app id, installation id, private key — never a stored PAT (see the fleet rule on minting App tokens). The App needs, at organization scope, Self-hosted runners: Read and write (organization_self_hosted_runners), plus repository Metadata: Read.

🚨 The org's existing App cannot be reused, and this was measured rather than assumed. Its declared permission set is contents:write, emails:write, metadata:read, pull_requests:write — no runner permission at all — and an installation token minted from it answers HTTP 403 Resource not accessible by integration on both GET /orgs/{org}/actions/runners and POST /orgs/{org}/actions/runners/registration-token.

Reusing it would also be the wrong shape even if it worked: that App holds contents:write across every repository in the fleet, and adding runner administration to it widens the blast radius of a single key. A dedicated App whose only permission is runner administration is the correct credential — it can be revoked without touching content sync, and it grants nothing else if it leaks.

A GitHub App's permissions are not writable through any API, and App creation goes through the browser-based manifest flow. Both are maintainer actions; no agent can mint this credential for itself, which is the same reason the fleet's PAT-elimination work landed on Apps in the first place.

Related

• Deployment — AKS — the cluster, and why every kubectl goes through az aks command invoke.
• Chart Ownership and the Runner Pool — the lane shape this implements the first half of, and the parts still outstanding.
• Reading CI Signals — why a skipped or absent check reads as satisfied, which is the failure mode this page's "registered but never picks up a job" mirrors.