Sharding the node-repo gate

.github/workflows/node-repo-gate.yml is the heavy tester gate every MeshWeaver.* content repo runs: it installs each package into a real mesh, compiles every NodeType, renders it, and executes its Tests layout area. MeshWeaver.Plugins runs it over 59 packages, serially, in one job.

This page records why it now fans out, what was measured to decide the shape, and the two invariants that make a fan-out safe to require.

The defect: a verdict nobody can read

The job carries timeout-minutes: 30. Measured across 28 consecutive gate jobs on MeshWeaver.Plugins (2026-09-03, 12:32-16:29Z), each classified by reading its duration and its verdict text, never its colour:

Of the 20 runs that gated the full set, 4 (20%) were cut at the cap and 7 (35%) came within two minutes of it. A cap cut reports as cancelled, which is byte-identical to a real cancellation — so the gate's verdict was unreadable exactly when it took longest. Two consequences, both measured:

• it blocked nothing. MeshWeaver.Plugins#1253 merged with this gate cancelled.
• it took the Tests-area evidence with it. The caller's ratchet job is if: always() && needs.test-repos.result != 'cancelled', so a cap cut silences the ratchet too. On the cap-cut run 33776132232 every other one of the 61 jobs finished; only the gate was cancelled, and Tests-area ratchet over the gate log and publish-bake were both skipped.

🚨 A failure on this gate is NOT evidence of a cap cut. Three of the four longest reds in that sample were genuine content verdicts. Attributing them to the cap inflates the "before" and overstates what sharding buys — read the duration and the verdict.

Why raising the cap was refused

Every CI job in the fleet is hard-cut at 45 minutes (see Reading CI Signals and AGENTS.md). So raising the cap buys 15 minutes against a moving tail — and a 45-minute cut still reports cancelled. It would make the gate slower to fail and no more readable, which is the band-aid shape AGENTS.md forbids: raising a bound instead of finding what is not completing.

What the time actually goes on

Broken down from run 33758875713 (green, 22.0 min, full 59 packages) and its gate-log artifact:

Of the gate's 1 167 s, 902 s attribute to individual packages and 265 s sits in the gaps between them (~4.5 s each). The per-install cost is the portal/nodeops tax of Bake Seal — NodeOps Saturation: the mesh's ONE node-CRUD execution hub, which every install goes through.

The fix: one job per shard, one container each

Each shard is its own docker run --rm, so shards share no mesh, no nodeops hub and no runner — the saturation does not follow them, and wall-clock falls close to linearly until the per-shard fixed cost (image pull, /app extract, the compile pass ≈ 2.5 min) dominates.

plan ──▶ gate (matrix: shard 1..N, fail-fast: false) ──▶ verify
 │                                                        │
 │ decides the affected closure ONCE                       │ folds the shards' logs into ONE
 │ and the shard matrix                                    │ and IS the required context

Modelled on the measurements above, for MeshWeaver.Plugins' 59 packages:

The model reproduces the measured 1-shard baseline to 0.1 min, which is the only reason to trust the rest of the column. It is opt-in per caller (shards:, default 1) precisely because the wall saving is bought with runner minutes.

What a shard installs versus what it gates

The gate boots a fresh mesh, so a package whose requires / shared=@Other/Source / nodeType: Other/Type names a sibling cannot install unless that sibling landed first. A shard therefore installs its slice plus the forward dependency closure of that slice, and gates only the slice — reusing the installed-but-not-gated shape an upstream package already has (PackageResult.Upstream), for the same reason: a verdict belongs to exactly one place.

That closure is computed by GateShardPlan from LocalNodeRepo.DependencyMap — the same map the install order is sorted by, deliberately not a second reader of those three edge kinds. A second reader is exactly how an edge silently disappears: MeshWeaver.Plugins' scripts/affected-modules.py is already a mirror of that contract and had 51 of its 52 requires edges dead for a while because it compared "Store@^1.0.0" against a bare package id.

In MeshWeaver.Plugins the graph is a shallow star — Store is required by 58 of 59 packages, mean closure 1.63, max 13 (Collaboration) — so the duplicated install cost is small.

Why equal package COUNTS, and no weight table

Measured per package from run 33758875713's gate log, every structural proxy a checkout can offer is anti-correlated enough to lose to plain equal counts:

Simulated makespan at 4 shards — equal counts 8.8 min, files 13.4, NodeType count 14.1, test-source bytes 16.6, and a perfect oracle 8.4. Equal counts lands within 0.4 min of the oracle at every shard count from 2 to 6.

So the plan carries no weight table at all. That is a feature, not a shortcut: contrast .github/scripts/shard-assign.sh, whose table drifted for three weeks unnoticed and cost ~170 s of long pole on every run, because an LPT loop balances the numbers, not the clock. There is nothing here to re-measure and nothing to go stale.

The slice is taken by stride over the dependency order rather than as a contiguous block: both give equal counts, and the stride spreads a run of adjacent heavyweights (the order is alphabetical among the ready set, so neighbours are unrelated) across different runners.

The two invariants that make a fan-out requirable

1. ONE stable context

verify is named Compile + render node repos (MeshWeaver from ACR) — unchanged — so <caller job> / Compile + render node repos (MeshWeaver from ACR) keeps meaning "the whole gate ran and was green", whatever the shard count.

🚨 The per-shard contexts must never be what a repo requires. Gate shard i/n appears and disappears with the shards input, and a required context that stops being produced blocks every PR forever. This is the same invariant, for the same reason, as All selected bundles built in node-repo-module-pack.yml.

verify runs if: always() and fails RED naming the shard when the fan-out did not succeed — a skipped required check reads as satisfied, so it must never skip.

2. The slices must be a DISJOINT COVER — checked from receipts

Every shard prints, before it installs anything, the line GateShardPlan.Describe builds:

shard 2/4: gating 15 of 59 discovered package(s) — A, B, …; installing 3 support package(s)
gated on another shard: Store, AI, Maps

That is the shard's receipt, and .github/scripts/gate-shard-merge.py folds all N of them to answer the one question no single shard can: was every discovered package gated, exactly once? It refuses a run where a shard's log is missing, two shards claim the same package, the union is short of the discovered total, the shards disagree about that total, a shard printed no plan line at all (it ran without --shard, so it judged the whole set), or a shard's summary is truncated. Each of those otherwise renders as a green wall — GitHub paints a skipped matrix leg exactly like a passed one. The script self-tests all seven cases (--self-test).

🚨 Why the shard logs are FOLDED and never concatenated

The gate log is not just a transcript: a caller's Tests-area ratchet reads it (scripts/check-test-suites.py --gate-log), and that parser takes the LAST === mw-plugin-test summary === block in the file — deliberately, so a re-run appended to one log reads as the current one.

Measured, by splitting run 33758875713's real gate log into 4 synthetic shard logs and running MeshWeaver.Plugins' own ratchet over each shape:

The concatenation is a false pass: a green tick over one sixth of the evidence, produced silently, by the very step meant to make sharding invisible. The fold merges the summaries into a single block with one terminal verdict, dropping the [support: …] copies whenever the owning shard's entry is present (a package cannot have two verdicts), and re-emits it under the canonical gate-log-<sha> artifact name — so the caller's ratchet is untouched.

Verified against the real thing, before a single CI run

The lane is workflow_call-only, so core's own CI never exercises it — the first honest test would otherwise be a satellite's pin bump. It was instead run locally against the pinned portal image and the real MeshWeaver.Plugins tree, composing the gate host exactly as the lane does (compose-gate-host.sh over memex-portal-ai@sha256:c4467638… + the tester CLI) and composing the same three module bundles (MeshWeaver.AI, .Markdown.Collaboration, .Maps) the caller passes as --module. The one deviation: the CLI is the branch's own build, because that is where --shard lives.

Over Store, Training, Video, Northwind, Chess (Chess depends on Training, everything depends on Store):

shard 1/2: gating 3 of 5 discovered package(s) — Store, Training, Video; installing 0 support …
shard 2/2: gating 2 of 5 discovered package(s) — Northwind, Chess; installing 2 support
           package(s) gated on another shard: Store, Training

Shard 2 installed Store and Training — it could not install Chess otherwise — and did not judge them: they appear in its summary as [PASS] Store (184 node(s), 0 type(s)) [support: installed, gated on another shard], while shard 1 reports [PASS] Store (184 node(s), 17 type(s)) with all seventeen verdicts.

Folding the two real logs and comparing the verdict SET against a --shard 1/1 run of the same five packages: 27 verdict lines each, identical. Every package once, all 21 NodeTypes (17 Store + 1 Training + 3 Chess) compiled, rendered and Tests-executed, ALL GREEN.

The earlier attempts are worth keeping, because both failures were the design working: a mount missing Training failed Chess's install with NodeType(s) not registered: Training/Tour, and a run without the MeshWeaver.AI module failed with NodeType(s) not registered: Skill — the same content-shaped-error-with-an-infrastructure-cause the lane's own comments warn about.

🍞 The prepared bake: compiled once, adopted by every shard

A shard does not install its mesh from source any more. The mesh a shard stands up is seeded from the bake (--seed), so the bake IS the prepared state a gate needs. Until this section every shard built it from nothing: the same mw-plugin-test compile over the same mount, the same source sha and the same framework identity, four times per run, and then gated a quarter of it.

Measured from MeshWeaver.Plugins main, read through the REST jobs API and each shard's own log lines (bake: reference set → bake: framework=… packages=… assemblies=… is the compile; from there to ALL GREEN is the gate):

The installs themselves are small next to that — 13-50 s of package installs per shard in run 40531711572. (The compile was 72 s when the table above was measured; why it grew to 9 and then 28-33 minutes is its own question, and this section does not answer it.)

So a run with more than one shard now has a bake job between plan and the shards:

plan ──▶ bake (once) ──▶ gate (matrix: shard 1..N) ──▶ verify
           │                 │
           │ compiles, keys,  │ fetches the hand-off, verifies it against ITS OWN key,
           │ seals, uploads   │ copies it into its own bake directory, gates its slice

• Same steps, not a copy. bake runs the shards' steps through a YAML alias (steps: *gate-steps) with GATE_ROLE: bake in its env, so the bake and the shards cannot disagree about how a platform is resolved, composed or compiled against. It stops after the compile and gates nothing.
• Keyed on everything the compile read. gate-bake-handoff.py key hashes every file of the composed gate host (the portal's /app and the tester), of the staged repository tree and of the external-module directory, the names of the shared frameworks (volume mode), and the identity, platform reference, source sha and compile arguments. Only .git and __pycache__ are skipped, by name: they differ between two checkouts of one commit and no compile reads them. A .git BELOW a tree's root is the one exception: the sweep skips the whole directory that holds one, so its PRESENCE is keyed (never the Git metadata behind it). A symbolic link is keyed by what it RESOLVES to, because the compile follows it: a file link by its target's bytes, a directory link by the target's contents (a link back into a directory already on the walk is recorded as a cycle, not walked again). A link out of every hashed tree whose target differs between two runners therefore changes the key.
• Equal key ⇒ adopt an own copy. The shard copies the hand-off into its own, empty bake directory and re-verifies the copy; nothing a shard does to its bake reaches the hand-off or a sibling.
• Different key ⇒ refreshed, loudly. adopt exits 3 naming every input that differs (a value, the first differing paths of a tree, a framework listing), and the shard compiles its own bake under a ::warning::. A stale bake is never used.
• Damaged hand-off ⇒ RED. A missing, extra or changed file, a link, an identity file that disagrees with the key, or a key document that does not hash to the key it claims is refused — never "compile instead". The manifest catches damage; the run-scoped artifact store's own sha-256 verification carries the bytes between the jobs.
• One platform per attempt. A shard of the same attempt takes the set bake compiled against instead of re-resolving a newer one: a set sealed during the bake would otherwise make every shard stale, and a refresh on top of the bake's compile would cross the 45-minute cap. A shard of a LATER attempt ("Re-run failed jobs") re-resolves as before and refreshes when the set moved. This applies only to a platform the LANE resolved. A caller's pins, or allow-unpinned tags (plan then hands the shards empty digests), are taken "as called", exactly as the bake took them, and the key decides whether the bake is adopted.
• One shard is unchanged. With shards: 1 no bake runs and the single shard compiles its own bake exactly as before. A bake that fails or is cancelled runs no shard, and verify names it as the reason.

The self-test (gate-bake-handoff.py --self-test, run by dotnet-test.yml) carries the negative controls: another identity, an edited source file, a changed module assembly and another shared-framework set each refresh by name; a byte changed, a bundle missing, an extra bundle, no manifest, a re-labelled key, a rewritten identity file and a symbolic link are each refused; another file-link or directory-link target changes the key, and a link cycle is recorded instead of walked; a sibling adopts the original bytes after another shard changed its copy; and every rule was mutation-checked (removing it turns at least one case red).

Not established yet: the after-numbers in CI. What the design predicts is one compile per run instead of one per shard — for the runs above, 3 × 9 min and 3 × 28-33 min of runner time per run — with wall-clock roughly unchanged, because the shards now wait for the one compile instead of each running it; that prediction is not a measurement until a run on this lane reports it.

The pin moves with the lane

--shard is a flag on mw-plugin-test, which ships in the tester image. A caller that sets shards: > 1 therefore needs a pin from a CD wave carrying it, and the shard job asserts the flag by name against the composed host before it gates — so a stale pin fails with a message naming the pin rather than Unknown argument '--shard' naming the tester's argv.

And this lane is workflow_call, pinned by each caller with a 40-char commit sha: a change here reaches a satellite only when that satellite bumps its uses: and platform-ref. With shards: 1 the command line is byte-for-byte the pre-fan-out one, so a caller that bumps the pin without opting in sees no behaviour change at all.

See also

• CI Content Bake — the lane family and the pinning contract.
• Bake Seal — NodeOps Saturation — the per-install tax being parallelised.
• Reading CI Signals — why a skipped or absent required context reads as green.