Adoption and the Sweep Count Different Things

On memex, 2026-09-08, one cold boot printed these two lines ten seconds apart, from the same process, on the same framework identity:

00:31:08  ShippedPrebuiltBundles: 78 prebuilt assembly(ies) from 38 shipped bundle(s) …
          are backed by the assembly store — 78 adopted now, 0 already current
00:31:18  DynamicTypePreWarmer: 204 of 209 dynamic NodeType(s) need building … — 5 already
          on the share.  framework=sb43f928 total=209 baked=5 pending=204 frameworkstale=201

It was filed as "78 adopted, the sweep's store probe counts 5" (#3703) — the natural reading, and the wrong one. The share was fine. Neither counter was lying. They were never the same population, and one of them was reading a projection its own process had already superseded.

The cost was real: 197 compiles instead of ~20, and it is what put four already-adopted Doc/** NodeTypes on the compile path where a short source-discovery pass could turn them into four false regressions and hold the portal out of rotation for three hours (#3663).

What each instrument counts

Two bundles may legitimately name one NodeType, so even the units do not convert. But the sharper point is the last row:

🚨 The bake report is not a census of the assembly store, and cannot be. The store is asked exactly ONE question per type — TryGetAssemblyPath(typePath, definition.LastCompiledVersion) — and the key comes out of the record. A reader whose record snapshot is behind asks the wrong key, gets a miss, and reports FrameworkStale no matter what is sitting on the volume.

So baked=5 and frameworkstale=201 are both statements about records, filtered through one store lookup each. Reading either as "what the share holds" is a category error, and it is the error the issue's own title makes.

Why 5 and not 0 — the number that looked unexplainable

#3703 closed its reading with "what is left unexplained by all three is the 5: not 0, not 78", and the answer is what turns this from a race into a defect. baked=5 is over all 209 types. The 78 the seeding pass had just adopted contributed zero of them; the 5 were types already current from an earlier boot. So the adoption's effect on the sweep was not reduced by a lag — it was absent, completely, for every type it touched.

That matters because it rules out waiting. A probabilistic index lag would have shown up as some of the 78 landing; a total miss is a missing ordering, and the only thing that can supply an ordering between two sources is a fact one of them already holds.

Why the sweep could not see its own process's writes

The seeding pass writes each adopted type's record through workspace.GetMeshNodeStream(path).Update(…) — the authoritative mutation API, routed to the type's owning hub. The sweep then reads meshService.Query<MeshNode>(MeshWideQuery.OfType(MeshNode.NodeTypePath)).Take(1) — a CQRS read.

Those are different sources with no ordering between them. CQRS and Content Access states the contract plainly: "Queries route through a read-side index — a cached projection that is eventually consistent … that window is long enough to break any pattern that requires read-your-writes", and "a query's answer for one path can be minutes old". On a cold boot the seeding pass issues 78 cross-hub patches in 35 seconds, and the enumeration ten seconds later is free to answer with the records those patches replaced.

And nothing in the classification can tell that apart from a genuinely stale record. A record naming the previous framework because nobody has updated it, and a record naming the previous framework because the reader has not caught up, are the same bytes. NodeTypeBakeStatus is pure and correct on the input it is given; the input was behind.

That is why the mitigations that suggest themselves are all wrong:

The fix: classify from the newer of the two facts

The process knows which records it wrote. That fact was being thrown away — the seeding pass emitted a count and nothing else.

NodeTypeAdoptionRegistry — the mesh-scoped singleton that already existed to stop the first-build kickoff from racing an in-flight adoption — now also carries a ledger of completed ones:

• RecordAdopted(path, observedVersion, definition) is called on the same branch that logs Prebuilt assembly ADOPTED …, never without it. observedVersion is the MeshNode.Version the adoption read before its own write.
• OverlayOnto(definitions, nodes) lays those stamps over an enumeration snapshot, and returns both the definitions to classify from and the list of paths it overrode.

The ordering rule is a fact, not a heuristic

An adoption stamps the version it read, and its own write bumps the node. Therefore:

The second row is what keeps this from becoming the stale serve the bake's conservatism exists to prevent, and it is pinned by a test that fails against an unconditional overlay:

Expected value to be NeverBuilt because the record the owner left is what decides; serving the
adopted bytes over it would be exactly the stale serve #3703's fix must not introduce,
but found Baked.

Nothing is retried, nothing waits, and no bound moved.

Making the two lines legible

A fix that silently corrected the sweep's input would be the next version of the same defect — the operator reading a boot still could not tell whether the enumeration was behind. So all three statements now name their own population:

• the adoption line says it counts assemblies (bundle entries), and that the sweep's counts are over NodeTypes judged from their records, so a difference between them is not a disagreement;
• the sweep's line says {Baked} need no build (record and share agree; … NOT a census of the assembly store) — it used to read "already on the share", which is what invited the reading in the first place;
• NodeTypeBakeReport.ClassifiedFromLocalAdoption counts the entries classified from this process's own write, and Summary appends fromlocaladoption=N whenever it is non-zero. A boot where the enumeration was behind now says so in the same line that reports the verdict.

The second truncation, one level down

The same population had a second way to shrink in silence. DynamicTypesOf selected the sweep's types with n.Content is NodeTypeDefinition d — a direct CLR pattern match on a payload that crossed a query boundary. A row whose $type the serving hub could not resolve degrades to a raw JsonElement, and such a NodeType vanished: out of total=, out of baked=, out of pending=, with nothing anywhere saying a type had been dropped. The report's denominator moved and the report read identically.

The read is now ContentAs<NodeTypeDefinition> — which recovers the JSON case rather than dropping it — and whatever still will not type is counted and named by path in a warning. Excluded because checked (a static NodeType with no compilable source) and never checked at all are deliberately different states with different counters; collapsing them would be the same defect again. See Controls That Cannot Fail.

Measured after the fix — the cold boot after a framework roll

The reading this fix was waiting for is a cold boot on a NEW framework identity, read off the bake-report entry /health publishes (system-side, past RLS, its own denominator — see NodeType Compilation). Both live portals rolled to c84c6c055 on 2026-09-12 and booted on framework sd608997; read 2026-09-13 07:06:01–07:06:38Z, six GET /health calls per portal. On memex.systemorph.com the six bodies split 3/3 across two replicas; on memex.meshweaver.cloud they split 2/3 across two replicas plus one empty response (0 bytes — a failed transport, no body to attribute, excluded from the sample). So the denominator below is two sampled replicas per portal; a portal running more than two has replicas this sample did not reach, and only Sample on the deployment record reads every pod:

All four sampled replicas read Healthy, which on this check means ClassifiedFromLocalAdoption == 0 — no sampled replica's enumeration snapshot predated its own adoptions. Against the boot this page opens with (baked=5 pending=204 frameworkstale=201 with 78 adopted), the seeding replica of the control instance classified 214 of 215 as baked with 59 stamps of its own; the one pending type is the standing previouslybroken. The first memex-cloud replica on the new framework still reports frameworkstale=55: a 353-type mesh where only module content ships in bundles, read by the first process on the identity, so 55 authored types had no bytes under the new tag anywhere yet — the compile that is supposed to happen, not the disagreement this page is about. Its sibling five hours later reads 350 of 353.

So the defect this page describes did not recur on the boot shape that produced it, on any sampled replica of either portal, and the confirming instrument is the one an unauthenticated curl can read. 🚨 The sample is two replicas per portal, not the deployment: repeated calls land on whichever replica the balancer picks, and a replica the six calls never reached is unmeasured, not clean.

The general rule

Before comparing two numbers, make each one state its population, its unit and its source. Two instruments that disagree are only evidence of a defect once you have established they were measuring the same thing — and an instrument that reads a projection must be able to say when its input was superseded, or a stale read and a stale subject are indistinguishable from the outside.

Where the code is

See also

• CQRS and Content Access — why a query is never the authoritative read for a node's content
• Controls That Cannot Fail — the family this belongs to
• Install-Time Prebuilt Adoption — the adoption lane
• NodeType Compilation — what the sweep is deciding about