Sealed Publication Generations

Sealed Publication Reads describes how a publication is written and read today, and ends with what is not closed. This page is that remainder: the layout that closes it, what each reader must do, and the order the migration has to land in.

Every phase is landed (1, 2, 3, 4 and 5 — phase 3's pin half by measurement, see "Where this stands"). Every section says which phase it belongs to, and that section says exactly what is live. The order the phases have to land in is a property of who publishes and what they pin, which is measured below rather than assumed.

The defect this removes

A publication is replaced in place. publish-bake-bundles.sh deletes _complete, uploads over the live files, and re-seals — deliberately, because unsealing first is what makes "sealed" mean "every listed bundle is here".

Interleave two publishers on one prefix and the sealed-skip's answer is stale for whichever loses the race: both unseal, both upload, and the last to seal writes a sentinel over a directory holding some of each one's bytes. That is one seal with one generation — self-consistent to every consumer, and wrong. The read-side generation cannot see it (there is nothing stale to refuse), the boot seeder cannot see it (the sentinel is present and every listed bundle exists), and the first symptom is dependency record mismatch — built against mvid:…, live is mvid:… on a portal that renders nothing.

🚨 The publisher's byte-level postcondition (#3496) turns that from a silent mix into a loud refusal, and it is not the same thing as preventing one. It re-reads every uploaded file's digest immediately before the seal and refuses when any is foreign. What it cannot cover is the interval between that last read and the _complete upload: a publisher overwriting a file inside that single write still lands under this run's seal. The exposure drops from a whole ~90-second publication to one file upload — a smaller window, still a window. A postcondition is not mutual exclusion.

🚨 And "give the prefix one owner" is not the answer either — that was measured. Of the overlaps actually observed on 2026-09-06, zero were between the two lanes and four were two runs of the same lane on one identity. A rule about which repository owns a prefix does not address a lane racing itself. The numbers are in Sealed Publication Reads → "How many writers, measured".

The layout

Each publication is written into its own directory, and a one-line pointer says which one applies:

<root>/<identity>/<source>/_current                  the pointer: one line, a bare directory name
<root>/<identity>/<source>/<generation>/…            one publication INSTANCE
<root>/<identity>/<source>/<generation>/_complete    its seal, still written strictly last
<root>/<identity>/<source>/<generation>/modules/…    its module set, + _index

<generation> is the publisher's publication token<repository>-<run id>-<attempt>, already minted by publish-bake-bundles.sh and already restricted to [A-Za-z0-9._-] so it is a legal bare name on every backend. It is unique per run by construction.

That one property is the whole design:

No pointer means the flat layout

A source directory with no _current is its own publication directory. That is exactly today's behaviour, it stays legal for as long as any reader needs it, and it is what makes the migration below possible at all. Resolution is opt-in by the writer, never by the reader.

Why a pointer and not an atomic directory rename — measured, 2026-09-08

The obvious design is to stage the publication and then rename it over the live one. It is not available on this store through this client, and that is a measurement rather than a recollection.

Read off azure-cli 2.90.0 — the same version the publisher's read-back query was measured against:

group commands
az storage file copy · hard-link · metadata · symbolic-link · delete · delete-batch · download · download-batch · exists · generate-sas · list · resize · show · update · upload · upload-batch · url
az storage directory create · delete · exists · list · show

There is no rename in either, directory delete is documented "Delete the specified empty directory", and there is no lease command under either group — so a lease per identity is not reachable from this lane either. And even in the REST API, where Rename Directory has existed since API version 2021-04-10, a rename cannot replace an existing directory: the swap would be rename-away plus rename-in, two operations with a gap in which the live path does not exist at all.

🚨 So the smallest write this store actually offers is one small FILE, and that is what the pointer is. A design that assumed an atomic directory rename would have reintroduced a partial-visibility window wearing a better story.

What is still a window, stated honestly

The pointer is one small file, and writing it is not atomic on every backend: az storage file upload is a create-then-put-range, so a reader can catch it empty or short.

That does not produce a mix. An unreadable, empty, escaping or dangling pointer resolves to the source directory — see the resolution rules below — so the worst case is:

phase a torn pointer read gives what the reader does
flat copy still present (phases 1–4; a prefix nothing has published since) the previous publication, whole serves it; correct, just not the newest
flat copy gone (since phase 5) a directory with no _complete "being republished right now" → 503 + Retry-After, which every consumer already waits out — and, for a reader that DECIDES on the reading rather than serving it, "cannot tell": see the reader contract phase 5 changes

So the trade is: ~90 seconds in which a mix can be sealed becomes the duration of one small file write in which a reader may be told to come back. The failure mode changes, not only its size — from "a sealed mix nobody can detect" to "read it again". Where the backend offers an atomic rename (Azure Files' Rename File, present in the REST API since 2021-04-10 though not exposed by every az storage file build), the writer phase should publish the pointer that way and remove even that.

The reader contract

One function resolves the pointer, and every path is composed under its result:

ShippedPrebuiltBundles.PublicationDirectoryOf(sourceDirectory, logger)

🚨 The one way to get this wrong is silent, so the API is shaped against it. A reader that resolves the pointer for the listing and then composes its file paths under the source directory serves the flat publication's bytes under the generation's token — the very mix the generation exists to prevent, wearing a token that says it is not. During the migration both paths exist, so it is a wrong answer rather than a missing one. That is why SealedPublicationOf and SealedModulesOf hand the resolved directory back on the reading (Directory) instead of leaving each caller to resolve a second time.

Resolution rules. It never throws; it falls back. Each of these resolves to the source directory:

pointer why it falls back
absent the flat layout — the normal state today
empty or blank being replaced right now; the previous generation still applies
unreadable (IOException) same, mid-write
., .., anything with a separator, anything rooted 🚨 a pointer is a NAME. It must never be able to address bytes outside its own source directory — refused and logged as a warning, because a publisher wrote something this reader will not follow
names a directory that is not on disk a retention sweep outran the pointer; logged as a warning

Generation retention

Follow Released Artifact Retention (#3842). The 30-day age window and release/consumer references supersede the earlier 24-hour previous-generation grace proposal.

Where it runs. PrebuiltBundleStore — the portal's own sweep, one level down from the identity rules it already applies, sharing that pass's abort discipline, its report-only mode and its ledger. It is deliberately NOT in the publisher: the publisher has no consumer inventory, no release markers, no adoption stamps, and it holds none of them at the moment it publishes.

The rule, ORed as KEEPs, applied only inside an identity the pass RETAINS. A generation is collected when all of these hold, and kept when any one fails:

keep when
the pointer _current names it — it is the live publication, however old
the pointer's own health _current could not be resolved to a directory on disk: unreadable, blank, refused, or dangling. Then every generation of that source is kept
age its newest write is inside PreWarm:PrebuiltBundleRetention:MinimumAge (at least 30 days)

There is deliberately no "it could not be read" rule, and it is not missing: a generation the sweep's walk could not enter contributes no files, so it is never identified as a publication and never becomes a candidate. Unreadable still means kept — it arrives by not being enumerated rather than by a branch, which is the stronger of the two. Nor is there a rule on whether the generation was sealed: a superseded one is collected on age whether it was sealed, torn, or abandoned mid-upload. The entry records which (HasSentinel) so an operator can tell a wave of abandoned publications — a different problem — from ordinary supersession, and that field decides nothing.

One walk per identity, not one per generation. The bytes, the newest write and the marker files all come from the single recursive enumeration the identity-level sweep already performs. That is not a micro-optimisation: the root is an Azure Files share where every enumeration is a network round trip, and per-generation walks would have made the cost grow with the number of generations — the exact quantity this cleanup exists to bound. It is why HasSentinel reports the sentinel's presence rather than verifying its listing the way the source-level IsSealed does; verifying would cost one file read per generation per pass to answer a question no rule asks.

A generation under a collectable identity is not listed separately: that identity's directory goes whole, and its byte count already includes them.

🚨 Why this is sound without a per-generation consumer inventory — the one thing that could make it unsound. A non-current generation is unreachable, not merely unfashionable. Every read of a published source directory composes under ShippedPrebuiltBundles.PublicationDirectoryOf(sourceDirectory) — the boot seeder, the same-major adopted fallback, SealedPublicationIndex, PublishedBundleCatalogue and ServedModuleBytes alike — and nothing anywhere enumerates a source's subdirectories looking for a publication. So the set a consumer inventory would have to protect is exactly {the generation _current names}, and both are kept above by construction. The sweep therefore uses the readers' own resolution (ShippedPrebuiltBundles.ResolvePublicationPointer, which is PublicationDirectoryOf with the fallback reason kept rather than discarded) instead of a second copy of those rules: a sweep that decided reachability differently from the readers would delete bytes a portal was still resolving, and the two would drift silently.

🚨 "Publication must establish protection before moving _current" is satisfied by AGE, not by a lease. A generation is protected from the instant its first byte lands, because it is younger than the window. A publication in flight can therefore never be collected, and the publisher needs no claim, no lock and no ordering with the sweep — which is what makes this implementable at all across two repositories that share no lock. UnsealedGrace is deliberately not consulted here: it is hours where the window is at least 30 days, so a branch for it could never fire, and a check that cannot fail is not a check.

🚨 A directory is a generation only on POSITIVE evidence that it is a publication — it carries source-commit.txt, repository.txt or the completion sentinel, or the pointer names it. It is not "every subdirectory except the ones I know about". The publisher's modules/ sits beside the generations in the flat compatibility copy, and under an exclusion list every bookkeeping directory added later would become collectable the day it was added, silently. Under positive identification it is simply retained, and the worst case is bytes that stay.

Who actually publishes — measured, 2026-09-07

The migration order is decided by this table, so it is a measurement rather than a recollection. Every row was read off the producing repository's own ci.yml (or main-cd.yml) on 2026-09-07.

producer prefix (bake-source) which publish-bake-bundles.sh it runs (platform-ref) behind core main
core CD plugins-bake (main-cd.yml) plugins this run's own core commit (needs.gate.outputs.sha) 0 — it IS the tip
MeshWeaver.Plugins ci.yml plugins aa40758329216d57dcefc2d8e8a52101efd5f225 231 commits
MeshWeaver.Reinsurance reinsurance 1b5350d547473a5e2ca81e793e774cc962acfeb3 284 commits
MeshWeaver.SocialMedia socialmedia 1b5350d5… 284 commits
MeshWeaver.Manufacturing manufacturing 1b5350d5… 284 commits
MeshWeaver.Education — (it calls no node-repo-publish-bake)

Three things follow, and each of them changes the plan:

  1. 🚨 plugins is the ONLY prefix with two producers. Every other prefix has exactly one, and Education publishes no bake at all. The coordination this migration needs is therefore one pair — core CD and MeshWeaver.Plugins — not five repositories. Everything else flips on its own schedule, one repository at a time, with nobody to coordinate with.
  2. 🚨 Core CD has no pin to move. It checks the platform out at its own gate sha, so the day the writer merges, core CD runs it. If the writer were unconditional, the plugins prefix would become a new-writer/old-writer pair that same day, against a MeshWeaver.Plugins 231 commits behind — exactly the half-migration this page exists to prevent. That is what makes the selector below mandatory rather than tidy.
  3. Nothing is near the tip. The nearest producing pin is 231 commits back and none of the four carries phase 1. A plan that assumes a pin will "have moved by then" is assuming something that has not happened in a month.

🚨 The correction: "past phase 1" is not a satisfiable precondition

This page used to say phase 2 was "every producing repo's publish-bake pin moves past phase 1". Measured against what phase 1 actually changed (a4109d422), that instruction is a no-op: it touched src/ — the portal image's readers — plus documentation and one comment block in the publish script. It changed nothing a pinned lane executes. A producer whose platform-ref moves past it runs byte-identical behaviour, so the condition can be satisfied by the whole fleet without bringing the migration one step closer.

What a producer must be past is the writer commit itself — and a writer that is on by default cannot be got past, because it takes effect the moment a pin reaches it. Hence the selector, and hence the order below.

The migration, in order

Phase 1 — readers tolerate the pointer (landed, a4109d422)

PublicationDirectoryOf plus every read routed through it: the boot seeder, PublishedBundleCatalogue, ServedModuleBytes, and the registry's four prebuilt routes. Behaviour on a share with no pointer is unchanged, byte for byte.

Nothing writes a pointer yet, so this changes nothing observable — which is why it ships with a suite that builds the generation layout by hand and asserts the readers serve it, including the arm that catches the compose-under-the-source-directory mistake.

Phase 2 — the writer LEARNS the layout, selected per caller (landed)

🚨 Historical, as this phase shipped: the selector defaulted to flat. Phase 4 moved that default to generation; everything in this section describes the mechanism, not today's default.

publish-bake-bundles.sh gains generation publishing behind an explicit selector: a publication-layout input on node-repo-publish-bake.yml, carried into the script as an environment variable, valued flat (the default) or generation. bake-scope.sh and carry-forward-bundles.sh resolve the pointer in the SAME commit, because node-repo-publish-bake.yml fetches all three at one platform-ref and no pin can carry half of them.

At flat the script must behave byte-identically to today, and that is provable rather than asserted: test-publish-bake-overlap.py runs the real script, and its three control cases (a settled publish, a republish of new content, an already-published skip) are the regression suite for the flat path. The new mode earns its own cases — two interleaved publishers each seal their OWN directory, neither directory holds a byte of the other, and _current names exactly one of them.

At generation the writer uploads into <source>/<publication token>/, verifies there (the #3496 postcondition still applies, now over a directory nobody else writes), seals it, also writes the flat copy so a portal image that predates phase 1 keeps working, and moves _current last. (Historical, as this phase shipped: since phase 5 the writer DISPOSES of the flat copy instead of writing it, once the pointer has moved and been read back.)

Every read that decides what is already published — the architecture marker, the sentinel, the source-commit marker, the module index — resolves the pointer first and reads inside the resolved directory, or the writer and the readers disagree about which publication is live. The resolution rules are the reader's, unchanged: an absent, blank, unreadable, escaping or dangling pointer means the source directory. carry-forward-bundles.sh must read the generation the listing came from, which is the shell analogue of the reader's If-Match.

🚨 Any new marker file is LISTED and UPLOADED before architecture.txt. That file is the LAST upload before the postcondition, and the overlap harness hooks its second publisher onto it — a marker written after it silently stops the harness detecting overlaps while every case still reads green. (repository.txt was added under this rule and says so in place.)

What actually shipped, and the one deliberate deviation

Phase 3 — the remaining readers resolve the pointer (landed), and every producer's pin reaches phase 2 (open)

Two halves, and the code half is done.

The two Azure-direct readers now resolve the pointer. compose-sealed-modules.sh (the --storage-target OIDC fallback) and node-repo-gate.yml's seed step download-batch composed their paths under the bare prebuilt-bundles/<identity>/<source>/ prefix, because phase 1 routed the readers the PORTAL IMAGE carries and neither of these is one. Both now call the same resolver by the same rules.

🚨 Neither Azure path had ever been executed by anything. test-sealed-module-compose.py runs the script with --registry-url only, and its gate case runs seed with TARGETS="" — so the whole download-batch block was dead to every harness in the repository. test-publication-pointer-readers.py executes both, against a share carrying a flat copy and a generation whose files share names and differ in bytes, so the verdict is which publication landed. Measured on the pre-change readers: 5 of its 8 cases fail, the three that pass being the no-pointer controls that pin today's flat behaviour as byte-identical. bake-scope.sh --self-test — one of the two proofs this page leans on for phase 2's reader half, and until now run by no workflow at all — is wired into the same lane.

🚨 And #4172 made this load-bearing rather than tidy. A downstream publication now seals only its OWN modules, so an upstream's module bytes are reachable through the upstream's own seal and nowhere else. There is no downstream copy left to fall back on, which makes compose-sealed-modules.sh the single path to them.

The pin half is still open. A producer past phase 2 can write generations and is still writing flat. Core CD needs no pin move; the satellites move theirs the way they always do.

Phase 4 — flip (landed: core's half of plugins, then the lane default for every prefix)

Set publication-layout: generation on every producer of one prefix. The multi-producer prefix is plugins and nothing else: core CD's plugins-bake and MeshWeaver.Plugins' publish-bake. Every other prefix has exactly one producer.

What is landed, and the precondition that was measured before it was

main-cd.yml's plugins-bake passes publication-layout: generation. From that merge, core's publications of prebuilt-bundles/<identity>/plugins are written into their own run-named directory, sealed there, and pointed at.

🚨 And then the LANE DEFAULT moved, which is what flipped the other five. The five single-producer prefixes — crm, education, reinsurance, socialmedia, manufacturing — were described here as "one PR each, in its own repository". They are not, and treating them that way was leaving the more important half undone: the thing that decided their layout was the publication-layout default, and while that default was flat, a node repo that said nothing published in place — including every node repo that will ever be created. The default is generation from this change; none of the six callers passes the input, so all six prefixes are flipped by it and no satellite PR is involved. What a per-repo PR would still buy is the same thing the MeshWeaver.Plugins caller buys: a declared layout rather than an inherited one.

The script's own ${BAKE_PUBLICATION_LAYOUT:-flat} fallback deliberately stays flat. It is the direct-invocation default — the overlap harness, a manual run — and the lane always passes the input through explicitly, so no lane run ever reaches it. Reading that line as "the fleet publishes flat" is the mistake it used to invite, and the comment above it now says so.

🚨 The prefix-ownership mechanism lives in the PUBLISHER, so "either order" is a claim about which publisher each producer is RUNNING — a producer resolving a publish-bake-bundles.sh that predates #4249 still refreshes the flat copy in place and never moves the pointer. That is the one thing that had to be true before flipping and it is checkable rather than assumed. Measured 2026-09-14 across all six node repos — Plugins, Crm, Education, Reinsurance, SocialMedia, Manufacturing — every uses: of a node-repo-*.yml lane is @main and every scripts-ref: is main. No pinned publisher and no pinned reader of this prefix exists in the fleet. The exception is core's own plugins-bake, which passes no scripts-ref and so resolves its scripts at platform-ref = the commit the run publishes; on a reconcile run that commit can be older than #4249, and such a run refreshes the flat copy without moving the pointer — which leaves pointer-following readers on the newer generation, the safe direction, and is the same flat-copy residual phase 5 removes.

What the satellite's own flip buys is therefore explicitness, not correctness: it replaces a discovered layout (announced by a ::warning:: on every publish) with a declared one. The same is true of a per-repo PR on any of the five single-producer prefixes now that the default has moved.

The third reader, and the false hold it was already producing

🚨 Phase 4 did not only expose a phase-5 problem; for one reader it created a live false red, and it had been live for plugins since #4269 before this page noticed.

check-release-availability.sh answers "is this upstream published for the identity I am about to build against?" by probing one file: prebuilt-bundles/<identity>/<source>/_complete. That is the prefix's sentinel — the flat compatibility copy's — and under the generation layout the two answers come apart on every publish:

publish_publication seals the generation the publication is complete
… moves _current the publication is live to every pointer-following reader
… then refreshes the flat copy, which publish_one_target begins by deleting its _complete the prefix has no sentinel for the whole upload-and-verify interval

So for ~90 seconds per target per publish, a gate probing the prefix answers false and reports no sealed publication under … — the one message that means an upstream has not published yet, for a publication that is sealed, live and pointed at. That message is not a neutral one: it is the wording that held MeshWeaver.Reinsurance 23 times in 24 hours (#3583), so a false instance of it costs a reader the same investigation as a true one.

Fixed by routing it through the same resolution as its three siblings, with the same fall-back: an absent, blank, unreadable, escaping or dangling pointer means the source directory, which is exactly this gate's previous behaviour — so the change can only ever add an answer it used to get wrong. test-publication-pointer-readers.py now executes three readers, and the fixture for this one is the window itself: the generation sealed, _current naming it, and the prefix's own sentinel removed. Against the pre-fix script that case fails with the false-hold message; its two siblings (nothing sealed anywhere ⇒ still refuses; a flat prefix with no pointer ⇒ still sealed) pass on both, which is what makes them controls rather than new behaviour.

🚨 THREE places print "no sealed publication", and their remedies are OPPOSITE

This is the half that matters more than the fix, because conflating these makes a genuine outage look transient — and the sentences differ by a preposition.

the message who prints it what it means does re-running help?
no sealed publication **under** <path on the share> check-release-availability.sh since the fix above: a REAL absence. The probe follows _current, so the refresh window can no longer produce it — and since phase 5 a pointer that EXISTS and cannot be followed is reported as CANNOT DETERMINE instead, so this sentence keeps meaning exactly one thing. Before the fix it could be either, and the output did not distinguish them no (post-fix). A run from before the fix may have been the window
no SEALED publication **at** <registry URL> … (404) node-repo-gate.yml's upstream_not_ready, from the registry probe — a definite 404, refused immediately on purpose: 408, 429, any 5xx and 000 are retried as transient, and everything else is a decision the registry has already made the upstream genuinely has not published for that framework identity no. Look at whether the upstream's own seal is blocked
no SEALED publication **under** <account>/<share>/<dir> compose-sealed-modules.sh the resolved publication carries no sentinel — pointer-resolved since phase 3 no; not this window

🚨 "Re-running helps" was never a property of the message — it was a property of the WINDOW, and the window is what the fix removed. A sentinel probe answers false for a real absence and for a refresh in flight identically, which is why the honest table says no everywhere now and leaves the diagnosis to the identity, below. An earlier draft of this table said yes for the first row; that would have told an operator to re-run a job that cannot pass. (Copilot's review.)

A live instance of the middle row on 2026-09-14: MeshWeaver.Education main asking for identity s3b3fb865f… (platform set 8596) while the last MeshWeaver.Plugins publication was s6b704cd02… (set 8581, 12:53Z), its seal blocked on a broken suite. Reading that as the share window would send a reader to re-run a job that cannot pass, which is strictly worse than not diagnosing it at all.

🚨 The identity in the message is the tell. A share-window red names an identity the fleet is currently publishing for; a registry 404 names one nothing has published for yet. Read the identity before the verb.

🚨 The residual phase 4 does NOT remove: a flat publication beside a generation one

(Phase 5 removed the refresh this section is about; what survives it is point 2 of "The residues, named" below — a flat publication on a prefix with no resolvable pointer, which the same guard keeps at zero.)

Raised by the review of the default move, and real. A generation publication writes its generation, moves _current, and refreshes the flat compatibility copy last. A FLAT publication of the same prefix writes only the flat directory and never touches the pointer. So if a flat run's whole publication lands after a generation run's seal, there is no overlap for the byte-level postcondition to refuse, and flat readers end on the flat run's bytes while pointer-following readers stay on the generation. That is the half-migration in its last surviving form.

What keeps it at zero is that the flat arm has no reachable writer, and that is enforced rather than observed:

a run takes the flat arm when reachable?
its caller passes publication-layout: flat NodeRepoLaneHostGuard.NoCallerInThisRepository_PublishesFlat refuses one in this repository, and no satellite passes the input at all (measured 2026-09-14). A satellite that wanted it would have to write the line in its own ci.yml
it resolves a live _current while asking for flat not the flat arm — #4249 promotes it to a generation and names the caller
its publish-bake-bundles.sh predates #4249 yes — core's plugins-bake on a reconcile at an old platform-ref. No code added to today's script can help this one: it is not running it. Such a run refreshes the flat copy and does not move the pointer, so pointer-followers stay on the newer generation — the safe direction

A check on today's flat arm (re-resolve _current before the final seal, refuse if one appeared) would therefore only ever fire for a population the guard above already keeps empty, which is why there is no such check. Phase 5 removes the flat copy and the residual with it.

🚨 Every reader fails safe to today's behaviour. ResolvePublicationPointer degrades an absent, blank, unreadable, escaping or dangling pointer to the source directory, where the flat copy still sits, still sealed — so the worst case of a pointer that cannot be followed is the pre-flip serve. The single reader that may not take that fallback is node-repo-gate.yml's seed, because download-batch recurses and flattens; it refuses instead, transiently, and a re-run reads the publication that now applies.

🚨 "In one change set" was stated here as the precondition, and it is not available. The two producers of plugins live in two repositories, which cannot merge atomically, and one of them publishes on every core build. So the flip must pass through a prefix where one producer writes generations and the other still writes flat — the question is only for how long. Requiring an atomic change set does not avoid that state; it just leaves it unhandled.

The layout belongs to the PREFIX, not to the caller (landed)

The publication-layout input's own description used to tell operators what happens in a mixed state and then tell them not to create one: "the new one moves the pointer, the old one replaces the flat copy in place and never touches it, so a pointer-following reader keeps serving its generation and never sees the old writer's NEWER publication — a stale serve with nothing red anywhere." That instruction was unenforceable, because the state is unavoidable.

A live _current is now what decides the layout. resolve_publication_dir already runs on every run whatever the caller asked for; when it resolves a generation, the run publishes a generation — announced as a ::warning:: naming the caller to flip. Two consequences, and the second is the one worth stating:

🚨 A flat caller never takes the prefix back, and that is deliberate. Deleting or re-pointing _current would invert a migration another producer already made, and it can move consumers backwards: the pointer is moved before the flat compatibility copy is refreshed (the ordering argued above), so a run that finds an older flat copy could skip, retire the newer generation, and expose the older bytes. Going forward is always safe; going back is not. So a flat caller whose content is already the live publication skips and changes nothing — asserted as its own case.

Setting publication-layout: generation on both producers is still what you should do: it makes the prefix's layout explicit rather than discovered.

🚨 The pointer only moves FORWARD, and this is the one thing the generation layout could get silently wrong that flat could not

Under flat, two overlapping publications write ONE directory, so the byte-level postcondition finds the other run's bytes and refuses. Under generation they write disjoint directories: there is no mix to refuse, and whichever run finishes last moves _current. When that is the run carrying older content, every reader is handed a publication that is complete, sealed, self-consistent and out of date — with nothing red anywhere. That is the failure this whole layout exists to prevent, reached from the other side.

The decision-time never-seal-backwards guard cannot see it. It reads the publication that was live when publish_to_target resolved LIVE, roughly 90 seconds before the pointer is written; a sibling that sealed in between is invisible to it. So the question is asked again, in pointer_moved_past_us, immediately before move_pointer:

🚨 This is a postcondition, not mutual exclusion — the same thing verify_publication says of itself, and for the same reason: there is no lease command under either storage group. What it buys is that the window shrinks from a whole publication to the gap between that read and a one-line upload, and that losing that race is no worse than flat's behaviour today. An unorderable pair keeps today's behaviour and says so, exactly as the decision-time guard does; refusing on an unanswerable comparison would mean a run whose content cannot be ordered never becomes live at all.

test-publish-bake-overlap.py executes it: the newer content publishes and points while the older run is mid-upload, and the older run must leave the pointer alone, say so naming both commits, count it, and leave the flat copy carrying the newer bytes. Against the publisher that lacks the check, 4 of those assertions fail and the pointer ends on the older publication.

🚨 The residual window is a pin bump inside one lane. Even a single-producer prefix has a moment where a run started before the flip is still in flight while a run after it writes a generation. The loser leaves the pointer naming a stale generation until that lane publishes again — which happens on its next merge, so it is self-healing and bounded by one publication rather than permanent. Worth knowing before reading such a serve as a defect.

Phase 5 — dispose of the flat copy (landed, 2026-09-17)

From here the mix is unrepresentable and the republish window is gone — for every reader, not only the pointer-following ones. What remains is the sub-second pointer write described above, and an atomic rename would remove even that.

The precondition, measured on 2026-09-17 — the census, with its denominator

reading
build Ops/Status/buildcr.meshweaver.cloud/memex-portal-ai:3.0.0-ci.8411. The commit read the 09-14 census owed: main-cd run number 8411 is c84c6c055, and a4109d422 (phase 1) is an ancestor. Past phase 1 on a COMMIT now, not on identity equality. (The mapping is the run number the image tag carries: run 8798 = 43915af5c, the roll memex was on the same day, confirms it.)
memex (memex.systemorph.com) mid-roll: replicas on 3.0.0-ci.8710 = afde4eabe and 3.0.0-ci.8798 = 43915af5c. Both past phase 1.
memex-cloud (memex.meshweaver.cloud) 3.0.0-ci.8411 = c84c6c055 on all four replicas. Past phase 1.
pearl now RUNNING (it was Provisioning on 09-14): 3.0.0-ci.8080 = 67cbbe0ee, which contains phase 1.
partnerre status: Planned, no estate: the record deliberately carries no host, cluster or pinnedImageTag until its first infra deploy, and its data lives in PartnerRe's own subscription. It reads nothing on this share.

🚨 State the denominator: those are the FIVE Hosting/Deployment records on the control instance, and the records are not provably the whole population — an install that self-updates from the registry and has stopped doing so appears in none of them. The instrument that would name a running image per replica is Sample, and on this cluster it is blind (#4218).

Why an install that is NOT in the records still cannot be reached — and it is a mechanism, not an inference

A publication is written under the framework identity the bake resolved, i.e. the current platform's. The question is therefore whether a pre-phase-1 image can resolve the same identity as a current one, because only then would a phase-5 publisher write where such a reader looks.

It cannot, and Framework Identity Churn is why: Directory.Build.props deliberately lets the SDK append the commit to AssemblyInformationalVersion, so the identity moves on every core commit — measured at ≥ 18 distinct identities across 43 merges, of which only 5 touched the full-MVID set by design. The churn that page treats as a cost is, for this question, a guarantee: an image predating a4109d422 carries a different commit, therefore different assembly bytes, therefore a different identity, and nothing publishes under it. Its flat publication stays on the share, untouched by any phase-5 writer, until retention collects it — and retention already treats a registered instance's own reports as references.

So phase 5 cannot dark-serve an old install by writing somewhere it does not look. What it can do is the next section.

🚨 Phase 5 is NOT "stop writing the flat copy" — the copy already on the share is the hazard

This is the part the phase's one-line description hides, and it inverts the failure the design table assumes.

The table above says a torn pointer read with the flat copy gone finds a directory with no _complete, i.e. "being republished right now"503 + Retry-After, which every consumer already waits out. That is correct only if the copy is actually gone. A phase 5 that merely stops refreshing it leaves the last flat publication sitting at the prefix, sealed and complete, while _current keeps moving past it. From then on a torn pointer read resolves to a publication frozen at the day phase 5 landed — self-consistent, sealed, and older every hour. That is a stale serve with nothing red anywhere: this issue's own failure mode, reached from the third side, and permanent rather than bounded by one publication.

So phase 5 disposes of the existing copy, and the disposal is the publisher's own, incremental and per prefix: on a generation publish, instead of refreshing the flat copy, dispose_flat_copy deletes the flat _complete first and then its files. Removing the sentinel is what turns the prefix from "a complete older publication" into "being republished", which is the state every reader already handles and the one the design table assumes. It needs no bulk sweep of the share, it happens once per prefix on that prefix's next publication, and a prefix that is never published again keeps its flat copy — which is exactly right, because it is the one an old identity's reader still needs.

What landed, in the order the code runs it

step and why it is in that place
the generation is sealed unchanged: the #3496 postcondition runs there and refuses, exactly as at phase 4
_current is moved unchanged
_current is READ BACK (pointer_is_live) and must resolve to THIS RUN'S sealed generation 🚨 never the upload's exit code. This share has reported SUCCESS for files it did not store (39 of 45, 2026-09-08) — and in exactly that case the PREVIOUS pointer survives, so a read-back that asked only "is SOME generation live" passes, counts the run as published, and disposes of the flat copy for a publication nobody points at (raised by the review of the phase-5 PR; reproduced as a negative control). The comparison is against $dest/$PUBLICATION
anything else live ⇒ which is established, never guessed If _current now names a publication whose content is NEWER (pointer_moved_past_us, the same compare API the pre-pointer check uses), a sibling won the pointer in that gap: nothing is wrong, that run owns the prefix and its own disposal, and this one is superseded — green, and it disposes of nothing. Anything else is our own write not landing: RED, and nothing is deleted
the flat _complete is deleted, alone the one write that turns a complete older publication into "being republished". If it fails, nothing else is touched and the target fails: a sealed flat copy that is no longer refreshed IS the frozen serve this phase removes
the flat files are deleted (publish-bake-files.py dispose) only files POSITIVELY identified as the flat publication — *.zip, modules/*.module.nupkg, modules/_index, the markers — matched per directory so a pattern can never reach into a generation. An unrecognised file is LEFT and named; the worst case of a narrow pattern is bytes that stay. A failure here is a ::warning::, not fatal: the prefix is already unsealed, so a left-over is storage, not a publication, and the next publication retries it
the flat _complete is read again, and removed again if it is back 🚨 THE ONE INTERLEAVING THE ORDER CANNOT PREVENT, and the review of the phase-5 PR found it: a producer still running a pre-phase-5 publisher refreshes the flat copy — unseal, upload, VERIFY, seal LAST — so its seal can land after this sweep's deletes and leave the prefix SEALED over an emptied set, which a torn pointer read would be served. There is no lease on this store (no lease command under either az storage group), so this is a POSTCONDITION, not mutual exclusion: re-reading the sentinel and removing it turns that state into "being republished", which every reader already backs off from. A failure to remove it is the one state phase 5 must not leave behind — fatal, and named

🚨 A SUPERSEDED run disposes of nothing. pointer_moved_past_us returns before the pointer moves, so such a run never reaches the disposal — and it must not: the newer run it lost to owns that prefix, and may be a producer still refreshing the flat copy (a reconcile at an older platform-ref) whose seal this run would otherwise tear out from under it.

"A reader is never left with neither" is read off the DELETES, not off the code's line order. test-publish-bake-overlap.py's fake share records, before every delete, what _current names, whether that generation is sealed, and whether the flat copy still is — so every line of that log must satisfy (generation sealed) OR (flat sealed), the first line must be the sentinel, and every line after it must show the flat copy already unsealed.

🚨 The reader contract phase 5 changes: a FAULTED pointer is no longer the flat copy

Every reader resolves _current and falls back to the source directory when it cannot follow it (absent, blank, unreadable, refused, dangling). Those five used to be one answer because the fall-back landed on the flat copy — a sealed publication. They are now two:

the pointer what the fall-back finds what a reader must do
absent the source directory IS the publication (the flat layout; a prefix nothing has published for yet) read it, exactly as before
exists and could not be followed nothing sealed — the copy was disposed of 🚨 "cannot tell", never "nothing sealed"

The readers whose "no sentinel" answer already backs off need no change and got none: the boot seeder compiles instead, the registry's prebuilt routes answer 503 + Retry-After, compose-sealed-modules.sh and the gate's seed refuse. The three whose answer was PERMISSIVE were fixed with this phase:

The residues, named

  1. A producer running a pre-phase-5 publisher re-creates the copy. Core's own plugins-bake on a reconcile resolves its scripts at platform-ref = the commit it publishes, so an older run refreshes a flat copy that a phase-5 run then disposes of again on the prefix's next publication. It is bounded by one publication and reachable only by a torn pointer read in between.
  2. A FLAT caller on a prefix with no resolvable pointer still publishes in place. The lane default is generation and NoCallerInThisRepository_PublishesFlat keeps the reachable population at zero; #4249 promotes a flat caller that resolves a live _current to a generation.
  3. MeshWeaver.Education's e2e/mesh/fetch-upstream.sh has an Azure-share FALLBACK lane that probes the prefix's own _complete and download-batches the prefix recursively. It is taken only when MW_REGISTRY_URL is empty, which that repo's own preflight refuses, and it has been flattening generations since phase 4 — so phase 5 turns a silent union into a loud refusal on a lane the repo does not run. Fixing it is a satellite change; it is named here so the next reader of that script knows which phase it predates.

If the publication moves to an OCI registry

The fleet now has its own registry (cr.meshweaver.cloud) and a program to push plugin bundles into it as OCI artifacts. It is worth writing down exactly what that does and does not close, because "content-addressed" is easy to read as "the race is gone".

What it closes by construction. Blobs and manifests are addressed by the digest of their own bytes. Two publications pushing byte-identical content collide benignly — same digest, a no-op — and two publications pushing different content get different blobs, which cannot overwrite each other. A blob is immutable once pushed; there is no partial overwrite to interleave. So the mix — a set holding some of each publisher's bytes — becomes unrepresentable at the byte level, which is the same property the generation directory buys, obtained more cheaply.

🚨 What it does NOT close: a tag is a mutable, last-writer-wins pointer. If the publication is addressed by tag, the defect simply moves from a directory prefix to a tag, and the migration inherits it. Three conditions close it, and they are the rule already in force for images via MW_IMAGE_DIGEST:

  1. Every bundle is pushed and recorded by DIGEST, never by tag alone.
  2. Each publication also gets an immutable, identity-qualified tag — so a publication can be named without that name being reassignable to different bytes later.
  3. The sealed set is a list of (package, digest) pairs, so a mixed set cannot be written at all: the set names exact bytes, and a publisher that did not assemble those bytes cannot produce that list.

🚨 Content addressing does NOT make two bakes of one commit converge. The bundle compile is not byte-reproducible — measured 2026-09-08, 40 of 45 files differed between two bakes of the same source commit — so two publications of one commit produce different blobs and therefore different manifest digests. OCI does not merge them; what it does is make each one a complete, immutable, self-consistent object, so the only contention left is which one the reference names. That is a last-writer-wins between two correct sets, which is a different and far weaker thing than a mix.

🚨 Two invariants the migration must carry across, or it reopens something worse than it closed:

And the refusal must not be weakened, in either world. On the share it reads "never seal a sentinel over another publication's bundles". In OCI terms it is the same sentence one level up: never publish a reference to a set you did not assemble. That refusal is the only reason this was ever visible instead of silently shipping a mixed set.

Where this stands

Verification

Related: Sealed Publication Reads · CI Content Bake · Plugin Build Contract · Bake Identity Mismatch

Reconnecting…
The connection to the server was interrupted. Trying to restore it…
Trying again…
The connection could not be restored. Reloading the page…
The server was updated. Reloading the page to pick up the latest version.