Refusing a Lost User Action

A person clicked a button. The framework accepted the click, threw it away, and wrote one Warning five seconds later that reads exactly like routine stream churn. Nothing retried, nothing surfaced, and the next thing that looked for the result saw an absence indistinguishable from "you never clicked."

That is issue #3566, and it is the reason IUserAction exists.

What was measured

MeshWeaver.Education run 34042620439, job 101512796719 — the Store Install button.

15:44:38.03  Click … click action done                                (no error)
15:44:38.06  Navigate to "/Store"                                     ← 20 ms later
15:44:38.205 Circuit connection DOWN … disposing per-circuit portal hub
15:44:38.41  ClickedEvent arrives at e2e-admin/Packages — sync/{id} already gone
15:44:43.41  warn: Dropping ClickedEvent for stream JRGdthy… : no synchronization hub
             found on this hub or any parent — the target stream is gone

InstallPackage was never invoked. Attempt 1 installed nothing, and nothing was red.

Two facts make this more than "the circuit was gone, so of course":

• The action does not need the circuit, and the framework already knows that. InstallPackage ends in RunAsSystem(...).Subscribe(...) — fire-and-forget. In the very same run the retry's install ran to completion 850 ms after its own circuit closed.
• It is not decided by how fast you navigate. The retry had a tighter window (4 ms click→navigate, circuit closed 90 ms after the click) and it won. What differed was load: the failing attempt's card locator took 2 939 ms to settle versus 905 ms. The outcome turns on server load at the moment of the click — rare in testing, reachable in production.

The denominator: Dropping ClickedEvent appears exactly once in that run — that click — and zero times in run 34080328179, where the same install succeeded.

The scope call, and why it went the way it did

The issue names two coherent answers and picks neither. Both were examined; the first turns out not to be implementable as written.

"Deliver it anyway" — measured, and false as stated

routing the event to the target hub without requiring a live sync hub would make the click land

It would not. Three things have to be true for a ClickedEvent to run, and the disposal takes all three away at once:

1. The handler is LayoutAreaHost.OnClick, registered on the per-stream sync/{id} sub-hub — it is the only ClickedEvent registration in the framework. The owner hub itself (e2e-admin/Packages) has none, so an event routed there would be Ignored, not run.
2. The action is control.ClickAction, a closure held in the EntityStore snapshot of the LayoutAreaHost that was just disposed with the stream.
3. The handler's own filter is Stream.ClientId.Equals(delivery.Message.StreamId) — it is scoped to the departed subscriber by construction.

Making the click land therefore means re-materialising a layout area for a subscriber that no longer exists: re-running the view function against state that has since moved, under an AccessContext nobody is holding. That is a different design with its own answer to "whose identity runs this", not a routing tweak. It is deliberately not what this change does.

"Refuse it visibly" — what shipped

The drop stays a drop. What ends is its silence, on both of the audiences that exist:

🚨 In the #3566 timeline itself the circuit was already gone, so the modal reaches nobody — and that is honest, not a gap. The person had navigated away; there is no live surface to write to, and inventing one (a cross-session notification for a lost click) would be a feature, not a fix. The other two ways into the same code path — a released read stream and a reaped sync hub on a client that is still there — do have a live sender, and those are exactly the cases where somebody is looking at a page that silently did nothing.

The one line that separates the two classes

public interface IUserAction : IRequest<UserActionAccepted>
{
    string ActionArea { get; }
}

ClickedEvent, BlurEvent and CloseDialogEvent implement it. Nothing else does, and nothing about routing or handling changes — it exists only so that DataExtensions.RefuseStreamMessage can ask one question:

• A data frame (DataChangedEvent & co) whose stream is gone is dropped, exactly as before. The only party that wanted it is the subscriber that has gone away, and its view went with it. NACKing every one would be pure teardown noise — the reason the framework deliberately never did.
• A user action whose stream is gone is refused.

That asymmetry is the whole design. It is also what makes the change measurable: the regression test asserts the refusal and asserts that a data frame on an identically-gone stream still produces nothing (DroppedUserActionIsRefusedTest).

Why ErrorType.Rejected

Not NotFound — PortalErrorReporting swallows a routing NotFound as benign churn, and would swallow this with it. Not ShuttingDown — that is the transient "the address may come back, ride it out" verdict, and this address is not coming back for this stream. Rejected is what it is: the framework explicitly declined to run the action.

Why the log line is Error

Log levels are a production cost model, not a debug dial, so a level is only ever raised with the trade stated. Here it is: a lost user action is work the platform accepted and threw away, and it is rare by construction — once in run 34042620439, zero times in the green run. Neither of the two volume complaints applies: StreamEndedEvent (the "we get tons of this" case, maintainer 2026-09-01) stays at Debug, and data-sync frames stay at Warning. Only the user-action class rises.

The sentence a person reads

error.userActionNotRun, resolved from the catalog against the acting user's AccessContext.Locale — the one carried by the delivery being refused — never an ambient culture, which on Blazor Server is the container's and identical for every simultaneous viewer. See Localization.

The ordering fix that followed

The visible refusal closed the silent-failure half, but it did not stop an accepted action losing a race with circuit teardown. That second half is issue #3986 and is now an acknowledgement protocol:

1. IUserAction is an IRequest<UserActionAccepted>.
2. The Blazor sync hub uses Observe to register the response callback before it posts the click, blur, or dialog dismissal.
3. The owner-side LayoutAreaHost posts UserActionAccepted only after its stream-scoped handler has accepted the action.
4. A circuit close reaches the sync hub's existing Quiescing phase and sees that callback as pending. It therefore keeps the stream subscription alive until the receipt lands, then disposes normally.

There is no retry, grace extension, timer, or second disposal gate. The receipt makes the accepted action part of the lifecycle mechanism the hub already drains. An action whose stream was genuinely gone before it arrived is still refused by the path documented above.

🚨 Step 4 was only half true, and the half that was missing is the one that loses the click

A pending callback is drained in Quiescing, and Quiescing is a phase of the HUB. So the acknowledgement orders ahead of the release only if the release is posted from a point that comes after Quiescing. It was not.

The release is one line — the UnsubscribeRequest that destroys the owner-side sync/{id} sub-hub, registered in JsonSynchronizationStream.CreateExternalClient. It was registered on the stream:

reduced.RegisterForDisposal(new AnonymousDisposable(
    () => hub.Post(new UnsubscribeRequest(reduced.StreamId), o => o.WithTarget(owner))));

and SynchronizationStream.Dispose() disposes its registrants synchronously, and deliberately before Hub.Dispose() — that ordering is #1613's own fix and is correct for what it was for (it is what removes the pending SubscribeRequest callback promptly). Its cost here is that the whole disposal ordering runs before the hub has a phase in which to wait. So the two teardown routes behaved differently:

The second route is not an edge: released read stream is one of the three ways into RefuseStreamMessage this page already names, and it is the one where the person is still sitting in front of the page.

The fix is where the line is registered, not what it does. It now goes on the stream's hub, so it runs from DisposeImpl in ShutDown — strictly after Quiescing — on both routes:

var release = new AnonymousDisposable(
    () => hub.Post(new UnsubscribeRequest(reduced.StreamId), o => o.WithTarget(owner)));
if (reducedHub is not null) reducedHub.RegisterForDisposal(release);
else                        reduced.RegisterForDisposal(release);   // no hub left to wait in

Nothing new waits, nothing is delayed "to be safe": the release simply sits behind the drain the hub already performs.

The sender is a surface, not a call shape — stream.SubmitUserAction(...)

The ordering above is only armed if the sender registered the callback, which Post does not do. So the acknowledged send is a named surface — UserActionSubmission.SubmitUserAction, an ISynchronizationStream extension — rather than an Observe incantation copied into every view that raises a click. It carries the acting user's AccessContext (a user action must; the sync hub has no identity of its own), owns its own subscription, and hands a refusal to the caller as the already-localized error.userActionNotRun sentence.

🚨 And a second thing the refusal was doing, which nobody had measured

A refusal is a DeliveryFailure posted back to the sender, and the sender of a click is the stream's own sync/{id} hub — whose ConfigureSynchronizationHub carries a blanket DeliveryFailure handler that answers OnError for anything that is not a transient ShuttingDown. So a bare Post of a click that cannot be delivered does not merely lose the click: it faults the whole synchronization stream, and every view bound to that mirror dies with it. Measured on a real fixture — the stream terminated with

DeliveryFailureException: Your last action (“ProbeArea/Button”) did not run — the view it was
sent from had already closed. Nothing was changed; please try again.

DroppedUserActionIsRefusedTest could not see this: it posts from the client HUB, so its refusal never reaches a stream's handler.

🚨 Registering the callback is not on its own enough, and assuming it was cost one wrong claim. HandleCallbacks runs FIRST in the rule chain and then the chain keeps running, so a matched response reaches the blanket handler as well — the fault still fired. What the match does leave behind is the flag the framework already uses for exactly this: PostOptions.CallbackDispatched, which PortalErrorSink has long consulted so a failure the call site's OnError handled is not also popped as a modal. The sync hub's DeliveryFailure handler simply never adopted it. It does now, as the same one-line filter:

(_, delivery) => !delivery.Properties.ContainsKey(PostOptions.CallbackDispatched)

An un-awaited failure — the subscribe protocol, an RLS denial, a NotFound — still faults the stream exactly as before. Only a failure somebody is already holding is left to them.

The order in which this was found is worth keeping: the "does not fault" half passed in a filtered run and failed in the full suite, because the test's fault probe was a bare Subject and the fault landed before the assertion window opened. A replay-backed subject made the observation honest, and the honest observation falsified the claim. ARefusedActionSurfacesToTheCallerWithoutFaultingTheView pins both halves.

The measurement

UserActionOutlivesStreamReleaseTest asserts both directions against real hubs and a real remote stream, with the owner-side sync/{id} sub-hub's own DisposalCompleted as the instrument:

The owed-work window is made deterministic rather than raced: the action names a stream id with no sync/{id} on the owner, so the owner holds it for SyncStreamOptions.SyncHubRegistrationGrace (400 ms in the test, well inside the hub's 2 s Quiescing budget) and then refuses — the real reaped-sync-hub shape. Falsified by re-registering the release on the stream and rerunning: AnAcceptedActionHoldsTheReleaseUntilTheOwnerAnswers fails at 200 ms — "Expected the observable not to emit … but it emitted ()" — while the other two stay green.

The sender half — where it was, and what moving it actually took

The Blazor senders live in MeshWeaver.Plugins, and until they moved the ordering above was armed but unused: a bare Post registers no callback, so Quiescing had nothing to drain and the release went straight through.

🚨 The list below was re-derived against Plugins main rather than taken on trust, and the denominator is what a reader needs. The instrument is not Stream.Hub.Post — that literal appears nowhere in the repo except inside one comment. The honest denominator is every construction of a type implementing IUserAction (ClickedEvent, BlurEvent, CloseDialogEvent — and those three are the whole set, so the sweep is closed):

grep -rn "new ClickedEvent\|new BlurEvent\|new CloseDialogEvent" \
     --include='*.cs' --include='*.razor' --include='*.json' .

🚨 State the denominator with the count, or the count is a claim about nothing. Measured on Plugins main 2026-09-11 (merge 05fde510): 19 constructions repo-wide, of which 8 in 7 production view files — the number the first pass guessed, reached the second time by a search that could have contradicted it. The remaining 11 are in .Test projects (Markdown.Collaboration.Test ×4, Persistence.Test ×3, Graph.Views.Test ×2, AI.Test ×1, Todo.Test ×1): they post from a test or CLIENT hub rather than from a view, so they are not senders and are correctly left alone. Zero outside src/ — no in-mesh Source/*.cs and no NodeType JSON constructs a user action, which is what closes the half of the sweep dotnet build cannot see. (An earlier revision of this page said six test hits. It was counting .cs under src/ with a narrower pattern; the number is 11.)

🚨 "Each already resolves the hub defensively and already stamps the circuit user's AccessContext" was wrong, and the last column is why it matters. Exactly ONE of the eight stamped an identity. Four posted from the portal hub rather than the stream's, where an ambient context happens to be present during an inbound Blazor activity — so moving them to stream.SubmitUserAction moves the sender to a hub that has no identity of its own, and passing the acting user explicitly is not a nicety there but the thing that stops PostPipeline failing closed. Three of those four are [JSInvokable] callbacks, i.e. DEFERRED: CircuitAccessHandler has already nulled the ambient context by the time the browser calls back, so the live AsyncLocals answer nothing and the durable ICircuitContextAccessor.UserContext is what has to answer.

The move is therefore hub.Post(evt, o => …) → Stream.SubmitUserAction(evt, ActingUser, SurfaceRefusal), against two new members on BlazorView that every one of the eight shares:

• ActingUser — AccessService.Context ?? AccessService.CircuitContext ?? ResolveCircuitUser(). The live AsyncLocals first (what an inbound activity set), the durable circuit user as the fallback every deferred call site needs.
• SurfaceRefusal(string) — reports the already-localized sentence to the circuit's PortalErrorSink, verbatim, and logs it. Not SurfaceError: that takes an exception, re-words it with a context prefix, and suppresses itself once the view is disposed — and a refusal arriving after the view is gone is precisely the case a person still on the page needs to see. (It is also not ErrorSink.Report: ErrorSink is PortalLayoutBase's injected field; a leaf view resolves the sink lazily through Services, because non-portal hosts such as the MAUI client never register one.)

The HubOrNull() guards came OUT rather than being kept: SubmitUserAction answers the hub-released case (#3321 step 3) itself, with the same catalog sentence, which turns the silent return those guards performed into the refusal this whole page is about.

🚨 There was no platform pin to move. MeshWeaver.Plugins has carried none since #3842 — its platform-ref job resolves the newest SEALED core set at run time (scripts/resolve-platform.py), and the repo variable MW_PLATFORM_REF is an incident FREEZE, not a pin. The sender half's real gate is therefore a core RELEASE: it cannot compile until a sealed set carries this commit.

The Plugins-side controls

Core's UserActionOutlivesStreamReleaseTest owns the ORDERING proof, measured against the owner-side sub-hub's DisposalCompleted. What Plugins owes is the SENDER's properties, and UserActionSubmissionFromViewsTest pins them by driving the real BlazorView.OnClick — through the real Blazor renderer, over a real remote stream whose owner is a real layout area host:

Each falsification was built and run, and each reddened on its OWN assertion. The tallies are the part worth keeping, because they are what separates a control from a duplicate:

The middle row is the reason (b) exists at all: a submission path that refused every click satisfies every defect-direction assertion on this page.

🚨 One sentence for three causes — the residue, and what it cost

Everything above was merged, deployed, and the issue was reopened anyway. Not because the fix regressed: because the refusal sentence could not say which of three things had happened, so the incident fingerprint that folds recurrences together folded a designed refusal and a live defect into one ticket.

The line, as it stood:

REFUSING ClickedEvent on area Overview/Actions/1 for stream iUuXw_4bgUmNKx3-U8N3eQ on hub
rbuergi/Requests/provision-pearl-20260914: the target stream is gone (disposed circuit, released
read stream, or never-created sync hub), so the action the user asked for did NOT run and never
will.

Three causes, one sentence, one fingerprint. Admin/_LogIncident/c3ea7263f217a7f3 carried three occurrences: 2026-09-10 22:08Z and 2026-09-11 09:34Z, both on pre-fix images, and 2026-09-14 22:14:51Z on an image that provably carries the fix (memex.meshweaver.cloud/api/version → c84c6c0550, and git merge-base --is-ancestor 1594bb31e4 c84c6c0550 → true). The third one's owner is a per-node request hub and its sync/{id} had never been registered on the activation that received the click — the third cause, which the client-side ordering fix cannot reach by construction.

So the bot reopened a fixed issue, correctly, on evidence that named nothing. Two triages were spent re-deriving which cause each occurrence was, from the surrounding facts, because the line itself could not say.

What the owner records, and what it is not

One SyncStreamActivationLedger per hub — an INSTANCE registered beside IWorkspace in the data plugin's services, so its lifetime IS the activation it answers about. Two booleans per stream id:

It holds no message, changes no routing, and is read only when a message is already being refused. Nothing retries, nothing waits, and no bound moves — the three things this page has ruled out since #3566 stay ruled out.

Two details of the recording are load-bearing, and both are the same mistake avoided twice:

• The unsubscribe is marked across the self-and-ancestors chain, for the reason the sub-hub search walks that chain at all — a sync/{id} can be registered on an ancestor of the hub RouteStreamMessage happens to run on. Marking only the routing hub would leave the ancestor's entry unmarked and report a real subscriber release as an owner-side reap. Marking creates nothing (it only updates an entry that exists), so a chain hop that never served the stream costs one dictionary miss.
• A registration REPLACES any previous disposition for that id, rather than keeping it. A stream id is deliberately reused across a re-subscribe (Resubscribe means "refresh MY stream"), and the constructor runs only when a genuinely new sub-hub is built — i.e. the previous incarnation is gone. Carrying its "released by the subscriber" flag forward would report the second life's owner-side reap as the first life's subscriber release: the same conflation, one level down, and invisible because both sentences are plausible.

That the ledger dies with the activation is not an implementation detail, it is the whole answer: a recycled owner starts from empty, and a subscriber still holding a stream id from the previous activation is then correctly reported as addressing something that no longer exists.

The three sentences

Each cause is its own literal log template, not a parameter on a shared one, so the split holds however a reader derives a fingerprint — from the template or from the rendered line.

🚨 And a fourth line that is not a cause. The ledger is bounded (one entry per distinct stream id; a long-lived owner on a written path mints fresh ones through the change-feed eviction cycle documented on Workspace._remoteStreamLeases). The moment it has pruned anything, an ABSENT stream id is no longer evidence of "never registered" — it is equally "registered, and long since aged out". So it says exactly that, with its own numbers:

this hub holds NO RECORD of the stream — its activation ledger has already aged N disposition(s) out and holds M, so 'never served here' and 'served and long since ended' cannot be told apart

A diagnostic that cannot fail to give an answer is not a diagnostic. The fourth line exists so a FULL ledger can never masquerade as the third cause, which is the one reading that would send the next reader hunting a reactivation that never happened.

🚨 And a FIFTH line, for when nothing was asked at all. A refusal routinely fires while a hub is tearing down, and a hub whose DI scope has closed hands back no ledger. That is not evidence that a stream was never served — it is the absence of evidence — so it gets its own sentence, with no numbers in it, because there are none:

this hub could NOT BE ASKED which end the stream met — its activation ledger was no longer resolvable, which is what a hub tearing down looks like

The same rule governs the ancestor walk: a searched hub with no ledger is skipped rather than read as a "no", and only the hub whose name the line actually carries can turn an absence into the third cause.

The measurement

RefusalNamesWhichEndTheStreamMetTest produces each cause by its own real route, against real hubs, and reads the refusals out of the host's own log:

The reaped case deliberately has no client-side stream behind it: one would re-subscribe on the owner's StreamEndedEvent and re-create a sync/{id} under the same stream id, so the reap would be undone by a race and the test would measure whichever won. With nothing to re-ask, the window is closed by construction rather than by a wait.

Falsified by reverting the attribution (the single shared sentence restored), rebuilding the test project and rerunning: Failed: 5, Passed: 1 — every cause test red on its own assertion, and the positive control stays green, which is why it exists: a "fix" that refused everything with a more descriptive sentence would satisfy all five of the others on its own. The pairwise "and NEITHER of the other two" assertions are the other half of that — they are what a re-merge of two causes into one sentence reds on.

🚨 The door the ordering never reached — the portal hub's own teardown (2026-09-16)

Everything above orders the release behind an accepted action. It was re-examined from the code on 2026-09-16 with one question — can an action the client accepted still be lost before it is delivered? — and two routes were driven against real hubs with the click provably still queued on the client-side sync/{id} hub when the teardown starts (the hub is busy: a large patch, a binding update). The busy queue is not raced: the test parks that hub's turn loop, submits the click behind the park through the real SubmitUserAction, starts the teardown, and only then lets the click leave. The owner-side action records whether its own sync/{id} had already been told to go when it ran, so a click that is refused and a click that runs on a handler being released are both red, without a timed "nothing happened" window.

So the ordering fix was right and the drop was one hop earlier. The release is not what lost the click on the second route: the portal hub closed its door to its own hosted hubs in the very phase in which it asks them to go down. A parent in DisposeHostedHubs refused all transit — in the child's route-up (HierarchicalRouting) and again at its own intake gate (tier 2 of MessageService.RefusesIntake) — on the stated ground that "the children are going down with it". But that phase is when the children are asked to go down: each child's ShutdownRequest queues FIFO behind the work it already accepted, and that work runs first. The click was accepted work with no door left.

The fix — a parent carries what its children already accepted

MessageHub.CarriesAcceptedWorkOfAHostedHub is one predicate, asked at both places the delivery was refused, and each clause is a reason:

• only while the parent is in DisposeHostedHubs — before it the gate is open; it cannot reach ShutDown until every hosted hub has completed, and a hosted hub cannot complete before its accepted backlog has been handed up, so the delivery is in the parent's queue before the parent's own ShutDown is even posted;
• only a request its originating hub holds a live response callback for — the receipt that hub's Quiescing drain waits on. A one-way IRequest posted without Observe, and fire-and-forget, keep their historical drop; a reply already has its own exemption, which is also how the owner's receipt gets back into the child's drain;
• only transit — a request addressed to the parent itself is new work for a hub that is going away and stays refused;
• only while the hosted hub handing it up is still below Quiescing — the acceptance fence, and it is structural, not a snapshot: routing runs inside that hub's own turn, and the hub leaves Started only by handling the ShutdownRequest its Dispose() posted FIFO. A delivery routed from below Quiescing was therefore queued ahead of that request (accepted before its teardown); one it takes on afterwards queues behind it, is routed from Quiescing, and is refused;
• only when the parent's own parent still routes — in a whole-tree teardown the request could only be dropped one hop later, and the requester is better served by today's immediate transient refusal than by waiting out its quiesce budget.

Nothing new waits and nothing is timed: the parent was already joined on its children's DisposalCompleted; it simply keeps routing for them while it is.

The measurement

UserActionQueuedBehindABusySyncHubTest (MeshWeaver.Layout.Test):

Each also asserts that the park released on the condition the test meant, not on its budget — a park that timed out would have let the click leave at an unknown point and made a green result meaningless.

Each falsification was built and run against this change, and each reddened on its own assertion:

🚨 Measured on the same route, and NOT changed here: the release itself is dropped

With nothing owed at all, disposing the client hub does not release the owner-side sync/{id}: measured on the same fixture, its DisposalCompleted did not emit within 12 s. The UnsubscribeRequest is posted by the PARENT from the child's ShutDown, and the parent is then in DisposeHostedHubs, where its own fire-and-forget post is refused at intake. So on the portal-hub teardown route the owner is never told to unsubscribe; its per-subscriber stream is reclaimed only by the unserved-subscriber eviction (EvictClientSubscriptions) when a later change finds the subscriber gone. The comment in MeshWeaver.Plugins' CircuitAccessHandler.OnCircuitClosedAsync — "each child's teardown posts UnsubscribeRequest to its owner node, so the owner-side mirrors close too" — is therefore not true on that route. It does not lose a click (the release is the last thing the child does), so it is recorded here rather than folded into this fix. It is a candidate contributor to the owner-side half of the Started sync/ population in #3432 — a candidate only: how often the Blazor components dispose their streams before the portal hub reaches DisposeHostedHubs (which sends the release through the still-open door) was not measured.

🚨 The door NEITHER announcer reached — the OWNER deactivating (2026-09-20)

Everything above is about the SUBSCRIBER's half: a click the client accepted, and the release or the portal-hub teardown that overtook it. #3986 was then reopened four more times, and the last batch falsifies its own filing.

The measurement that does not fit the filed cause

memex-cloud, one pod, occurrences 5–8 of Admin/_LogIncident/c3ea7263f217a7f3:

16:47:18.478  REFUSING ClickedEvent on area Catalog/Categories/Cat-Education
              for stream b98AWu3uVUS05xCcA9XQeA on hub Store …
              Sender: sync/b98AWu3uVUS05xCcA9XQeA~portal/ASZ-lU6GbnZJU__d0Japk2GPwZqsjg214pwfwTV5YY0
16:47:20.144  … same area, same stream, same sender
16:47:28.175  … same area, same stream, same sender
16:47:30.464  … same area, same stream, same sender

Four clicks, one stream, one sender, twelve seconds. Each line is written 5 s after the message arrived (the registration grace), so the arrivals span 16:47:13–16:47:25. A subscriber whose circuit is gone does not click four times over twelve seconds — and one whose STREAM has been released posts nothing at all: SubmitUserAction answers locally off HubIfHeld and never reaches the owner. The client half was alive and working; the OWNER had no sync/{id}. The person kept clicking because the page still rendered.

The 2026-09-10 occurrence has the same shape with one click (Catalog/Categories/Cat-Insurance, the same Store hub), and the two middle ones name a per-node request hub and a per-node exercise hub — rbuergi/Requests/provision-pearl-20260914, AgenticOffice/03-Rechnung/Exercise/VierFehlerarten — the addresses that go idle and deactivate. All eight are owner-side.

Why the subscriber was never told

A stream's end has TWO announcers, and for a deactivating owner both were keyed to an event that does not happen:

1. JsonSynchronizationStream's per-stream StreamEndedEvent is deliberately SUPPRESSED once the owning hub is disposing — "a hub must speak only for itself, and never while it is dying", because a dying owner reaching up the hub tree resurrects the Orleans activation it is retiring (OrleansGrainTeardownStragglerTest). It delegates that case, in terms, to the second announcer.
2. Workspace's RecycleAnnouncement has neither problem: it snapshots the client-subscription registry while the hub is whole, resolves a non-router carrier that OUTLIVES the hub, and posts only after DisposalCompleted. But it was hung on MessageHub.HandleDispose — on a message-routed DisposeRequest.

MessageHubGrain.OnDeactivateAsync calls hub.Dispose() directly and posts no DisposeRequest, and this codebase calls that "the largest single source of direct Dispose() in the mesh" (#4888). So on the commonest teardown there is, neither announcer spoke. RecycleAnnouncementTest even pinned the silence as intended, with the reason "only a message-routed DisposeRequest is a RECYCLE — an address that is coming back. A direct Dispose() is a teardown". That reasoning is right for HostedHubsCollection disposing its children and false for a grain deactivating on a live silo: the address IS coming back, and the subscribers are live mirrors in other hubs, circuits and pods that are NOT going down with it.

Nothing else covers it, which HandleDispose's own comment had already established for the routed case and which holds verbatim here: the recycle re-arm needs an in-flight SubscribeRequest to be NACKed (nothing re-asks, so nothing is NACKed) and the change-feed latch needs a WRITE (a deactivation is not one). The mirror kept replaying its last snapshot, and every user action it sent afterwards was refused "NO sync hub for this stream was EVER registered on the current activation" — the third sentence of the classifier above, describing a platform drop exactly as designed.

The fix — the goodbye hangs on the TEARDOWN, not on the request

MessageHub.AnnounceRecycleUnlessAnAncestorIsTakingUsWithIt is the FIRST statement of Dispose(), so it runs on whatever turn started the teardown and before IsDisposing flips — which is what keeps the two things the announcement reads available (the registry, and a resolvable parent). HandleDispose no longer announces; it ends in Dispose() on the same turn, so a routed recycle behaves exactly as before and still gets exactly ONE goodbye.

The guard is the fact that actually decides whether telling a subscriber to re-ask is right, and it is answered in two independent places, neither a guess:

• Is an ANCESTOR taking us with it? IsShuttingDown is already true then — HostedHubsCollection.CloseCreation freezes the whole subtree the instant the ancestor's own Dispose() starts, strictly before it disposes its children. The carrier (our parent) is going down too, the address is not coming back, and a re-ask is the resurrection (1) exists to prevent. Silent, exactly as before.
• Does a carrier outlive us at all? Workspace.AnnounceRecycleToClientSubscriptions already resolves one and returns without posting when there is none — which is what keeps a ROOT hub's host teardown silent (its parent resolves to itself) without MessageHub knowing about hosts.

🚨 The first question is asked TWICE, because its answer is only final at delivery. The read at the top of Dispose() is unsynchronized with an ancestor's CloseCreation on another thread — a window that existed unchanged when the read sat on HandleDispose. Closing it with a lock would mean holding one across hubs around a callback. It needs none: Workspace's deferred Announce() asks the CARRIER IsShuttingDown when the owner's DisposalCompleted fires. A cascade that raced the first read has by then frozen the carrier too — it is this hub's parent, or a sibling under the same router — so a shutting-down carrier means the tree is going, and the goodbye is declined.

🚨 The seam's contract changed with it (RecycleAnnouncement): Announce is invoked by whichever thread STARTS the hub's own teardown — the hub's turn for a routed request, the caller's thread for a direct Dispose() — so an implementation must be thread-safe and must not assume a hub turn. The one real implementation already was: it snapshots a ConcurrentDictionary and resolves a parent hub.

Nothing new waits, nothing is timed, nothing polls and nothing retries. The announcement was already event-driven off DisposalCompleted; only the event it hangs from moved.

The measurement

The end-to-end fixture is the production shape rather than a simulation: host/1 is reached through RouteAddressToHostedHub with HostedHubCreation.Always, so disposing it and then addressing it again IS deactivate-then-reactivate — a new activation with no sync/{id} for a stream a live subscriber still holds. Every wait is on the event that settles its step (the owner's own DisposalCompleted, the mirror's next emission, the click action's own signal), and the file contains no interval of its own at all: the negative assertion's window is the host's SyncStreamOptions.SyncHubRegistrationGrace, READ from the hub rather than set or written as a literal — a number the test invented would either be shorter than the framework's (an assertion that cannot fail) or a guess about a machine's speed.

Falsified by moving AnnounceRecycle() back onto HandleDispose alone, rebuilding both test projects and rerunning:

The two that stay green are the positive controls a wrong fix would fail: announcing from BOTH sites reds the routed test on announcements == 2, and dropping the IsShuttingDown guard reds the cascade test.

🚨 One trap the end-to-end hit first: the test tree's log floor is Warning

The first run of AClickAfterItsOwnerDeactivatedStillRuns was RED with the fix in place, for 36 seconds, because its re-subscribe assertion reads an Information line and test/appsettings.json floors every category at Warning — and filtering happens in the LoggerFactory, BEFORE a provider is handed the record. The sink saw nothing whatever the framework did. The fix is a provider-scoped AddFilter<SubjectLoggerProvider>(null, LogLevel.Trace): this test's sink sees everything, the console sink keeps the tree's floor, and no src-tree level moves. A sibling test in this family (RefusalNamesWhichEndTheStreamMetTest) never hit it because every line it reads is Error.

What this still does not do

It does not make a click that ALREADY missed its sync/{id} run — the refusal above is still the answer for one in flight when the activation went. What it ends is the stranding: the subscriber learns within one round trip and the NEXT click lands on a live handler, instead of every click for the rest of that page's life being thrown away.

What this deliberately does not do

• Any retry, resubscribe or widened grace. The issue rules all three out and so does this: an event that is genuinely undeliverable is not made deliverable by polling for a stream that is gone, and moving the 5-second grace only moves the cliff.
• The harness half. A Store install click with no outcome check and no wait — unlike its sibling install() helper, which retries in rounds — stays in MeshWeaver.Education#275.
• Making a lost action run, now that the cause is named. Naming the cause is an ATTRIBUTION change: it tells a reader which of the three they are looking at, and gives the third one — an owner-side per-node hub deactivating under a live subscriber — a ticket of its own instead of reopening this one. A resubscribe-before-accept fix for that route would belong there, and it would still not be a timer, a retry or a widened grace.

Related

• Error Propagation & Wedges — why fire-and-forget traffic is not NACKed during teardown, which is the rule this change makes one exception to and states.
• Stream Liveness and the Hub Reference — how a stream and its sub-hub come apart, which is the state this page starts from.
• Hub Disposal Model — the Quiescing callback drain that now retains accepted user actions until their owner-side receipt lands.
• Localization — the catalog and the explicit-locale rule.