A release refused at the source

The rule. A hub past DisposeHostedHubs refuses every post but its own ShutdownRequest / DisposeRequest and a correlated reply. That refusal is right for an event — nobody awaits one, and the receiver recovers from the loss through a fresh snapshot, a re-subscribe, a change feed or a heartbeat lapse. It is wrong for a release, because a release is the only thing that ever frees state the RECEIVER holds: there is no requester to NACK, no retry to trigger and no later probe that discovers the loss. So "nobody is waiting" is the reason a release must go out, not a reason it may be dropped.

This is the third asymmetry at the same gate. The first two are in Refused replies during teardown: a refusal is advice to the sender, which is the wrong party for a reply. This page is the one that leaks memory instead of stranding a caller — and it leaks it silently, in another hub, with nothing to grep.

Filed as #3432, whose title stood for two weeks as population MEASURED, cause NOT established.

1. What holds a sync/ hub — settled before this page

A cross-hub subscription builds two SynchronizationStreams and therefore two sync/{id} hubs: the client's, hosted by the subscribing hub, and the owner's twin, hosted by the owner (Sync hub population shows the arithmetic — 6 925 Started + 1 495 Dead sync hubs against 8 461 streams, 0.5 % apart, so there is no separate hub population to explain). Each retains roughly 390 KB: its own Autofac ILifetimeScope and its own TypeRegistry.

Three earlier causes have landed, each removing a way the population GROWS:

This page is about the other direction: not how one is created, but why the one thing that ends it never arrives.

2. The only thing that ends an owner-side stream

Workspace says it in as many words: "only an UnsubscribeRequest disposes a server-side stream". JsonSynchronizationStream.CreateExternalClient registers that release, and since #3986 it registers it on the stream's hub rather than on the stream, so it runs from the hub's ShutDown phase — strictly after Quiescing, whose whole job is draining the response callbacks an accepted user action holds. That ordering is deliberate and is pinned by Refusing a lost user action.

// src/MeshWeaver.Data/Serialization/JsonSynchronizationStream.cs
var release = new AnonymousDisposable(
    () => hub.Post(new UnsubscribeRequest(reduced.StreamId), o => o.WithTarget(owner)));
if (reducedHub is not null)
    reducedHub.RegisterForDisposal(release);
else
    reduced.RegisterForDisposal(release);

🚨 Read the two hubs in that snippet. The release is REGISTERED on reducedHub — the client-side sync/{id} hub — and POSTED through hub, the subscribing hub that hosts it. Those are different hubs at different run levels, and on one of the two teardown routes the difference is the defect.

3. Two teardown routes, and only one of them has an open door

The second row is not a race. DisposeHostedHubs is the phase that disposes the hosted hubs, so the child's ShutDown — which runs the release — can only ever execute while its parent is in it. And at that run level MessageService.PostImplGeneric's teardown guard refuses the post:

if (hub.RunLevel >= MessageHubRunLevel.DisposeHostedHubs
    && message is not ShutdownRequest and not DisposeRequest)
{
    // … a correlated reply is handed to the parent; everything else:
    return ((IMessageDelivery)delivery).Failed("Hub is shutting down", ErrorType.ShuttingDown);
}

An UnsubscribeRequest is neither of the two exempt types and carries no RequestId, so it takes the last line. Nothing downstream ever sees it — it is refused before the post pipeline and before ScheduleNotify, so there is no intake trace, no NACK and no log line above Debug. The owner keeps its per-subscriber stream and its sync/{id} hub at RunLevel=Started, and only an unserved-subscriber eviction can ever reach it — which needs a later change on that node, so for a node nobody writes again it is never.

🚨 CarriesAcceptedWorkOfAHostedHub does not cover it, and correctly so. That exemption (#3986) carries a hosted hub's accepted work OUT through the disposing parent, but it is scoped to a request its originating hub holds a live response callback for — the receipt the child's Quiescing drain is waiting on. A release is fire-and-forget: nothing is waiting, which is exactly why the existing clause cannot see it.

4. The category the guard was missing

The guard's own reasoning for refusing fire-and-forget is sound and must stay — forwarding every event out of a disposing hub is the storm shape. What it did not distinguish is who recovers from the loss:

So the third column gets the marker interface IReleasesRemoteState (src/MeshWeaver.Messaging.Contract/IReleasesRemoteState.cs), and UnsubscribeRequest implements it. Implementing it is a narrow claim — no other mechanism in the system ever reclaims what this message releases — and deliberately NOT a way to make an ordinary event survive a teardown.

5. The carrier is the parent, and one hop is the whole rule

The remedy is the primitive this file already has, applied to a third case. NackThroughParent states it: "our own Post would re-enter this same gate and be dropped" — so hand it to the parent, which is alive. The refused-reply path does exactly that, and the release now does too:

if (message is IReleasesRemoteState
    && ParentHub is { } releaseParent
    && releaseParent.RunLevel < MessageHubRunLevel.DisposeHostedHubs)
{
    releaseParent.Post(message, _ => opt);
    postFate?.Add($"RELEASE_FORWARDED_THROUGH_PARENT runLevel={hub.RunLevel} parent={releaseParent.Address}", Address);
    return delivery;
}

Why one hop is enough, by construction rather than by a bound. On this route the parent IS the hub disposing us, and it cannot reach its own ShutDown until every hosted hub has signalled DisposalCompleted (see MessageHub.CarriesAcceptedWorkOfAHostedHub), so it is demonstrably still routing. In a whole-tree teardown the parent is going too — and then so is the receiver, which is about to drop everything anyway, so there is nothing left to leak and nothing to escalate to.

opt is passed through unchanged, which is load-bearing. UnsubscribeRequest is ICorrelatedBySender: the owner keys its per-subscriber stream on the subscriber that OPENED it, and that subscribe was posted from this same workspace.Hub. Re-stamping the sender as the parent would leave the owner holding a subscription opened by one hub and released by another.

6. What this is NOT

• Not a sweeper, a cap, a weak reference or a timer. Nothing scans, nothing expires, nothing retries. The population drains because the message that drains it is delivered.
• Not "dispose harder". Teardown still lets accepted work finish; the subscribing hub's own teardown is not held open waiting for the owner to answer, and the second test arm below fails if it ever is.
• Not a change to the release's ORDERING. It still runs from the sync hub's ShutDown, strictly after Quiescing, so an accepted user action still orders ahead of it (#3986).
• Not a widening of the guard. An unmarked fire-and-forget event is still refused, with the same transient ShuttingDown classification, and the control arm measures that.

7. The negative control — both directions, no window

Two tests, and neither can pass on no evidence.

SubscriberTeardownReleasesTheOwnerSyncHubTest (test/MeshWeaver.Layout.Test) builds a population and drains it: three remote layout-area streams on one subscribing hub, each with its owner-side sync/{id} live and holding the area's handlers, then disposes the subscribing hub — never a stream.

ReleaseLeavesATearingDownHubTest (test/MeshWeaver.Messaging.Hub.Test) pins the framework contract on the production route — a disposable on a HOSTED hub, so it runs in the parent's DisposeHostedHubs — and reads the POST's own verdict rather than inferring from a wait:

🚨 The control's "never arrived" is decided by ORDER, not by a timer. The unmarked event is posted FIRST and dropped at the source, so once the sink has handled the release there is no queue left that could still deliver it. A negative assertion with a window would have to choose one, and on CI TestTimeouts.Quick exceeds the 30 s methodTimeout.

Rules

1. A message that is the only thing which frees state elsewhere implements IReleasesRemoteState — and nothing else does. An event whose loss the receiver recovers from keeps the historical refusal.
2. Never re-derive "is my hub past the gate?" at a call site. The guard's predicate lives in one place; a second copy in another assembly is two lists that have to stay in step.
3. When a farewell must leave a disposing hub, the carrier is the parent — the same one NackThroughParent and the refused-reply path use. One hop, and no walk: past the parent, the receiver is going down too.
4. A leak fix asserts that the population DRAINS, with a stated denominator, and shows that it does not before the change. A test that watches one object cannot tell "the release arrived" from "that one happened to go away".