Live Mirrors and the Change Feed

A mirror is the cross-hub half of GetMeshNodeStream(path): when hub A reads or writes a node owned by hub B, A opens a SynchronizationStream to B. That costs a SubscribeRequest, an initial-state round trip, a sync/{streamId} sub-hub on both ends, and a registry entry on the owner. Once open it is live — the owner fans every change out to it.

IMeshChangeFeed is the other invalidation signal. The persistence layer publishes one event per Created/Updated/Deleted, and three components listen. This page is about what the third one did, why it looked like a defect, why it was load-bearing anyway — and how #1174 replaced it.

🚨 FIXED 2026-09-15 — the fix is the last section of this page. A versioned Updated no longer evicts: the workspace records the version it announced and keeps the mirror, and the next write's base must reach that version. The sections between here and there describe the behaviour before the fix and are kept because the measurement that rules out the liveness gate is still the reason the fix has the shape it has.

Every cross-hub write ended its own streams (before #1174)

Workspace.EvictForPath dropped every cached remote stream whose owner matches the changed path — including the mirror the writer itself is using, on the writer's own write. A write leases that mirror only for the duration of its Observable.Create subscription, so:

1. write N leases the mirror for P, posts its patch; the owner commits and persistence publishes Updated for P;
2. EvictForPath("P") removes that very mirror from _remoteStreamCache and parks it;
3. write N settles and releases its lease;
4. the parked stream now has no declared holder, so ReclaimIfUnheld disposes it — which posts UnsubscribeRequest to the owner, kills the client sync/{id} hub, and makes the owner dispose its own twin;
5. the owner's healthy-owner announcement then posts StreamEndedEvent back to a subscriber whose sync/ hub step 4 has just destroyed.

The next read or write resolves a fresh mirror: another SubscribeRequest, another initial-state round trip, another pair of sync/ hubs. Per write.

Measured on the monolith harness — one activity node, one live reader, six progress writes:

Those step-5 announcements are the Dropping StreamEndedEvent … the target stream is gone lines that #2776 was filed on.

The two arithmetic traps

#2776 was filed as "the activity node's owner ends its streams ~5 s in", and two hypotheses — a hub recycle and the hosted-hub drain cap, which is also 5 s — were built on that reading. Both are wrong, and each is wrong for a reason worth keeping.

1. The drop diagnostic is written five seconds after the event it describes. A StreamMessage whose sync/{id} sub-hub is not registered is not dropped on arrival: RouteStreamMessage holds it for SyncStreamOptions.SyncHubRegistrationGrace — 5 s — waiting for the sub-hub to appear, and only then logs the drop. So a line at T describes an event at T − 5 s. In #2776 the streams ended at +0.12 s; the ADVANCE_WITHOUT_HANDOFF beside it is a different 5 s bound (QueueAdvanceBound) measured from the write's own start, which was also +0.12 s. Two unrelated five-second bounds counting from one instant produced a coincidence that looked like a mechanism. The drop line now states its own age.

2. The sender rules out a recycle. There are two emitters of StreamEndedEvent. The recycle announcement (Workspace.AnnounceRecycleToClientSubscriptions) is deliberately posted by a carrier — the parent hub or its spokesman — because a dying hub must never speak for itself. The healthy-owner announcement (JsonSynchronizationStream's disposal registration) is posted by the owner and refuses to fire once RunLevel > Started or IsDisposing. #2776's two events name the activity node itself as sender, which is provably the healthy-owner path: at that instant the owner was Started and not disposing. No recycle, no teardown, no drain cap.

Why the obvious fix is wrong

Three components listen to this one broadcast, and two of them already refuse to act on a healthy subscriber:

So the obvious change is to give EvictForPath the same rule — skip a mirror that is still StreamLiveness.IsUsable. It was implemented and measured, and it is wrong.

The eviction is, incidentally, what keeps a cross-hub writer's BASE current with respect to writes the OWNER makes for itself. The per-path update queue hands a predecessor's locally-computed node to its successor (_pendingSelfWrites), but that only carries this cache's writes forward. An owner-side write — an activity's messageCount, a sealed log segment — reaches the mirror only through the asynchronous fan-out. Evicting on the change event forces the next write to resolve a fresh stream and therefore to diff against a freshly-fetched authoritative snapshot.

Controlled measurement, StaticRepoImportActivityWriteCountTest.AppendCost_DoesNotGrowWithTheLengthOfTheActivity, same machine, DOTNET_PROCESSOR_COUNT=4 (the CI-race repro):

And on CI with the gate on, the same test turned 80 append calls into 99 writes with 44 OWNER_NACK_REENQUEUE and MergeGuard: refused stale/reordered cross-hub write to 'messageCount' (changed since the writer's base) — refusing every other version. Both shapes are one mechanism: a stale base.

Note what this means for the two rules quoted above. They are right for their layer: the shared handle and the subscription itself must not be rebuilt on every write. EvictForPath sits below them and is doing a different job than its own comment claims — it is not (only) cache hygiene, it is the writer's freshness barrier. That is why removing it broke writes rather than merely changing their cost.

The fix: a write base held to the announced version (#1174)

MeshChangeEvent already carries Version, so a write can wait for the mirror to reach the announced version instead of throwing the mirror away to force a fresh snapshot. That is what landed, in two halves.

The workspace (Workspace.OnOwnerNodeChanged) answers a commit on a mirrored owner in one of two ways, decided by what the commit announced, never by how the mirror looks:

🚨 The floor is a field of the mirror, not an entry beside the cache. The handler raises it on each instance it finds in the cache for the owner, and the write reads it off the very stream it acquired (Workspace.AnnouncedVersionFor(stream)). So a floor cannot outlive its mirror and cannot reach the next one: whatever removes a mirror — eviction, the mesh-node cache's idle release, a fault, a recycle — removes its floor with it, and a new mirror starts with none, which is right because it hydrates from the owner's current state, at or past every version announced before it. The first cut kept a per-owner map instead, and review found the window that shape cannot close: a removal landing between "the cache still mirrors this owner" and "record the floor" left a floor with no mirror behind it, which the next mirror inherited (AnnouncedFloorLifetimeTest reproduces it deterministically through a seam in exactly that window, and the same inheritance through the faulted-stream path — both red against the map, both green now).

The write path (MeshNodeStreamHandle.RebaseSource, the same seam the #1910 conflict rebase uses) reads the mark once per attempt and requires the base to satisfy Version > refused && Version >= announced. A mirror that already carries the version passes at once; one that is a moment behind is waited for (the owner's own fan-out delivers the version — no round trip); the wait is bounded by ConflictRebaseBound, and it never parks — at the bound the write proceeds on the state it has, exactly as before. With no mark (a cold path, or a feed that never reaches the workspace) the read is byte-for-byte the old mirror.Take(1).

🚨 The fallback evicts only on PROOF. When the bound elapses, the writer evicts the mirror — the old freshness barrier, paid once — only if the mirror carried a node below the floor and nothing at or above it. A mirror that carried nothing at all is still hydrating, not behind; evicting it would throw away the one subscription about to deliver and make the next write hydrate yet another, which is the churn this fix removes, re-entered at exactly the moment the owner is slowest.

🚨 This is the opposite of the liveness gate above. Liveness asks how the mirror looks and lets a healthy-but-behind one through, which is what lost the 25-message batch. The version floor asks the mirror to prove it carries the announced commit — and the announcement is the same event that used to trigger the eviction, so every commit the old barrier covered is still covered.

Measured

The real-mesh arm also asserts that the floor was in force (the workspace's "Mirror … kept" line fired for the later commits) and that the owner's fanned-out version met the announced one every time (no MIRROR_BEHIND) — the second is the property the whole design rests on, and only a real owner can show it. Negative controls: with the source change set aside both measurements go red as shown in the "before" column, and with either refinement removed (Created evicting; proof-only eviction) its own arm goes red.

What it means for #1174 and #2776

Each of those per-write hydrations had to complete inside the writer's 30 s base-state bound while the mirror it replaced was being torn down on the same owner address, and a node like {user}/_UserActivity/{user} — written on every cold page load, at version 6498 in production — paid that thousands of times. That is the only mechanism in the code that predicts #1174's concentration on hot paths. The census from #4342 is what will confirm it on a rolled replica: an occurrence after this fix would have to come from a mirror that was NOT rebuilt per write. The Dropping StreamEndedEvent … the target stream is gone lines of #2776 were the same churn seen from the owner, and stop with it for every write-driven eviction.

The #3432 retention (The Evicted-Stream Retention) is untouched as a mechanism, but it is now reached only by the events in the table that still evict — rare ones — rather than once per write.

What #2776's other half was

Its visible failure was a write that reached no verdict in 31 s. That silence has a separate, already-fixed cause — #2882: the write registered its response subject after posting, so a warm owner's sub-millisecond ack could land before anything was listening for it, and the trail could only ever say REGISTERED_AFTER_POST (exactly what that trace says). Fixed in 4d231170a.

See also

• The MeshNode Stream Cache — the layer above, and the write path whose base freshness this page is really about.
• Debugging Message Flow — how to read a stalled hub trace without mistaking a bound for a mechanism.