Durable Streams Are Mesh Nodes

Read first: Orleans Stream Pub-Sub Durability (the defect and why a durable PubSubStore did not close it) and Pod-Hub Delivery (the transport swap that made the stream a fallback). This page is the design those two pages end on. Maintainer direction, 2026-08-27, recorded on #2320 / #2322: "we can essentially use mesh nodes for durable streams" — do not evaluate Microsoft.Orleans.Streaming.AdoNet.

Status: design, with the first slices landed. Everything under Where it stands names the PR that shipped it or the issue it is waiting on.

The one idea

"Use mesh nodes for durable streams" is not one replacement — it is three different answers, because the memory stream carries three different kinds of traffic, and durable means something different for each:

A durable stream provider would have bought the first two rows a property they do not need and left the third and fourth exactly where they are. That is why the provider decision went the other way, and why the memory stream can be retired without a successor of the same shape.

What the memory stream still carries — measured on main, not remembered

Registered once, silo.AddMemoryStreams(StreamProviders.Memory) in OrleansServerRegistryExtensions.cs. Every remaining user, from a grep of GetStreamProvider / GetStream< across src/:

Two facts that change the shape of the design, both verified from source:

1. No production process hosts hubs as an Orleans client. UseOrleansMeshClient has exactly two callers, both test fixtures (OrleansMeshTestBase, OrleansDocumentationTest). Memex.Portal.Distributed is a co-hosted silo; the monolith, LocalMesh and the bake host run no Orleans at all. So "a client cannot host a grain, therefore it keeps the stream permanently" — the standing justification for the fallback — describes the test rig, not the fleet.
2. The database's own cross-process feed is live in production. AddPartitionedPostgreSqlPersistence registers PostgreSqlChangeListener and the IHostedService that opens its LISTEN mesh_node_changes session (PostgreSqlExtensions.cs, pinned by ChangeListenerWiringTests since #1814/#1816). Every commit on any mesh_nodes table fires notify_mesh_node_changes(), which already de-duplicates no-op updates. The feed carries {path, op, node_type} and surfaces as IStorageAdapter.Changes (DataChangeNotification, with an identifier-only descriptor). Several code comments still say this session "is not started in the partitioned wiring"; they date from before #1816 and are wrong — see Stale claims below.

1 · Data synchronization — already node-backed, nothing to build

A SynchronizationStream mirror does not need the transport to be lossless: every patch carries BasedOnVersion, a gap is detected on arrival, and the mirror re-requests from the node — whose Version is the durable, monotonic revision counter. That IS the durable stream, and it is why the earlier "frame loss" storms (#1384, #2641) were resync storms rather than data loss.

What the transport owes this row is only honest classification: a fault that is a lifecycle transition (a silo departing, a grain-directory handover, a container disposing) must reach the mirror as ErrorType.ShuttingDown, which the resubscribe latch rides out — never as a terminal Failed, which tears the mirror down. #2518 (directory instability), #2647 (scope teardown) and #2645 (attach retry) are that work. Kept, unchanged.

2 · Request / response — retire the stream from the routing leg

A request whose target is not live should fail in milliseconds, transiently. The stream fallback does the opposite twice over: a publish into a stream with no live subscriber succeeds and discards (the subscriber probe narrows this but fails open by design), and a publish into a stream whose queue grain is wedged or whose producer never registered stalls for 30–60 s (#2322, #2320, #2406 — all three are Orleans-internal, and there is no MeshWeaver line to change on that path). Durability cannot fix a request that should not have been queued.

The design is the roll plan's release N+2, made precise:

• PodHubNotHere after the directed call is answered with a transient NACK, not a publish. RoutingGrain.BuildPodHubRoute's FallBackToStream() arm becomes PostFailure(…, ErrorType.ShuttingDown, TargetUnserved: true) — the same verdict the subscriber probe already produces for "no silo serves this hub" (#1742), now reached in one hop. SynchronizationStream and MeshNodeStreamCache ride it out; a requester gets its answer inside the directed call's own budget instead of the stream's.
• The owner's claim gets a DERIVED lifetime. OrleansRoutingService.AttachPodHub used to make six attempts over ≈3 s and then give up at Debug — after which a silo-hosted hub kept the stream forever, invisibly (the only signal was the router-side fallback line). The claim now retries with its capped backoff until one of two real terminals: the hub's registration is disposed, or IHostApplicationLifetime.ApplicationStopping fires (the gate GrainWhileRunning already expresses). Once the initial budget is exhausted on a process that can host grains the line is Warning naming the hub — abnormal, and the fleet has no clients for which it would be noise. This is the #2426 rule applied to the claim: no cleanup message a restarting process would never send, only lifetimes derived from the hub and the host.
  • A third terminal, and it is derived too: IMPOSSIBILITY. A process that cannot host a grain can never win the claim — PodHubGrain is [PreferLocalPlacement], so from a cluster client the activation lands on some silo with no local route and answers false, for ever. There the initial budget is the end and the give-up stays at Debug, because that is the expected permanent outcome. Retrying it would not be a lifetime, it would be a poll that cannot converge — and a measurable one: every attempt makes the silo log [POD-HUB] Attach … landed on a silo that has no local route at Information, i.e. one line per hub per backoff interval, which is the storm shape #2426/#2546 exist to remove. The discriminator is Orleans' own: ILocalSiloDetails is registered by DefaultSiloServices and by nothing else.
• The stream stays only for CLIENT-hosted address types, by declaration. The fallback is gated on the address type being declared client-hosted (MeshBuilder.AddClientHostedAddressType), never on the grain answering "not here". In production no address type is declared client-hosted, so the router never publishes. The Orleans test rig declares all four built-in stream-routed types — it hosts a hub of each on its cluster client (client/{id} from GetClient, the client host's own root mesh/{guid}, portal/{guid} in the documentation/graph/markdown tests, and the client's cache hub) — and keeps its stream until the rig hosts its hubs on a silo, at which point AddMemoryStreams and PubSubStore go with it.
• The verdict is ShuttingDown + TargetUnserved, and the owner-side eviction had to be re-gated to see it. DataExtensions.HandleTargetUnservedFailure — the #2426/#2546 fix that stops an owner fanning changes out to a dead subscriber forever — required TargetUnserved && ErrorType == NotFound. That second test was redundant belt-and-braces from the era when the only producer of the stamp was the stream leg's subscriber probe; left in place it would have made the new one-hop verdict inert, silently re-opening the leak for every dead circuit in the fleet. The gate is now the STAMP alone, which is what DeliveryFailure.TargetUnserved's own contract always said ("only the router … may stamp this"). The two facts are complementary rather than contradictory: the subscriber rides ShuttingDown out and re-asks, while the owner drops the server-side half it can no longer push to.
• StreamMessageSizeGuard (#1890) needs no code change, and did not get one. The guard exists to turn a silent drop into a loud, NACK'd refusal: an oversized payload on the memory stream succeeds at the publish and dies inside PersistentStreamPullingAgent's non-convergent retry loop, naming only a queue id. On the directed call the wall is Orleans' MaxMessageBodySize and crossing it throws, which BuildPodHubRoute's TerminalCallFailure already turns into a classified DeliveryFailure naming the address, the delivery id and the sender — the outcome the guard exists to produce. Retargeting the constant itself (plumbing SiloMessagingOptions into RoutingGrain plus its own refusal shape) is a separate change and is not required for correctness here; the guard stays on PostToStream, which after this slice is reachable in production for nothing at all.

The N+2 gate, and why these two slices shipped without waiting for it

The roll plan's gate was: a full rolling deploy with none of [ROUTE] Pod-hub grain for {Address} is not attached — falling back to the stream publish in Loki. That gate measured a risk that the two slices above jointly remove, so it was satisfied by construction rather than by observation:

• The gate's worry is the window "the owner exists, but its claim has not landed" — in which removing the fallback would drop a delivery to a live hub. Slice 2 does not drop it: it ANSWERS it, with a transient NACK inside the directed call's own budget. Every consumer on that path already has recovery machinery armed for exactly this verdict (SynchronizationStream's resubscribe latch, MeshNodeStreamCache.IsTransientOwnerFailure, MeshNodeStreamExtensions' paced retry), which is why the verdict is ShuttingDown and not NotFound. The pre-change alternative was strictly worse for the same window: a publish that succeeds and discards when nobody is subscribed, or stalls 30–60 s on a wedged queue grain — the failure with no signal at all.
• Slice 1 closes the window rather than surviving it. The claim now retries until it lands, so "owner exists, claim not landed" resolves by retry instead of persisting for the life of the process. Before slice 1 it could persist forever, which is precisely why the gate was needed.
• And the gate's own instrument was unreliable in the direction that matters: the line it counts is Information, emitted per delivery, and its absence was never proof (the page said so). A Warning naming the hub, emitted once per claim that has not landed, is a strictly better instrument — and it is what slice 1 adds. The fallback line survives, but after slice 2 only a DECLARED client-hosted type can reach it, so in the fleet it cannot be emitted at all.

Residual risk, stated plainly: an address type that is genuinely client-hosted in some deployment and was never declared would go from "works over the stream" to "NACK'd transiently". No such deployment exists — UseOrleansMeshClient has only test-fixture callers — and the symptom would be loud (a windowed Warning naming the address) rather than silent.

3 · Cross-silo change notifications — the storage layer's feed is the durable channel

This is the only memory-stream user whose loss is permanent, and it is also the one with a ready-made durable substitute that the platform already operates, monitors and backs up.

Before the storage relay there were two parallel cross-process channels for the same commit:

write ──commit──► pg_notify('mesh_node_changes', {path, op, node_type}) ──LISTEN──► IStorageAdapter.Changes ─► synced queries re-run,
                                                                                                    MeshDataSource reconcile re-reads
      └─post-commit─► IMeshChangeFeed.Publish(MeshChangeEvent)  ─local Subject─► consumers
                                └─► Orleans memory stream "mesh-{kind}" ─► PathCacheInvalidatorGrain (other silos) ─► PublishLocal

The first core slice collapses the cache-invalidation leg onto the storage feed while retaining the Orleans broadcast during the additive rollout:

1. The notification contract carries NodeType and Version. The PostgreSQL trigger already sends node_type beside path and op, and has NEW.version in hand. DataChangeNotification gains NodeType and Version as optional members. Typed in-process producers derive the hints from their entity immediately; older and path-only backend payloads leave them null. Core lands first, the trigger change in the PostgreSql adapter second; the relay below tolerates a payload without them by re-reading the node before any consumer that filters on type sees the event. That compatibility read has a five-second bound. An error or silence still emits a path-only version-zero invalidation, so one backend fault cannot wedge the relay or leave the replica's exact-path cache untouched. A read that finds NO row emits nothing: the row is gone, the delete that removed it is self-contained and was relayed on arrival — and a Created/Updated with no node and no version AFTER that Deleted would read as a retype to "(none)" to NodeTypeRebindWatcher and recycle a hub the delete is already tearing down. Same rule as the per-node hub's own reconcile.

   🚨 The compatibility read is coalesced per path, through the ONE coalescer (ReReadCoalescing in MeshWeaver.Mesh.Contract: a 50 ms Throttle that always emits the LAST trigger of a burst, then Concat so reads on a path are serialised) — the same operator and the same window the per-node hub's own reconcile in MeshDataSource has used since #1440. The relay first shipped reading ONCE PER NOTIFICATION, ahead of any coalescer, and the plugins suite's read-storm guard (CrossProcessChangeFeedTest.AnEntitylessBurst…, which measures that window by name) went red on the first set carrying it: 200 entity-less notifications on one path cost 801 reads against a query-layer baseline of 600 — 200 relay reads plus the one coalesced reconcile read, 201 where the contract is "a handful" (< 20). That is the #223 shape — a notification storm turned into a read storm — on every replica, for every bulk import under the older notifier (#4139). A second coalescer with a second window is how this comes back with a different number: both callers reference ReReadCoalescing.Window.

   The newest notification on a path always wins. Every notification on a path joins that path's group; self-contained ones (node, hints, or a delete) are relayed on arrival, and the coalescer fires with the LAST notification of the burst once the path has been quiet for the window — a read is owed only when that last one is not self-contained, so a burst that ends on a hinted update or a delete reads nothing. A read in flight is overtaken by ANY later notification on its path and then says nothing: the later one is either self-contained (already relayed, newer than anything the read could return) or starts its own coalesced read. So a read result is published only while it is the newest information about its path, and a path-only fallback for a read that faulted or timed out can never land after a newer event — which is what kept the remote-stream resubscribe gate (a version-zero event announces received + 1) from refreshing a healthy stream. A path's group lives (window + read bound) past its last notification: with overtaking at most one read is ever pending per group, it starts at the window and is over by the bound, so a notification during it joins the same group and overtakes it, and one after the group closed finds nothing in flight. Reads on a path never overlap; an idle path holds no state.

   The PostgreSQL listener today carries a backend-private ChangedNodeDescriptor (path + node type) as the entity and sets neither hint, so every production NOTIFY takes the coalesced read on every replica until the plugins module sets DataChangeNotification.NodeType and Version from the payload — the follow-up #4104's body names. The relay classifies that descriptor silently: a foreign entity is the feed's designed shape, not a fault (the first relay logged it at Error, once per NOTIFY per replica).

2. A relay, not a grain. The mesh-scoped InProcessMeshChangeFeed owns one StorageChangeFeedRelay, which subscribes IStorageAdapter.Changes and relays each notification into the process-local IMeshInvalidationFeed. Each process therefore receives its own database listener's copy directly. A monolith or LocalMesh process that shares a database with another process gains cross-process invalidation too. The old Orleans broadcast classes remain binary-compatible during the additive rollout; their PublishLocal entry point now feeds only invalidators. After the PostgreSQL payload upgrade ships, the dead wrapper registration and PathCacheInvalidatorGrain can be deleted without a mixed-version gap.

3. Cache invalidation and logical delivery are separate. A direct IMeshChangeFeed.Publish reaches both logical subscribers and the writer's local invalidators. A storage or Orleans echo reaches only IMeshInvalidationFeed. This boundary is load-bearing: logical subscribers include AccessGrantNotifier (which can send mail) and InstanceSyncCoordinator (which can write to another instance). Sending a database echo through the logical feed would run those effects once per replica, while PostgreSQL can additionally echo the writer's own commit. Cache invalidators are idempotent and deliberately run in every process; logical effects retain the publisher's single delivery. Both channels serialize concurrent publishers. Invalidation subscribers are isolated per callback, so a broken cache misses its own event and cannot prevent later caches from receiving the same commit.

4. Loss semantics are strictly better. A NOTIFY is missed only inside the listener's own reconnect window (a 5 s retry loop, logged at Error), which is the window the synced queries already accept and the reconcile-on-start pattern in Event Subscriptions already covers. There is no rendezvous grain to time out, no queue grain to lose its RAM, and no membership handover in the path — the three mechanisms behind #2320, #2322 and #2406.

5. Backend-agnostic by construction. Cosmos and Snowflake already feed Changes with Entity = null; the in-memory and file-system adapters feed it in-process. The relay does not know which one it is on.

The production proof that selected the relay

On 2026-09-12 the public two-replica portal held two incompatible answers for the same path after a successful plugin publication. An exact read of Hosting/PlatformBuilds/plugins on one request returned the old v7 row and source SHA, while the children query returned a newly-created v1 row with a different storage identity and the current SHA. The new row had committed and its Created notification had been emitted; one process's exact-path cache had simply never seen it.

The reason is structural twice over. The default Orleans composition calls AddPartitionedInMemoryPersistence first; that reaches AddMeshCatalog and registers IMeshChangeFeed as the process-local feed, so the later Orleans wrapper's TryAdd is ignored. Even if that order were reversed, OrleansMeshChangeFeed publishes every kind at stream id Guid.Empty, and PathCacheInvalidatorGrain is an ordinary grain at that same key. Orleans grain identity is a cluster singleton, although the class comment claimed one activation per silo. Its handler calls a process-local InProcessMeshChangeFeed, so only the process hosting that one activation is invalidated. No retry, registration reorder or second publish can guarantee delivery to every process. Subscribing each process to the storage notification stream removes both impossible fan-out assumptions.

4 · At-least-once work items — the _Inbox node is the durable stream

Where a message genuinely must outlive the process that will handle it, the node-backed stream already exists and is documented: Webhook Inbox. Its contract, restated here because it is exactly the contract a durable stream needs:

• an entry is a write-once node at {target}/_Inbox/{id} — its own existence is its lifetime;
• the consumer is a live children query taking Initial | Added | Reset, processed with Concat (one at a time) — Initial is the replay-on-start leg, so an entry written while the consumer was down is delivered when it comes up;
• the ack is a delete, under system identity, on every outcome including "unverifiable" and "irrelevant" — a poison entry is dropped, never looped; every action is idempotent because replay is the normal case, not the exception;
• the consumer lives on an always-on hub. A drain armed on an on-demand per-instance hub runs only while somebody happens to be looking at that node — Plugins#777 is precisely that defect (Hosting/Deployment's inbox watcher stopped consuming platform-build announcements until a Deployment page was opened). A fleet-wide consumer belongs beside a hub warmer or in a host-level hosted service, and wants the positive liveness signal "the inbox is non-empty and ageing".

The derived-lifetime rule, applied

#2426 found a server-side subscription that only an explicit UnsubscribeRequest could dispose — immortal by construction, because a portal that restarts never sends one. Every row above is checked against it:

Where it stands, and the order of work

How to check a live cluster

# 🚨 Must be EMPTY, always, not merely after a roll: only an address type DECLARED client-hosted
# can reach this line, and production declares none. A hit means somebody added a declaration.
{namespace="memex-cloud"} |= "falling back to the stream publish"

# THE instrument now — a hub whose claim has not landed, once per claim, naming the address.
# The claim keeps retrying, so a matching "landed after its initial budget was exhausted" line
# for the same address is the resolution; one without it is a hub still on no transport at all.
{namespace="memex-cloud"} |= "Pod-hub claim for" |= "did not land"

# the transient NACK that replaced the publish — windowed to one Warning per address per 60 s
{namespace="memex-cloud"} |= "was refused: no silo in this cluster is currently serving that hub"

# the database feed is up on every pod — one line per pod per LISTEN (re)connect
{namespace="memex-cloud"} |= "PostgreSQL LISTEN started on mesh_node_changes"

# the stream-provider failure family this design retires
{namespace="memex-cloud"} |~ "RegisterAsStreamProducer failed|memorystreamqueue.*Enqueue"

Stale claims this page corrects

Code comments and doc pages in core (BuildCoordinationExtensions, BuildNodeType, BuildProtocolDriver, RegistryUpdateReconciler, Build Coordination, Plugin Update on Green Build) stated that PostgreSqlChangeListener is "registered and never started in either partitioned-PG overload"; the PostgreSql adapter's PostgreSqlPathRoutingAdapter (in MeshWeaver.Plugins) still says "the pg_notify LISTEN fallback is disabled for partitioned PG". That was #1440 as filed on 2026-08-13; it was the middle leg of the #1814 outage and was fixed by #1816 — the partitioned overload now registers the hosted service that opens the session, and ChangeListenerWiringTests fails the build if it ever stops. The conclusions those comments draw (read the durable witness, never wait to be told — BuildProtocolDriver, BuildNodeType.ArbitrateDurably, ObserveBuildGo, RegistryUpdateReconciler) remain correct for a different reason: a NOTIFY is delivered to a live LISTEN session on the same database and is never replayed, so a mirror that activates after the write — or a deployment on another database — is still not told. Every core occurrence is corrected with this page; the adapter's one is corrected in its own repo.

Related

• Orleans Stream Pub-Sub Durability — the defect, the durable PubSubStore, and the two residuals this page answers
• Pod-Hub Delivery — the transport swap and the N+2 gate
• Event Subscriptions — the live + reconcile-on-start pattern
• Webhook Inbox — the _Inbox node contract
• Error Propagation & Wedges — "an undeliverable delivery must surface as a DeliveryFailure, never as silence"
• Issues: #1742, #2320, #2322, #2406, #2426, Plugins#777