Read first: Asynchronous Calls and Orleans Task Scheduler. This page is the I/O-edge counterpart to those two — where the actor model meets real, blocking work.

The problem

Every hub is an actor running on a single-threaded, turn-based scheduler — the Orleans grain scheduler for the root hub, TaskScheduler.Default for every other hub. That single-threading is a guarantee about state, not a claim that the process has one thread: the same process owns the multi-threaded .NET ThreadPool.

Genuine I/O at the leaves — a file read, a blob download, an HTTP call, a Roslyn compile, a Process.Start — must therefore satisfy two requirements:

1. Run off the hub scheduler. A bare await inside a handler captures TaskScheduler.Current and queues its continuation back onto the hub's single turn — blocking the action block, or (across hubs that share a scheduler) deadlocking. The work has to be handed explicitly to the ThreadPool.
2. Be bounded. Without a cap, a mesh of thousands of per-node hubs can each subscribe to the same kind of I/O at once, issuing thousands of concurrent file handles or sockets. This exhausts the resource and — for sync-blocking work — triggers ThreadPool thread-injection that starves the very pool Orleans' grain turns rely on.

Postgres had half of this already: Npgsql's connection pool (MaxPoolSize, sized per role) is a real concurrency governor for DB work. What it never provided is requirement 1 — the old Observable.FromAsync(work, Scheduler.Default) sites did not get the round-trip off the hub scheduler, because FromAsync's scheduler argument schedules notification delivery, not where the function is invoked (see "The hybrid governor" below). File system, blob, HTTP, compile, and process carry no pool of their own and had neither half. IIoPool supplies both, uniformly: off-scheduler by construction, and bounded per resource class.

IIoPool routes all I/O leaves off the hub scheduler onto bounded ThreadPool workers, with per-resource concurrency caps.

🚨🚨🚨 ABSOLUTE: Observable.FromAsync is NEVER tolerated

Observable.FromAsync(...) is FORBIDDEN everywhere in src/ — no exceptions. Not for storage, not for Postgres, not for "it already runs off the scheduler", not for a one-off. There is exactly one place the call may appear in the entire codebase: sealed inside IoPool (the primitive). Anywhere else it is a defect to be removed. Every genuine async/blocking I/O leaf goes through IIoPool.

A bare Observable.FromAsync only schedules notification delivery. It invokes the function's synchronous prologue on the subscribing thread — which is the hub/grain scheduler when the subscribe happens mid-handler — and applies no concurrency bound. That is the entire bug class this primitive exists to kill.

// ❌ FORBIDDEN — runs the prologue on the subscriber (hub) thread, unbounded
=> Observable.FromAsync(ct => httpClient.SendAsync(req, ct));

// ✅ REQUIRED — routed through the resource-class pool: off the hub scheduler, bounded
=> _httpPool.Invoke(ct => httpClient.SendAsync(req, ct));

Pick the method by leaf kind:

There is no "out of scope" residue. If you find yourself typing Observable.FromAsync, stop: the answer is an IIoPool call (or, for an idempotent one-shot, the promise-cache below). The only FromAsync that survives a review is the one inside IoPool itself.

Promise-cache for idempotent one-shots

For work that should run at most once and then be observed by many (schema provisioning, a connect handshake, a container-ready probe), hold the eager pool.Run(...) observable in an instance PromiseCache<TKey, TValue> — or PromiseSlot<TValue> when there is only one — never static:

// PostgreSqlPartitionStorageProvider.EnsurePartitionProvisioned — the canonical example.
// First caller kicks the CREATE SCHEMA off on the per-adapter pool; every later subscriber
// replays the cached completion. No Observable.FromAsync at the call site.
private readonly PromiseCache<string, Unit> _provisioned = new(StringComparer.OrdinalIgnoreCase);

public IObservable<Unit> EnsurePartitionProvisioned(string @namespace) =>
    _provisioned.GetOrAdd(schema, _ =>
        _ioPool.Run(ct => EnsureSchemaAsync(def, ct)).Select(_ => Unit.Default));

// The keyless variant — McpRemoteMeshClient's connect handshake.
private readonly PromiseSlot<McpClient> _connect = new();
private IObservable<McpClient> Connect() => _connect.GetOrCreate(() => _pool.Run(ConnectAsync));

pool.Run is ReplaySubject-backed (see IoPoolExtensions) — eager, single-run, replays to all. That is the "promise pattern": the cache entry is the promise. (pool.RunBlocking is the same for a sync-blocking leaf.)

🚨 Why this is a type and not a ConcurrentDictionary

A ReplaySubject latches terminals, OnError included. So the older recipe — the eager observable in a bare ConcurrentDictionary<key, IObservable<T>> — turned one transient fault into a permanent one: the entry replayed that same exception to every later subscriber for the life of the process, and nothing ever re-attempted. Replay(1).AutoConnect(1) and Replay(1).RefCount() latch identically — one already-terminated subject behind the connectable.

That is not a corner case; it shipped eight times before it was fixed in the recipe (#1369), and its worst instance made a partition permanently un-provisionable after a single connect blip — every later write 42P01-ing until the pod was restarted. PromiseCache exists so the next one-shot someone writes gets the cure for free.

What the type guarantees, and what you must not undo:

Internally the eviction is attached with Do, never a bookkeeping Subscribe: subscribing would be the AutoConnect(1) first subscriber and would connect chains nobody asked for. Invalidate(key) exists for a real domain invalidation (the partition was dropped) — not for test isolation, which a mesh-scoped instance never needs.

Contract pinned by PromiseCacheFaultEvictionTest (test/MeshWeaver.Hosting.Test) and, end-to-end against a real Postgres, PartitionProvisioningFaultRecoveryTests.

The primitive

IIoPool (in MeshWeaver.Mesh.Threading) is the single sealed boundary between the hub schedulers and the I/O. It is hidden inside the leaf adapters — public signatures stay IObservable<T>; callers never see a pool.

public interface IIoPool
{
    // Genuinely-async leaf (blob, HTTP, async file, DB round-trip).
    IObservable<T> Invoke<T>(Func<CancellationToken, Task<T>> io);

    // Sync-blocking / CPU leaf (File.ReadAllBytes, Roslyn compile, Process).
    IObservable<T> InvokeBlocking<T>(Func<CancellationToken, T> work);

    // IAsyncEnumerable leaf (partition objects), bridged to a bounded observable.
    IObservable<T> InvokeStream<T>(Func<CancellationToken, IAsyncEnumerable<T>> source);

    int CurrentInFlight { get; }   // diagnostics / tests only
}

All three return cold observables: the work runs on Subscribe, a pool slot is taken only on Subscribe, and released when the operation completes, errors, or is unsubscribed. This keeps the MeshWeaver.Mesh.RequireSubscribe semantics accurate — a never-subscribed leaf never takes a slot and never runs.

The hybrid governor

The concurrency cap is enforced two ways, chosen per leaf kind:

This design is "compatible with how Orleans wants us to pool": it reuses the ThreadPool the framework already uses and merely puts a governor in front of it — no custom OS threads that Orleans can't see or coordinate with.

Named pools and caps

Pools are keyed by resource class and resolved lazily from IoPoolRegistry (a mesh-scoped singleton, disposed with the mesh — no static state). Caps come from IoPoolOptions, with sensible defaults that a host can override via AddIoPools(o => o with { Blob = 64 }) without any call-site change.

Defaults below are the values on IoPoolOptions — read that record for the reasoning behind each one, which is often not "how much parallelism can this resource take".

Note the prefix-shadowing order in MaxConcurrencyFor: pg-read: is tested before pg: (and sf-read: before sf:), because the read prefix also starts with the write prefix.

Routing is an ISOLATION boundary, not a throttle. RoutingGrain is [StatelessWorker(1)] and non-reentrant, so a silo has exactly ONE routing turn — and Orleans' request timeout applies to callers waiting on a grain, never to the turn itself. Anything the turn does inline is therefore unbounded by construction and blocks every other message the silo needs to route. Prod (2026-08-07) had one RouteMessage turn executing for 06:00:22 with NonReentrancyQueueSize=541; Orleans' own diagnostics showed the work item still Running with Total processed frozen — i.e. RouteMessage had never returned, it was blocked in its own synchronous body. The cure is structural (you cannot time out a synchronously blocked thread): RouteMessage captures its activation-bound handles and hands the composed route to this pool via SubscribeThroughPool, so a leg that never terminates costs one slot and nothing else. See issue #1028.

Hidden inside the interfaces

A leaf resolves its pool from the mesh-scoped registry — for a hub-attached leaf via hub.ServiceProvider.GetService<IoPoolRegistry>(), and for an adapter constructed at a composition root via a required IoPoolRegistry constructor parameter (see the ledgerless-Unbounded rule below — an optional parameter with a ?? IoPool.Unbounded fallback is the shape that let a whole mesh's file I/O escape the teardown drain). Each leaf reads uniformly:

// HTTP leaf (McpRemoteMeshClient) — was Observable.FromAsync(async ct => …).
// Connect() is the promise-cached one-shot handshake (pool.Run, ReplaySubject-backed);
// each call composes off it with SelectMany, so the handshake runs once for all callers.
public IObservable<MeshNode?> Get(string path)
    => Connect().SelectMany(client =>
        _pool.Invoke(async ct => Parse(await client.CallToolAsync("get", …, ct)
            .ConfigureAwait(false))));

// CPU / process leaf — InvokeBlocking on the dedicated limited-concurrency scheduler
=> _compilePool.InvokeBlocking(ct => RunRoslynScript(…));   // KernelExecutor
=> _processPool.InvokeBlocking(ct => RunTestsCore(…));      // MeshPlugin.RunTests

IoPool.Unbounded is a stateless offload onto the ThreadPool with no cap — and, crucially, no ledger: its CurrentInFlight is unconditionally 0, so I/O on it is invisible to every teardown drain and quiescence hold (see "IoPool.Unbounded is LEDGERLESS" below). It is an immutable constant, not a cache. Pools come from the mesh-scoped IoPoolRegistry (registered by MeshBuilder.AddIoPools()), resolved from the owning hub's ServiceProvider or injected via the leaf's constructor — and for adapters constructed at composition roots the registry is a required constructor parameter with a loud failure when the provider lacks one, never an optional parameter with a silent ?? IoPool.Unbounded fallback (that shape is how the FutuRe mesh's entire file I/O escaped the teardown drain, issue #613).

Streaming an agent response into a cell — the precise process

Invariant: the thread hub must never block. A blocked thread turn stops answering GetData / GetPermission / tool-call responses for its own output cell — so the response never renders and the round wedges (GetDataRequest@{thread}/{cell} pending for tens of seconds, GetPermissionRequest timing out). That is the entire "harness doesn't work after submit" symptom. Therefore the streaming round runs in the I/O pool, never on the thread turn. The pool is not an optimisation here — it is the mechanism that keeps the actor's single turn free while the multi-second LLM enumerable drains on a bounded ThreadPool worker.

An LLM round is the archetypal InvokeStream leaf: IChatClient.GetStreamingResponseAsync(...) returns an IAsyncEnumerable<ChatResponseUpdate> — a genuine async I/O source that must run off the thread hub's scheduler and be bounded, exactly like a blob download or an HTTP call. It is never consumed with a bare Task.Run(async () => await foreach …) on (or launched from) the hub turn: that runs the enumerator's continuations under the grain scheduler, and a tool call that needs the same scheduler to answer then deadlocks against the in-flight await foreach. That is the "harness hangs after submit" failure — the thread hub stops answering GetData/GetPermission for its own output cell.

The correct path is exactly three steps, and the output cell is the rendezvous: the pool writes it, the GUI reads it, and neither blocks on the other.

1 — Resolve the output cell and mark it streaming. The round's last entry in MeshThread.Messages is the assistant output cell; its path is {threadPath}/{ActiveMessageId}. Confirm the last cell is the output (assistant) cell, take that as the streaming target, and flip its Status to Streaming so the GUI renders a live cell:

// thread.ActiveMessageId is the canonical handle; the full output path derives from it.
var output = $"{threadPath}/{thread.ActiveMessageId}";   // the last (assistant) cell in Messages
workspace.GetMeshNodeStream(output).Update(node =>
        node with { Content = ((ThreadMessage)node.Content) with { Status = ThreadMessageStatus.Streaming } })
    .Subscribe(_ => { }, ex => logger.LogWarning(ex, "mark-streaming failed for {Path}", output));

2 — Stream in the pool, writing each chunk to the cell's sync stream. Consume the LLM IAsyncEnumerable through IIoPool.InvokeStream (off the hub scheduler, bounded — never Task.Run), and fold every chunk into the output cell via GetMeshNodeStream(output).Update(...). The owning cell hub serialises the writes on its single-threaded action block (no race, no clobber), and the grain scheduler stays free to answer the round's tool-call responses:

var acc = new StringBuilder();
ioPool.InvokeStream(ct => chatClient.GetStreamingResponseAsync(messages, options: null, ct))
    .Sample(StreamingSampleInterval)        // one cell write per sampled tick, NOT per token
    .Subscribe(
        update =>
        {
            acc.Append(update.Text);
            workspace.GetMeshNodeStream(output).Update(node =>
                    node with { Content = ((ThreadMessage)node.Content) with { Text = acc.ToString() } })
                .Subscribe(_ => { }, ex => logger.LogWarning(ex, "stream write failed for {Path}", output));
        },
        ex => SetCellStatus(output, ThreadMessageStatus.Error),
        () => SetCellStatus(output, ThreadMessageStatus.Completed));   // terminal: flip Status once

3 — The GUI subscribes to the same cell stream. The Blazor view databinds the output cell with GetMeshNodeStream(output) (or GetRemoteStream<MeshNode>), rendering Content.Text as it grows and reacting to the terminal Status. It reads the exact node the pool is writing — the cell is the single source of truth, so there is no second channel to reconcile.

Why this is deadlock-free, point by point: the enumerator runs on a pool ThreadPool worker (step 2), never the grain turn, so an in-flight tool call still gets the scheduler. The cell writes go through the owning hub's serialised action block via the stream handle — a non-blocking cross-hub patch, not a synchronous wait. The GUI only reads (step 3). Three actors, one cell, no one blocks another.

🚫 The anti-pattern this replaces. Task.Run(async () => { await foreach (var u in client.GetStreamingResponseAsync(…)) cell.Update(…); }) looks offloaded, but it (a) is unbounded — N concurrent rounds spawn N enumerators with no governor — and (b) bypasses IIoPool, so it is invisible to the pool's diagnostics and cancellation, and any synchronous wait on the output cell from the hub turn (e.g. a sync-handshake read of a cell that isn't reachable yet) still wedges the hub. Route the enumerable through InvokeStream; the offload, the bound, and the cancellation come for free.

🚨 A tool call runs INSIDE the leaf — so its Task must observe the token

The round in the previous section holds one gate permit for its whole duration, and a tool call happens inside that await foreach. So the tool's Task<string> is not a detail of the agent loop — it is the thing standing between Drain() and a real join.

Agent tools are Task-returning by contract (AIFunctionFactory needs a Task-returning delegate), and the usual shape bridges an observable to it with a TaskCompletionSource. That bridge is the one sanctioned Task boundary here — but the token that is bound into the tool's CancellationToken parameter must be able to settle it. It is the round's token: linked to the user's Stop (executionCts) and to the pool token that IoPool.Drain() cancels.

var tcs = new TaskCompletionSource<string>(TaskCreationOptions.RunContinuationsAsynchronously);
// Everything the wait holds — subscriptions AND the cancellation registration — in one bag,
// released by whichever terminal fires first.
var pending = new CompositeDisposable();
void Settle(Func<bool> set) { if (set()) pending.Dispose(); }

pending.Add(cancellationToken.Register(() => Settle(() => tcs.TrySetCanceled(cancellationToken))));
pending.Add(source.Subscribe(r => Settle(() => tcs.TrySetResult(r)), …));
return tcs.Task;

Why a timeout is not a substitute. delegate_to_agent had a 10-minute backstop on one of its two completion paths and none on the other. Ten minutes is 20× the 30 s DrainTimeout, so from teardown's point of view a backstop that generous is indistinguishable from no exit at all: the parked continuation keeps its permit, Drain() sits out its budget, reports a leaked leaf, and the scope is disposed (and collectible node ALCs unloaded) over live code. The user-visible half of the same defect is that Stop does nothing — the round is parked in a Task the Stop cannot reach.

Cancel rather than resolve an error string: ThreadExecution's catch (OperationCanceledException) when (executionCts.IsCancellationRequested || poolCt.IsCancellationRequested) classifies that as the graceful shutdown/stop it is, so the round settles Cancelled instead of writing a false "#147 streaming exceeded the maximum round duration" into the user's response cell.

Pinned by DelegationCancellationTest (unit) and DelegationDrainJoinsParkedToolCallTest (integration — a delegation that never resolves, asserting DrainAll() == 0), the sibling of AiPoolDrainJoinsRoundTest for a round parked on the model call.

Scope — storage and Postgres are pooled too

Earlier guidance carved storage and Postgres out of the pool and left them on plain Observable.FromAsync. That carve-out is rescinded — there is no exemption. FromAsync is never tolerated (see the absolute rule above), so storage / file-system / Postgres leaves go through IIoPool like everything else.

Per-PROVIDER pools: a WRITE gate of 1 and a READ gate of 16, and the split is not optional. The Postgres backend's writes run on pg:{provider}, capped at 1; its reads run on pg-read:{provider}, capped at IoPoolOptions.PostgresRead (16). Both names are the PROVIDER's (pg:Postgres, pg-read:Postgres) — PostgreSqlPartitionStorageProvider resolves each once from Name, and PostgreSqlPathRoutingAdapter hands the same pair to every per-schema adapter it materialises. So these are two process-wide gates, not one pair per schema, and the 16 + 1 below is the aggregate. The naming + caps live in IoPoolNames.PostgresAdapterPrefix / IoPoolOptions.MaxConcurrencyFor.

🚨 Read the pairing correctly. Only a dedicated single-connection data source makes "the gate and the driver pool are the same size" literally true — PostgreSqlChunkedContentVectorStore is the one place that holds (MaxPoolSize=1 alongside its cap-1 pg:vector pool). The partitioned provider deliberately does the opposite: every per-schema adapter shares ONE NpgsqlDataSource (MaxPoolSize=50 in the portal), because minting a data source per (schema, table) leaked a pool per hub and exhausted the server. There the two caps are a budget, not an identity — 16 reads + 1 write = 17 concurrent connections, comfortably under 50. That budget only holds if both pools are actually wired and each operation is filed on the right one.

What the caps actually cost, measured

The queue-wait distribution was read for the first time on 2026-09-16, on memex.systemorph.com, after 828 minutes of uptime:

🚨 The cap-1 write gate is not the constraint; the cap-16 read gate is the one that queues. The write pool served 3.4 admissions a minute and never made anything wait a full second. The read pool served 38,500 a minute and puts 77% of them in the [100 ms, 1 s) bucket — it queues routinely. That is the reverse of where a year of reasoning about these caps was aimed (the recursive-delete drain), and it is why the reading had to be taken rather than argued.

Neither number moves on this. The write pool has no queueing to relieve. The read pool's queueing has no diagnosis: InvokeStream holds one slot for an entire enumeration, so a long-held slot and a too-small cap produce the same mean and are different problems — and the caps are a connection budget, so spending the headroom needs a cause, not a symptom. Denominator: ONE portal, ONE pod, ONE process lifetime, and not the portal where the delete timeouts were logged.

How to take the reading. IoPoolRegistry.Snapshot() enumerates the pools that EXIST — name, cap, in-flight, queue depth, distribution — and mints none. 🚨 Never read a pool by resolving it: Get is a resolver and answers an unknown name by CREATING that pool, which then reports itself, brand new, as idle. IoPoolQueueReport.Describe is the formatter that keeps "not measured", "measured, nothing queued" and "these pools had work queued" as three different sentences.

Both halves have to be wired, and reads must not be filed on the write pool. This is not a style point — it was issues #1310/#1312/#1313/#1316. The Postgres backend resolved its cap-1 pg:Postgres pool, used it for provisioning, and then never passed ioPool: to the adapters that perform every actual write, so each per-schema adapter fell back to IoPool.Unbounded. Compounding it, eight read-shaped operations (Read, ReadMany, Exists, FindBestPrefixMatch, ResolvePath, GetPartitionObjects, GetPartitionMaxTimestamp, ListPartitionSubPaths) were filed on that write pool rather than the read pool. Net effect: the hottest read path in the portal — per-node-hub activation seeds, URL resolution, write-guard probes, the per-path read fan-out inside StorageAdapterMeshQueryProvider — ran with no bound at all against a 50-connection data source, and memex-cloud duly reported "the connection pool has been exhausted (currently 50)". Keeping reads off the cap-1 pool is also what makes that pool safe: a read issued from inside a write would otherwise be a same-pool re-entry on a cap-1 gate, the one documented way to deadlock an IIoPool. PartitionAdapterIoPoolWiringTests pins both the wiring and the read/write filing.

The cost concern that originally justified the carve-out (a SubscribeOn hop on every hot read under a constrained CI ThreadPool) is real — the answer is to size the per-adapter pools correctly, not to fall back to bare FromAsync. The migration is finished: every query/storage leaf is pooled (see "The sweep is complete" below), and new code (e.g. PostgreSqlPartitionStorageProvider.EnsurePartitionProvisioned) is pooled from day one.

Edge cases — the review checklist

These four properties must hold for every leaf that uses IIoPool:

• Pool the leaf, never the orchestration. IIoPool is injected into leaf adapters only. Orchestration layers (PersistenceService fan-out, version-writing decorator, query providers, MeshQuery) compose adapter observables and hold no slot — a fan-out of N reads acquires N leaf slots and never nests behind a held one. Per-resource pools mean a FileSystem leaf never blocks on a Blob leaf while holding a FileSystem slot. Never let an IIoPool call resolve an observable that itself acquires the same pool — that is the one way to deadlock it.

• Cancellation and release. Every shape releases its slot in a finally. WaitAsync(ct) makes acquisition itself cancellable — a dispose before the slot is granted throws before the in-flight increment, so no slot leaks. The subscription's ct flows into the leaf, so file/blob calls cancel on unsubscribe.

• Disposal. IoPoolRegistry disposes each IoPool, which disposes its SemaphoreSlim. The limited-concurrency scheduler borrows ThreadPool threads — nothing to dispose.

• No sync-context capture. .SubscribeOn(TaskPoolScheduler.Default) + .ConfigureAwait(false) everywhere; InvokeBlocking dispatches via the scheduler-bound TaskFactory, never TaskScheduler.Current. Safe even when Subscribe is called inside a grain handler.

• 🚨 The pool carries NO AccessContext — it is an identity hole, by design. .SubscribeOn(TaskPoolScheduler.Default) re-runs the whole subscribe (gate wait + the leaf's synchronous prologue + every ConfigureAwait(false) continuation) on a bare ThreadPool worker. That worker has no AsyncLocal baton — AccessService.Context reads as null/whatever the pool thread last held, NOT the originating user. The pool is the threading primitive (get IO off the action-block thread, bounded); it is deliberately not an identity primitive. Contrast the mesh write/read primitives (MeshNodeStreamCache.Update / GetMeshNodeStream / GetQuery), which wrap the cold pipeline in CarryAccessContext(accessService) — capturing the caller's identity synchronously on the caller's thread and re-stamping AsyncLocal on each emission. So: never read or write a mesh node from inside an IIoPool leaf and expect RLS to see the user — the leaf runs identity-less. Genuine external IO (subprocess, HTTP, blob) doesn't touch AccessService.Context and is fine. If a pooled leaf genuinely must do an authorized mesh operation, capture the identity before the pool boundary and pass it explicitly, or — far better — do the mesh read/write through GetQuery / GetMeshNodeStream (which carry the baton) and keep only the non-mesh IO in the pool. This is why the Copilot model catalog moved off ioPool.Run(... ListModelsAsync ...) and onto workspace.GetQuery(...): the picker is per-user, and only the GetQuery path stamps the subscriber's identity.

Disposal is reactive — Dispose() fires, the mesh drains

There is no DisposeAsync() and no IAsyncDisposable anywhere — not in the public API, not on any hub or resource. The whole shape is deleted. An await DisposeAsync() inside hub-reachable code (or a shared-scheduler fixture) captures TaskScheduler.Current and queues its continuation back onto the very turn that is tearing down → the turn never drains → deadlock. Disposal is synchronous + reactive instead:

• Dispose() (IDisposable) fires and returns. It does the synchronous unregister (e.g. IRoutingService.RegisterStream returns a synchronous unregister handle) or kicks off reactive teardown — it never blocks on it. IDisposable is the only disposal interface a hub or resource implements.
• Asynchronous teardown is observed through IObservable<Unit> DisposalCompleted (the shape IMessageHub uses), not awaited. Dispose() triggers it; completion is a reactive emission.
• Someone has to subscribe. DisposalCompleted is cold — with no subscriber the completion is never observed. That subscriber is the mesh teardown, never the individual hub or call site: the mesh's disposal merges its hubs'/resources' DisposalCompleted and waits for them all to emit before it completes. Hub-reachable code only exposes DisposalCompleted; it never subscribes to or drains its own.
• Let them all finish. When the teardown owns several reactive completions, drain them together so they all emit before it returns — never an awaited loop that serialises (and can deadlock on) each one.

🚨 The mesh teardown drains THREE things, not one

DisposalCompleted is necessary but NOT sufficient. It drains the hub's action blocks and in-flight message round-trips — but I/O offloaded through IIoPool runs on the ThreadPool, independent of the action block, and DisposalCompleted knows nothing about it. If the teardown disposes the service scope after DisposalCompleted but while an IIoPool operation (or any other async cleanup) is still in flight, that continuation resolves a service from the dead Autofac scope and throws ObjectDisposedException: …LifetimeScope… has already been disposed — unobserved, it surfaces as an xUnit "catastrophic failure" that aborts the whole run.

So the mesh teardown awaits all three, in order, before the scope is disposed:

1. IMessageHub.DisposalCompleted — action blocks + message round-trips. (Resources enqueue their async cleanup onto the AsyncDisposeQueue during this synchronous-Dispose() phase — Dispose() must never block, so async cleanup is queued, not run inline.)
2. IoPoolRegistry.DrainAll() — offloaded ThreadPool I/O. GRACE, then CANCEL + JOIN — never wait alone, never cancel first. The drain first lets every in-flight leaf finish on its own: it waits while anything is still outstanding, under IoPoolOptions.DrainGrace (8 s) per step, and every completion restarts the clock — so a leaf that is going to finish is never cancelled (a write that would have landed in 50 ms lands), and a leaf that reaches the gate while the drain is running extends it rather than consuming someone else's grace (see the admission count says OUTSTANDING, never PROGRESS). 🚨 The grace is deliberately not the gate: re-acquiring permits here would compete with the queued leaves for them, steal their turn and then cancel them at the gate — the very outcome it exists to prevent. Only a leaf that outlives a whole grace with the pool making no further progress is wedged: its call site is captured, and then the pool cancels. 🚨 Do not use the wait-only WhenDrained(timeout) in place of that cancel: a live change-feed leaf never completes on its own, so a polled wait times out and lets the scope dispose while the leaf is still running — its ThreadPool thread then dereferences a collectible node ALC's freed metadata after unload, a native use-after-unload SIGSEGV. DrainAll() cancels so the stream leaves and the wedged leaves stop, then joins, and returns the count it had to leak — and, separately, the leaves it had to cancel (IoPool.LeavesCancelledAfterGrace / CancelledLeafSites), which the teardown logs at Error because that work did not finish (Teardown Layers). 🚨 The cancel runs on its own thread, never on the joining one. CancellationTokenSource.Cancel() executes every registered callback synchronously on the caller, and this token's callbacks are not bookkeeping: SubscribeThroughPool registers one per live pooled subscription that runs that subscription's whole downstream teardown, and every gated leaf links its subscriber token to the pool's, so cancelling also resumes each leaf's gate wait into its observer. Cancelling inline therefore ran arbitrary application teardown on the MESH TEARDOWN thread with no budget over it — the drain timeout covers only the gate join that comes after — so one clean-up leg that would not finish parked teardown silently and forever (#2394: a whole test assembly killed at its 8 min cap with no test named). IoPool now issues the cancel on a dedicated thread and joins it first, ahead of the gate join and under the same budget, because the gate join's meaning depends on the cancel having landed ("once the pool token is cancelled, no NEW leaf can take a permit"); a cancel that does not finish inside the budget is added to the residual. Dispose() issues it the same way and never joins — its wait lives on Disposed.
3. AsyncDisposeQueue.DrainAsync(timeout) — the queued async cleanup. A TPL ActionBlock drains it; DrainAsync Complete()s the block and awaits the remainder (bounded), so it converges even under continuous influx — a version-target wait would not (the queue is a message stream / endless messages). DrainedVersion advances once per item, the test hook.

🚨 A residual with NO site is the CANCEL join — and a subscriber can park it

IoPool.Drain() reports three things under one number: gate permits it could not re-acquire, blocking leaves still running, and a cancel that did not return. The first two always name a site (every leaf registers one on entry); the third is not a leaf and had none. So a trace that read

DISPOSE_IOPOOL_DRAIN_DONE elapsed=30016ms leakedIoLeaves=1 pools=[Query=1]

— Query=1 with nothing in brackets — meant exactly one thing, _poolCts.Cancel() on the Query pool never returned, and nothing on the page said so. It now reads Query=1 [IoPool.Drain: the pool token's cancel did not return within the budget — …].

What parks a cancel. Cancelling the pool token runs, inline on the IoPool-cancel thread, one callback per live SubscribeThroughPool subscription: inner.Dispose(); observer.OnCompleted(); — that subscription's whole downstream teardown. Two facts about the libraries underneath turn a race into a deadlock:

1. CancellationTokenRegistration.Dispose() blocks until a callback executing on another thread has finished (WaitForCallbackIfNecessary; only the callback's own thread is exempt). Unregister() never waits.
2. Rx operators forward from their timers under their gate. Throttle.Propagate runs ForwardOnNext inside lock (_gate), and Throttle.OnCompleted takes the same gate; Take(1) completes and disposes upstream synchronously — still inside that gate.

So a consumer shaped Query(...).Throttle(1 s).Take(1) whose timer fires as the drain cancels holds the operator gate while its upstream disposal waits in Dispose() for the drain callback, and the drain callback waits in Throttle.OnCompleted for the operator gate. Two locks, two threads, no exit. The drain reports the cancel residual after its budget, RSS is flat the whole time (parked, not computing), and the two threads stay deadlocked for the life of the process.

The occurrence. MeshNodeLanguageServiceTest went DIRTY at teardown on 2026-08-28 (#2578, #2616), 08-30 (twice), and 09-03 (Plugins #1260 attempt 1, shard 3) — always the same shape: the test body PASSES in ~1 s (1018 ms / 1044 ms in the two traces that survived), the drain then spends its whole 30 s, and the residual is Query=1 with no site. The consumer is CompletionUsageIndex.EnsureFresh(), whose Throttle(1 s) lands on exactly those ~1 s bodies; a faster machine ends the test before the timer fires, which is why twenty local runs never reproduced it. #2598 fixed a different leaf (the first-in-process script-reference build, on the Compile pool) on the strength of the same anonymous 1; the failure recurred unchanged four hours later.

The rules this leaves behind:

• Never put a raw CancellationTokenRegistration where a subscriber disposes it. The subscription's disposable unregisters (drainReg.Unregister()); the drain callback stays exactly-once through its latch and completes on its own. More generally: never dispose a registration from a thread that may hold a lock the callback needs — in Rx that is every operator gate.
• A request-scoped subscription needs an OWNER. EnsureFresh()'s query used to be fire-and-forget with a 30 s Timeout — a second, unowned 30 s budget coinciding with the drain's. It is held in a SerialDisposable and disposed with the hub (hub.RegisterForDisposal(...), which runs in DisposeImpl — strictly before DisposalCompleted, hence before DrainAll()); the same for the completion memory store's debounced save. Three lifetimes exist for any deferred work — the caller's (WaitAsync(ct)), the owner's (the hub or service that started it), and never (a process-wide memo) — and "no caller cancels this" must not silently become "nothing ever does".
• Read a residual by its site. A bracketed site is a leaf; no site is the cancel join, now labelled. Query=1 and Compile=1 are different bugs, and so are Query=1 [MeshQuery…] and Query=1 [IoPool.Drain: the pool token's cancel did not return…].

Pinned by IoPoolDrainCancelJoinTest (deterministic: a TestScheduler-driven Throttle fired from a thread the test owns, with the two interleavings the deadlock needs made explicit).

🚨 IoPool.Unbounded is LEDGERLESS — I/O on it does not exist to this drain

The three-phase drain only covers what it can see, and IoPool.Unbounded is invisible to it by construction: its CurrentInFlight is unconditionally 0, its Invoke is a bare Observable.FromAsync(io).SubscribeOn(TaskPoolScheduler.Default), and it lives outside the registry — so IoPoolRegistry.DrainAll() never enumerates it, TotalInFlight/WhenDrained never count it, and there is nothing to cancel, dispose, or join. The same blindness applies to every other quiescence hold built on the ledgers (e.g. the silo's routing-quiescence hold). Phase 2 therefore reports a clean drain while the unbounded I/O is still running — and that straggler is exactly the teardown SIGSEGV shape: a leftover Rx emission enters full hub construction after the scope is disposed and faults on a span into the unloaded collectible ALC.

This is not hypothetical. FileSystemStorageAdapter took its registry as an optional parameter with a ?? IoPool.Unbounded fallback, and every production construction site silently dropped it — the storage-adapter factory (the path every config-declared FileSystem data source takes) and both PersistenceExtensions registration sites. Result: every file-system-backed mesh (the FutuRe sample, the one file-system-data-source-backed space) ran all of its file I/O on the unbounded pool, and issue #613's exit=139 teardown crash recurred with zero failing tests in the trx — the drain had nothing to join.

The rule. An adapter or service whose I/O must be drainable takes IoPoolRegistry as a REQUIRED constructor parameter — the compiler then names every dropping site, which an optional parameter never does ("check the call sites, not the declaration"). A DI factory or registration lambda resolves the registry from the provider and fails LOUDLY — a thrown error naming the missing registration and how to add it (AddIoPools(), which MeshBuilder calls by default) — never a silent ?? IoPool.Unbounded. A deliberate IoPool.Unbounded use must be written out explicitly at the call site with a comment saying why bare-ThreadPool is correct there; a pure test convenience is the only acceptable answer, and fixing the test to use a real registry is preferred.

The only sanctioned await is that single three-phase drain at the boundary — the mesh teardown, the same in tests and in prod (the silo's mesh disposal at shutdown). Capture the mesh-scoped teardown services before Dispose() (never resolve DI once disposal has begun), then drain all three, bounded:

// ✅ The one drain, at the mesh-teardown boundary (test mesh OR prod silo shutdown).
//    Either call the canonical helper:
await mesh.TeardownAsync(TimeSpan.FromSeconds(15));   // MeshWeaver.Mesh.MeshTeardownExtensions

//    …or, if you drive Dispose() yourself, do the phases by hand:
var ioPools = mesh.ServiceProvider.GetService<IoPoolRegistry>();        // capture BEFORE Dispose()
var disposeQueue = mesh.ServiceProvider.GetService<AsyncDisposeQueue>();
mesh.Dispose();
// 🚨 ObserveCompletion, never Rx's ToTask bridge (forbidden repo-wide, 2026-08-30) and
//    never a bare `await disposalCompleted` either: both resume this method INLINE on the
//    hub's own disposal thread, which then has to run phases 2 and 3 while the mesh is
//    trying to finish tearing itself down. ObserveCompletion completes with
//    RunContinuationsAsynchronously, so the disposing thread is released immediately.
using var phase1Deadline = new CancellationTokenSource(TimeSpan.FromSeconds(15));
await mesh.DisposalCompleted
    .Catch<Unit, Exception>(_ => Observable.Return(Unit.Default))
    .FirstOrDefaultAsync()
    .ObserveCompletion(
        ex => logger.LogWarning(ex, "disposal faulted AFTER the wait settled"),
        phase1Deadline.Token);                                             // phase 1
var leakedIoLeaves = ioPools?.DrainAll() ?? 0;   // phase 2 — cancel + join, NOT a polled wait
if (disposeQueue is not null)
    await disposeQueue.DrainAsync(TimeSpan.FromSeconds(15));               // phase 3
// ONLY NOW dispose the service scope.

"Only drainage of async pipelines is allowed": the await lives at that one three-phase drain, the work stays reactive. Same principle as IIoPool — the async boundary is pushed to the edge and bounded; it is never an ambient await mid-flow. Full order + failure mode: Mesh Lifecycle. See also Asynchronous Calls.

🚨 Ambient context does not cross the pool. Work handed to IIoPool runs on a pooled thread whose ExecutionContext is not yours: an AsyncLocal you set upstream is not readable inside the leaf, and a value written inside the leaf is not visible to the caller. Capture what the leaf needs into the closure before handing it over — see AsyncLocal Across Scheduler Hops and, for identity specifically, AccessContext Propagation.

🚨 The pool releases its gate on an ADMISSION COUNT, never on "is anyone running?"

Dispose() must not block (a synchronous 30 s join parks a pool thread while the leaves it waits for need pool threads to observe cancellation — a starvation deadlock on a 4-vCPU runner). So it cancels, returns, and lets the last caller out release _gate / _poolCts / the blocking-idle signal. That makes "who is still using them?" the load-bearing question, and the obvious answers are all wrong:

• CurrentInFlight is raised only once WaitAsync has granted a permit. A leaf between the grant and the increment holds a permit and counts as zero, so disposal releases the gate and the resumed leaf calls Release() on it (issue #2146).
• A _disposed flag read at the top of the entry point is read when the cold observable is built, while _poolCts.Token and _gate.WaitAsync are touched on subscribe. Everything the pool returns is cold, so that gap is arbitrarily wide — and a chain built during a render and subscribed as teardown lands threw ObjectDisposedException out of the reactive chain (#2134 / #2135).

IoPool therefore counts admissions, not executions: every path that may touch those primitives — each of the four entry points, Drain(), and Dispose() itself — brackets its whole reach in TryEnterGateRegion() / LeaveGateRegion(), and disposal completes only at zero. Entry is publish-then-recheck (increment, then re-read the disposal flag; Dispose publishes the flag before it reads the count), so of the two check-then-act orders at least one side always observes the other: either disposal defers, or the caller is refused and answers OperationCanceledException. A leaf the pool will not run is a CANCELLATION, never an ObjectDisposedException — that is the contract the region exists to keep, and it is why there is no catch (ObjectDisposedException) anywhere in the file. Adding one would hide a region that was never entered.

🚨 The admission count says OUTSTANDING, never PROGRESS — the grace needs a second counter

The drain's grace is a stall bound, not a budget: every completion restarts it, so a burst of ten short writes drains in ten completions rather than one budget, and work that keeps finishing is never cancelled. The obvious way to implement that — and the way it was implemented — is to take a baseline of the admission count and wait for it to fall below it.

That reads progress off a counter that moves in both directions. _gateUsers is a live census: it rises on an arrival exactly as far as it falls on a completion. And arrivals during a drain are routine rather than exotic, because the pool creates them itself — Invoke and InvokeStream defer their prologue to the ThreadPool (SubscribeOn), as does SubscribeThroughPool's setup leaf, so a leaf whose Subscribe() returned before the drain enters its gate region after the drain has taken its baseline. (InvokeBlocking is the exception: its region is taken on the subscriber's thread and spans the whole leaf.) Each such arrival then cancels out a completion one for one, and the predicate cannot fire until every arrival has also finished:

The per-completion grace had silently become one total budget for the whole queue. The pool made progress four times in that window and the drain called it a stall, then cancelled a leaf 400 ms from the end of work it was going to finish — accepted work discarded, which is precisely what the grace exists to prevent and a contradiction of what the teardown contract promises: teardown lets accepted work finish and NAMES what it had to stop. This did neither; it discarded the work and reported a stall.

So progress is now counted, not inferred. LeaveGateRegion increments a monotone _admissionsCompleted (published after the census decrement, so a drain that observes the completion also observes the settled count behind it), and the two counters answer the two different questions the loop actually asks:

var seen = Volatile.Read(ref _admissionsCompleted);
while (Volatile.Read(ref _gateUsers) - 1 > 0)          // is anything still OUTSTANDING?
{
    var progressed = SpinWait.SpinUntil(
        () => Volatile.Read(ref _admissionsCompleted) != seen,   // has anything FINISHED?
        _drainGrace);
    if (!progressed)
        break;                                          // a whole grace, nothing finished ⇒ wedged
    seen = Volatile.Read(ref _admissionsCompleted);
}

A leaf that reaches the gate mid-drain now extends the drain — it is outstanding, so the loop keeps going — and never consumes the grace of the leaf that finished before it. Nothing about the cancel changed: a leaf that outlives a whole grace with nothing finishing is still wedged, still cancelled, and still named in CancelledLeafSites.

How it was found, and why load was not the explanation

It surfaced as a 1-in-5 failure of IoPoolTest.Drain_restartsTheGraceOnEveryCompletion_… — Expected 3 … but found 1 — on a machine at load ~29 across 18 cores. Saturation is the condition, never the cause. It changes nothing about the predicate; it only widens the window in which a prologue lands on the far side of the baseline, which is what makes an arrival available to mask a completion. Raising the grace, lengthening the leaves or widening the test's margin would have bought slack against the load and left the defect exactly where it was.

IoPoolDrainGraceTest reproduces it on an idle machine instead, by arranging that ordering structurally rather than waiting for load to arrange it: a test seam (IoPool.OnDrainGraceBaselineTaken) runs on the drain's own thread the instant the baseline is taken, subscribes the queued leaves there, and waits until each is provably at the gate before the grace clock starts. The queue then outlasts one grace by construction — three 800 ms leaves against a 2000 ms grace, and Task.Delay never fires early — so the red is structural and load can only make it redder. Its sibling test is the control that keeps the fix honest: a leaf that reaches the gate after the baseline and then wedges is still cancelled and still named, so the fix cannot have degenerated into unconditional patience.

🚨 The residue: a leaf between Subscribe() and its prologue is invisible, and that is separate

Closing this closes the masking, not the whole window. Between Subscribe() returning and the ThreadPool running the prologue, a leaf is counted by nothing at all — the drain cannot extend a grace for work it cannot see, and if the outstanding count reaches zero in that gap the grace simply ends. Three of the four entry points have such a window, and the consequence is not the same in each:

That window is the pool's own making and is structurally evident in the code, but it was not the mechanism measured here and is not closed by this change. It is tracked as #4555, separately and deliberately: closing it means moving the admission onto the subscriber's thread, which is a change to the subscribe path — the same path #4530 / #4545 are editing — rather than to the drain. Both defects can produce the same observable symptom, a queued leaf cancelled during a drain, which is exactly why they are worth keeping apart.

🚨 The gate permit is the DRAIN'S SIGNAL — so a leaf publishes it LAST

Drain() joins by re-acquiring every permit and then reports 0, which the contract spells as "no pool thread is still running". CurrentInFlight is the pool's own statement of the same fact. Those two must never be able to contradict each other, and with the old ordering they could: the shared exit path released the permit before it decremented _inFlight.

// before — the permit is visible one Interlocked op too early
_gate.Release();
Interlocked.Decrement(ref _inFlight);

The defence written beside it was that the leaf is "two interlocked ops from done, no user code", so nothing dangerous can still be running at the moment Drain observes the permit. That is true about ALC safety and beside the point about accounting — and it quietly assumed the releasing leaf owns the instant after Release(). It does not. Measured on #4448, the unwind of a cancelled leaf runs on a .NET TP Worker while Drain() sits in _gate.Wait() on the mesh-teardown thread: the cancel resumes the leaf's continuation on the ThreadPool, not inline on the IoPool-cancel thread, so the two race on every drain that cancels a leaf. The leaf normally wins by a mile — one lock xadd against Drain's remaining loop — which is why it surfaces about once per fleet-week, and why IoPoolTest.Drain_cancels_in_flight_leaves_and_joins_synchronously failed on core #4417 (run 34970182190, shard 3, 2026-09-15) with Expected value to be 0 … but found 1, on a diff that cannot reach IoPool. That is the ONE failure this explains: the sibling Dispose_doesNotBlockOnASlowPooledSubscriptionTeardown in #4448 is a terminal that did not arrive, takes a path that never calls Drain(), and stays open.

Decrementing first turns the permit into a real happens-before edge: the decrement is a full fence, Release() publishes under the semaphore's lock, and Drain's Wait() acquires that same lock — so a permit Drain holds proves the accounting behind it is already settled. No spin, no re-read, no second signal.

// after — the accounting is settled before the signal is published
Interlocked.Decrement(ref _inFlight);
_gate.Release();

The ordering inverted because its original reason was removed elsewhere. #2135 released first because the exit path itself called TryFinishDisposal(), which disposed _gate the instant that decrement took _inFlight to zero — the last leaf out disposed the semaphore and then released it (seen in prod as a failed Comments render on memex-cloud). #2146 then moved that decision onto the admission counter, and TryFinishDisposal now runs only from LeaveGateRegion() and returns immediately unless _gateUsers is zero. All three callers of the exit path sit inside their own region whose finally runs strictly after it, so _gateUsers ≥ 1 throughout and the gate cannot be disposed there in either order. The hazard was gone; the ordering it forced was not. This is the general shape worth remembering: an ordering justified by an invariant somewhere else becomes a defect the moment that invariant is enforced by something better, and nothing points at it.

Reproducing it

The window is two instructions wide and cannot be forced from outside the class — a sweep of 3,840 cancelled-leaf unwinds on an idle 18-core box reproduced it zero times, as did 26 class runs under DOTNET_PROCESSOR_COUNT=2 and full CPU saturation (#4448). The mechanism is instead proven by widening the window in place: insert Thread.Sleep(50) between the two statements and

• with Release() first, Drain_cancels_in_flight_leaves_and_joins_synchronously fails every run with the exact merge-queue message, but found 1;
• with the decrement first, the same test passes with the same 50 ms delay still in place.

That is also why this page carries the analysis rather than a sweep-style guard test: a probabilistic test that reproduces nothing on the hardware it runs on is a verification step that cannot fail.

The sibling failure is NOT this, and the difference is the thread

IoPoolTest.Dispose_doesNotBlockOnASlowPooledSubscriptionTeardown — a terminal that did not arrive within 5 s, #4448's original subject — takes a path that never calls Drain() and is not explained by the ordering above. Probing which thread delivers that terminal settles what it cannot be:

[run 0: terminal on 'IoPool-cancel', Dispose took 0 ms]   … 5 of 5 runs identical

Dispose() → Thread.Start() → _poolCts.Cancel() → the SubscribeThroughPool drain registration → inner.Dispose(); observer.OnCompleted() → the subscriber's handler is entirely on the dedicated IoPool-cancel OS thread. No ThreadPool work item appears anywhere in it, and the subject the test awaits is an AsyncSubject that has already latched, so the await replays without needing a scheduled continuation either.

So .NET ThreadPool starvation is ruled out for that assertion by construction, not merely by experiment — which also explains why a DOTNET_PROCESSOR_COUNT=2 run could never have reproduced it, while the very same regime is the right instrument for the Drain failure above, whose leaf demonstrably unwinds on a .NET TP Worker. Two sibling tests in one class, and the thread is what tells them apart. What is left standing for the open one is OS-level — and the section below settles which half of it is ours to fix.

🚨 Teardown must not MINT the thread that delivers the terminal

Drain() and Dispose() used to call new Thread(...).Start() per call. That put OS thread creation on the critical path of every pooled subscription's terminal: nothing downstream is delivered until that thread exists and is first scheduled. Two independent reasons that is a defect and not merely a cost:

• It is a blocking call inside the method whose contract forbids blocking. Thread.Start() takes the runtime's thread store lock and can queue behind a GC suspension. Dispose()'s own comment, three lines above the call, is "DISPOSE MUST NOT BLOCK" — the same shape as the inline Cancel() that #2394 removed from it, just one layer down.
• The latency it adds is unbounded and invisible. Every in-process instrument reads healthy while it elapses: Dispose returns in 0 ms, the awaited subject is a latched AsyncSubject, no gate permit moves, CurrentInFlight/CurrentlyWaiting are both 0.

So the canceller is now started with the pool and parked on a one-shot latch — a CancellationTokenSource's wait handle, because the wait is raised once, never resets, and lasts the pool's whole lifetime, which is the one wait a slim spin-then-block primitive is documented not to be for. Drain()/Dispose() raise that latch instead of minting a thread; Drain() joins on a ManualResetEventSlim the canceller sets once Cancel() has returned, waited on with the drain budget exactly like _blockingIdle, in place of Thread.Join. The requester takes the gate region before raising the signal, exactly as it did before Start(), so _poolCts is still provably alive when the canceller wakes; the canceller publishes the completion event before handing that region back, because handing it back can complete disposal and dispose the very event a Drain() would be waiting on. The thread exits as soon as the one cancel it exists for has run, and IoPoolRegistry creates pools lazily by name, so the cost is one parked stack per resource class actually in use, for as long as it is in use.

Two consequences of owning a thread from the constructor, both of which had to be paid. A thread is a GC root, so a pool that nobody disposes no longer merely leaks a semaphore that the collector reclaims — it parks a thread for the process's life. ConcurrentDictionary.GetOrAdd does not promise its value factory runs once, so IoPoolRegistry.Get now records what it built and disposes any candidate that lost the race, which wakes that canceller and lets it exit. And the thread is started with Thread.UnsafeStart(), never Start(): Start captures the starting thread's ExecutionContext and flows it for the thread's whole life, and a lazily-resolved pool is started from whatever caller first touched that resource class — a hub turn serving a viewer, say. A captured context would run every pooled subscription's downstream teardown under that user's AsyncLocal identity, AccessService.Context included, and pin it until the pool died. Identity- neutral is both correct and what the old shape gave for free, since it created the thread on the mesh-teardown thread, which carries none.

What the widened window proves — and what it does not

The race is two thread-state transitions wide and 26 class runs across three load regimes reproduced nothing, so the mechanism is established the same way #4466's was: by widening the window in place and watching the verdict flip. Two wideners, because there are two distinct latencies and only one of them is the pool's to remove:

A is the discriminator: creation cost moved off the teardown path entirely. B is the honest half: it reproduces the merge-queue failure verbatim, including the message, and the fix does not change it — because "the OS did not run the thread" cannot be designed away while the cancel is forbidden to run on the caller. What the change does is reduce that residue from create a thread, register it with the runtime, and have it first-scheduled to wake a thread that already exists, which needs no thread store lock, no stack allocation and no GC-safe-point transition.

So if this assertion ever fails again, the reading has already narrowed: IoPoolTest.Dispose_doesNotBlockOnASlowPooledSubscriptionTeardown now asserts CancellerIsAlive before the disposal it measures, and TheCancellerThreadIsStartedWithThePool_NotAtTeardown pins the property on its own. A green line there with a red terminal below it means the thread existed and the OS did not wake it — a host-level verdict, not a pool defect, and still never a reason to widen the 5 s bound.

One forensic note on the occurrence, because it was read the other way round

The original triage read the failure as "the pooled subscribe landing ~3.4 s late on a busy ThreadPool, followed by Within(5 s) expiring", on the strength of 718 tests in 333 s on that shard. test/xunit.runner.json sets maxParallelThreads: 1 and parallelizeTestCollections: false, so that assembly runs one test at a time — the shard's 718 tests are never concurrent demand, and only one IoPoolTest case ran in that job at all. The 8.39 s hole in the runner log (17:54:16.97 → 17:54:25.36, with no other line in it) is this test plus the preceding class's mesh teardown, not evidence of parallel load. The test's own precondition also passed, which rules the ThreadPool leg out directly: source.HasObservers is only true once source.Subscribe(observer) has run, so the subscribe had already landed before Dispose() was called.

🚨 A drain callback registered LATE runs on the SUBSCRIBER — so it is registered disarmed, before the leaf

The canceller thread (#2394, #4448) decides where _poolCts.Cancel() is called. It says nothing about a callback that is registered after the cancel, and CancellationToken.Register on a token that is already cancelled registers nothing at all: it runs the callback synchronously, on the registering thread (#4524).

SubscribeThroughPool registers one such callback per subscription — inner.Dispose(); observer.OnCompleted();, the subscription's whole downstream teardown — and it registers it inside the subscribe, on the subscriber's thread: a hub action block, a grain turn, OrderedRouteDispatcher.DrainNext. The refusal that should keep it out is read when the cold observable is built (_draining), and the admission region only refuses once _disposing is set, which Drain() never does. So a leg built a moment before a drain and subscribed as its cancel lands is admitted, reaches Register on a cancelled token, and runs its teardown on the subscriber. Two orderings reach it, and the second is worse:

The fix changes the order, not Drain()'s admission. The registration is created first and disarmed; armed is published (with Interlocked.Exchange) once Register has returned; only then is the setup leaf started. That publication is the synchronisation point. A callback that finds itself unarmed ran before it — inline inside Register, or on the canceller at any moment up to the Exchange, including after Register returned — so the cancel was requested before the setup leaf exists, and that leaf cannot miss it: its linked token is created cancelled, its gate wait throws, and its error arm delivers OnCompleted from a pool thread. The CTS's state transition and the arm are both full fences, so a callback reading armed == 0 and a leaf reading the token as uncancelled cannot both happen. Starting the leaf after the registration also removes the second row: the source is never opened without an armed registration covering it. The two producers share one exactly-once latch, so the hand-off is explicit; the producer that takes it delivers the terminal in a finally, so a throwing source Dispose() cannot leave the latch taken and the observer unterminated.

Neither shape the issue sketched closes the window. Refusing new regions in Drain() still admits a subscribe that was already past its region check when the grace expired and the cancel ran. Checking IsCancellationRequested before registering is check-then-act against the same cancel — and "route the terminal through the canceller" has nothing to route to, because that thread exits once its one cancel has run.

Proven by widening, not sweeping. IoPoolLateDrainRegistrationTest builds the leg while the pool is alive, lets Drain() complete — it joins the cancel — and only then subscribes from a dedicated thread, so the entire drain sits inside the window. The one other producer of a terminal, the setup leaf, is parked at a test seam (IoPool.OnSubscribeSetupLeafStarting) until Subscribe() has returned; without it the test would pass whenever the pool thread happened to win the race. Against main's ordering (seam only) it fails every run with Did not expect value to be 10 — the subscriber's own thread id; with the fix it passes, and reverting the fix lines and rebuilding turns it red again.

What this did not cover was the other way a terminal reaches the caller's thread: a leg the pool refuses. That was filed as #4530 and is the section below.

🚨 A REFUSED leg terminates off the subscriber's thread as well

A refusal is what every entry point answers once the pool is terminal — Cancelled<T>() for a leg built after Drain()/Dispose(), and the admission region's own observer.OnError(...) for one built while the pool was alive and subscribed after disposal began. Both ran on Rx's immediate scheduler, so the terminal was delivered inside the caller's Subscribe() call, on the caller's thread. Measured on main, 50 subscribes per cell from a dedicated thread:

So an admitted leaf's terminal already came from a pool thread and only the refusal was handed back to the caller — the one thread this pool exists to keep work off.

The consequence is not symmetry, it is stack depth. OrderedRouteDispatcher.DrainNext subscribes the next leg for a destination from the previous leg's terminal, and says why that is safe: "No recursion depth to worry about: every leg is subscribed through the pool, which hops to a thread-pool thread, so a leg can never complete inside its own subscribe call." Once refusals are inline that premise is false, and the drain walks the destination's whole backlog by recursion. Measured (OrderedRouteDispatcherDrainRecursionTest, 32 legs queued behind an in-flight head): completions ran at stack depths 31 → 248, about 7 frames per leg, 65 DrainNext/Enqueue frames under the last one — all on the pool's single IoPool-cancel thread, inside _poolCts.Cancel(). A destination's backlog is deepest exactly when a saturated silo goes down, which is when this path runs.

The fix is the scheduler, not a gate. Cancelled<T>() throws on TaskPoolScheduler.Default, and the admission region's refusal is scheduled there too (RefuseOffSubscriber, whose returned disposable is the scheduled item, so an unsubscribe cancels it). That is the same scheduler every admitted leaf already terminates on. After the change the first two rows above read 0/50 on the subscriber's thread at 7–12 µs, and the recursion test's spread collapses.

What it costs, measured rather than asserted. A refusal takes 4.4 µs at the median instead of 0.5 µs (p95 9.8 µs) — inside the 13–30 µs an admitted leaf's cancellation already costs. With every ThreadPool worker deliberately saturated, 41 of 50 refusals arrived within 500 ms and the slowest took 291 ms: a refusal now queues for the pool the way the work it refuses would have.

🚨 And a delayed refusal is not free at teardown, which is worth stating precisely. The POOL joins nothing on it — no permit, no admission region, so Drain()/Dispose() neither wait for it nor report it. A CONSUMER does: RoutingGrain.Dispatch releases its RoutingQuiescence slot from the leg's .Finally, and RoutingQuiescenceSiloParticipant holds the silo stop until that count reaches zero. So under a saturated ThreadPool a refusal can delay that hold, bounded by the hold's own 30 s budget, after which it names the residual and the silo proceeds. That is the trade: a bounded delay under saturation, against an unbounded stack and downstream teardown on a hub turn. A dedicated thread would dodge the ThreadPool and buy back the worse half — every refusal's downstream teardown serialised behind the slowest one, which is the shape #2394 is about — so the shared pool is the deliberate choice, not the convenient one.

🚨 One cell in that matrix was a different defect, fixed separately as #4545 — the section below.

🚨 A blocking leaf the POOL cancels FAULTS; one the SUBSCRIBER cancels stays silent

InvokeBlocking starts its work with a token linked to the pool's, so a drain, a dispose and an unsubscribe all land the task on IsCanceled. The continuation's arm named one of them — "Unsubscribed before completion — silent teardown" — and stayed silent for all three. The other two are the pool ending the caller's work:

• a leaf still queued on the limited-concurrency scheduler when Drain() cancels the pool, and
• a leaf built before a drain and subscribed after, whose task is created with an already-cancelled token, so its delegate never runs at all.

Measured on the second: 0 terminals in 50 subscribes — the one cell of the matrix where a subscriber got neither OnNext, OnError nor OnCompleted. That is the #1789 shape: the .Finally that releases a route slot and advances a FIFO never runs, and a caller with a bounded wait sees only its own timeout.

The terminal is a FAULT, and the choice is forced by what callers do with a completion. InvokeBlocking<T> emits one value and completes, so an empty completion does not read as "cancelled" — it reads as "the IO ran and produced nothing", and this codebase folds exactly that into a value:

A cancelled read that completed empty would be read as a successful negative answer by all three. The siblings agree: Invoke/InvokeStream already fault with TaskCanceledException when the pool cancels them, and every refusal answers OperationCanceledException — so a cancellation-shaped fault is also the only terminal that keeps the three value-producing entry points identical. SubscribeThroughPool is the one surface that COMPLETES on a drain, and it is not a counter-example: a change feed's terminal carries no value, nobody reads a result out of it, and its job is to run the .Finally bookkeeping (#1789).

The discrimination is explicit, because the task cannot tell you. The subscription's disposable publishes unsubscribed before it cancels, and the arm reads it: the subscriber's own cancellation stays silent (after a dispose nothing may be delivered), the pool's gets OperationCanceledException.

🚨 And WHERE that terminal is delivered decides whether the drain covers it. The continuation runs before its own finally hands the leaf's region back, and TryFinishDisposal refuses to complete disposal while any region is open — so delivering inline there puts the subscriber's teardown strictly inside the window Disposed closes, which is the window a caller waits on before releasing the mesh and unloading collectible ALCs. Scheduling it away would put it after that join. Measured, same scenario both ways (a leaf queued behind a parked head, then Dispose()):

always scheduled:  head released → Disposed fired → queued OnError      (terminal ~200 µs AFTER the join)
delivered inline:  head released → queued OnError → Disposed fired      (terminal INSIDE the join)

So the arm delivers inline, and schedules only in the one case that cannot be delivered there: a token already cancelled when the continuation was ATTACHED runs it on whoever subscribed, and a terminal must never run on that thread (#4530). Nothing of that leaf ever ran — no delegate, no slot — so it is a refusal in all but name and takes the refusal's path, with the refusal's stated trade. IoPoolCancelledBlockingLeafTest.ACancelledBlockingLeafsTerminal_RunsBeforeDisposedReportsTheJoin pins the ordering, and fails on the always-scheduled shape by the microseconds above.

The whole matrix, measured after the change (50 subscribes per cell, from a dedicated thread): 16 of 16 cells deliver a terminal, 0 of 16 on the subscriber's thread, medians 7–17 µs. The previously-silent cell answers OperationCanceledException at 9.7 µs. The only OnCompleted left is SubscribeThroughPool on a drain, which is the deliberate exception above. Pinned by IoPoolRefusedLegTerminatesOffSubscriberTest (the matrix, now including that cell) and IoPoolCancelledBlockingLeafTest (the queued-then-drained path, and the unsubscribe that must stay silent).

🚨 ADMISSION is taken on the subscriber's thread, so "accepted" means one thing

Three of the four entry points deferred their whole prologue to the ThreadPool (SubscribeOn) and took their admission region inside it. Between a caller's Subscribe() returning and that prologue running, the leaf was counted by nothing — not _gateUsers, not _inFlight, not CurrentlyWaiting. The caller believed the work was queued; the pool did not know it existed (#4555).

Drain() waits while anything is outstanding and gives it a grace. A leaf in that window is outstanding to nobody, so the drain never enters the grace at all, cancels the pool token, and the leaf then arrives to find itself cancelled — too late even to be counted in LeavesCancelledAfterGrace, which is captured before the cancel. Measured on main, subscribing one leaf and draining immediately: 50 of 50 for Invoke and 50 of 50 for InvokeStream lost the work, terminated the subscriber with a cancellation, and reported LeavesCancelledAfterGrace == 0. That is accepted work discarded in silence — the exact outcome #3291 forbids (teardown lets accepted work finish and NAMES what it had to stop).

The fix is where the region is taken, not a new counter. Invoke, InvokeStream and SubscribeThroughPool's setup leaf now enter the region synchronously, on the subscriber's thread — which is what InvokeBlocking always did — and the leaf claims it when its prologue starts:

if (!TryEnterGateRegion())
    return RefuseOffSubscriber(observer);          // a refusal still leaves on a POOL thread (#4530)

var regionOwner = 0;                               // 0 unclaimed · 1 the leaf · 2 an unsubscribe
// prologue:   if (Interlocked.CompareExchange(ref regionOwner, 1, 0) != 0) throw …   // already released
// unsubscribe: if (Interlocked.CompareExchange(ref regionOwner, 2, 0) == 0) LeaveGateRegion();

The CAS is the whole protocol, and it exists because only one of the two may release: a leaf that has started still touches _gate and _poolCts, so an unsubscribe releasing the region under it would reopen the hole #2146 closed. A leaf that never started is released by the unsubscribe, so a subscription dropped before its prologue cannot leak a region and park Disposed forever.

What does not change: the work still runs on a pool thread (the SubscribeOn is untouched — only the admission moved), refusals still leave off the subscriber's thread, and the region still ends with the setup leaf for SubscribeThroughPool, never with the long-lived subscription — holding it for a live change feed would park disposal behind every feed routed through the pool.

What it costs. Disposed now waits for a leaf that has been accepted but whose prologue has not run, where before it could complete and leave that leaf to be refused on arrival. That is the correct direction — the pool waits for work it accepted — and it is bounded by the ThreadPool getting to the prologue. Pinned by IoPoolAcceptedWorkIsAccountedTest, which parks a prologue on IoPool.OnLeafPrologueStarting (so the leaf is inside the window by construction, not by racing it) and then asserts both halves of the contract: the drain spends its grace, and the leaf it has to cancel is counted. On main it fails at Expected 0 to be greater than 0.

Applied to (current scope)

The sweep is complete. Observable.FromAsync no longer appears anywhere in src/, test/, samples/, or memex/ — the only occurrence is sealed inside IoPool. The former "migration debt" query/storage sites (PostgreSqlMeshQuery family, Cosmos, file-system adapters) are now pooled; orchestration that isn't an I/O leaf (layout view generators, message-delivery and routing bridges) was rewritten as pure reactive composition (Observable.Create / Defer + Task.ToObservable()), never FromAsync.

Cross-references

• Asynchronous Calls — rule 9, "the async boundary lives at the real I/O edge".
• Aggregating Providers — "the async boundary lives at the I/O edge", pool-at-the-edge.
• Orleans Task Scheduler — per-hub schedulers and the SubscribeOn offload this complements.