An Unreachable Store Is Not a Refusal

A store that could not be REACHED means the operation was never EVALUATED. That is a different outcome from "the operation was evaluated and refused", and the two demand opposite responses from the caller:

Collapsing the second into the first is not a cosmetic reporting bug. Where a caller mints its target id per attempt, a false "refused" makes it give up on the id it was using and start over with a new one — so the retry lands as a duplicate rather than as the same write. That is the shape #2229 is about, arriving through the reporting layer instead of through a stale query.

One rule, three consumers

There is exactly ONE definition of "the data store could not be reached", and it lives in MeshWeaver.Data.Contract — the one assembly all three consumers can see:

// src/MeshWeaver.Data.Contract/StorageFaults.cs
public static bool IsTransientConnectFault(Exception? ex)

It is typed on the BCL System.Data.Common.DbException surface rather than on NpgsqlException, because the storage adapters live in provider packages that core cannot reference. Every ADO.NET driver derives its faults from DbException and carries DbException.SqlState, which is enough.

Two copies of that rule would drift silently, in either direction — a fault the fan-in retries but a create reports as a defect, or an outage a create excuses that the fan-in never retried. Both failures are invisible from outside. That is why TransientStorageFaults.IsTransientConnectFault and StoreReachability.IsStoreUnreachable are both thin forwards, and why what stays local to each layer is the policy (the retry budget, the frame copy, the rejection reason), never the classification.

The ladder: retry ONCE, then answer

provider observable
  └─ RetryTransientConnect ── 250 ms → 500 ms → 1000 ms ── the last error surfaces
       └─ the consumer's terminal arm ── CLASSIFY and ANSWER.  Never retry again.

🚨 A fault that reaches a handler's terminal error arm is one whose retry budget is honestly spent. Retrying it there would aim a second, unbounded-in-aggregate retry at the resource that is already the bottleneck — and a database unreachable for 21 seconds outlives 1.75 seconds of budget no matter how many layers spend it. The correct answer to a spent budget is to report the condition accurately, not to spend more of someone else's. See Error Propagation & Wedges for why the retry that "recovers from a state that shouldn't happen" is the shape that takes portals down.

The same reasoning rules out the other two reflexes:

• Do not downgrade the log. An availability failure stays at Error, where an operator sees it. What changes is the wording: it names the store, not the operation.
• Do not call it transient in a sense that promises recovery. Nothing fires when a database becomes reachable again, so a caller told "this will resolve itself" waits forever. Unavailable says retry is meaningful, not a retry will happen.

What the node operations answer

NodeCreationRejectionReason.Unavailable already existed for a different unreachable dependency — a security-policy read that starved (#1446) — and its documented meaning is exactly right here: "The create was NOT evaluated … an availability failure, not a verdict." The 2026-09-02 change is the wire between the classification and that value, on both create verbs:

// MeshExtensions.HandleCreateNodeRequest — terminal onError arm
if (StoreReachability.IsStoreUnreachable(ex))        // ← condition, so it outranks the type tests
    Respond(CreateNodeResponse.Fail(
        StoreReachability.DescribeNotAttempted($"Node creation at '{node.Path}'"),
        NodeCreationRejectionReason.Unavailable));
else if (ex is InvalidOperationException) { /* ValidationFailed */ }
else if (CancellationClassifier.IsCooperativeCancellation(ex)) { /* Unavailable, #2152 */ }
else { /* Unknown — a genuine unexpected failure */ }

Three properties of that ordering are load-bearing:

• The condition outranks the type. An InvalidOperationException wrapping a driver connect fault is still the store being unreachable; answering it ValidationFailed would tell the caller their request was invalid because a database was down. CancellationClassifier makes the same argument in the other direction — a cancellation carrying a TimeoutException cause is a fault, not a cancellation, despite the type.
• The bulk verb carries the same branch. CreateNodesRequest is what every installer and static-repo import travels; a guard on one create verb and not the other is a guard on neither.
• …but below its partial-landing branch. "Nothing was written" is half of what this answer asserts, and it is FALSE once a batch has committed a window (Postgres writes each WriteMany window in its own transaction). A partially-landed batch keeps its own, louder report.

The falsification boundary

The classification is only worth anything if it can say no. These are deliberately NOT matched and keep their loud "unexpected failure" treatment:

CreateWhenTheStoreIsUnreachableTest pins both directions: the connect timeout must answer Unavailable, and 42P01 travelling the identical path must still answer Unknown. Without that second case, "answer Unavailable for every exception" would pass every other assertion in the file.

Partition bootstrap must preserve a failed read

A child create first reads its partition root, its creator grant and the GitSync ownership record. The storage adapter already returns null when a partition schema does not exist. The bootstrap must therefore preserve any exception from these reads. Previously its two read helpers caught every exception and returned null; a connect timeout while reading an existing root made the bootstrap attempt to create that root again. The nested create's error then reached the child as a ValidationFailed refusal. It reaches the same nested root-create boundary named by the Store/Core and Signature/DeepSignCredential release failures in #6055/#6056. The original root-read outcome was swallowed, so those incident logs cannot establish why each read failed.

AnUnreadableExistingPartitionRoot_DoesNotTriggerAFalseHeal creates a real partition, faults only the next root read, and checks that the child gets Unavailable without a false root repair. It failed before the change with ValidationFailed from the nested root create and passes with the read fault propagated to the create handler's existing availability classifier.

🚨 "Unreachable" does not mean the store was down

The classification above is about what this process could reach, and that is deliberately all it claims. A connect timeout is not evidence that the database was unavailable — the identical exception is produced when this process is the one that stalled, because the connect timeout is wall-clock and does not pause for GC.

That is not a theoretical caveat. It is what actually happened in the incident that produced #3050 / #3051, whose bodies both concluded "database-host unavailability … high confidence". Measured over the 3 h 5 min of the affected pod's retained log:

The coverage figure is the control, and it is what makes this more than "both lines are logged when the process resumes": if the database had genuinely been unreachable, its timeouts would be spread across the other 98 % of the window. They are not. Only 11 of 329 (3.3 %) fall outside even a 5-second shadow of a stall.

The same pod was at 96.4 % of its managed-heap hard limit with 38-second thread-pool delays, and was evicted from the Orleans cluster ~4 minutes later for exactly that reason — see Reading a Silo Eviction, which is the same host condition wearing a different incident number.

So the reading rule is:

• The answer the fix above produces is correct either way — "this process could not reach the store, so the create was not attempted" is true whichever side stalled, and it is all the handler can honestly assert.
• The investigation the answer points to is not. Before opening a ticket against the database, check the calling process's own .NET Runtime Platform stalled / Thread Pool is exhibiting delays lines in the same window. The cheap discriminator is the adjacency above; the cheaper one is whether other clients of the same database timed out at the same moment.

Known edge: the in-process exception mapping still collapses the two

IMeshService.CreateNode and HubNodePersistence.CreateNode turn a failed response into an exception, and both map Unavailable and Unknown to the same InvalidOperationException:

return Observable.Throw<MeshNode>(r.RejectionReason switch
{
    NodeCreationRejectionReason.ValidationFailed  => new UnauthorizedAccessException(…),
    NodeCreationRejectionReason.NodeAlreadyExists => new InvalidOperationException(…),
    _                                             => new InvalidOperationException(r.Error ?? …)
});

So an in-process caller distinguishes the two only by reading the message; a caller that reads the response directly (MCP, the API, a test) gets RejectionReason and does not have to.

This is a genuine gap, and it is left open deliberately rather than patched here. There is no node-operation "unavailable" exception type today, and NodeUpdatePipeline documents why the obvious alternative is worse: Unavailable sits on the InvalidOperationException side of its fork on purpose, because mapping it to UnauthorizedAccessException "would tell the caller they lack a permission they may well hold". Introducing a third type is an API change with a call-site sweep attached — a design, not a branch — and it should be taken together with the same question for the delete, move and upsert verbs, whose terminal arms have not been swept for this condition at all.

Who re-issues an Unavailable create (#7380)

Unavailable tells the caller that re-issuing the same request is meaningful. Until #7380 nothing said who does it. A NodeType release re-cut has its own re-issue (Release Failure Classes, #7072), but every other writer's create was simply lost unless that writer retried. During the 2026-10-22 database restart, the creates that were eventually made again were made by an unrelated trigger 76 minutes later. Notifications and other bookkeeping writes were never re-issued at all.

The platform now re-issues them, once:

1. The refused create is recorded as an intent. The create handler's store-unreachable arm answers Unavailable as before. It also records a PendingWrite in the mesh-scoped PendingWriteLedger. The intent holds the node exactly as requested (same id), the resolved requester, and the AccessContext the request was delivered with.

   • The intent cannot go to the store that just refused it. With Mesh:PendingWrites:JournalDirectory set to a volume that is not the database, each intent is also a journal file. The file is written before the intent counts as recorded and removed once the intent is settled. A process that starts reads the journal, so an intent survives a restart.
   • Without that key, an intent lives for the process. It still outlives the caller and the activation that was refused, which a retry inside the caller would not.
   • Each path holds one intent. A second refusal of the same path replaces the first.

2. The recovery watcher re-issues it when the store answers again. PendingWriteRecovery is armed on the mesh hub. It acts on StoreReachability.StoreAnswersAgain: a declared change-feed gap (the LISTEN connection resumed) or a commit the store announced. Each piece of evidence takes every pending intent out of the ledger and re-issues it. The re-issue is a CreateNodeRequest carrying the original CreatedBy, posted with the original AccessContext, so the owner authorises it exactly as it would have authorised the first attempt.

   • There is no timer, no poll and no widened bound. While the store stays away, no evidence arrives and nothing is re-issued.

3. Once, then a verdict. Each re-issue settles as one of three verdicts, and every verdict is logged and published on PendingWriteLedger.Outcomes:

   Verdict	Meaning
   Landed	the create went through
   AlreadyPresent	another writer, often the caller itself, created the node meanwhile
   Dropped	the re-issue was refused again, for any reason; reported at Warning, naming the path and the writer

   The re-issue is marked CreateNodeRequest.ReissuedAfterUnavailable, so a second Unavailable is never recorded again. A pending write is re-issued once and never retried for ever.

What is never re-issued. A create that was judged is never re-issued: refused for permission, for validation, or because the path exists. Only the store-unreachable arm records an intent.

Not covered yet: the bulk create (CreateNodesRequest), whose unreachable arm may have landed part of the batch, and the update, delete and move verbs. Each needs its own decision about what "the same request" means.

Evidence: AnUnavailableCreateIsReissuedOnceTest (MeshWeaver.Hosting.Test), using the fault-injection store's RefuseWritesAsUnreachable and a declared feed gap.

With the recording removed from the handler, the first test fails: no intent is ever recorded.

The trail

All three were reported as separate incidents with different top frames, and all three are the same sentence: the store could not be reached, and the layer that found out said the wrong thing about it. When the next one arrives — and it will, from a verb not yet swept — the fix is another consumer of the same rule, never another copy of it.

See also

• CQRS — Queries vs. Content Access — why a stale negative on a may-not-exist path produces duplicate data, the other road to #2229
• Error Propagation & Wedges — surface errors to a real sink; never swallow, never retry a "shouldn't happen"
• Debugging Postgres — when the store really is the problem
• Unanchored Security Reads — #3031's other half, and why the Catalog render died inside the call that enumerates schemas