Why an operation request can look inert

A field report on 2026-09-06 (#1320) said the agent-facing half of operation requests does not work: a request created through the API — the MCP create tool, a well-formed OperationRequestContent with requestedAction: "Preview" — "was created. It never previewed. steps: [] forever, no error, no log, nothing to grep." The decisive difference it named was the missing Script child, and the conclusion drawn was that "the executable half is generated by the creation flow, not by the request node's own watcher".

That conclusion is wrong, and the page you are reading exists so nobody re-derives it. The Script child has exactly one author — OperationRequestControlPlane.Run — and the API path works. What was true is that the control plane could stop watching, permanently and silently, and a request whose watcher has stopped is indistinguishable from one nobody has picked up yet.

The measurement

Measured on the live memex.systemorph.com portal, 2026-09-06, with no browser involved at any point — a request created purely through the MCP create tool (rbuergi/Requests/probe-1320, a plan of one Note step that touches nothing):

So: create → preview → approve → run, entirely through the API, sub-second at every step. The Script child (nodeType: Code, createdBy: system-security) is generated by the node's own watcher on the Preview/Approve transition, exactly as the issue's own suggested shape proposed — it already worked that way.

The request in the field report tells the same story from the other side. Its version history is v1 at 10:35:37 (the create, requestedAction: Preview), v2/v3 at 11:15:20 and 11:15:32 (the reporter forcing Preview → None → Preview), then v4–v6 — the control plane's own three writes, whose content timestamps read 15:59:11–12, and whose Script child carries createdDate: 15:59:11.286Z. Nothing ran for four hours and forty-four minutes, and then the whole preview ran in 1.3 seconds. That is not a missing mechanism. That is a mechanism that was not subscribed, and then was.

🚨 A MeshNodeStreamHandle.Update does not re-stamp lastModified, so the version rows v3–v6 all carry v3's timestamp. Read the content's own startedAt/previewedAt and the child node's createdDate for when a run actually happened — the version row's lastModified is the node's field, not the write's clock.

The defect: the recovery arm was itself a write

The watcher was one subscription for the life of the per-node hub:

workspace.GetMeshNodeStream()
    .Where(node => node is not null)
    .Select(node => Process(hub, workspace, node!)
        .Catch<Unit, Exception>(exception => Stamp(workspace, node!.Path, c => c.Cleared() with
        {
            State = OperationRequestState.Failed, Error = exception.Message,
        })))
    .Concat()
    .Subscribe(_ => { }, exception => logger?.LogWarning(exception, "watcher stream failed"));

The Catch looks like the guard that keeps the pipeline alive, and its comment said so. It is not: Stamp is a cold write to the mesh, so the recovery can fail on its own. When it does, the error leaves the Catch, Concat forwards it, and the subscription's onError runs — which ends the sequence. From that moment the hub is up, the node is writable, patches land and bump the version, and no action on that request is ever processed again. There is no resubscribe, and there is nothing on the node saying so: the terminal Failed stamp is precisely the write that failed. The only trace is a single LogWarning in one pod's log.

That end state is byte-for-byte the field report: requestedAction still Preview, state absent (so Proposed), steps: [], scriptHash absent, no activityPath, no Script child, no error. And it is cured by the one thing the fix does — a fresh hub. Which is what 15:59 was.

What the control plane guarantees now

OperationRequestControlPlane.Watch is the composition, extracted so the invariant is pinned by a test rather than by review:

• A pass cannot end the watcher. The recovery gets a recovery: a failed pass is recorded on the node, and a failure of that write is reported and contained.
• Containment is not swallowing. Every arm reports. A pass that failed and was recorded logs at Warning (the approver can read the error on the page). A pass whose failure could not be recorded logs at Error, naming the consequence — the request still reads as pending — because that is the only place the information now exists.
• The subscriber's onError still means something. It is reachable only for a fault of the node stream itself, which is genuinely terminal for that hub, and it logs at Error saying that no further action on the request will be processed until the hub restarts.
• Passes still run strictly one at a time (Concat) and in order — they all write the same node.

Two cases in OperationRequest/Test/OperationRequestTests.cs hold it: AFailedPass_WhoseRecordingAlsoFails_KeepsTheWatcherAlive (the trigger after a failed pass whose recording also failed is still processed) and AFailedPass_IsRecorded_AndTheWatcherCarriesOn (the ordinary path: recorded once, reported once, order preserved). Reverting the inner Catch alone turns the first one red and leaves the other 26 cases green.

What this deliberately does NOT do

It does not declare an interrupted run dead. A request stamped Previewing/Running whose hub then goes away — a portal roll, a grain deactivation, a pod eviction — is still stuck: OperationRequestContent.ShouldRun() is false for both in-flight states, so no action, not even Reject, is ever processed again. That is a second, independent defect and it wants a different fix from the obvious one:

The run is not owned by the control-plane hub. Run dispatches an ExecuteScriptRequest to the Script child's kernel and follows the resulting activity; the kernel keeps going when the request's hub restarts. So a fresh hub that finds an in-flight state must re-attach to ActivityPath and keep folding its frames — not stamp Failed, which would tell the approver the deletion stopped while it is still deleting. Only an in-flight state with no ActivityPath was genuinely never dispatched.

Filing that as its own change is deliberate: getting it wrong makes the page say one thing while the mesh does another, which is worse than the wedge.

Two answers that read like a pass (#1922)

A second field report, #1922, came off one approved request on memex.systemorph.com, 2026-09-15 (rbuergi/Requests/provision-pearl-20260914). Nothing wedged and nothing errored. Two separate things said something that was not true, which is the harder failure: a wedge is at least visible.

1. A failed run whose page still said "Running…"

The run went silent after 10:45:06Z — its kernel host hub was disposed mid-run (MeshWeaver#4422, fixed by #4423). The silence watchdog fired on schedule at 11:05:06Z and stamped:

state:   Failed
error:   "The operation has timed out."
message: "Running — not yet visible /  after 30 min"   ← the last in-flight line, unchanged

A Failed request whose page reads Running… is unreadable by anybody — a person, or an agent polling the node — and The operation has timed out. names no operation, no budget and no next step. Both are fixed: the watchdog throws OperationRunSilentException carrying its budget (a distinct type, so a silent follow no longer reads like the 90 s dispatch timeout), and OperationRequestFailure.Failed is the ONE terminal stamp for a failure outside the kernel's own verdict — it clears the in-flight Message and, for a silence timeout, says what the watchdog can actually establish and no more: no progress reached this request for N min, last heartbeat …Z; either its run stopped reporting or the request could not record what it reported.

🚨 The watchdog deliberately sits AFTER the writes, so a frame whose write never completes starves it exactly as a silent kernel does — and it must, because a follow is one pass of the control plane and Watch runs passes one at a time. That is why the wording claims the weaker thing. Follow is pure and driven across its real budget on a virtual clock by Follow_FailsOnlyAfterAFullBudgetWithNothingReachingTheRequest.

2. A read that could not tell "not there" from "not visible to me"

The same run created Deployments/pearl-provision-20260914, its create step logged created, and every 20-second Plan.Find poll after it answered null — so the request's follow printed not yet visible for thirty minutes and the plan never failed. Two independent defects made that possible, and both are now closed at their own level.

The read had no viewer. Every DSL read went out as a bare MeshQueryRequest.FromQuery(query). A null UserId does not mean unfiltered; it means work out who is asking, and when nothing can be worked out the read is evaluated as Anonymous — the narrowest viewer there is — which for a path in a private partition returns nothing at all. The identity is ambient, and the platform's own note on that surface says it "does not depend on the caller's ambient context surviving whatever scheduler, pool or change-feed hop lies between the call and the storage provider". So the DSL's three most consequential answers were all unfalsifiable in the same way: Find reporting a node is not there, DeleteAt reporting absent — nothing to delete and skipping a delete the plan promised, and CreateAt deciding to create a node that already exists.

OperationPlan.Read(query) is now the one request every DSL read issues, and it declares RequireViewer(): an unresolvable viewer throws QueryIdentityUnresolvedException naming the query instead of returning the empty set. It stamps no viewer of its own — a run whose identity resolves reads exactly as before, with exactly the rights it had. It deliberately does not stamp AsSystem(): this file is text a script can compile, and an RLS bypass one call away in a shipped prelude is a different feature from a diagnosable read.

And created was never established. CreateAt reported created because IMeshService.CreateNode returned — the shape a verification step that cannot fail. The platform has a name and a page for when that is not evidence: Durable But Unreadable — a write acknowledged, versioned state: Active, and permanently invisible to every reader — and its conclusion is that a mint-time read-back is the only acknowledgement worth trusting. CreateAt now does that read-back (Appears, waiting on the live query for CreateReadBack, never re-asking on a timer), and OperationPlan.Created decides the outcome from what it found. A create the read-back cannot see fails the step, naming the node, saying the write was ACCEPTED, saying how long it waited, and naming the next read — rather than reporting success and leaving every later step to say not yet visible.

The budget exists for the index trailing the store, not as a retry: the query is live, so it answers the moment the node appears. Over the budget is not "slow", it is the inconsistency.

What this does NOT fix: why that node is unreadable

The store's own defect is untouched, and it is not this module's. Re-measured 2026-09-16, as a stamped global admin on the same portal:

That is the three-seam signature of Durable But Unreadable exactly, and it is a third confirmed instance — the first in a system partition on the control instance. Two hypotheses are already ruled out by core's own write-up and hold here too: the untyped-content degrade leaves a node in the listing (every degradation seam logs and returns the node; nothing throws), and the index/point-read split would leave the point read working. The root cause needs the partition schema inspected — main_node, partition_access and user_effective_permissions for deployments against a sibling that works — and that is tracked on core, not here. Do not repair it by restoring a version: a restore takes the same write path and can land the same way.

Seven other control planes carry the same shape

This is not one type's mistake — it is the shape every RequestedX watcher in the repo was written in, copied faithfully, comment and all ("A failed transition must never wedge the pipeline"). A source scan over every *ControlPlane.cs / *Watcher.cs outside src/ finds the recovery-arm-is-a-write pattern in:

Each is Process(…).Catch(ex => Stamp(…Failed…)) → .Concat() → .Subscribe(_ => { }, ex => LogWarning), so each can lose its watcher to a failing write and leave its own node reading as merely pending — an order stuck before fulfilment, a provision stuck at Requested, an enrollment that never grants.

They were left alone in the #1320 change to keep it to the type the issue is about. The fix for each is to route it through the same Watch invariant; the reason to do them together is that a shared helper needs a home both Store/* and Essentials/* can see (Store/Core/Source, which Essentials/OperationRequest already shares), and moving it there is a decision about the Store package's surface rather than about this bug.

Reproducing and measuring locally

The in-mesh test bodies run on this laptop — no Docker, no mesh — through the platform's own gate runner, which compiles each package's Source/ and Test/ with Roslyn and executes every *Tests case:

mw-plugin-test build <plugins-repo-root> Essentials \
  --module <…>/MeshWeaver.AI.dll \
  --module <…>/MeshWeaver.Markdown.Collaboration.dll \
  --module <…>/MeshWeaver.Payments.Stripe.dll

The three --module arguments are not optional: without them four Store NodeTypes fail to compile against a reference set that has no module bundles in it (MeshWeaver.Payments missing, then StripeWebhooks/StripeGateway), Store goes RED, and every package that requires it — Essentials included — is reported blocked, having compiled and tested nothing. A blocked package is not a pass.