A supervisor that re-settles the thread it gave up on

Measured 2026-09-21 on LinkedIn/_Thread/bug-report-from-carson-via-claude-over-m-1475 (memex.meshweaver.cloud), filed as #2229, first half fixed in #2236.

What it looked like

One thread, supervisorRetries: 2, one pending user message from 2026-07-28 never ingested (ingestedMessageIds: []). Its supervisorNote had reached 27,927 characters holding 45 nested copies of the supervisor's own diagnosis — innermost the original Parked … never ingested, then 44 (Stale) layers wrapped around it, one every ~900 s from 08:20Z to 19:50Z. Summary was "Error: " + supervisorNote, exactly, and the whole string was copied into the thread's Admin/Threads row (lastFailure 27,993 chars).

Growth rate: +607 bytes per sweep, ≈58 KB/day, unbounded, on a field every reader of that row loads.

Four defects, and how they compose

A — three submissions in 563 ms for one thread

ThreadSupervisor is an IHostedService: one per process, and the mesh ran five replicas. The terminal branch decided whether to file from thread.SupervisorFeedbackPath read out of its own sweep snapshot, with no claim and no version pin. Three replicas reached that branch in one 60 s tick and each filed — two became GitHub issues in a satellite repository, the third was orphaned. SupervisorFeedbackPath is single-valued and last-writer-wins, so the node remembers one of them.

The tell that it was three snapshots rather than one event: the third submission quotes node version 214980 where the other two quote 215216.

B — the diagnosis read a field the settle writes

Settle writes the diagnosis into both Summary (as Error: {diagnosis}) and SupervisorNote, in one with. Diagnosis embedded Summary as last error:. So the composition Diagnosis ∘ Settle's-write grew its own input on every application — neither line wrong when read alone, which is why no test saw it.

The guard that should have stopped the second application was a case list:

if (thread.SupervisorFeedbackPath is not null && verdict.Kind == VerdictKind.Parked)
    return …;   // inside a block whose `case Parked:` and `case Stale:` share one body

Stale fell through. The kind was never what decided it; whether it had been filed was.

C — the settle re-arms the round it just gave up on

Settle writes ResetExecution(), which sets Status = Idle and deliberately preserves PendingUserMessages — "the inbox queue — the next round drains anything still pending". That write reaches a cold address, activating the hub, whose submission watcher sees Idle + pending and claims → StartingExecution without stamping a timestamp. The round does not advance; 900 s later the sweep reads that as Stale; defect B re-settles; round again. The supervisor is the loop's clock — supervisorLastActionAt moved 19:35:47.84 → 19:50:52.10, i.e. 904 s.

This is why "SETTLED" is not a terminal state today, and it is a design question rather than a bug to patch: the remedy changes what happens to a real user's unanswered message.

D — the staleness gauge is blind to the writes actually happening

Classify measures quiet from node.LastModified. On this node ~250–640 writes/s did not advance lastModified: it stayed at 19:50:52.997 across reads at 19:47, 19:49:56, 19:50:10 and 20:04:19 while version climbed by ~300,000. So the gauge read "unchanged for 15 minutes" about a node being written several hundred times a second, and classified it Stale on that basis.

🚨 The write storm is NOT the supervisor. 182,624 writes landed in a window sitting entirely between two settles. The emitter is unestablished: the code names a candidate shape (the submission claim oscillating StartingExecution → rollback → Idle → re-claim, and RollClaimToIdle) but that is a hypothesis, and the one instrument that would close it — a Logs instance action — failed to answer. Do not read a still-climbing version as the B fix having failed.

What #2236 changed, and the two traps in doing it

The invariant, not the case list

internal static bool AlreadyFiled(MeshThread thread) => thread.SupervisorFeedbackPath is not null;

and the branch choice is now one pure function, Choose(thread, verdict, cfg), with the filed arm ahead of Wake, Relaunch and Settle.

🚨 The ORDER is load-bearing, and putting the guard in the terminal branch is not enough. cfg is the live ThreadSupervisorStatus node, so an operator can raise MaxRetries at runtime — and a thread already given up on and filed at SupervisorRetries == 2 then matched the Relaunch arm the moment the cap went to 3, and was recycled for a thread nobody was waiting on. Report stays ahead of the filed arm on purpose: a filed thread the pool has picked up is live activity, and suppressing that row would hide it. So the arm covers the branches that act, which is what the invariant is about — it is not the case-list-over-kinds that caused the defect.

🚨 The provenance test is the CO-WRITTEN PAIR, not "has the supervisor written here"

The first attempt asked SupervisorNote is null, and it was a regression:

• SupervisorNote is not settle-specific — the wake and the relaunch write it too;
• a round's own terminal write sets Summary while clearing ExecutionStatus.

So on round fails → supervisor wakes it → next sweep Parked/Stale, that form dropped a genuine provider error and emitted (the thread carries no error text — nothing ran far enough to write one). A confident false statement, which is worse than the verbose true one it replaced — and invisible to a control that leaves SupervisorNote null, which the first control did.

Because Settle writes the two fields in one with:

var supervisorsOwn = thread.SupervisorNote is not null
                     && thread.Summary == $"Error: {thread.SupervisorNote}";
var roundError = supervisorsOwn ? null : thread.Summary;

That is "is this Summary still the one Settle wrote?". Any later writer — a round's terminal write, a wake, a relaunch — breaks the pair, and a broken pair means Summary is the round's and is read. It inspects no text for supervisor-ish wording; it asks whether two fields are still the pair one write produced.

Why Summary keeps being written

Thread.Summary is a contract, not a scratch field: a delegating parent reads it as its tool-call result, and every terminal write is required to carry one (a failed round's starting Error:). A settle that wrote none would hand a delegating parent an empty result for a thread that genuinely failed. The fix is to stop reading it back, not to stop writing it.

Two things this cost, stated because they are the reusable lessons

It did not only make the note grow — it SUPPRESSED. A non-empty Summary short-circuits ahead of ExecutionStatus, so for any thread the supervisor had ever settled, every later genuine failure was reported with the supervisor's stale sentence instead of the real error. That was not in the original report; the control found it. Verbose-but-true and silently-wrong are different severities, and this was both.

The identity is not the fingerprint. summary == "Error: " + supervisorNote holds after one correct settle, so asserting its absence would have forced the contract break above. The defect's fingerprint is growth under iteration — Diagnosis(SettleProjection(t)) ⊃ Diagnosis(t) — which is why the live node carried 45 nestings and not one. The same identity is the right discriminator for whose Summary this is; it is simply not evidence of the loop.

How to test a fixed point

The defect lives in the composition, so Settle's inline write is lifted to a pure SettleProjection and Diagnosis made internal — the seam Classify already has, for the same reason. ThreadSupervisorSettleLoopTest then iterates the pair rather than inspecting either line.

🚨 A test of the helper is not a test of its use. The first version asserted AlreadyFiled directly, which passes even if nothing consults it and cannot see where in the branch order it is consulted. Testing Choose — the single decider Act switches on — closes both.

Controls that make the suite non-vacuous, verified in both directions against builds that differ only in the fix under test:

The cases that pass on both sides are the controls: a round's own error still reaching the diagnosis, an unfiled thread still actionable, and the other three arms still firing.

Still open

AlreadyFiled keys on SupervisorFeedbackPath, which is set only when filing succeeded — the failure path leaves it null. So on exactly the threads whose filing is broken, the guard keeps answering false and the terminal branch is re-entered every sweep. B(i) therefore does not subsume B(ii); both are load-bearing, and #2229 stays open for A, C and D. A claim-atomicity defect of the same shape as A is reportedly filed separately — if it is one root, A wants that mechanism extended rather than a second guard built beside it.

Delivery note: ThreadSupervisor is an IHostedService, one per process, so none of this arrives by recycling a node — it needs a new image. And a thread already in the cycle stays in it until its pending message is drained or marked, because ResetExecution()'s preservation guarantees the re-claim.