Reading a Recurrence Reopen

The recurrence bot reopens a closed incident issue and writes "Reopened — this fault is happening again." That sentence carries two claims, and they fail independently:

1. the fault was seen after the close — otherwise the reopen reverses a person's judgement on evidence that person already had;
2. the fault was seen recently — otherwise "again" is a statement about a counter, not about the system.

Only the first has a fix in flight (MeshWeaver.Plugins#2014: issue.ClosedAt is { } closedAt && incident.LastSeen > closedAt). The second is unwritten, and it is the one that mattered on 2026-09-17.

The wave, and its residue

Between 07:51:00Z and 07:53:59Z on 2026-09-17 the bot reopened a burst of closed core incident issues. It is filed twice — #4577 (~130 across a wider 07:45–07:55Z window, backlog 43 → 164) and #4610 (~51 in the narrow one, backlog 46 → 97).

🚨 Those are ONE burst measured at two times, not two waves — do not add them. #4577 was written at 08:54Z and #4610 at 15:37Z, and #4610's window sits inside #4577's; the backlog figures differ because re-closing had been going on in between. A third reading at ~17:30Z finds 102 open and 51 from the burst — which matches #4610's count exactly, by a different instrument (it counted updated_at inside 07:40–08:15Z; this counts reopened events). Reading the two tickets as ~181 issues of damage overstates it by roughly threefold.

Denominator, measured 2026-09-17T~17:30Z: of core's 102 open issues, 51 carry a reopened event inside that three-minute burst. Every one of the 51 was closed before 2026-09-17 — 50 by rbuergi, 1 by github-actions[bot] — so all 51 are in scope, and the rest of the wave has already been unwound by earlier sessions.

Eight further issues carry a 2026-09-17 reopen outside the burst (#2387 05:33, #4222 06:42, #4326 08:54, #3712 09:13, #1198 11:04, #1840 13:55, #1561 16:04, #1140 16:06). #1198 was reopened by a human, not the bot. None of the eight is part of the wave and none was touched.

The finding: the residue passes the first predicate, 51 out of 51

Taking, per issue, the bot's own Last seen: from the very comment it posted alongside the reopen, and the closed event that reopen reverted:

So the residue of this wave is already clean by Plugins#2014's rule, which is the only rule that authorises re-closing. Nothing here may be re-closed on the evidence available, and nothing was. That is not the same as saying the 51 are live.

The second axis: how old is the evidence?

Measuring the newest occurrence any comment on the issue reports, against the reopen instant:

The oldest is 37.4 days (#1112, #1135 — last seen 2026-08-10, reopened 2026-09-17). That is the dishonesty in the count, and no close predicate anybody has proposed reaches it: these faults were seen after their close, and then stopped.

The honest triple

Awaiting delivery is empty here by construction. The newest close in the population is 2026-09-08T21:04Z, while memex.systemorph.com runs afde4ea (2026-09-15 21:50Z) and memex.meshweaver.cloud runs c84c6c0 (2026-09-12 09:30Z). Every fix that closed a wave issue is in both running images by at least four days, so no wave issue can be excused as delivery lag. (Whether a given close carried a fix at all is a separate question, answered per issue below; the delivery argument only establishes that if there was one, it is running.) The genuine awaiting-delivery cases on 2026-09-17 are the two reopened outside the wave whose fixes merged on 2026-09-16, after both images: #4616 and #4464.

The three that are live

Each has an occurrence on 2026-09-17, reported from memex-portal-deployment-6c967c7c4c-* — the currently-running replica. 🚨 What each close actually asserted differs, and that decides what the recurrence means — so the three are listed with their closing verdict rather than lumped together:

These three are the wave's entire signal. Reading the reopen wave as noise loses them; reading it as 130 regressions buries them.

🚨 And "recurring against its own fix" is a claim to check per issue, not a property of the bucket. Only #1549's close carried a stated, deployed fix; #2765's close explicitly identified no code defect, and #2299's carried no verdict at all. A live occurrence is a live occurrence either way — but only one of these three is evidence that a fix failed.

The third predicate nobody has written: the close races the roll

🚨 15 of the 51 have their ONLY post-close evidence within 17 hours of the close, and nothing in the 8–37 days since:

#2152 (+0.0 d) · #3466 (+0.0 d) · #2171 (+0.1 d) · #2182 (+0.1 d) · #2679 (+0.1 d) · #3051 (+0.2 d) · #2444 (+0.3 d) · #1112 (+0.4 d) · #1135 (+0.4 d) · #1175 (+0.4 d) · #2233 (+0.5 d) · #3647 (+0.5 d) · #2153 (+0.6 d) · #2876 (+0.6 d) · #3126 (+0.7 d)

Measured: the gap, and the silence after it. Inferred, not measured: that these are the old replicas still emitting between the merge that closed the issue and the roll that delivered it — a fix closes its issue when it merges, while the pods keep running the previous image for hours. Confirming it needs a pod→image mapping the incident comments do not carry.

If that reading is right, the predicate is not "was it seen after the close" but "was it seen on an image that contains the fix" — and it would take these 15 out of the wave without weakening the rule for the three that are live. It also explains why Plugins#2014's predicate, which is correct, leaves a residue that still reads wrong: a close and a delivery are different events, and the reopen rule compares against the earlier one.

The mirror image: dormancy is only evidence if the trigger happened

The section above is about a close that races the delivery — post-close lines coming from replicas still running the old image. This is its inverse, measured on 2026-09-19, and it bites the closing side rather than the reopening one.

Three teardown faults were closed partly on dormancy and all three reopened within a day:

Every reopen is correct by the post-close predicate — each occurrence genuinely postdates its close. The mistake is upstream, in the reasoning that closed them: one carried the words "15 days with no occurrence".

🚨 All three fired inside one seven-minute window, 05:05–05:12Z, and that window is a roll. Their subjects are teardown by construction — ObjectDisposedException unregistering a grain from the directory, a mesh hub resolving from a disposed Autofac scope. They fire when pods stop, and they are silent when nothing is stopping them. Fifteen quiet days meant fifteen days without a roll, not fifteen days of health.

🚨 Correlation with a roll window is NOT the same finding as the mechanism. Two more issues reopened across the same period — #1422 (Release-snapshot cleanup, whose reopen is a cascade racing a concurrent delete) and #1449 (a PostgreSQL CreateNode Unicode failure) — and it is tempting to sweep them in. Their records identify neither as teardown- or boot-shaped, and #1449's occurrence (09-18 15:35:29Z) is not in the window at all. Firing during a roll is evidence; needing a roll in order to fire is the claim, and only the three above carry it. An earlier draft of this section asserted all five, which is the very over-reach the page exists to warn about.

The predicate: silence is evidence only over a window in which the fault's trigger actually occurred. For a roll-triggered fault, a dormancy argument has to name the rolls it survived. Without that it is the same shape as a sweep's zero with no denominator — a number that reads like an answer and measured nothing.

Classifying by the timestamp is cheap and splits the backlog usefully. Of fifteen old incident issues checked the same day, three more sit in the roll window (#1126, #1840, #2833 — two compile-at- boot, one shutdown quiescence), while four fired at 08:54–09:03Z, after the image then running was built (#1246, #2307, #2480, #3045).

🚨 But "fired after the image was built" does NOT imply "fires on that image", and for a teardown fault it implies close to the opposite. The pod that emits a teardown fault is the pod being REPLACED — it is running the image the roll is replacing, by construction. #2480 is exactly that shape: its subject is the mesh drain at silo shutdown, and its 08:59Z occurrence sits right after an 08:32Z roll, so the likeliest reading is the old replica tearing down, not the new one failing. The timestamp separates "during a roll" from "between rolls"; deciding WHICH image was running still needs the pod→image mapping, which the incident comments do not carry.

So the classification gives three buckets, not two: silent-between-rolls, fires-during-normal- operation, and fires-at-teardown-on-the-outgoing-image — and only the middle one is evidence about the image now serving.

A fourth check that is not a predicate: the fingerprint may have been RE-ADDRESSED under the issue

Every check above asks what does the evidence on this issue mean. This one asks the prior question — is this issue still the thing the evidence lands on — and on 2026-09-19 the answer changed for three of core's oldest bot tickets at once.

Between 11:41:37Z and 11:44:05Z on 2026-09-19 a new incident-identity function (MeshWeaver.Plugins#1796) re-addressed the fleet's high-volume fingerprints: a mega-fingerprint that had been folding several different log SHAPES was split per shape, the old node was marked Superseded, and each surviving shape either inherited the old issue or filed its own. Read off the nodes on the control instance the same afternoon:

Three consequences, each of which reverses a reading a triage pass would otherwise make.

1. "It will just be reopened" stops being true for a superseded node. The fold comment says it in so many words — "Those nodes are superseded and will not fold, file or comment again." The recurrence pressure a session declines to close against may already be gone, and the only place that is written is content.status / content.supersededBy on the node. The issue's own history cannot show it: the last fold comment looks exactly like every earlier one.
2. The residue that held a ticket open may have MOVED to another ticket. #3659 was held open on 2026-09-17 for one operator item — ClaimsDeepfield, 82 of 83 declared nodes absent on memex-cloud, unhealed for three weeks. That shape is now its own fingerprint with its own, correctly-titled issue (#4812), still folding as of 09:09:51Z. Closing #3659 no longer buries it. So check every successor's issueNumber before concluding a ticket still owns its residue — and before concluding that closing it would lose something.
3. Inheriting an issue and filing a new one are indistinguishable from GitHub. Both arrive as an ordinary bot comment on some issue. The binding is written only on the node (issueNumber, reporterFingerprint, foldedFrom).

The fingerprint's namespace is not the deployment its NEWEST samples came from

content.namespace is one value on a node whose samples accumulate for weeks, and a split can hand a successor a namespace that no longer matches the sample you are reading. It is worse than stale: one fingerprint can span deployments while that field holds a single value. 4ff70ec1124b1b3c carries namespace: memex and its pod list contains both …-6cd5d8f887-2kcwk (in Ops/Status/memex's roster, 3.0.0-ci.8968) and …-69956b6dbc-{v5r29,s246c} (in Ops/Status/memex-cloud's, 3.0.0-ci.8969). No single value can be right for that node.

🚨 A pod name IS decisive — but only once you resolve it against a roster, never by reading it. Every portal runs a Deployment called memex-portal-deployment, so the name carries no cluster; what separates them is the replicaset hash, and which hash belongs to which deployment is knowable only from Ops/Status/<deployment>'s replicas[]. Measured 2026-09-19: memex-cloud was on …-69956b6dbc-* (3.0.0-ci.8969, commit c25f86ae85b7…) and memex on …-6cd5d8f887-* (3.0.0-ci.8968, 96f88406b4…). So a sample naming …-69956b6dbc-s246c is memex-cloud's even though the node it sits on says namespace: memex — and an earlier draft of this paragraph had it backwards, inferring the pod's cluster from the node's namespace, which is the error the paragraph exists to name.

The second discriminator, and the only one left when the pod is gone or the namespace is not being scraped, is a record version quoted in the log line. #3659's 05:29:06Z sample says taken over Plugins/DefaultViews v48 (written 2026-09-19T00:24:55Z). Read the same hour on both portals, Plugins/DefaultViews is v31 (written 2026-09-18T21:31:10Z) on memex.meshweaver.cloud and v54 (2026-09-19T07:50:02Z) on memex.systemorph.com. memex-cloud never held a v48, so that sample is the other portal's — and an ancestry argument about memex-cloud's running image says nothing about it. The same trick works on any line that names a node version, a module version or an installedAtUtc.

The positive control for "no recurrence since the roll" lives on OTHER tickets

A close review at 2026-09-19T10:35Z could not establish "not seen since the roll" for twelve issues, because no incident anywhere in the readable population had folded a sighting since 07:28:51Z — a reading indistinguishable from a stopped watcher, and correctly refused as evidence. By 15:41Z the same instrument had folded sightings from pods in Ops/Status/memex-cloud's own roster. 🚨 What qualifies each row below is the POD, never the incident node's namespace — for the reason in the section just above, two of these three nodes carry namespace: memex while the pod they name is in memex-cloud's roster:

(Ops/Status/memex-cloud sampled 18:53:13Z lists exactly …-69956b6dbc-{gx6z6,ndnxt,s246c} on 3.0.0-ci.8969; Ops/Status/memex lists …-6cd5d8f887-{2kcwk,lvmpf} on 3.0.0-ci.8968. That is what makes the pod decisive and the namespace field not.)

🚨 The control does not have to be the fingerprint you are judging. Any fold naming a pod from the current generation establishes that ingestion is alive for that generation, which is exactly what converts a silence on your own fingerprint from "unusable" into evidence. Sort the readable Admin/_LogIncident population by lastModified descending and read the newest nodes' content.lastSeen and content.samples[].pod — a node rewritten without its occurrences advancing is triage bookkeeping and is NOT such a control.

The pod→image mapping the section above wants DOES exist, on the deployment record

"Deciding WHICH image was running still needs the pod→image mapping, which the incident comments do not carry" — they do not, but Ops/Status/<deployment> does, whenever kube-state-metrics is scraping the namespace (notScraped: false; it flips, #4218). Its replicas[] carries pod, image, generation, commit, startedAt and restarts per replica, so a sample's pod name resolves to an image and a start time in one read. Measured 2026-09-19T18:53:13Z for memex-cloud: three replicas on 3.0.0-ci.8969 / c25f86ae85b7…, started 08:34:53 / 08:39:58 / 08:39:59Z.

That closes the teardown ambiguity in the direction the section above leaves open: a sample whose pod STARTED before the sample is the incoming replica, not the outgoing one. #1246's 08:57:30Z and 08:59:06Z sightings are from …-69956b6dbc-gx6z6, started 08:39:58Z — eighteen minutes earlier — so they are the NEW replica reporting, and the fault is on the image that roll delivered. 🚨 The same read also warns you when the node's own namespace disagrees with the roster a pod appears in, which is the mis-attribution above.

How to read one, until both predicates exist

• A reopen means "re-read the record". It does not mean a fix regressed (#1122 says so, and this wave is the measurement behind it).
• Read Last seen:, never the counter. Still occurring: 705 more since the last update is a comparison of accumulated totals; it stays positive forever once an issue is closed while its record keeps history. Three of the 51 say Last seen: 2026-09-17; the rest say August or early September, with the same headline.
• Compare it to three timestamps, in this order: the close (was it even after?), today (is it recent?), and the image the reporting pod was running (could the fix have been in it?).
• 🚨 And before closing one on quiet, ask what the fault needs in order to fire. A teardown or boot fault is silent by construction while nothing rolls, so dormancy across a roll-free window is not evidence — see the section above for five issues closed that way and reopened within a day.
• A burn-down or "zero issues" reading across 2026-09-17T07:51Z is comparing two different populations. Say which side of the wave a count was taken on.
• Read the incident NODE, not only the issue. content.status, content.supersededBy and content.issueNumber on Admin/_LogIncident/{fingerprint} are the only place it is written whether this issue is still what the evidence lands on. A superseded node folds nothing further, and a successor may have carried the residue to a new ticket.
• Name the deployment from the pod resolved against Ops/Status/<deployment>'s roster, or from a record version quoted in the line — never from the incident's namespace, which holds one value for a fingerprint whose samples span deployments, and never from a pod name read on its own (every portal's Deployment has the same name).
• Take the positive control off a DIFFERENT fingerprint. "Not seen since the roll" needs one fold from a current pod somewhere in the readable population; without it the silence is indistinguishable from a stopped watcher.

Appendix — the 51, one row each

A negative age means the newest occurrence is NEWER than the reopen: the fault carried on after the bot reopened the issue.

Live — seen 2026-09-17 (3)

Recently live — quiet 5–7 days at the reopen (7)

Quiet 8–37 days at the reopen (41)

Method

Read-only over the GitHub REST API: /issues?state=open (paginated, pull_request filtered out), then per issue /issues/{n}/events for the close/reopen pair and /issues/{n}/comments for the bot's own Last seen:. Last seen is taken from the reopen comment for the first axis and from the newest comment of any kind for the second, so a fault that resumed after the reopen is counted as live. Running images read from /api/version on both portals. No issue in the wave was closed, reopened, commented on or labelled.

See also

• Reopening on Image Provenance — the third predicate this page's 15-of-51 finding points at, designed: the evidence chain, the one link still missing, and the fail-safe for an image that cannot be resolved

• Operating from the portal, not the cluster — where the pod and image facts come from

• Router Traffic Detection — a detector that had the opposite problem: it named the rule broken without naming who broke it