Read first: Asynchronous Calls (everything here is reactive +
IIoPool-bounded, neverasync/await) and Access Context Propagation (the runner writes as system). This engine supersedes the formerScheduledActionRunner; it is the general form of the same reconcile-on-startup pattern.
What it is
An EventSubscription is the durable record of "when THIS trigger fires, run THAT continuation" — and it survives a reboot. It is a MeshNode at Admin/EventSubscription/{id} (the always-present Admin partition), and a single background EventSubscriptionRunner drives every one of them by two complementary paths:
- Live — reacts to the event as it happens (the mesh change feed for CRUD; an
Observable.Timerfor a timer; a node-stream watch for a status change). - Reconcile-on-startup — a live query over the outstanding subscriptions re-evaluates each against current state whenever the set changes (including at boot). So a trigger that fired while the runner was down — the invitee signed up during a deploy, a timer came due mid-restart, a sub-thread finished while the portal was rolling — still fires. The durable state is the subscription node itself (
Pending → Fired), so nothing is lost and nothing double-fires (the terminal write gates re-entry; every continuation is idempotent).
Because the mesh node is the durable source of truth (Postgres), durability is free — we do not need Orleans reminders or durable streams (they aren't wired, and the monolith host has no Orleans at all; the reconcile query is host-uniform).
The shape
EventSubscription (project MeshWeaver.Mesh.Contract) is flat + enum-discriminated — it serialises through the mesh content serializer like any node content (no polymorphic-content risk), and new trigger/continuation kinds are added as new enum values APPENDED at the end + nullable fields (additive). A module's own effect needs no member of its own: it uses the generic DeliverToNode (see "A module's own continuation: DeliverToNode" below):
| Group | Field | Meaning |
|---|---|---|
| Trigger | TriggerType |
NodeChange | Timer | NodeStatus |
| NodeChange | TriggerNodeType, TriggerKind, MatchField, MatchValue |
fire when a node of this type is created/updated/deleted and its field matches |
| Timer | FireAt |
fire once at/after this instant (a past time fires on the next boot) |
| NodeStatus | WatchPath, StatusField, RestingValues, RequireActiveFirst |
fire when the watched node's status enters a resting value |
| Effect | ContinuationType |
GrantSpaceAccess | AddToGroup | PostThreadMessage | PublishSocialPost | RunScript | DeliverToNode |
TargetPath, SubjectId, Role, Pin |
what the continuation does (and to whom) | |
DeliverTo |
[DeliverToNode] the node whose own hub acts on the firing | |
| Lifecycle | Status, CreatedBy, CreatedAt, FiredAt, LastError |
runner-managed (Pending → Fired \| Failed \| Cancelled) |
Trigger 1 — react to a CRUD event (the email invite)
The flagship case: invite someone to a Space by email. If they already have an account, grant now. If not, write an EventSubscription that grants (and pins) the moment a User with that email is created — so access lands automatically on sign-up, surviving any restart in between. This is exactly what SpaceInviteService writes:
var subscription = new EventSubscription
{
// Deterministic id per invitee+space → a re-invite upserts the SAME subscription (idempotent).
Id = $"grant_{Slug(email)}_{Slug(spacePath)}",
TriggerType = EventTriggerType.NodeChange,
TriggerNodeType = "User",
TriggerKind = MeshChangeKind.Created,
MatchField = "email",
MatchValue = email,
ContinuationType = EventContinuationType.GrantSpaceAccess,
TargetPath = spacePath,
Role = "Editor",
Pin = true,
};
EventSubscriptionOps.CreateSubscription(meshService, subscription).Subscribe();
When the invitee onboards, their User node is created → the change feed fires the subscription live; the reconcile path is the safety net if the sign-up happened during downtime. The continuation (GrantSpaceAccess) creates the {space}/_Access/{user}_Access assignment and pins the Space — both idempotent create-or-updates, so live + reconcile can never double-grant. The subject is the triggering node's id (a User node's path IS the userId).
The group twin is AddToGroup (written by GroupInviteExtensions.InviteToGroup / the bulk InviteAllToGroup behind the group's "Invite by Email" dialog): on sign-up it creates the {group}/{user}_Membership node, and — when the invite chose a Role — additionally the {group}/_Access/{user}_Access assignment (groups are not publicly readable, so the grant is what lets the new member see the group; Admin makes them a group manager). Same idempotent upserts, same deterministic id per invitee+group.
Trigger 2 — react to a timer
Fire a continuation at (or after) a time. The runner schedules one Observable.Timer per pending Timer subscription; a FireAt already in the past fires immediately on the next startup — restart-safe at-least-once without any external scheduler. A timer carries no triggering node, so the continuation subject is on the subscription (SubjectId):
var subscription = new EventSubscription
{
TriggerType = EventTriggerType.Timer,
FireAt = DateTimeOffset.UtcNow.AddDays(7), // e.g. grant a trial role in a week
ContinuationType = EventContinuationType.GrantSpaceAccess,
SubjectId = userId, // no trigger node → subject is explicit
TargetPath = spacePath,
Role = "Viewer",
};
Trigger 3 — react to a node reaching a resting status
Fire when a watched node's status field leaves "running" and reaches a resting value — the exact shape of "wait for a reply, then continue". The runner watches the node via the self-healing SubscribeWithReEstablish (re-establishes on a transient fault, and terminally stops without a storm when the watched node is gone), tracks whether it first saw a non-resting (active) state (RequireActiveFirst, so an initial replayed-resting of a node that never ran doesn't fire), and fires when the status enters RestingValues:
var subscription = new EventSubscription
{
TriggerType = EventTriggerType.NodeStatus,
WatchPath = subThreadPath, // the delegated sub-thread
StatusField = "Status",
RestingValues = ["Idle", "Cancelled", "Done"], // "not running any more"
RequireActiveFirst = true, // saw it Executing first → this is a genuine finish
ContinuationType = EventContinuationType.PostThreadMessage,
TargetPath = parentThreadPath, // continue the parent
};
This is the durable backbone of delegation: a parent agent delegates to a sub-thread and continues when the sub-thread finishes. The in-memory wait is the fast path for the same-process happy case — DelegationTool.WaitForDelegationResult, an IObservable<string> folded off the sub-thread's node stream (Scan for "saw it running, now resting" → Take(1).Timeout(...)), bridged to the tool's Task<string> return only at the model-facing tool boundary. The EventSubscription is the reboot backstop — if the portal restarts mid-delegation, the runner reconciles the subscription, sees the sub-thread already resting, and continues the parent. Nothing is lost. (See Thread Operations.)
Asking a question vs. done. A sub-thread that finished a task and one that asked the user a clarifying question both currently reach
Idle— the thread status alone can't tell them apart. TheRequireActiveFirst+ summary-presence heuristic covers the common case. An explicit round disposition (something likeCompleted/AwaitingInput/Failed) would make the resting trigger fire only on genuine completion; it is a proposal, not a type that exists today.
A module's own continuation: DeliverToNode
In-mesh source cannot register a root-DI IEventContinuationHandler, so before DeliverToNode it could not own a continuation at all. DeliverToNode is the ONE generic continuation for that: the runner writes an EventDelivery (subscription id, the subject's id — for a sign-up, the new user's id — the time, and CreatedBy) onto the content of the node named by DeliverTo, as System, through the node's own stream. The node's own hub, which belongs to the module's node type, decides what the firing means. The node must exist when the subscription fires.
new EventSubscription
{
TriggerType = EventTriggerType.NodeChange,
TriggerNodeType = "User",
MatchField = "email",
MatchValue = email,
ContinuationType = EventContinuationType.DeliverToNode,
DeliverTo = "Admin/TierInvites/someone_example_com", // its content declares EventDelivery? EventDelivery
CreatedBy = inviter, // the receiver re-checks this person's right
};
The receiver re-checks EventDelivery.CreatedBy before it acts: the firing comes long after the request was written, and the person who asked may have lost the right in between (policy authorize-as-caller-execute-as-system). The Store's tier invitation (MeshWeaver.Plugins, Store/TierInvite) is the first receiver. A delivered subscription is Fired once the delivery is written; what the module then does is reported on its own node.
How the runner resolves a continuation. The platform's members run in the runner; DeliverToNode delivers (and fails, named, without a DeliverTo); everything else goes to the exactly-one DI-registered IEventContinuationHandler claiming it (PublishSocialPost reaches MeshWeaver.Social that way). A value nobody claims — including a number this build has never heard of — is logged by name and ends Failed with the value on LastError; it is never run as a continuation that means something.
Why an enum member, not open string constants (yet). EventContinuationType is persisted and module-extended, so policy open-vocabulary-string-constants says it should become a static class of const string. That conversion is a BINARY break for modules already compiled against the enum (MeshWeaver.Social's handler declares the enum type; MeshWeaver.Plugins' ContentIndexBackfill calls the enum-typed setter), so it is a planned, coordinated rebuild, not a side effect of a feature. Appending a member is binary-safe: every earlier member keeps its number.
Why one runner, reconciled on startup (not Orleans reminders)
- Durability is already in the node. The subscription is a Postgres-backed MeshNode;
Pending → Firedis the durable checkpoint. The reconcile query gives exact replay forCreatedtriggers and at-least-once for timers/status — for free. - Host-uniform. Orleans reminders/durable streams are not wired (the silo uses memory streams; the monolith has no Orleans), so a reminder-based design would split behaviour between hosts and need net-new reminder storage — more surface, not less. The reconcile substrate runs identically in both.
- No wedges. Every read/write runs under
ImpersonateAsSystem(the runner has no ambient identity); every timer/watch is idempotent per id, self-disposing on fire, and disposed with the runner; the node-status watch terminally stops (never resubscribes) when its node is gone — the anti-storm rule.
Back-compat
EventSubscription generalises the former ScheduledAction (same field names for the NodeChange + GrantSpaceAccess case). On startup the runner migrates any legacy Admin/ScheduledAction/{id} nodes into Admin/EventSubscription/{id} (a lag-robust live query; a failed migration releases the id so a later emission retries) — so no in-flight invite is dropped. ScheduledAction is kept only for that deserialization + migration; nothing creates it any more.
Testing
test/MeshWeaver.Graph.Test/EventContinuationDeliverToNodeTest covers DeliverToNode writing the new user's id onto the module's node, DeliverToNode without a node failing by name, an unknown value failing under its own number, and every earlier member keeping its number. test/MeshWeaver.Graph.Test/EventSubscriptionRunnerTest covers the NodeChange grant (fires when the matching user is created), the legacy migration, the Timer (a past-due timer fires + grants), and the NodeStatus trigger (a watched node flipped Running → Idle fires). SpaceInviteServiceTest covers the invite → subscription write.