Payments

This node anchors the Store's payment webhook inbox.

🚨 The payment provider is an OPTIONAL MODULE, and the Store binds an abstraction. Signature verification, event parsing and checkout-session creation live in MeshWeaver.Payments.Stripe, shipped by the Stripe Payments package; what stays on this side is what the Store DOES with a verified delivery. The seam is the platform's IPaymentProvider β€” see The Payment Provider Contract. Its wiring β€” the three configuration keys and this endpoint β€” is unchanged and is written up in the module's guide; the reasoning is in the module design.

🚨 A portal that mounts NO payments module is a portal that does not sell β€” not a broken one. Every surface below answers that as a state: a checkout refuses with a sentence in the viewer's language, the payment-path audit reports out of scope rather than healthy or broken, and a delivery that arrives is LEFT in the inbox (nothing can authenticate it, which is the same recoverable arm as a missing signing secret β€” never a rejection, which would mean a check failed). Before Plugins#1328 the Store's node types named the module directly, and a deployment without it could not compile them at all.

When a delivery cannot be verified

A delivery whose signature does not verify but which names an order is kept at Store/Payments/_Rejected/{id}, with the same id, as a Store/RejectedDelivery node. It is not deleted, because deleting it left a buyer's paid order sitting at PendingPayment with an empty inbox β€” a state indistinguishable from "Stripe never called", which needs the opposite response (#1069).

A Store/RejectedDelivery holds:

Field What it is
delivery the inbox's delivery, verbatim: receivedAt, contentType, headers (the Stripe-Signature header included) and the raw body
rejectedAt when the Store rejected it
rejectionReason why it was kept, in words an operator who cannot read the portal's logs can act on
receivedSignature the signature header that failed (the signing secret is never recorded)
resolution absent while the delivery is unresolved; once recorded: outcome, note, resolvedBy, resolvedAt and the requestPath of the maintenance request that recorded it

🚨 Until Plugins#3563 the retained node was the inbox's WebhookEvent with rejectedAt / rejectionReason / receivedSignature written beside its fields β€” and those three were lost. The stored node is typed, so the store kept only what WebhookEvent declares; the retained node of #3563 said nothing about its rejection except its description, and had nowhere to record that anybody had looked at it. Since #3563 each of them is a declared member of the record's own type. A node retained before that is still a WebhookEvent: it reads as an unresolved entry whose rejection fields are its description and its persistence time, and resolving it converts it to a Store/RejectedDelivery.

So an unresolved entry in _Rejected means: someone may have paid and not been served. Two checks can refuse a delivery, and the provider answers them as one bit:

Compare the signing time in receivedSignature (t=…) with the delivery's receivedAt before you touch the secret β€” changing a correct secret breaks a working endpoint. Then fix whichever check failed and replay the delivery from the Stripe dashboard.

🚨 The replay window is measured at ARRIVAL, never at processing (#3544). The watcher hands the provider the receivedAt the inbox stamped when the request came in (the processing time only when a node carries no stamp). Before that fix it handed over the processing time, so an authentic delivery that waited more than five minutes in _Inbox β€” behind a restart, a backlog, a slow settle ahead of it β€” was rejected whatever its signature, and the rejection named the secret.

Resolving a retained delivery: ResolveRejectedDelivery

Once a person has compared the delivery with Stripe and it needs nothing more, they record why. A global admin creates a Store/Maintenance node under Admin/Maintenance/… with:

Field Value
task ResolveRejectedDelivery
requestedAction Run
deliveryPath the entry's id, or its full path Store/Payments/_Rejected/{id}
outcome why it needs nothing more β€” Replayed (replayed and processed), NotAPayment (an expired, unpaid checkout; a test event), Stale (settled another way), or your own word: the set is open
note what you checked (optional)

The watcher authorizes the request's author as a global admin, then records the resolution as System β€” Store/Payments is a system-owned subtree no person holds Update on β€” with the author as resolvedBy. Nothing is deleted: the delivery and its rejection stay, with the reason beside them. The task refuses, and the request lands Failed naming why, when nothing is at the path, when the node is not a retained delivery, and when the entry is already resolved (the first recorded reason stands; it is never overwritten). "Already resolved" is judged on what the atomic update LEFT, not on the read before it: when two requests both saw the entry open, the one whose update finds the other's resolution in place fails naming it, so neither reports a resolution the entry does not carry. A retry of the same request β€” the control plane re-running a request whose acknowledgement was lost after its update landed β€” is Done, not refused: the standing resolution carries that request's own path (requestPath), so it is this request's act, and nothing is written again (#3609). Identity is the request path, never the instant; each attempt stamps its own.

The payment-path audit counts only unresolved entries (below), so a resolved delivery no longer keeps the Payments (live) row red, and no longer hides a new rejection behind an old one.

Two deliberate limits:

_Rejected is covered by the same Public/Anonymous deny as the rest of Payments (CatalogGate.SensitiveSegments): readable by a Store admin, never public.

When a delivery is held β€” re-driven by what it waits on, never by a restart

An AUTHENTIC delivery cannot always be acted on the moment it is read. Its order or coupon claim may not be visible yet (the index trails the store), or acting on it may fault (the owning hub is being re-activated, a write times out). Before MeshWeaver.Plugins#2064 such a delivery stayed in _Inbox until the inbox query's next Initial or Reset, which in practice meant the Store hub's next activation. A buyer who had paid waited for a restart.

Every authentic delivery now names a prerequisite (PaymentInboxWatcher.PrerequisiteOf):

Delivery Prerequisite Must exist first?
a checkout (completed or expired) the order it names yes
a payment-setup delivery the coupon claim it wakes yes
a renewal, failed charge, ending or renewal change the buyer's plan record (Subscriptions.RecordPath) no: the activation creates it

Existence comes first, from a listing (PaymentInboxWatcher.ReadPrerequisite). The listing is the same one the hold's watch uses: run as System, with the System viewer stamped on the query, so the gate and the watch always agree about a private order or claim. Both use MeshQueries.ObserveNodeAnswered, which says nothing until the query has answered, so an unrelated change to one of the node's descendants that arrives before the query's first snapshot is never read as "absent". A node the listing shows is then read from its owner, and the owner's version becomes the hold's baseline. If the listing shows a node that its owner does not have (a stale index row of a deleted order or claim), that is a fault hold, never confirmed absence. Otherwise the same stale row would read as an appearance and re-drive the delivery in a loop. A point read of an absent path faults and opens the storm breaker on it, which is why existence is never decided that way. A NO is a hold waiting for the prerequisite to appear. A fault (of that read or of the work behind it) is a hold carrying the fault. Both arrive as one PaymentDeliveryHeldException (PaymentInboxWatcher.Gated). A delivery that has not authenticated is never held. It is rejected, discarded or left exactly as before.

A hold is re-driven reactively (PaymentInboxWatcher.Drain). It subscribes to the prerequisite's path through the existence-safe query transport (MeshQueries.ObserveNode), as System AND with the System viewer stamped on the query, because an ambient identity can be lost across the query scheduler hop and a private order or claim would then read as absent.

The attempt reads the prerequisite BEFORE it acts, and when it faults it reads it again, so the attempt's own writes are part of what it saw. That reading is the hold's baseline, and the hold is re-driven by the first observation that is NEWER than it:

"Newer", never merely "different": the watch reads the query index, which can trail the owner, and a trailing index must not look like a change.

A hold is released when any later attempt of the same delivery STARTS, so a Reset queued behind a pending attempt cannot leave a hold behind that re-enqueues a delivery that has since completed.

There is no timer-based retry and no polling. Nothing re-attempts on a clock, except for ONE last attempt at the hold's deadline, described next. The re-driven attempt goes through the same one-at-a-time Concat as every inbox delivery.

A hold is bounded and ends loudly. The deadline is the delivery's ARRIVAL (receivedAt, which is durable, so a restart neither extends a hold nor restarts it) plus HoldLifetime (24 hours). At the deadline the delivery is attempted once more. An attempt that is held at or after its deadline is dead-lettered:

The entry is unresolved, so the payment-path audit counts it, and ResolveRejectedDelivery (above) records how it was settled. The audit reports these entries in a finding of their own (PaymentPathAudit.UnresolvedDeadLettered), naming what each one was held on. They never count as deliveries that did not authenticate, because that finding sends the operator to the webhook secret. heldOn is what tells it apart from a signature failure: this delivery authenticated, and the webhook secret is fine. Fix what it waited on, then replay the delivery from the payment provider's dashboard. If the retention write itself fails, the delivery stays in _Inbox rather than being lost.

Nothing is processed twice by accident, and processing twice is harmless anyway. A path holds at most one registration. A newer hold replaces it. A re-drive removes it before it re-enqueues the delivery. The inbox's own snapshot of the same delivery (a Reset) supersedes it. Underneath, every handler is idempotent:

The replay window is still judged at ARRIVAL (#3544): a re-driven attempt hands the provider the delivery's own receivedAt (PaymentInboxWatcher.ReadAtArrival), so a delivery re-driven hours later authenticates exactly as it would have on arrival.

One deliberate difference from before: a setup delivery whose claim cannot be found used to be deleted at once, as a wake with nothing to wake. It is now held, because "not found" can equally mean "not indexed yet". A claim that really was deleted therefore dead-letters after a day. That is a harmless entry: the claim's own control plane re-reads its setup on its next emission in any case. Resolve it as Stale.

The holds live in the Store hub's drain subscription and are never static. A re-opened drain starts with none, and its Initial snapshot re-attempts whatever is still in _Inbox. A delivery left because this portal has no provider or no signing secret is not a hold. It needs configuration, which is read at start, so it still waits for a restart.

Pinned by Store/Catalog/Test/PaymentInboxRedriveTests.cs.

What "fulfilled" means

Fulfilment is entitle β†’ stamp β†’ install, in that order and in one chain (OrderFulfilment), shared by both writers β€” the verified webhook (PaymentInboxWatcher.Fulfill) and the amount-zero branch of OrderControlPlane. The buyer gets the entitlement and the package's material copied into their own space (DependencyInstall.InstallAfterAcquisition β€” dependencies leaves-first, then the package into {buyer}/{packageId}), so a card payment ends the way a coupon redemption always did.

When Stripe never calls at all β€” the alarm, and how to read it

Everything above assumes a delivery arrives. The failure mode that took days to identify (#1109) is the one where none ever does: a Checkout session completes at Stripe, no webhook is sent, and on this side the order stays at PendingPayment with both _Inbox and _Rejected empty. Nothing errors, nothing logs, and every surface on the mesh reports health. The only way anyone found out was by buying something and finding nothing.

Two configurations produce exactly that picture, and neither is visible from the mesh:

  1. No webhook endpoint registered at Stripe for this instance's URL β€” including one registered in the other mode. Test and live endpoints are configured separately, so a live-mode endpoint never fires for a sk_test_… checkout, and vice versa.
  2. The portal does not allowlist the target β€” WebhookInbox:Targets is fail-closed, so POST /api/hooks/Store/Payments answers 404 and stores nothing, however correct the Stripe side is.

Store/Maintenance β†’ PaymentPathAudit measures both, as cases in that type's Tests area β€” the same operator health board ModulePinAudit runs on. The same secret key that creates Checkout sessions can read the account's webhook endpoints (GET /v1/webhook_endpoints), and β€” the property that makes the check answer the real question β€” a key lists only the endpoints of its own mode. So a sk_test_… key asking "is there an endpoint for my URL" is asking "is there a TEST-mode endpoint for my URL", which is precisely the unknown #1109 could not close from this side.

Three states, never two

Verdict What it means Rendered
Delivering measured: an enabled endpoint for {Commerce:BaseUrl}/api/hooks/Store/Payments, subscribed to every event pass
NotDelivering measured: absent / registered in the other mode / disabled / missing an event fail, naming the mode, the URL, what the account does list, and the remedy
Undetermined cannot tell β€” Stripe unreachable, a 401, a truncated listing, or no Commerce:BaseUrl to compare against fail

🚨 The remedy follows the failure (Plugins#2789). Only an ABSENT endpoint (none for this URL in this mode β€” including one registered only in the other mode) gets the provider's remedy: add an endpoint, store its signing secret, restart. An endpoint that EXISTS is repaired, never replaced: a disabled one is re-enabled, and one that lacks events is EDITED to list every required event (an update may replace the event list rather than add to it). Both keep the endpoint's signing secret, so nothing changes on the portal and no restart is needed β€” and the line says not to add a second endpoint or replace the stored secret. It used to print the add-an-endpoint remedy for all three, and followed against an endpoint that only lacked invoice.payment_failed that remedy registered a second endpoint for the same URL (every event delivered twice) and swapped the stored secret, so every delivery from the existing endpoint failed verification and landed in Store/Payments/_Rejected. The wording lives in the audit (PaymentPathAudit.MissingEventsRemedy, DisabledEndpointRemedy), not the provider's single PaymentDeliveryPath.Remedy, so the core contract did not change.

Undetermined failing is the point. An unreachable Stripe, a 401 and an account with genuinely no endpoint all answer "nothing found"; only keeping the third state separate tells them apart, and a health check that cannot is the defect, not the fix. (Same rule as ModulePinAudit: a source that cannot be READ is not a clean bill of health.)

A fourth value, NotSelling, is out of scope rather than health: a portal with no Commerce:Stripe:SecretKey creates no Checkout sessions, so it has no payment path that can be broken. It says so in those words and never reads as "payments work" β€” and it is why the audit reaches nothing at all (no HTTP call, no mesh read) on a CI mesh.

It also sees the deliveries that DID arrive

On a portal that sells, the same run reports:

Each container is read with a row budget (MeshQueries.SnapshotQueryLimit, #3609): _Rejected keeps every resolved delivery for good, so an unbounded, content-bearing read grew with every rejection the instance ever saw. A listing that FILLS the budget is reported as a finding of its own β€” the counts above it are a floor, and an unresolved delivery may be among the unread rows.

Ages, never wall clocks: a duration is the same number in every time zone, so the report cannot show container UTC to a viewer in ZΓΌrich.

The demanded set and the documented set are the SAME set

The events in the first bullet of this page are not prose beside the code β€” they are the set the audit measures an endpoint against, and a test holds the two together (StripePaymentProviderTest.TheDemandedEventsAreTheOnesTheRunbookTellsAnOperatorToConfigure): this document is embedded in the module's test assembly, the events are read out of that one bullet β€” the line beginning - **Stripe endpoint**:, the line that instructs the configuration β€” and the result must equal what the provider demands. Reword that bullet away and the guard reds for a missing anchor rather than passing on a line it could not find.

That guard exists because they drifted. The demanded set carried only the two checkout events from the self-serve plan checkout landing until Plugins#1530 β€” through two rewrites of the array, with every test green, because the only coverage asserted the array against a literal copy of ITSELF. The consequence was not academic:

Adding an event to this page without adding it to the provider (or the reverse) now reds. Whatever the demand becomes, the operator is told the same thing.

What was actually measured β€” the #1109 record

The definition of done for #1109 is one test-card purchase reaching Fulfilled with installedAt set on the public instance. It is not met, and nothing below claims it is. What is settled is why, and the answer is not in this repository.

The denominator. Every purchase attempt anyone has recorded against this instance, and how far it got:

when order got to fulfilled
2026-08-03 sglauser/_Orders/order-e9d4535a… (cs_test_a1YeWyz…) PendingPayment no
2026-09-01 sglauser-private/_Orders/order-14887283… (cs_test_a1FNcrqr…, 900 CHF) PendingPayment no
2026-09-02 sglauser-private/_Orders/order-bdcf93f6…, paid at the provider after the #1069 deploy PendingPayment no

Store/Payments/_Inbox and _Rejected were read directly (not through the search index) on every one of those days and on 2026-09-08: empty every time.

🚨 An empty container is not by itself proof that nothing arrived, and this record must not be read as if it were. Both containers are drained by design β€” a processed event is deleted, and before #1069 a REJECTED one was deleted too β€” so "empty" is exactly as consistent with a delivery that came and went as with one that never came. That indistinguishability is #1069. What the observations do establish, each from something other than an absence:

established by
No delivery has ever been processed successfully a processed checkout.session.completed stamps its order Fulfilled and writes an entitlement. No order has ever left PendingPayment, and no entitlement exists.
No delivery has arrived since 2026-09-02 from the #1069 deploy onward an unverifiable delivery naming an order is RETAINED at _Rejected/{id}. A purchase completed at the provider after that deploy produced nothing there.
None can arrive today the 2026-09-08 audit measured no test-mode endpoint for this instance's hook URL at all.

What is not established, and cannot be from this side: whether some delivery arrived before the 2026-09-02 deploy and was silently dropped by the deleting watcher. The endpoint listing is a reading taken on 2026-09-08 β€” it says what the account holds now, not what it held in August. Nothing turns on the difference (the remedy is the same either way), but the claim "nothing has ever been delivered" is a stronger one than the evidence carries, and this page does not make it.

🚨 The order counts are what the mesh could be made to SHOW, not a census. Orders live in each buyer's own partition, so an identity without access to that partition reads zero β€” and reads it identically whether there are no orders or merely none it may see. Anyone re-establishing this number must say whose identity took it.

The exact step that does not complete. Not "the webhook" β€” the subscription:

PaymentInboxWatcher.AddPaymentInboxWatcher
  β†’ mesh.Query<MeshNode>("path:Store/Payments/_Inbox scope:children")
      .Where(change => change.ChangeType is Initial or Added or Reset)   ← never emits an Added

Nothing arrives, so ProcessEvent never runs, so Fulfill never runs, so OrderFulfilment.Complete's stream.Update(… Status = Fulfilled …) is never subscribed and the order stays where the buyer left it. Every step downstream of that emission is exercised and green β€” the fulfilment chain is covered live by "Paid (live): a card payment INSTALLS, not just entitles" β€” so nothing between the delivery and the entitlement is what is broken.

Everything upstream of it has been excluded, in order β€” each by a positive observation rather than by an absence. The signature computation is pinned to a vector computed outside this codebase (#1154). The target IS allowlisted (POST {} to the hook answers 200; a real node that is not allowlisted answers 404). A signing secret IS configured β€” the probe delivery that 200 stored was discarded, and Decide reaches Discard only when a secret is present. A wrong secret is out, because since #1069 that RETAINS a node under _Rejected/{id} and a purchase completed after that deploy produced none. That leaves one arm, and on 2026-09-08 the audit measured it:

The payment provider lists NO webhook endpoint at all for this instance's hook URL in TEST mode β€” the mode this portal's own credential charges in, and the only mode whose endpoints can fire for its checkout sessions.

So the remaining step is an account action, and it is the whole of what is left. In the provider's dashboard, in test mode: add an endpoint for {portal}/api/hooks/Store/Payments subscribed to every event named at the top of this page; put its signing secret in Commerce:Stripe:WebhookSecret; restart the deployment (the vault is read only at pod start); then replay the already-paid session β€” fulfilment runs off the session metadata and is idempotent, so an existing order completes without a second payment.

The audit never writes: it lists endpoints and reads two nodes, and it is rendered casually and runs as a CI gate. Registering a missing endpoint, or adding an event it lacks, is a separate, deliberate act β€” the EnsurePaymentEndpoint maintenance task, run by a global admin β€” so the audit's verdict stays a reading, and the write happens only when someone asks for it (policy payment-endpoint-governed-task, Plugins#1881).

Registering the endpoint: EnsurePaymentEndpoint

A global admin creates a Store/Maintenance node under Admin/Maintenance/… with task: EnsurePaymentEndpoint and requestedAction: Run. The watcher authorizes the invoker as a global admin and runs the task; its summary lands on the node.

  1. Dry run first (the default). Without execute: true the task asks the provider for its plan β€” create, or add which events, to which endpoint β€” and writes nothing.
  2. Execute β€” in a NEW request. A dry run lands its node Done, and a Done request does not run again (only a Failed one is retried by setting requestedAction: Run). So create a second Store/Maintenance node with the same task and requestedAction: Run (without the action a request is never picked up), plus execute: true and confirm set to the mode this portal's key charges in (test or live). A missing or different confirmation is refused, so an instance holding the wrong key is never wired to the wrong account.
  3. It acts only while the payment-path audit reads NotDelivering. Delivering is an answer (nothing to do); Undetermined β€” an unreadable or truncated listing β€” is a refusal, because it cannot prove an endpoint absent or incomplete. One exception: the audit reads Delivering off ONE endpoint for this URL, so a listing that also shows a second endpoint for it, or an endpoint for another host, is handed to the provider, which refuses it by name (rule 6 below) β€” the task never reports "nothing to do" over a doubled or shared account.
  4. No endpoint for this URL β†’ create. The idempotency key is written to the node before the create is sent, so a run killed after it and retried replays the provider's original answer instead of creating a second endpoint. The endpoint carries this instance's URL (from Commerce:BaseUrl, never a value typed on the node), every required event, and the instance and request node in its metadata. Its signing secret is shown once, in the node's PaymentEndpointSecret area, to the person who ran the task, and stored nowhere β€” not on the node, in its log or in any log line. Paste it into the deployment record page's Set Key Vault secrets dialog as Commerce:Stripe:WebhookSecret, then Restart the deployment. The secret lives only in the memory of the request node's hub: a recycle of that hub before you store it loses it for good (the request has recorded its endpoint, so a re-run replays nothing) β€” then run DeregisterPaymentEndpoint and a NEW EnsurePaymentEndpoint request.
  5. An endpoint exists but lacks events β†’ edit it. The task sends the endpoint's current events plus the missing ones (Stripe replaces the list rather than appending), and reads the answer back. The signing secret is kept: nothing changes on the portal and no restart is needed. This is the remedy of Plugins#2789, performed instead of printed.
  6. Refused, naming why: two endpoints for one URL; endpoints for OTHER hosts in the same account (every endpoint receives every event, so a shared account exposes each instance's buyers to the others); a disabled endpoint (re-enabling is a person's act at the provider); a portal that sells nothing or declares no public base URL.
  7. Outcome UNKNOWN, never "nothing written". A write that was SENT and got no answer β€” a timeout, a dropped connection, a 5xx β€” may have been applied by the provider, so the run reports Indeterminate (core PaymentEndpointOutcome), names the call, and stamps the node Failed with "Outcome UNKNOWN β€” the provider may have applied the change". Only a 4xx is a definite rejection, and only that is a Refused (nothing written). A removal interrupted midway names the endpoints it had already deleted. The way out depends on the call, and the refusal names it: an interrupted CREATE is re-run as the SAME request, executed (requestedAction: Run on the Failed node), which replays the create under its recorded idempotency key β€” past the audit, which by then may read Delivering β€” so the signing secret is recovered rather than lost when the create had succeeded and only its answer went missing. The provider replays a recorded FAILURE under that key too (a 5xx included), so a re-run that ends the same way is not run a third time: a dry run in a NEW request shows the listing, and the way on is a NEW EnsurePaymentEndpoint request (a fresh key) β€” after DeregisterPaymentEndpoint when the listing shows this instance's endpoint; an interrupted UPDATE re-run adds only the events still missing; an interrupted DEREGISTRATION re-run deletes whichever of this instance's endpoints remain. A dry run to see the account's listing (the truth) goes in a NEW request, so it does not consume this one. A request that has RECORDED an endpoint never creates another under its key: if that endpoint is gone, the run refuses and asks for a NEW request, because re-sending the key would replay the deleted endpoint's answer.

DeregisterPaymentEndpoint (same gates, no audit precondition) deletes only the endpoints whose metadata names this instance, and reports any other endpoint for the URL as untouched. It is the way out of the named state "endpoint exists, secret not recovered" β€” a create whose answer was lost past the task's 23-hour replay window (StripeEndpointRegistration.ReplayWindow, an hour inside Stripe's 24-hour key lifetime) β€” followed by a NEW EnsurePaymentEndpoint node. The request that hit that state records it (endpointSecretLost) and stays Failed: re-running it answers the same failure and sends nothing, because its endpoint is by then listed and complete and a fresh reading would report "nothing to register" over a secret nobody holds.

The task is the portal acting on a person's explicit request with the key it already charges with; no agent calls the provider with an account key, and a CI mesh (no key) never reaches a call.

What it is careful not to do