One provider key per function

The incident it answers. One OpenRouter key served every model call of the control instance — reviews, bug-fix attempts, triage and chat — under one $900/day limit. Dispatched work spent it, OpenRouter answered 403 Key limit exceeded (daily limit), and every review round was refused with it until the next UTC midnight: no pull request could pass its gate. A deployment can now give each function its own key, so a function's limit is its own.

The functions

CredentialFunctions — an open vocabulary of string constants, not an enum:

Function Whose rounds (the lane the shared admission seated them on — AI/AgentAdmission)
Review reviews — the internal review (internal-review, the PR steward's review rounds and the review ensemble's halves)
Dispatch bugfix, express (a red main), babysitter (the babysitter, the PR fixer, platform builds), triage — and every sub-thread of those rounds, which the admission seats on the parent's lane
Chat interactive — a person's own chat, and any round whose lane is unknown: the default

A deployment re-maps a lane with AI:CredentialFunctions:Lanes:{lane} (env AI__CredentialFunctions__Lanes__babysitter=Babysitting) and may name functions of its own; a function with no key of its own uses the shared key.

The configuration keys

Key env (pod) What
OpenRouter:Review:ApiKey OpenRouter__Review__ApiKey the key the review rounds spend
OpenRouter:Dispatch:ApiKey OpenRouter__Dispatch__ApiKey the key dispatched work spends
OpenRouter:Chat:ApiKey OpenRouter__Chat__ApiKey the key chat (and everything else) spends
OpenRouter:ApiKey OpenRouter__ApiKey the shared key — unchanged; every function WITHOUT a key of its own uses it

The same {Section}:{Function}:ApiKey shape works for any provider section; only OpenRouter's usage is read on the status page.

How a key gets there. Create the keys in OpenRouter (one per function, each with its own daily limit), then enter each through the instance's write-only Set Key Vault secrets… on its Deployments/<id> record page and declare it in the record's keyVaultSecrets.secrets by its env key ({ "key": "OpenRouter__Review__ApiKey" } — the vault object follows the fleet naming rule, <prefix>OpenRouter-Review-ApiKey). Nothing here is ever typed into Key Vault by hand. On the next boot ProviderCredentialSeed carries each one onto the provider node, encrypted at rest, as ModelProviderConfiguration.FunctionKeys[{Function}] on Provider/OpenRouter (the EU route, Provider/OpenRouterEU, borrows them through credentialFrom exactly as it borrows the shared key). Until a function's key is entered, that function keeps using the shared key — nothing breaks before the keys exist.

🚨 A function key converges on its configured secret. The shared key is administered on the node (Settings → Models) and configuration only fills it when absent. A function key has no node-side editor, so its one administered home is the vault secret: a changed secret IS the rotation, and the next boot re-writes the node (ProviderCredentialSeed.DecideFunctionKey). A function the configuration stops naming is LEFT on the node — removing a key because a pod booted without its secret would silently move that function back onto the shared key. To retire one, set its secret to the shared key's value.

Adopting it on an instance: object first, declaration second

A record must never declare a vault object the vault does not hold yet: the whole CSI mount fails and the next pod never starts (Memex#202). Adoption is therefore three ordered steps, never one change:

  1. Write the objects. On the control instance, open the instance's Deployments/<id> record page, choose Set Key Vault secrets… and paste each function's key into a NEW object <prefix>OpenRouter-<Function>-ApiKey. The values are write-only.
  2. Confirm them. Run a SecretStatus action that NAMES the new objects in its secretObjects (<prefix>OpenRouter-<Function>-ApiKey, one per function). Do not continue until it reads every one of them as present and enabled. A status action that names no objects reads only what the record already declares (InstanceActionPlan.SecretScope), so it never sees an object from step 1 and cannot confirm it.
  3. Declare them. The record's keyVaultSecrets.secrets maps OpenRouter__<Function>__ApiKey to each object. The record-change Reconcile then rolls the instance onto them.

A function with no key of its own keeps the shared key, so declaring only some of the functions is safe. A lane can be given a function of its own by configuration alone. For example, the control instance runs triage under Triage (AI__CredentialFunctions__Lanes__triage=Triage, Memex#899) rather than Dispatch, so a triage key (OpenRouter__Triage__ApiKey) holds only triage when it is spent. Until that key exists, triage falls back to the shared key, never to the Dispatch key. The declaration half for control is Memex#900, which stays a draft until step 2 has passed.

How a round is served its function's key

  1. The shared admission seats the round on a lane (ThreadDispatchPool); ThreadExecution turns that lane into the round's function (CredentialFunctions.OfLane) and sets it on the round's chat client BEFORE the agents are built (AgentChatClient.SetCredentialFunction).
  2. The agent build hands it to the provider factory as an ARGUMENT (ChatClientAgentFactory.CreateChatClient(AgentConfiguration, string?) — never instance state: a factory is a singleton concurrent rounds share) and the factory resolves the key OF THAT FUNCTION: ChatClientCredentialResolver.ResolveForFunction(model, function) — the provider's own key for the function, then (for a route) the referenced provider's, and only then the shared key. CredentialResolution.Function says whether a function key served; CredentialIdentity names the credential (Provider/OpenRouter#Review, or Provider/OpenRouter for the shared key). The OpenAI-wire factory (OpenRouter) honours it, including its configuration fallback while the catalog is still cold.
  3. A client built outside an agent's round (a utility call) has no function and uses the shared key.

The admission closes ONE key, not the provider

A refusal no load change cures — a key's daily limit, credit, a rejected key — used to close every pool of the provider until midnight. With separate keys that would still couple the functions: a spent Dispatch key would close the very pools the reviews run in. So each round's seat carries the credential it runs with (Seat.Credential), and such a refusal closes that credential (AdmissionState.ClosedCredentials): its rounds wait (or move to an alternative on another key), ONE probe goes through when nothing runs on that key, and a clean probe reopens it — while every other key's rounds go on starting on the same provider and the same models. A closed key with work waiting is a provider-closed starvation signal. With one shared key (no function keys) every lane carries the same credential, so the refusal still holds every lane — the coupling only separate keys remove. When the credential cannot be told, the old rule applies: every pool of the provider closes.

Where the spend is shown

The Daily budget & health section (/Provider/AiProviders, platform admins only — Per-provider daily budget) lists, beside each key holder's row, one row per function key: OpenRouter 🔑 Review, with the key's own limit, today's usage and what remains as OpenRouter reports them (GET /api/v1/key → limit, usage_daily, limit_remaining, read with that key, every five minutes while the section is open). A spent key, or one whose usage cannot be read (the HTTP status, never the key), is red.

What this does not establish

Where the code is