Prompt caching across providers

An agent thread re-sends its whole prefix on every model call: the tool definitions, the system prompt, and the history. A long bug thread does that tens of times per round and for dozens of rounds. Prompt caching lets the model provider keep that prefix and charge a fraction of the input price when the same bytes come again.

Every model-call path

Caching is a property of the WIRE, not of the caller. Every caller — an agent round, the triage classifier, the pull-request intake and fixer threads, the agent dispatch, the review ensemble's dedupe call, node-description and pull-request-description writers, the icon generator, the inbox triage — gets its chat client from an IChatClientFactory, and the factory picks one of the wires below by the model's provider. Core (MeshWeaver) makes no model call of its own. Embeddings are out of scope.

Every wire's read count ends up in the same place: UsageTokens.SplitCache reads both the typed CachedInputTokenCount and the AdditionalCounts keys, and the round stores the result on the response cell (ThreadMessage.CacheReadTokens), the token-usage records, the round's cost and each model call's ModelCallTiming.CachedInputTokens.

Direct Anthropic and Claude on Azure Foundry

AnthropicChatClient speaks the Messages API itself, so it places the breakpoints explicitly — three of the four a request may carry, all {"type": "ephemeral", "ttl": "1h"}:

1. the last tool, which caches the tool schemas (the provider renders tools → system → messages);
2. the system prompt, which caches tools and the agent's instructions;
3. the last block of the last message, a rolling breakpoint. It caches the whole conversation so far, so the next tool-loop step within the same round reads the history from the cache and pays only for what was appended. It does not carry over to the next round: earlier rounds are rebuilt from stored text (without their tool_use/tool_result blocks or the per-turn clock), so the next round's history no longer matches the cached bytes and is written afresh; across rounds only the static prefix (markers 1 and 2) is read from the cache. Making the history reusable across rounds is tracked separately. A message held as a bare string is turned into a one-block array first, because only a block can carry cache_control. Moving the marker forward does not change the cached bytes, and the provider looks back up to 20 blocks from the new marker for the previous entry; one tool-loop step appends two.

Before this, only the first two were placed, with the default five-minute lifetime: the static prefix was cached, but every call re-paid the entire history at the full input price, and a thread resting more than five minutes lost even the static prefix. The one-hour lifetime follows the same reasoning as the OpenRouter policy below. All three markers carry the same lifetime, so the rule that a longer lifetime must come before a shorter one holds by construction. Pinned by AnthropicPromptCacheShapeTest.

OpenRouter

This package does two things on every OpenRouter request so that the cache is written once and then read.

What is sent

Both are added only for an OpenRouter endpoint (openrouter.ai or a subdomain, such as eu.openrouter.ai). Another OpenAI-compatible gateway may reject an unknown top-level field, so it gets neither. A request that already carries its own cache_control keeps it.

One hour, not five minutes

The default lifetime of a cached prefix is five minutes. Our threads rest longer than that between rounds: a bug-pool thread waits 20 minutes, and a person reads before answering. With five minutes every such round paid the whole history again. With one hour:

• a cache write costs 2× the input price instead of 1.25×, but only for the part that is new this round;
• every read within the hour costs about 0.1× and starts the hour again, so a thread worked at least hourly keeps its prefix.

The hour holds only on the provider that wrote the cache. The OpenRouter session that keeps a thread on that provider expires after 10 minutes without a request. A thread that rests longer, like the 20-minute bug-pool wait, is routed afresh on its next round. For a model served by one provider nothing changes. For a model our EU routing allows on several providers (Opus on Bedrock eu-west-1 and on Vertex europe), that round can land on the other provider. It then pays the full history at input price plus a fresh write there. So the saving above is certain only for rounds less than 10 minutes apart, or for single-provider models. Beyond that it depends on which provider the round lands on.

One session per thread

The id is derived from the thread's path (ProviderSession.For). It is the same after a restart, a roll or a change of model, so nothing has to be stored for it to work. It is a hash, so no user name, issue title or space name reaches the gateway. It is still recorded on the thread (Thread.ProviderSessionId) the first time a round runs, so OpenRouter's activity log can be matched to the thread.

The id travels from the round (AgentChatClient) to the client in the round's ChatOptions, under ProviderSession.OptionsKey. The round puts it there only when its chat client answers IProviderSessionClient, which only an OpenRouter session wire does. Any other provider (Anthropic, Ollama, Copilot, a non-OpenRouter OpenAI endpoint) never sees the key. The wire still takes it off the options before the request is built, so the internal key never reaches any wire.

Without the id OpenRouter falls back to a hash of the messages, which a growing thread changes every round. A model that our EU routing allows on several providers, such as Opus on Bedrock eu-west-1 and on Vertex europe, could then land each round on a different provider, whose cache holds nothing of the thread.

What each upstream provider does

The table is OpenRouter's documentation as read on 2026-10-03. Prefixes below a model's minimum (1,024 to 4,096 tokens) are not cached, without an error. Caches are separate per model, so a thread that moves to another model starts with an empty cache there.

What cannot be done

The cache is the model provider's, keyed by the exact bytes of the prompt. It cannot be read out, stored by us, or written back later. Sending the same prefix again is the only way to create or refresh an entry. A cache we own is possible only for a model we serve ourselves.

Not used yet

• cache_control on individual blocks on the OpenRouter wire: up to four breakpoints, for example a fixed one at the end of the shared system prompt besides the automatic one at the end of the history. (The direct Anthropic client already places them, see above.)
• OpenAI's prompt_cache_key on the OpenAI-direct and Azure OpenAI wires: a routing hint, like OpenRouter's session_id, that keeps requests sharing a prefix on the same cache machine. Caching there is automatic without it; the session id only reaches a wire that answers IProviderSessionClient, which today is the OpenRouter wire alone.
• OpenAI's own prompt_cache_options (mode and lifetime) and prompt_cache_breakpoint on GPT-5.6 and later.

Is it working

The response reports cached_tokens and, where the provider writes, cache_write_tokens. Microsoft.Extensions.AI hands the read count to the engine as the typed UsageDetails.CachedInputTokenCount, and UsageTokens.SplitCache turns it into the cache-read subset that the round stores on the response cell (ThreadMessage.CacheReadTokens), on the token usage records and in the round's cost.

Each model call in the round's timing ledger (ModelCallTiming) also records:

• CachedInputTokens: the part of that call's input served from the cache;
• ResponseId: OpenRouter's generation id (gen-…). GET /api/v1/generation?id=… on the account then names the upstream provider, the cached tokens and the charged cost of that one call.

A long tool loop whose calls all show no CachedInputTokens is paying its whole prefix every call.

On the direct Anthropic client the write count IS read (cache_creation_input_tokens, carried as CacheCreationInputTokens), so a healthy loop shows each call reading everything up to the previous call's rolling marker and writing only what the last step appended. A write near the full conversation size on every call means the prefix changes upstream of the marker.

cache_write_tokens is not read yet: the OpenAI SDK does not model that OpenRouter field. Writes are therefore priced as plain input in our cost figures, which for an Anthropic one-hour write (2×) slightly understates the bill.

The October 2026 under-reporting

Until 2026-10-20 SplitCache read only the AdditionalCounts key that older adapters used (InputTokenDetails.CachedTokenCount). Microsoft.Extensions.AI 10 moved the count to the typed property, so every OpenAI-wire route (all OpenRouter models, Claude and GLM alike) recorded zero cache reads. The gateway was caching all along. Measured live on eu.openrouter.ai through the engine's own client stack: from the second call on, GLM-5.3 read 8,512 of 8,592 prompt tokens from the cache and Claude Haiku 4.5 read 7,707 of 7,710. Because the reads were missing, the token counts shown on threads were the full prompt totals with no cache line, and every cost figure priced the cached prefix at the full input rate. The provider bill was lower than what the engine showed.