The catalog's first frame β€” what it may wait for, and what it may not

/Store on the public instance showed Loading the store… for over two minutes before a single card appeared (#1177). The same page on a local install painted immediately. Nothing was broken: the NodeType compiled, the hub was healthy, the packages were there, and every other area on the same node rendered in about two seconds.

The rule this cost us

🚨 Every arm of the catalog's first-frame join is SEEDED, unconditionally. A feed may take as long as it takes; the page frame may not wait for it. "Still loading" is something a feed says β€” it is never inferred from a clock, and never from the fact that some earlier render registered a query.

What was measured, and what each measurement eliminated

Read-only, against the public instance, 2026-09-02, plus the portal logs from the public instance.

Measurement Result What it rules out
GET /Store (page shell) 200, TTFB 0.18 s the HTTP path, the shell, TLS, routing
get @Store warm instant cold grain activation, hub health
get @Store/area/Plans complete in ~2 s the node, the hub, the process, the NodeType assembly, the cluster
get @Store/area/Catalog timed out at 35 s, three separate attempts β€” this is the defect
get @Store/data/StorePackage 132 packages, instant the package feed, and with it the whole "the GitHub enumeration is slow" theory
StoreManifestSource git polls fail in 2–50 ms (Octokit.AuthorizationException: Bad credentials, all five repos within 50 ms of each other) "the cloud's git sources are live and the first render waits for the fetch"
StoreManifestSource registry poll fails, 503 instance-key resolution same
[CrossSchema] SLOW on nodeType:Store/Plugin 1.0–7.7 s per pass across 196 partition schemas, re-running every 2–5 s nothing β€” this is the cost that was on the critical path

Two things in that table are worth stating plainly, because the first diagnosis of this bug was built on their opposites:

The arm that was not seeded

StoreCatalogLayoutAreas.Catalog joins six arms with CombineLatest, which emits only once every arm has emitted. Five were seeded (isAdmin, installStatus, provisionRun, the grace timer, and β€” via its own StartWith β€” the inner real-node batch). The plugin feed was seeded conditionally:

var warm = IsWarm(hub, PluginsQueryId);          // hub.GetQuery(id) is not null
var plugins = Seeded(ObservePlugins(host), warm); // warm ? feed : feed.StartWith([])

IsWarm asks whether a synced query is registered under that id. It is not a question about data, and three facts turn it into the bug:

  1. MeshNodeStreamCache.GetQueryRaw inserts the stream into its registry before anything subscribes β€” registration precedes the first emission.
  2. The registry entry then persists for the life of the process (Replay(1).AutoConnect(1); the entry is only ever removed by fault eviction). So after the very first render on a pod, IsWarm answers true for every visitor, forever.
  3. warm was also the branch that withheld the seed.

So from the second render onward, the catalog's first frame was gated on ObservePlugins delivering β€” and delivering means a mesh-wide nodeType:Store/Plugin query (1.0–7.7 s across 196 schemas) whose shared snapshot is then handed to each subscriber through WrapWithPerUserRls β†’ FilterByReadPermission, which probes CheckPermission(node.Path, user, Read) once per node in the snapshot, sequentially, and emits nothing until the whole snapshot is filtered. Eighty-seven plugin nodes, each probe potentially its own AccessAssignment scope:Subtree and PartitionAccessPolicy scope:Children fan-out β€” the two most frequent fanned-out queries in the whole fleet log (1,857 and 458 occurrences in ten minutes). That is the two minutes, and the page was blank for all of it.

The warm branch was not careless: it was written to stop a "Nothing published yet" flash on a store that is full. That is a real problem with a better answer β€” see below.

Why this is NOT a location-declaration problem

nodeType:Store/Plugin fans out across every schema because every plugin root is its own partition root. There is no path or namespace to pin it to; the query is legitimately mesh-wide. Declaring an instance location for it would be the same false premise that #1127 was closed on. The fix is not to make the query cheap β€” it is to keep it off the first frame.

What the fix is

The page now paints its frame, hero and category tiles on the first frame; the cards fold in when the feed answers. Nothing about the feed's cost changed.

What is still slow, and where it belongs

The per-user RLS filter on a shared synced query is O(nodes) permission probes per subscriber, serialized ahead of that subscriber's first emission, and the probes themselves fan out across every schema. That is MeshWeaver.Graph.Contract/SyncedQueryDataSourceExtensions.FilterByReadPermission and it is core's, not this repo's. It is filed separately. Until it is cheaper, the rule at the top of this page is what keeps it off a user's screen.

The extension shelf's buy/install row: why it still has no reading (#4399)

The extension shelf (Store/area/Extensions) painted ? cards with no name or description. That half is fixed: ExtensionCard builds its card with MeshNodeCardControl.FromNode (#4330), and a card paints its row caption before its package hub answers (#4805). On the public instance, get @Grok/area/ExtensionCard now carries the package's title, description and icon.

The latency half is still open. On the public instance the row (InstallCta, CoverCta) arrives about one second after its base frame when opened alone, and in clusters of 1.5, 2.0, 3.2 and 5 s when a shelf opens many at once. No test rig reproduces this: not the monolith, not one silo, not two silos, and not the Postgres-backed rig. Core #7740 added the instrument that splits the delay into compose time and the owner's update queue. That instrument is the FIRST FULL FRAME line, written by LayoutAreaHost at Information when an open is slower than Layout:SlowFirstFrameThreshold.

The line could not reach Loki. The portal's Logging:LogLevel is an allow-list (Hosting/Configuration), and MeshWeaver.Layout.Composition.LayoutAreaHost was not on it. The line therefore fell under the "MeshWeaver": "Warning" catch-all and was dropped before stdout. On the instance where the delay was timed, a control Logs action read 0 lines over the 24 h after the image carrying the line had rolled. That window included renders of CoverCta, InstallCta and ExtensionCard on the AI-threads packages. Both image hosts now pin that one type to Information, and ShippedLoggingLevelsGuard holds the pin in place, with a negative control on a sibling category.

Two earlier eliminations do not stand. Each is reopened for a different reason:

Next reading: after this host configuration rolls to the instance being measured, render the AI-threads shelf, or run get @<package>/area/InstallCta on a few of its packages. Then run a control Logs action for FIRST FULL FRAME on that instance's Deployments/<name> record, and check the Logs action names the SAME record the timings came from. The line says whether the second is compose (base β†’ composed) or the owner's update queue (composed β†’ applied).