Document ingestion

Step 3 of the document-ingestion end vision (core Doc/Architecture/DocumentIngestionEndVision): gaps G7 (InitializeRequest), G9 (exactly once per content) and G15 (chunks and the vector index in the owning partition). It runs on a storage connection. Where the resulting index lives is the declared Indexing:Backend option — PostgreSQL by default (Index backend).

The flow

InitializeRequest ──▶ {P}/_Ingest/{name}            StorageConnection   Active/Done → Ingesting (one run)
                                                                         Attached/Parked → remembered (step 2)
      walk {root} ──▶ claim {P}/_Ingest/{name}/{sha32}  IngestDocument  Claimed   (or link the path onto it)
                         its own hub ──▶ Parsing ──▶ extract ──▶ index + _DocumentPart nodes ──▶ Indexed (parseCount + 1)
 every document terminal ──▶ the connection reads Done
Rule How it holds
One run per request burst IngestionRuns.Initialize is pure and idempotent; the connection's hub applies requests one at a time, so of N concurrent requests exactly one starts a run.
Never fails without its table On an Attached/Parked connection the request sets initializePending; when the provisioning write resumes the connection, StartIfPending starts the run in the same update.
Identity is the content The document node id is the first 32 hex digits of the original's SHA-256. A second path with the same bytes is LINKED (paths), never parsed.
The claim is the creation Creates are serialized on the mesh's node-CRUD hub; a claim that loses the race links instead.
The document's hub owns "parsed" Claimed → Parsing happens inside that hub's own node update (IngestDocumentNodeType.BeginParse); only the activation that made the transition parses; parseCount moves only on completion. A Parsing left by a dead activation is taken over once by the next one.
A written node is not a live hub The walk reads each document after claiming it, and the connection's hub wakes every unfinished document once per activation — so a run survives a recycle.
Originals are read-only The document hub reads the bytes through the connection's store with a read-only collection and checks the SHA-256 still matches the claim.
Only an identity holding Update on the partition starts a run InitializeRequest reads CheckPermissionOutcome(partition, caller, Update); denied and undetermined both refuse, naming why. The root is plain path segments only (., .., \, : refused), so a walk never leaves the container.
A run cannot wedge An unreadable or over-size file is skipped, counted and named with its reason on the connection's log (coverage); a walk that fails as a whole returns the connection to Active, so the next request starts a new run. Every IO edge of a parse is bounded: opening the container and reading an original (2 min, one budget over both), extraction (5 min), each chunk's embedding (1 min — that chunk is unindexed with the timeout as its error), the vector-index write (2 min — no chunk counts as indexed) and each part node's write (1 min). Past the read, the extraction or a part write, the document is Failed — never left in Parsing on a live hub, where no takeover would ever fire. A failure of the hub's own begin or record write ends that parse, never the document watcher: it is written as Failed, and only if that write is refused too does the document stay in Parsing (logged at Error) until its next activation takes it over. Failed is terminal for the RUN, not the content: the next run that finds the content claims it again (IngestionRuns.Relink), so a transient parse-time error never excludes a content for good; within one run a failed content is never re-claimed. A parse reads its container as System, so it reads only the connection the document LIVES UNDER (its parent path); a document whose content names another connection is refused, never read (IngestionPaths.ConnectionProblem).
Bookkeeping is O(changed), not O(all) One live query feeds both watches; its fold (DocSet) keeps running counts and names only the documents a change made pending.
Chunks live in the owning partition Vector rows are keyed (collection = the document's path, file = its name), so search_chunks and the mesh search's content fold resolve them under {P}/_Ingest/{name}/{sha32}; each chunk is also a _DocumentPart node (core #6013's document_parts table) under the document.

Proven

src/MeshWeaver.ContentCollections.Indexing.Graph.Test/DocumentIngestionTest.cs, on a real mesh with an in-memory vector index and a deterministic bag-of-words embedder:

Mutation-checked: letting a hub re-parse an Indexed document turns the replay case red.

Open