Content indexing coverage

The rule: every file in a content collection is either fully indexed — all of its text, chunked whole — or recorded with the reason it is not. A file the indexer cannot read still gets its Document node ({collection}/_Documents/…), carrying indexingGap (the reason) and chunkCount: 0, and the same sentence is on the indexing activity's log line: 'decks/q3.pptx': NoText (0 chunk(s)) — no text extractor for '.pptx' files.

Find every file the index cannot serve with a search for nodeType:Document under the collection and read content.indexingGap.

This page is the audit behind that rule: it was measured against the code of src/MeshWeaver.ContentCollections.Indexing* (this repository) and MeshWeaver.ContentCollections (core), not against other documentation. The flow itself is Document ingestion for attached storage, and the upload → ContentIndexingObserver → ContentIndexingService.IndexFile path for content collections.

Formats

Caps

Skips — where a file is not indexed, and where that is recorded

Which writes reach the pipeline

The pipeline indexes a file when it is ANNOUNCED. A single upload is announced per file (IContentUploadObserver.OnUploaded, raised by the caller). A write that puts many files there at once is announced ONCE (IContentUploadObserver.OnFilesWritten, raised once the write committed — by the owning collection, or for a sync by its producer after the last delivery answered — under the writer's identity — core MeshWeaver#7190, Doc/Architecture/ContentIndexingActivation → "What a bulk write announces"). Deletes are raised by ContentCollection itself for every delete path. A write path that announces nothing is indexed only when the daily backfill (below) walks its collection.

One announcement is at most ONE indexing activity. ContentIndexingObserver.OnFilesWritten queues the write on a serial pump; its gate (FilesToIndex) reads each named file and keeps only those whose bytes the index does not already serve. A file counts as served when its stored hash matches AND, where the document branch writes a Document for it, that Document exists (ContentIndexingService.Serves), or when its Document records a gap for exactly these bytes (one listing of {collection}/_Documents). Current chunks with a MISSING Document (a Document write that failed after the chunks were committed, or an index older than the document branch) stay pending, so IndexFile's hash-gate repair writes the Document. The kept files are indexed as one activity (IndexFilesActivity); when none are kept, NO activity opens. So a boot re-import, which re-sends every content file of every synced repository, opens an activity only for a repository whose content actually changed. The serial pump means two announcements never index at the same time. That is the opposite of the per-file burst Doc/Architecture/ContentRoute503 (core, Cause B) describes, and it is why a sync is never announced per file.

Every file the activity indexes is looked up AGAIN once its chunks are committed, and on a late failure too; when it was deleted meanwhile its index is removed once more, and when it now holds DIFFERENT bytes (a newer version was written and indexed while this run held the old one) the current bytes are indexed again, so a late commit of the old version never stays in the index. The single upload, the bulk write, the re-index and the backfill share that one per-file step, so a delete or a rewrite that lands while a sync is still extracting or embedding never leaves the deleted file, or the superseded version, searchable. Bulk writes are indexed one batch at a time, and a batch that fails does not stop the batches behind it.

A write whose file the index cannot read is still recorded (indexingGap) and does not fail the activity. A gap recorded for unchanged bytes is not re-attempted on the next write. The daily backfill retries it.

Re-index and backfill

ContentIndexingObserver.ReindexAll walks every file of the named collections as one cancellable activity and runs IndexFile for each. It is hash-gated, so a run indexes exactly the files the index does not hold yet — which is the backfill for files uploaded before indexing existed — and re-attempts every recorded gap (a recorded file has no stored hash). force re-extracts everything.

It is reached from the Space settings tab Content Indexing (Re-index all content, Rebuild), for the Space's own content collection only.

The daily backfill

ContentIndexBackfill.Run is the fleet-wide, hash-gated backfill. It enumerates every Space (nodeType:Space partitions:all, giving {space}/content) and every mail copy that carries attachments (nodeType:MailMessageCopy content.hasAttachments:true partitions:all, giving {copy}/attachments), page by page in path order. It then visits the collections one at a time:

• PendingFiles, the same hash gate the write path uses, lists the files the index does not serve. Here a recorded gap counts as NOT served, so every gap is retried once a day.
• When a collection has a pending file, ReindexAll runs for it as its own activity. A collection that is current opens nothing.

The run reports one line per collection: current, reindexed (with the pending count and the activity), no-collection, or failed (with the reason). The line follows the re-index activity's TERMINAL status: an activity that ended Failed is a failed line carrying its activity and reason, and one that was Cancelled stops the run. Collection paths are de-duplicated ordinally: mesh paths are case-sensitive, so two paths that differ only by case are two collections. When an enumeration reaches its page bound, the report says TRUNCATED instead of reading as complete.

Authorized as the caller, executed as System (policy authorize-as-caller-execute-as-system). The caller is captured when Run is called. The platform (System, or trusted infrastructure that set no context) passes. An anonymous context (an empty ObjectId) is refused before anything is read. A person must be a global admin, checked before the enumeration is subscribed, and an unanswered check fails closed. The run itself executes as System on behalf of that person, because it reads every partition.

The operation. The Indexing/Script/content-index-backfill Code node (Run button; its activity lands under Admin) runs ContentIndexBackfill.Run(Mesh), so a cancel is the activity's RequestedStatus. It reaches it BY KEY: GetRequiredKeyedService<Func<IMessageHub, IObservable<string>>>("content-index-backfill"), registered with the pipeline. A Code node compiles against the portal image's reference set, which a host type added after that image does not belong to, so a script that named ContentIndexBackfill failed CS0103 in the Code-node gate. A delegate of framework types resolves on every image. A cancel that disposes a running re-index is recorded: the child activity ends Cancelled and never stays Running.

The enumeration visits nodeType:(Space OR Store/Plugin), the package roots Store/Catalog lists, plus the mail copies, plus every collection holding a Document that records a gap (nodeType:Document content.indexingGap:* partitions:all; the collection is the Document path before /_Documents/). The third enumeration reaches a gap wherever it was recorded, such as a node picture in {node}/content on a node that is neither a Space nor a package root. It pages by KEYSET (sort:path + path:>"{last}"), never by offset. The Postgres fan-out loads Limit rows and the merge layer applies an offset afterwards, so offset page two came back empty. One unreadable file neither stops the gate nor the walk: it is attempted and recorded FAILED while the readable files are indexed. The recorded-gap listing filters for gaps IN the query, so its limit cannot be spent on indexed Documents.

The daily schedule is the platform's repeating EventSubscription, Admin/EventSubscription/content-index-backfill: a Timer firing at 02:00 UTC, RepeatEvery one day, with a RunScript continuation on that Code node, which EventSubscriptionRunner fires as System. ContentIndexBackfillSchedule, a hosted service the PostgreSQL module registers behind the same activation gate as the pipeline, CREATES that subscription once. It never overwrites it: a restart does not reset its next slot, and an administrator's edit or cancellation stands. Because the schedule outlives a switch-off, the keyed backfill delegate checks the same activation gate first: on a deployment whose indexing is not active it answers Content index backfill: SKIPPED — … and never fails.

Open

• The bulk-write seam needs core MeshWeaver#7190 on the platform the module is BUILT against. ContentIndexingObserver.OnFilesWritten implements the core interface member only when it is compiled against a core that declares it. A module built against an older core still runs on a newer one, but the core's empty default answers instead and nothing is indexed. That lasts until the module is rebuilt against the new platform set, which the sealed-set rebuild does. Until then, imports and syncs are covered by the daily backfill.
• A split sync whose LATER delivery is refused never announces the files its earlier deliveries wrote. The daily backfill indexes them.
• Node pictures: OCR. A picture is captioned by the IImageDescriber when one is configured. Without one, it is recorded (no text extractor for '.png' files). Text IN the picture (OCR through IOcrProvider, MeshWeaver.Plugins#4204) is not added yet: it lands with #4204, which is still a draft.
• Storage connections (InitializeRequest, Document ingestion) are not part of the daily backfill. A run is started per connection.
• Plain-text formats outside the extension list (.yaml, .log, source code) are recorded, not indexed. Sniffing valid UTF-8 in the default branch of TextExtractor would index them; it sits in the code #3826 rewrites, so it follows that change.
• #3826's bounds should set ExtractedDocument.IndexingGap ("not indexed: {bound} exceeded") wherever they cut, so a partial file is recorded on its Document and not only noted in its text.
• Document view. indexingGap is on the node's content; the Document page does not show it yet.

Proven

• BulkWriteIndexingTest (real mesh, real file-system collection):

  • One write of three files (one .pptx) is ONE activity, which succeeds and records the gap. The same write announced again opens NO activity.
  • A re-write indexes only the file that changed: one activity, and one summarizer call.
  • The OnFilesWritten seam indexes what it is given.
  • A node picture with no describer is recorded with the gap, and the activity succeeds.
  • The backfill indexes a never-announced file, then reports the collection current with no activity.
  • The backfill retries a recorded gap.
  • A re-index that fails is reported failed, with its activity and reason, never reindexed.
  • Two collection paths that differ only by case get a line each.
  • A file whose chunks are current but whose Document is missing is indexed by a write (the Document is repaired); written again, it opens nothing.
  • The schedule is created once, and a second Ensure leaves it untouched.

• ContentIndexBackfillAuthorizationTest (a mesh WITHOUT the public admin grant): a person who is not a global admin is refused before the enumeration is subscribed, and so is an anonymous caller. A global admin runs it (the positive control).

• DeleteDuringIndexingTest: a file deleted while it is being indexed leaves no chunks behind, through the single upload AND the bulk write, on success and on a late failure.

• MeshWeaver.Plugins#4431: ChatClientImageDescriberTest / ChatClientSummarizerTest (a client that times out fails as TimeoutException, and the indexer fails the FILE with it), IoLeafTimeoutTest (a cancelled token stays a cancellation — the negative control), DocumentWithoutSummarizerTest (empty → readable with no summarizer replaces the gap Document; text together with a gap is indexed and carries the reason; a summary timeout on changed bytes is recorded on the Document and retried by the next index).

• NodePictureIndexingTest: with a describer configured, a new picture is captioned and chunked. Announced again, it opens no activity.

• Negative controls, run locally:

  • Gate disabled (every named file pending): four cases fail — the re-write, the changed-file, the backfill-current and the picture re-announce cases.
  • Recorded gaps ignored: the boot re-import case fails with "found 1 item(s)".
  • Caller check removed: the refusal case fails.
  • Anonymous treated as the platform: the anonymous refusal case fails.
  • Case-insensitive de-duplication: the case-variant case fails.
  • Every re-index read as reindexed: the failed-re-index case fails.
  • Hash-only gate (no Document check): the missing-Document case fails.
  • No re-check in the shared walk: both bulk-write delete cases fail.

• Core MeshWeaver#7190 proves the announcements themselves (ContentBulkWriteAnnouncementTest, PictureUploadIsAnnouncedTest). Against that core branch, hub.RaiseContentFilesWritten was checked to dispatch to ContentIndexingObserver.OnFilesWritten and index the file.

• IndexingGapRecordedTest (unit): unsupported format, failed extraction removing the previous version's index, a failed forced rebuild of unchanged bytes keeping the earlier index, a file replaced by an empty version, an empty image description, and the fully-indexed positive control.

• MeshDocumentSinkTest.UnreadableFile_IsRecordedAsADocumentNode_NamingTheGap_AndDropsTheOldParts (real mesh): the Document node carries the gap and the previous version's parts are pruned.

• ContentIndexingPipelineTest.ReindexAll_RecordsTheFileItCannotIndex_WithTheReason (real mesh, real re-index activity): the log line names the reason and the node records it.

• IngestionSkippedNoteTest: the storage walk names each skipped file with its reason, bounded.

• PartWritesAreBoundedTest (real mesh): a 24-part document's writes reach the owner exactly 4 at a time (a creation validator parks each part create and records the peak), and once released every part lands. Negative control, run locally: with the writes built through hub.Observe again, the ceiling assertion fails (more than 4 part creates in flight at once).

Negative control: with the recording reverted, five of the unit cases, the node case and the re-index case fail — the re-index log then reads 'decks/q3.pptx': NoText (0 chunk(s)). and no node is written.