A MeshWeaver coding model from verified tokens

Decision (2026-11-06): do not fine-tune a model on the documentation. Use the Mac Studio fleet to generate tokens, and keep only the ones the platform's own gates verify. Every NodeType that ships Test/*.cs is a task with an executable answer key, and core's run-node-tests.py is already the judge the PR gate uses. Verified samples become the training set. The scorer is also the eval harness, so the same files measure today's models and any model trained later.

The prototype lives in devtools/coding-model/ (tasks.py, generate.py, score.py, stdlib-only).

1. Why verified tokens and not the docs

• The platform moves faster than weights. Conventions and APIs change weekly. Text learned from the docs freezes one vintage and then writes retired patterns confidently. A reward computed against the CURRENT reference set re-learns the platform each time it is re-run.
• The docs teach too little. A few thousand doc nodes teach style. They do not teach "reactive, never async", "framework controls only" or "read node.Content in any shape". The compiler and the tests do.
• The judge already exists. run-node-tests.py compiles a NodeType's own unit the way the mesh does and EXECUTES its tests. The PR gate trusts it, so a reward built on it inherits that trust.

2. Tasks — mined, never written by hand

tasks.py walks every NodeType through core's canonical compile-check.py, using the same discover_nodetypes / resolve_sources the gate uses, so a task's unit is the unit the mesh compiles. Only types with a Test/ folder are mined: a task without an executable check has a guessed reward.

Prompt context: the rest of the unit, up to 60,000 characters, plus two things calibration showed are required:

• Files that use the masked file come first. With smallest-first ordering, ChessGame.cs came back without its Notes property, because the 500-line layout file that reads it had been cut.
• Every type the rest of the unit declares is listed, including types in files the budget cut, with "do not declare these again". Without the list, 2 of 6 samples re-invented a class they could not see and the unit collided (CS0101).

Split by SOURCE-SHARING GROUP, never by file. Files of one module share helpers, so a file-level split leaks answers into training. Splitting by the owning module is not enough either: a NodeType may compile another module's sources (Governance/Standard pulls Store/Core/Source/…), which would put a held-out Store file verbatim into a training prompt. Modules are therefore joined whenever one's NodeType compiles the other's sources, and the split is a hash of the group. --eval-modules holds out every group containing a named module. Measured: one large group (Approvals, Developer, Essentials, Games, Governance, Hosting, MyAi, Providers, Store) holds about two thirds of the tasks and hashes to train. The eval set therefore comes from the self-contained modules today. Holding out part of the big group would require excluding the training tasks whose context contains an eval target, which is a refinement for later.

Measured on this repo (2026-11-06): 918 tasks from 45 modules (569 file + 305 tests train, 31 + 13 eval), mined in 20 s. The first count, 1,091, also included *TestsArea.cs wrappers, which declare no cases; calibration rejected them, so they are no longer mined. Satellite repositories have the same layout and mine with --root.

3. Reward — the platform's gates around untrusted code

score.py builds a sandbox per candidate. Its top-level entries are symlinks into a BUILT checkout, except the candidate's module, which is cloned with APFS cp -c (copy-on-write, nearly free) and has the candidate written over the target file. Core's run-node-tests.py then runs on that one NodeType with MW_REPO_ROOT set to the sandbox.

The candidate is untrusted code, and it is executed. A model trained against this reward is optimised to raise it, so the scorer assumes the candidate is hostile (all from the review of #5048):

• Confinement: macOS sandbox-exec, enforcing:

  • No network.
  • No reads under the home folder except what the gates need: the checkout's TRACKED top-level entries (untracked files in its root are neither linked nor readable), the reference set, the gate scripts, dotnet and the NuGet global-packages folder. Only that folder, never ~/.nuget/NuGet/NuGet.Config, which can carry private-feed credentials.
  • Writes denied EVERYWHERE, then re-allowed narrowly: the sandbox, the standard /dev nodes, and the one shared directory below. A prefix deny-list had left /opt/homebrew and /Volumes writable. The dotnet CLI home is redirected into the box, and the NuGet cache is read-only via NUGET_PACKAGES, so one candidate cannot poison the next.
  • One shared exception: /tmp/.dotnet, where .NET keeps named mutexes regardless of TMPDIR. Without it dotnet run dies on NuGet's first-run mutex, and the worst a candidate can do there is make others wait.
  • A clean environment and a private TMPDIR.

  Measured with a probe candidate whose module initializer tried four things: reading ~/.gitconfig, reading an untracked file in the checkout root, and writing into ~/.nuget/packages and ~/.dotnet. All four were refused with UnauthorizedAccessException. The scorer refuses to run on a host without sandbox-exec (--unconfined is for trusted input only). The profile file lives outside the box, so the candidate cannot rewrite it.

• Whole process group: each evaluation runs in its own process group, killed on timeout. The harness starts dotnet run, which starts the candidate, so killing only the parent left an infinite loop running.

• Verdict source: the verdict comes from the harness's LAST aggregate line and must agree with its exit code. Text anywhere in the log does not count: a test can print 999 passed · 0 failed before it throws.

• Tamper refusal: any candidate touching the harness's channels scores a hard 0. This covers the summary marker, Environment.Exit, redirecting the console, killing processes and AssemblyLoadContext. 🚨 The runner shares a process with the candidate, and the harness believes the last ##NODETESTS-SUMMARY line in the runner's output. A candidate that prints a fake one and exits would therefore be believed. The deny-list stops the direct route, but an obfuscated one (reflection) remains possible. Before RL at scale, core's harness should report its verdict out-of-band, for example from a separate process that inspects results the candidate cannot write. Until then, spot-audit reward-1.0 rows.

• Strict compile: any positive reward, mixed results included, also needs the canonical compile-check.py verdict for the type, including the configuration lambda and with warnings as errors. The test harness checks neither. The strict check runs in a FRESH sandbox that the executed tests never touched, and only a normal exit with a report written by that run counts. Otherwise a test run could plant a verdict or delete the module root the check reads.

• tests tasks: the case count with the target emptied is the baseline, and the candidate must add cases of its own. Otherwise an empty file "passes" on the type's other suites.

• Re-scoring: a row is reused only under the same evaluation fingerprint: the timeout and confinement mode, the checkout commit, the checkout's BUILT module assemblies (src/<P>/bin/<cfg>/net10.0/<P>.dll, by content, because the harness binds them), the reference set's CONTENT (not its path), the gate scripts and the scorer. Re-scoring against a new platform set, or a reference set rebuilt in place, re-runs everything. generate.py likewise resumes only for the same model and the same prompt fingerprint. Both drop an incomplete last record left by a killed run before resuming.

• Reward: 0 if the type does not build strictly, if no test ran (an absent failure is not a pass), or if a tests candidate adds no case. Otherwise the pass rate, multiplied down for each convention lint the candidate ADDS over the original file.

• Lints are relative to the original: calibration flagged three original files. Two were legitimate: an async leaf behind IIoPool, and a chess board drawn with Controls.Html. The third was a property called Result, which is not a blocking wait. An absolute lint would have punished the house standard, so the .Result pattern was dropped.

• Calibration is a gate on the tasks: score.py --reference scores each original file, and a task whose original does not score 1.0 is dropped. In the first run 25 of 26 Chess/ThreeBody originals scored 1.0. The 26th failed because the tasks had been mined at a newer commit than the checkout scored against; the scorer now warns when the two differ, so always mine from the checkout you score against. With the full hardening on (read and write confinement, strict compile in a fresh box, tests baselines), 24 of 26 score 1.0 in 84 s with 6 jobs. The other two are the *TestsArea.cs wrappers, correctly rejected because they add no test case, and they are no longer mined. After the third review round (global write-deny), 24 of 24 score 1.0.

• The checkout must be BUILT. The harness binds the registry-served module assemblies (MeshWeaver.AI, MeshWeaver.Markdown.Collaboration, MeshWeaver.Maps) from src/*/bin, and refuses without them. A fresh worktree has none; --root points at one that has.

• Cost: 2–3 s for the tests alone. About 3.2 s effective per candidate with 6 parallel jobs once confinement, strict compile and tests baselines are on (M5 Max). The strict compile runs only for candidates that could earn a positive reward, and baselines run concurrently per task. A single evaluation, strict compile included, takes 16–19 s. Under parallel jobs on a loaded host (another session's compiler server and a VM busy) the mean rose to 63 s, which is contention, not the profile. Measure on an idle Studio before sizing the fleet. Scoring is still far cheaper than generating.

4. The fleet — Macs make tokens, control places the work

Each Mac Studio runs an OpenAI-compatible model server (Ollama /v1, mlx_lm.server or llama.cpp) and scores its own candidates, because .NET runs natively on Apple silicon. A generation job is (task shard × model × k samples). The prototype's --endpoint list is phase 0. In the fleet the same body runs as a job placed by the control instance's queue on machines enrolled through the back-lease, which already models placement with per-model memory reservations. This is deliberately not a second distributor: every activity is enqueued on control.

Measured (2026-11-06, qwen3-coder:30b on Ollama, one M5 Max laptop, 2 concurrent requests): prompts of about 12k tokens arrived whole (no silent context truncation), the 2nd sample of a task reused the cached prompt (cached_tokens ≈ the whole prompt), and output ran at 38–40 completion tokens/s. The Studios' rates are measured the same way, from the rows generate.py writes. Order the work so a task's k samples land on the same server, because the cached prompt is most of the cost.

5. From tokens to a model

1. Rejection sampling / expert iteration (start here). Keep candidates with reward 1.0, de-duplicate them, and fine-tune an open coder with LoRA (MLX on a Studio is enough at 30B-class with LoRA). Re-sample with the new model, re-score and repeat. This is offline RL with a binary reward and needs no GPU cluster.
2. Repair trajectories. A failed candidate plus its compiler diagnostics is a prompt, and a passing fix is its answer. This teaches the loop an agent actually runs: write, compile, read the error, fix. The diagnostics are already in every score row.
3. Online RL (GRPO) only if 1–2 plateau. It needs a rollout-and-update loop that rented GPUs suit better than the Macs. The reward function is unchanged.

Teachers: use open-weight models for training data (Qwen, GLM open weights). A closed model's outputs can be used for evaluation, but check that provider's terms before training on them.

6. Staleness is a re-run, not a retrain

Every row records the commit it was mined at. When a new platform set seals, the stored verified candidates are re-scored against it. Rows that no longer compile are dropped, which tells you exactly which APIs moved, and the next expert-iteration round re-learns them. Do this at least on every MAJOR.

7. Confidentiality

The model memorises its training code, so it is as private as its most private source. Mine only repositories that pass the confidential-term gate. Keep the model and its datasets internal, and never serve it from the public instance.

8. First baseline (2026-11-06)

qwen3-coder:30b, no retrieval, prompt as in §2:

Round 3 drew 30,626 completion tokens in 1,049 s (29 tok/s aggregate, 2 concurrent requests; these prompts were longer than in round 2).

What the failures say, read from the diagnostics in the score rows:

• Types the masked file itself must declare were left out: GameRowSummary, GameAnalyzer, ExplorerMoveStat, PositionEval. Other files use them, so the task is determined, but the model did not infer that it owes them. Next prompt change: state the types the file must declare. A task spec is allowed to give the interface.
• Plausible framework types and members that do not exist: MeshHub, UiAction, StackControl.WithChildren, Controls.DataGrid<T>, RenderingContext.Hub. This is the gap a MeshWeaver model has to close. Before training, measure the cheaper control arm: the same prompt plus the public signatures of the framework types the unit uses.
• The relative lint fires where it should: three failing candidates added Controls.Html markup that their originals do not have.

9. Next steps

1. Calibrate all 1,091 tasks on one built checkout and drop those that fail.
2. Freeze the eval split. Measure each arm on it (local Qwen, Qwen + API signatures, GLM, a frontier model with the MCP) and record the table here.
3. Wire generate.py + score.py as a back-lease job kind on control, then run the first train shard on the Studios.
4. First expert-iteration round (MLX LoRA), scored on the frozen eval split.