📝 Experiment_001_checkpoint_9.mdv4.6.2 · 2026-10-28

Experiments 000, 001, 002 — Checkpoint (9)

Operator: Andrew. Written 2026-10-29 23:50 Sydney. Supersedes Experiment_001_checkpoint_8.md (same content plus session narrative; kept, never overwritten). Earlier checkpoints _7 to _2 and Experiment_001_checkpoint.md also kept. Status: recorder 1.3.0 (Experiment 002 prefill injection) built and delivered, tested only against a scripted stand-in server, not yet run on the real 4B. No experiment has run since checkpoint 7. Resume: paste this file into a fresh conversation and attach NaturalRecorder_v1.3.0.zip, Exp000_queries_v1.0.0.zip, Hotwords_v2.1.0.zip, Memstore_Clone_v1.0.16.zip. For the experiment definitions, the 000/001 results and risks 1 to 40, use Experiment_001_checkpoint_7.md (risks 1-13 in _2, 14-21 in _3, 22-24 in _5, 25-29 in _6, 30-40 in _7; this file adds 42 to 47). Newest file wins. Terms: LLM1 = Qwen3-4B (chat model). LLM2 = second module / second 4B copy on 8082. RAG = hotwords.py pseudo. Pool = top 50 candidates at one step. Prefill = tokens placed at the start of the assistant's turn. Greedy = always take the most likely token (same prompt, same reply).

1. Delivered: natural_recorder.py 1.3.0

NaturalRecorder_v1.3.0.zip → NaturalRecorder/ (natural_recorder.py, NATURAL_RECORDER_MANUAL.md, both 1.3.0). New run options:

Option Meaning
--inject-top N Inject the N hottest terms retrieved for the query (needs --hotwords or --inject-terms-file).
--hotwords PATH, --hotwords-python PYTHON hotwords.py and its interpreter (Clone: python3.6). Runs pseudo --file - --format json once per query. HOTWORDS_DB picks the store.
--inject-source rag|query|both rag (default) = related terms from stored arrays; query = the query's own array; both.
--inject-terms-file PATH Ranked terms, one per line; with --inject-top N the first N.
--inject-text TEXT Literal text (the "ceiling"). Not combinable with the term options.
--inject-frame TEMPLATE Wrap terms, e.g. "On the Clone, regarding {terms}:" (needs {terms}). Default: bare list.
--inject-step N 0 (default) = very start of the reply; N = forced after N model tokens.

Saved in qNNN_rNN.json under inject: applied/reason, terms, terms_available, source, frame, retrieval, text, ids, tokens, applied_at_step, natural (the model's own pool at the injection point: first injected token's id, text, rank and probability in the top 50 or null, natural top 5), reply_text_with_inject, reply_with_inject_sha256. reply_text/reply_sha256 = the model's own tokens only (comparable with a natural run). Steps file holds only the model's chosen steps.

Unrequested changes (listed at delivery): injected text tokenized with special tags off; an injected run's replies kept out of pairs.jsonl (queries only; risk 36); a reply that cannot be injected (no terms retrieved, or it ended before the step) runs natural and says why.

Stand-in tests passed: natural run unchanged; hotwords top 2 at step 0; terms file at step 3; literal text; frame; no-terms case; reply-ended-first case; four usage errors. In the test store the retrieved terms were runs, centos, end-of-life ("runs" is filler: risk 35). Not run on the Clone's Python 3.6. Release card: all rules done or not needed.

2. Environment changes

3. Findings and corrections worth keeping

  1. Greedy is deterministic, so the baseline is noise-free: any A-versus-injected difference comes from the injection. The formal repeat check (--repeat 3 --max-steps 600, then 20) is still not done; the claim rests on the 8080/8082 comparison and the repeated Q3 runs.
  2. After an injection the free-running reply diverges, so tokens stop lining up. Per-token comparison needs A's reply fed back as the path under the injection (--force-path, not built).
  3. The vocabulary (about 152,000 entries) is fixed per model family, inside the GGUF file; quantising and fine-tuning do not change it; ids mean something only within one vocabulary. A word may be one token or a chain (" CentOS", "CentOS", " Cent"+"OS"). Whether the 0.5B shares the 4B's vocabulary is unchecked.
  4. Rare and private names split into fragments and carry no associations; names that are also words ("Clone", "Fin") pull to their common sense. A bare injected name is only a pointer: the relation must go with it.
  5. Withdrawn: "learning the deepest secrets of the LLM". What is known is specific to one 4B with greedy decoding (the 50 candidates cover nearly the whole distribution; about half the steps are at 99% or more; a fact the model lacks sits near 1e-5; boosting moves the frame, not the fact). Only outputs are watched, not internals.
  6. Design answers given: no vector database is needed at this size (embeddings only if the Q3 shared-vocabulary problem persists, then one vector per array in plain Python/SQLite); Graph RAG's graph is a knowledge graph (entities and relations), not backprop's computation graph; protecting weights in a future weights-changing phase means weight checkpoints plus a regression set (the approved entries), not graphs. Experiments 000 to 002 never touch weights.

4. Proposals (none built, none agreed)

  1. Metrics, in order: (1) token lift along A's path under the injection (log-probability change per token; most lifted, most suppressed; the fact words' lift is the headline); (2) pool presence of the fact tokens and their rank; (3) first divergence step; (4) fluency gap (mean log-probability of the model's own tokens); (5) echo (injected words that recur) against influence (retrieved but not injected words that appear) against baseline (neither); (6) suppression. 3 to 5 come from saved files; 1 and 2 need --force-path.
  2. Controls: inject the global top words (hotwords.py top --words into --inject-terms-file), unrelated to the query; and add a few of the operator's own off-the-likely-path queries beside the plain ones.
  3. Builds: 1.3.1 = per-term token spans recorded at injection time plus a relate block matching the reply's words to the hot terms by spelling key (appears or not, first step). 1.4.0 = compare command, --force-path, vocab command, corrected word-start flag.
  4. vocab command: dump every id with its piece and flags (leading space, letters, digits, special, partial multi-byte) via /detokenize (about 152,000 calls, run once, kept as a file). Shows how fact words and names split, gives a tokenizer-aware word-start flag (risk 28), tests the multi-byte theory for the "no candidates" fault, and may give an outside rarity reference from id order (believed true for Qwen, unverified). No-code first look: curl -s http://127.0.0.1:8080/tokenize -d '{"content":" CentOS 6.10 end-of-life MAOIs SSRIs","with_pieces":true}'.
  5. Nudge and pinned names for hotwords.py: manual nudge TERM +0.3 in a side table applied only in pseudo (second ladder lever: the order of the same retrieved terms); event log, heat table frozen and recorded per run; reinforcement only from the operator's approvals and corrections. Pinned names = importance set by the operator, applied as a gain only when the name or an alias matches the query, with aliases and the relation attached.
  6. Prior-art search on the combination (full pool per step, deterministic baseline, unaided-facts pass test, pre-written fail criterion, operator-set importance). Offered, not done.

5. Risks (new; adding to 1 to 40)

  1. A moving heat table breaks comparability: freeze and record it per run.
  2. Echo through reinforcement: signals must come only from the operator, never from the model's output.
  3. A pinned name that fires on every query is the B leak again: fire only on a match.
  4. Alias gaps: fuzzy skips short words and digits; nicknames need an alias list.
  5. The video job on the Clone's four cores distorts timings and can stall runs on 8082 (extends 29 and 40).
  6. 1.3.0 is unproven on the real 4B: untested are tokenization at the boundary between the prompt's final newlines and the injected text, the real server's cache behaviour, and Python 3.6.
  7. The id-order rarity idea is unverified.

6. Open items

7. Standing operator rules (carry forward)