đź“ť Experiment_001_checkpoint_8.mdv4.6.2 · 2026-10-28

Experiments 000, 001, 002 — Checkpoint (8)

Operator: Andrew. Written 2026-10-29 23:42 Sydney. Follows Experiment_001_checkpoint_7.md (kept, never overwritten; _6, _5, _4, _3, _2 and Experiment_001_checkpoint.md also kept). Status: recorder 1.3.0 built and delivered (Experiment 002's prefill injection), tested only against a scripted stand-in server, not yet run on the real 4B. No experiment has been run since checkpoint 7. The rest of the session was design talk (section 6), nothing built from it. Resume: paste this file into a fresh conversation and attach NaturalRecorder_v1.3.0.zip, Exp000_queries_v1.0.0.zip, Hotwords_v2.1.0.zip and Memstore_Clone_v1.0.16.zip (or the newest Memstore tree). This file is written to stand alone for the work ahead; for the full text of risks 1 to 40 and the 000/001 results, Experiment_001_checkpoint_7.md is the reference (risks 1 to 13 are in _2, 14 to 21 in _3, 22 to 24 in _5, 25 to 29 in _6, 30 to 40 in _7; this file adds 41 to 50). Where files differ, the newest wins. Terms: LLM1 = the chat model (Qwen3-4B). LLM2 = the second module that would weight or add words (also the name of the second 4B copy, "4b2", port 8082). RAG = choosing stored arrays that match the query (here hotwords.py pseudo). The pool = the top 50 candidates at one step. Prefill = tokens placed at the start of the assistant's turn so the model reads them as already said. A token = the unit the model reads and writes (a piece of text from its fixed vocabulary of about 152,000 entries), not quite a word. Greedy = at every step the single most likely token is taken, so the same prompt always gives the same reply.

1. The aim and the experiments (unchanged)

Standing aim: find out whether a second module (LLM2 plus retrieval) can move LLM1's next-word candidacy inside the query's subject, so a corrected mistake is not repeated.

2. Environment (as in checkpoint 7, with changes)

3. What happened this session

  1. Checkpoint 7 and the four zips were read. Claude critiqued the prefill plan before building (section 6, item 1).
  2. The operator's "shall we get back into … knobs and levers?" was taken as the go for recorder 1.3.0. He did not say the word "go"; this is Claude's reading.
  3. Recorder 1.3.0 was built, tested against a stand-in server and delivered (section 4).
  4. A long design discussion followed (section 6): Graph RAG versus this design, whether a vector database is needed, dynamic hot-word values, pinned names, the vocabulary, metrics, prior art.
  5. Claims withdrawn or corrected in the session: "learning the deepest secrets of the LLM" (nothing opens the model's inner workings; only outputs are watched); the earlier suggestion that a knowledge graph or "graphs retained for integrity" had been proposed in this chat (not found in any checkpoint; unconfirmed).

4. Delivered: natural_recorder.py 1.3.0

NaturalRecorder_v1.3.0.zip unzips to NaturalRecorder/ (natural_recorder.py 1.3.0, NATURAL_RECORDER_MANUAL.md 1.3.0). Earlier zips (1.2.1, 1.2.0, 1.1.0, 1.0.0) are kept. New options on run:

Option Meaning
--inject-top N Inject the N hottest terms retrieved for the query. Needs --hotwords (or --inject-terms-file).
--hotwords PATH, --hotwords-python PYTHON hotwords.py and the interpreter for it (on the Clone python3.6). Its pseudo --file - --format json runs once per query (the query goes in on standard input).
--inject-source rag|query|both Which terms: rag (default) = the related terms from stored arrays, because the query's own words are already in the prompt; query = the query's own array; both.
--inject-terms-file PATH Ranked terms, one per line, hottest first; with --inject-top N the first N.
--inject-text TEXT Literal text (the "ceiling": the stored sentence). Cannot be combined with the term options.
--inject-frame TEMPLATE Wrap the terms, e.g. "On the Clone, regarding {terms}:" (must contain {terms}). Default: the bare list "term, term, term".
--inject-step N 0 (default) = the very start of the reply; N = forced after the model has chosen N tokens.

Saved per reply (qNNN_rNN.json, key inject): applied/reason, terms, terms_available, source, frame, retrieval (arrays and weighted terms), text, ids, tokens, applied_at_step, natural (the model's own pool at the injection point: the first injected token's id, text, rank and probability in the top 50, null when absent, and the natural top 5), reply_text_with_inject, reply_with_inject_sha256. reply_text and reply_sha256 hold only the model's own tokens, comparable with a natural run. prompt_tokens excludes the injection; inject_tokens counts it. The steps file holds only the steps the model chose. run.json settings carry the inject options.

Changes the operator did not ask for (all listed in the delivery): injected text is tokenized with special tags switched off; an injected run's replies are left out of pairs.jsonl (queries only) so they cannot reach hotwords.db (risk 36); a reply that could not be injected (no terms retrieved, or the reply ended before the step) runs natural and says why. The --inject-frame option was proposed in the critique and covered by the go.

Tests (stand-in server only, a scripted chain "The Clone runs Apple OS" / "…CentOS 6.10, which is end-of-life" when "CentOS" is in the context; hotwords.py pseudo run for real on a small test store): natural run unchanged; hotwords top 2 at step 0; terms file at step 3 (reply became "The Clone runs CentOS 6.10, which is end-of-life", natural probability of " CentOS" recorded as not in the top 50); literal text at step 0 (natural probability 0.95, rank 1); frame; injection beyond the reply's end (reported, runs natural); no terms retrieved (reported, runs natural); four usage errors. In the test store the retrieved terms for the Q3-style query were runs, centos, end-of-life: "runs" is filler (risk 35 in miniature). Compiled on Python 3.13; not run on the Clone's Python 3.6; no f-strings or newer syntax used.

Release card (as delivered): version/date bumped done; manual history done; version shown at start done; zip with never-reused version done; xmllint not needed; unrequested changes listed done; legacy zipper not needed.

5. Lessons learned (new this session)

  1. Greedy is deterministic, which makes the baseline noise-free. Any difference between an A run and an injected run is attributable to the injection. The formal repeat check (--repeat 3 --max-steps 600, then 20) is still not done; the claim rests on the 8080/8082 comparison and the repeated Q3 runs. Sampling mode varies by design.
  2. Free-running replies diverge after an injection, so tokens stop lining up. Per-token comparison needs the fixed-context method: feed A's own reply back in as the path under the injected prefix and score each token (needs a --force-path option; not built).
  3. The vocabulary is fixed per model family and never changes in use. It is built before training; quantising and fine-tuning do not touch it; the same GGUF file gives the same ids on 8080 and 8082. A token id means something only inside one vocabulary. A word may be one token or a chain; " CentOS", "CentOS", " Cent"+"OS" differ. Whether the 0.5B shares the 4B's vocabulary is still unchecked.
  4. Rare and private names split into several fragments and have no associations in the model; names that are also words ("Clone", "Chip", "Fin", "Clarion") are pulled to their common sense. A bare injected name is only a pointer; the relation must go with it.
  5. Predictive text is the same family, scaled down: a short ranked list of next words. Older keyboards used the last word or two; the 4B reads thousands of tokens. A phone's learned dictionary is a personal hot-term table with the same weakness: it knows what you use often, not who matters.
  6. The model follows the likeliest reading. In this session Claude missed or over-read the operator's wordplay about five times (planet, Clarion, "models", "baseline", "monarosexual"). The operator's queries are off the likely path; tagging planted probes with "(probe)" helps; leaving some untagged is also data.
  7. Fuzzy matching handles a one-letter typo ("baseline"/"bassline": 8 letters, 1 edit, factor 0.9) but not short words or any word with a digit.
  8. Honest framing: the mechanism is next-token prediction; what has been learned is specific to one 4B with greedy decoding (the 50 candidates cover nearly the whole distribution, about half the steps are at 99% or more, a fact the model lacks sits near 1e-5, boosting moves the frame not the fact). It is not a window into the model's internals.

6. Design discussion and proposals (none built, none agreed beyond what is stated)

  1. Critique before building 1.3.0 (accepted in effect): the pass test counts only connected facts the model supplied that were NOT injected; a bare term list may be continued as a list, so a framed variant runs beside it; run the clean-store pseudo test on Q3 first (the shared-vocabulary problem: Q3 shares only "Clone" with entry 1); switch the store stage off first.
  2. Which RAG design: closest to Naive RAG with hot-term arrays instead of embeddings, ending not in a prompt template but in a change to the next-word pool (or, for 002, a prefill). Useful pieces from the others: Corrective (a relevance grade = the match floor and the fail criterion), Adaptive (decide whether to retrieve = the context definer; stops leakage into controls), Hybrid (a second kind of retrieval). HyDE would feed the model's own confabulation back (risk 26).
  3. No vector database is needed at this size (8 entries now, hundreds later). It would earn its place only when query and memory use different words for the same thing (Q3) and the store is large. The fix would be one small-embedding vector per array with matching in plain Python/SQLite; no server. Decide after the clean-store pseudo test on Q3.
  4. Graph RAG's graph is a knowledge graph (entities and relations, filled by an extractor at indexing time, queried at question time); backprop's graph is the transient computation graph. Keeping history to protect weights during a future weights-changing phase is done with weight checkpoints and a regression set (the approved memory entries make one); elastic weight consolidation is the related technique. Experiments 000 to 002 never touch weights.
  5. Nudging hot-term values over time (proposed): decay with age (query-time, stored arrays untouched); reinforcement from the operator's approvals and corrections only (never from the model's own output: echo, risks 14 and 43); demotion for terms retrieved and injected without effect. Keep an event log (term, time, reason, size) and compute values from it; freeze the heat table during a run and record it (a moving table breaks comparability, risk 42). It adds a second lever to the ladder: the order of the same retrieved terms (e.g. CentOS first instead of fourth). Smallest first step: a manual hotwords.py nudge TERM +0.3 stored in a side table and applied only in pseudo.
  6. Pinned names (importance set by the operator): names of the nearest and dearest should sit at the top of the heap whatever their count or rarity; nothing in the text supplies importance. A pin is a gain, applied only when the name or an alias actually matches the query (never a standing injection: risk 44), with aliases (fuzzy matching will not link a nickname to a full name: risk 45) and the relation attached, so one cue brings the cluster (a graph node with importance on it). This mirrors the operator's NE-salience idea (affect sets the weight, not frequency).
  7. A vocab command for the recorder (proposed): dump all ids with their pieces and flags (leading space, letters, digits, special, partial multi-byte), via /detokenize one id at a time (about 152,000 calls, a few minutes; kept as a file). Would show how the fact words and names split, give a tokenizer-aware word-start flag (risk 28), test the multi-byte theory for the "no candidates" fault, and possibly give an outside rarity reference from the id order (byte-level vocabularies merge the commonest pairs first, so the id roughly tracks frequency: believed true for Qwen, not verified; answers risk 16). A no-code first look: curl -s http://127.0.0.1:8080/tokenize -d '{"content":" CentOS 6.10 end-of-life MAOIs SSRIs","with_pieces":true}' (needs with_pieces in the build). Test words to try: long chemical names, 25I-NBOMe, 2C-B-FLY, 5-MeO-DMT, CentOS6.10 with no space, camelCase/snake_case versions of an identifier, Cyrillic look-alike letters, zero-width characters, emoji with joiners, long digit strings.
  8. Metrics (proposed order): (1) token lift along A's path under the injection (log-probability change per token; the most lifted and most suppressed; the fact words' lift is the headline); (2) pool presence of the fact tokens (enter the top 50? at what rank?); (3) first divergence step between A and the injected free run; (4) fluency gap (mean log-probability of the model's own tokens); (5) echo: injected words that recur (echo) against retrieved-but-not-injected words that appear (influence) against neither (baseline); (6) suppression (what the injection pushed down). Metrics 3 to 5 can come from saved files; 1 and 2 need --force-path.
  9. Controls to add: inject the global top words (hotwords.py top --words into --inject-terms-file), unrelated to the query: if they move the reply as much as query-matched terms, the effect is a general nudge, not retrieval. Include a few of the operator's own off-the-likely-path queries next to the plain ones.
  10. Proposed builds: 1.3.1 = per-term token spans recorded at injection time (term, ids, pieces, position) plus a relate block that matches the reply's words back to the hot terms by spelling key (appears or not, first step); 1.4.0 = compare command, --force-path, vocab, corrected word-start flag. Also a manual nudge for hotwords.py with pinned names and aliases (item 5, 6). Prior-art search on the specific combination (full pool recorded per step, deterministic baseline, unaided-facts pass test, pre-written fail criterion, operator-set importance) is offered, not done; the operator's "nobody else would think of it" was not accepted as stated (the parts are known: kNN-LM, contextual biasing, prefilling).

7. Problem & Resolution

Problem: Experiment 002 had no tool; reweighting inside the pool cannot reach facts the model does not hold, and the 4B confabulates smoothly about the operator's systems. Resolution so far: recorder 1.3.0 puts retrieved terms at the start of the reply and records what the model did with them, including the natural probability of the first injected token; the metrics to read it are designed (section 6, item 8) but not built; nothing has run on the real server.

8. Risk Analysis (new; adding to risks 1 to 40)

  1. Off-the-path queries. The model takes the likeliest reading, so an unorthodox query meets a weaker baseline than a plain one; a stress test, but results will not transfer to plain queries unless both are run.
  2. A moving heat table breaks comparability. Freeze and record it per run.
  3. Echo through reinforcement. Raising a term because the model used it amplifies what the model already says. Signals must come only from the operator.
  4. A pinned name that fires on every query is the B leak again. It must fire only on a match.
  5. Alias gaps. Fuzzy skips words under 5 letters and anything with a digit; nicknames and variants need an alias list.
  6. Name/word collisions and bare names. "Clone", "Clarion", "Fin" are pulled to their common meaning; a bare injected name is only a pointer.
  7. Contention from the video job on the Clone's four cores distorts timings and can stall runs on 8082 (extends 29 and 40).
  8. Names in run files. Injected runs save terms and replies in the run folder (mode 600, on the operator's box); pinned names would also sit in hotwords.db in plain text.
  9. 1.3.0 is unproven on the real 4B. Only a stand-in server was used. Unverified: tokenization at the boundary between the prompt's final newlines and the injected text; whether the real server's cache behaves as in the stand-in; the Clone's Python 3.6.
  10. The id-order rarity idea is unverified (section 6, item 7); until checked it is a hypothesis, not a reference.

9. Code Change

Delivered this session (all kept, never overwritten): NaturalRecorder_v1.3.0.zip (new). Earlier: NaturalRecorder_v1.2.1.zip, v1.2.0, v1.1.0, v1.0.0; Exp000_queries_v1.0.0.zip; Hotwords_v2.1.0.zip (unchanged); Memstore_Clone_v1.0.16.zip (unchanged). Nothing else delivered; this checkpoint is the only other file.

10. Open items

11. Standing operator rules (carry forward)