📝 Experiment_001_checkpoint_7.mdv4.6.2 · 2026-10-28

Experiments 000, 001, 002 — Checkpoint (7)

Operator: Andrew. Written 2026-10-27 16:55 Sydney. Follows Experiment_001_checkpoint_6.md (kept, never overwritten; _5, _4, _3, _2 and Experiment_001_checkpoint.md also kept). Status: Experiment 000 (natural baseline) is run and analysed. Experiment 001 style reweighting was tried through --boost and shown to be a risky lever that cannot reach facts outside the pool. The operator has chosen the way forward for Experiment 002: inject weighty tokens at the very start of the reply, before any reply token is chosen. Nothing for that is built yet; the operator has not said "go". Resume: paste this file only into a fresh conversation and attach NaturalRecorder_v1.2.1.zip, Exp000_queries_v1.0.0.zip, Hotwords_v2.1.0.zip and Memstore_Clone_v1.0.16.zip (or the newest Memstore tree). This file is written to stand alone. Older checkpoints are needed only for the full text of risks 1 to 29 (checkpoint 2 holds risks 1 to 13, checkpoint 3 holds 14 to 21, checkpoint 5 holds 22 to 24, checkpoint 6 holds 25 to 29); this file adds risks 30 to 40. Where files differ, the newest wins. Terms: LLM1 = the chat model (Qwen3-4B). LLM2 = the second module that would weight or add words; it is also the name of the second 4B copy ("4b2", port 8082). RAG = choosing stored arrays that match the query (here hotwords.py pseudo). The pool = the top 50 candidates at one step. Prefill = tokens placed at the start of the assistant's turn so the model reads them as already said.

1. The aim and the experiments

Standing aim: find out whether a second module (LLM2 plus retrieval) can move LLM1's next-word candidacy inside the query's subject, so a corrected mistake is not repeated.

2. Environment and tools (everything needed to run, in one place)

3. What happened this session

  1. Launcher memory display fixed and confirmed on restart (entries show; 1.0.4 waits up to 30 s for the memory: line).
  2. Five entries added (ids 4 to 8), nine queries drafted, Experiment 000 set up (A and B).
  3. Runs analysed. The recorder's end-of-reply token (<|im_end|>, id 151645) has empty text in the candidate list, so 1.0.0 never saw it and replies ran on to max_steps. 1.1.0 finds stop ids through /tokenize.
  4. "No completion_probabilities" failures after ##/### headings: 1.1.0 retries with other seeds and saves the server's raw answer. Cause unconfirmed (suspect a partial multi-byte piece such as an emoji).
  5. --boost added (1.2.0) and extended to phrases and numbers by tokenising and boosting the tokens (1.2.1, minimum 3 letters on a phrase's first token).
  6. hotwords.db explored: the operator added the follow-up sentence "YES, Clone is best placed for that function: it is the orchestrating server, and its /bin is the master for shared scripts." (array #25) and pseudo was re-run to see what it retrieves.
  7. Boost runs on Q3 (port 8082, greedy): Linux, EOL, end-of-life, Linux + end-of-life, and Linux + "CentOS 6.10" + end-of-life.
  8. The operator concluded that messing with reply tokens is RISKY and asked for the original approach: context and token injection (section 6).

4. Results

Experiment 000: A against B

Boost runs on Q3 (A's reply is the reference)

hotwords.db

5. Lessons learned

  1. Reweighting only works on words already in the pool. A word absent from the top 50 cannot be boosted (CentOS, 6.10, end-of-life never appeared). Facts the model does not hold need adding, not reweighting.
  2. Small boosts cascade. Three overrides at 3% to 8% raw probability rewrote a 150-step reply. A standing weight acts at every step and compounds.
  3. The direction of a boost is not controlled. EOL became "Epic Games"; Linux became a generic distro, not CentOS. Boosting a word moves the frame, not the fact.
  4. A fluent wrong answer is the worst failure. The 4B confabulates smoothly ("Apple Clone OS"). Fluency (mean probability) cannot be the pass test on its own; correctness against the entry must be.
  5. Memory in the prompt works too well and too bluntly (B). It overrides, parrots, changes style, and leaks into unrelated queries. It is not a fair target without relevance filtering; a filtered B (memstore.py inject WORDS --limit) is the fairer one.
  6. Tokenise phrases before boosting them. A number such as 6.10 is several tokens; boosting the first piece alone lifts unrelated words. The first piece of a phrase needs a minimum of 3 letters or every E/C is lifted.
  7. Match by normalised piece text. ' Linux', 'Linux' and '**Linux' are one word.
  8. The end-of-reply token has no text in the candidate list; identify it by id.
  9. Take probabilities with a retry. The server sometimes returns no candidates (after headings); retry with another seed and keep the raw answer.
  10. Greedy runs are repeatable and the two 4B copies interchangeable, so A and B runs can be compared directly.
  11. Retrieval is by shared words. A question only retrieves a memory that shares vocabulary; the RAG stage as built does not change the prompt.
  12. The store stage pollutes hotwords.db with every chat query and reply. Switch it off (enabled=no in stages.control) before experiments; the operator decides.
  13. A weighty, early, context-side push is the plan, not a late, light, per-step one. The operator's direction: tokens before any reply token is chosen, and weighty.
  14. Process lessons: the operator ran an old recorder file (v1.0.0 banner) and got "unrecognized arguments": every delivery shows its version on start, so a mismatch is visible at once. Unzipping creates a folder. A command that reads a run must name the right folder (the wrong folder was read twice). pkill -f and ps | grep patterns can match the shell's own command line and kill it; use a bracketed pattern or a PID.
  15. Claims withdrawn or corrected: "the order of work is settled" for storing B replies (entries or the operator's follow-up sentence may be better sources); "expect little or no effect" for the EOL boost (it was wrong).

6. Direction chosen for Experiment 002 (agreed in principle, not built)

The operator's words: "approach this the original way, by context and token injection", then "token injection prior to any reply tokens selected?", then "but we need WEIGHTY tokens".

7. Problem & Resolution

Problem: no way to see whether a second module can move the model's next-word candidacy; reweighting inside the pool was untested, and the 4B invents facts about the operator's systems. Resolution so far: the natural baseline is recorded and repeatable; memory in the prompt (B) is shown to dominate and leak; reweighting (boost) is shown to change the frame without reaching facts, and to cascade; the next test is a weighty prefill (section 6).

8. Risk Analysis (new; adding to risks 1 to 29)

  1. Cascade from small boosts. A boost applied at every step changes the whole reply (Linux run). Keep any weight to one early point.
  2. Uncontrolled direction. A boosted word shifts the frame to its commonest neighbour (Linux to Ubuntu, Debian, Fedora; EOL to Epic Games).
  3. Fluent confabulation passes a fluency check. A reply with high mean probability can still be wrong. The pass test is correctness against the stored entry.
  4. B leaks into controls and overrides, so it is not a clean target; B is not relevance-filtered.
  5. A prefill that is a bare list of terms may be continued as a list, or restated, putting the weight into repetition. The stored-sentence ceiling invites parroting.
  6. Retrieval overlap limit and filler in hot terms weaken the prefill: a query that shares no vocabulary with the memory retrieves nothing, and filler words score into arrays.
  7. The store stage writes every chat into hotwords.db, which echoes the model's own invention back (extends risks 14, 22, 26). Switch it off before experiments.
  8. Wrong-file and wrong-folder runs. An old script or the wrong run folder gives an answer to a different question (happened twice). Check the version banner and the folder name.
  9. The "no candidates" fault after headings is mitigated, not understood; a retry changes the seed and so the path in sample mode.
  10. Unbuilt prefill may be mistaken for a pass. If the injected words are the fact, a reply repeating them proves nothing. Only unaided connected facts count.
  11. Four cores shared by two 4B copies and go-betweens: timing is noisy, and a request to 8082 during a run evicts its cache.

9. Code Change

All deliveries are kept; versions are never reused.

10. Open items

11. Standing operator rules (carry forward)