Experiments 000, 001, 002 — Checkpoint (2)

Operator: Andrew. Written 2026-10-27 04:44 Sydney. Supersedes Experiment_001_checkpoint.md (that file is kept, never overwritten). Status: design only. No program written or delivered. Resume: paste this file into a fresh conversation. Next step is build step 1 (the recorder). Related file: MEMSTORE_CLONE_CHECKPOINT_2026-10-27_2.md (global parameters: /root/memstore, Python 3.6 on the Clone, Qwen 4B on 8080 and 0.5B on 8081, memstore corrections, memtok).

1. Aim

Find out whether a second module (LLM2 plus retrieval) can move LLM1's next-word candidacy, strictly inside the query's subject, and whether that is worth having compared with plain retrieval in the prompt. Candidacy = a word's standing in the pool at the moment of selection (eligibility after the sampler's cuts, then weight).

2. The three experiments

Experiment 000 — background variation (control and archive)

• Part A, archive: 10 or more queries drawn at random from the operator's transcript archive, of different kinds (short fact, how-to, pharmacology, opinion, pronoun follow-up, off-subject). Each goes through LLM1 natural. Each pair is saved with the full record (raw top 50 per step, context definition, settings). Screen queries for secrets before saving.
• Part B, repetition: pick one query (mid-length reply, some spread in its candidates) and run it about 20 times under three conditions:
  1. fresh context, greedy decoding: expected identical. If not, the server is non-deterministic, and the "greedy is repeatable" assumption fails;
  2. fresh context, fixed temperature, different seeds: the pool's natural spread;
  3. same query repeated inside one growing conversation: repetition proper.
• Measures: exact-match count, first divergence step, word-level difference between consecutive replies, movement of the top 50 at the same step.
• Run LLM1 directly on port 8080 or with memory injection off. The go-between injects memstore facts, which would contaminate "au naturel".
• Needs only the recorder, so it can start first. Purpose: the amount replies vary when nothing is changed, so later differences can be judged against it.

Experiment 001 — reweight only (no new words)

• LLM2 adds no words. It reweights LLM1's own top-50 candidates, weights taken from what retrieval finds for the subject.
• Blend: for candidates the support table supports, new value = (LLM1 + weight) / 2 on probabilities normalised over the 50; unsupported candidates unchanged; renormalise.
• Two outputs per query: LLM1 normal and LLM1 influenced. Both free-running, plus a fixed-context comparison (blended candidacy scored along the normal reply's path, so divergence cannot muddy it).
• Weighting version 1 is counting (a word's lift = frequency in the retrieved passages relative to ordinary text). No LLM2 model pass yet.
• Influence applied only at word starts. Falls back to natural when fewer than MIN_OVERLAP candidates are supported or no passage clears the similarity floor.
• Measures: candidacy shift per step, subject containment, where replies differ. A null on replies is a valid result.

Experiment 002 — adding words, heavier weights

• Reverses the 001 rule "no new words". Words from outside LLM1's pool can be injected, with a raised weight: blend = (1 - α)·p1 + α·w, α above 0.5.
• Injection rules: only the next word of an active phrase pointer (anchored by the word before it), only at word starts, at most N_NEW (start 5) per step, boost capped by MAX_BOOST (start 5.0 on the raw score, about 150 times the probability, as in memtok).
• Runs per query: natural; 001 reweight at α 0.5; injection at α 0.5; injection at 0.7 and 0.9. Changing one thing at a time, so the first run where it breaks shows why.
• Measures: injected-pick rate, picks outside the natural top 50, fluency gap (LLM1's average probability per word on its natural reply against each influenced reply), in-subject share of picked words, copying (longest verbatim run shared with the retrieved passages). The α at which fluency collapses is itself a result.
• Depends on 001 (recorder, context definition, retrieval, support table with phrase arrays).

Order: 000, then 001, then 002.

3. Components and decisions

• Context definition (rule-based, no model, Python 3.6 standard library only): inputs are the current query, the last six turns, and the standing rules (memstore corrections, read-only). Subject = top 3 to 5 words or 2-to-3-word runs, scored by rarity (how uncommon in the forum text) plus recurrence in recent turns. Pronoun-only queries inherit the previous subject and are marked inherited. Intent from a small table (question word, command verb, statement). Earlier turns scored in scope or background by overlap with the subject, discounted by age. Retrieval terms: subject phrases plus top terms from in-scope excerpts, capped at about 12, phrases kept as ordered word lists.
• Saved to /root/memstore/ctxdef/<id>.json, one per run, plus an append-only index line per run. Fields: id, generator version, query (capped, with hash), subject (head, terms with scores and a why block, inherited, confidence), intent, in-scope turns with excerpts of at most 200 characters, background turn numbers, standing rules, retrieval terms, settings (including the rarity table hash), content hash. Mode 600 and folder 700 because queries can hold secrets. Nothing here is touched by sync or evict. A top score under a floor marks the definition low_confidence, and the run then goes natural.
• Background frequency table: built once from the forum text, used for both the subject's rarity and the support table's lift.
• Words and phrases are arrays of embeddings: a word is one vector, a phrase is an ordered array of two or more, never averaged into one. A phrase influences a next-word choice through a pointer: when the reply so far ends in the phrase's first word, its next word is "due" and inherits the phrase's lift. Version 1 matches words exactly. Soft matching by cosine similarity comes later.
• LLM2 in 001 is code (retrieval plus counting) behind a fixed interface: input = context definition plus the 50 candidates, output = one weight per candidate. A model pass replaces counting at build step 4 and must beat it. The 0.5B (port 8081) is the first candidate, subject to the tokenizer check. Weakness to expect: it is weak at long lists, and it is the same family as LLM1, so its opinions are correlated. The 4B is the fallback.
• Reading LLM2's scores: llama-server returns only the top N. Request n_probs set to a few hundred and look up the 50 candidate ids. Candidates outside get a floor value. (Corrects an earlier suggestion to read exactly the 50 ids.)
• Driving LLM1: one request per word, prompt as token ids (so the chosen word is exactly the one fed back), n_predict 1, n_probs 50, prompt cache on, raw probabilities (temperature 1, no top-k, top-p 1, min-p 0). Check whether the build returns probabilities before or after the sampler.
• Stages kept separate and each writes its own file under /root/memstore (context definition, retrieval passages with similarity, support table, per-step records for each run, comparison report), tied by one run id and a manifest listing each stage's output id, settings and versions. This allows walking a fault backwards, replaying a later stage from saved earlier output, and swapping stages.
• Bundling into the go-between is deferred. Build the experiments as a standalone script reading queries from a file. The go-between would have to become a generator, which is a large change to working code. Move the modules in later if the results justify it.
• Do not bundle a second model inside the first (adapter or extra head). That is a training project and is not needed.
• Retrieval needs an embedding model plus vector search. Whether the 0.5B can serve chat and embeddings from one instance is untested. A small dedicated embedding model is the safer choice. Still open (memtok checkpoint, question 4).
• Fits with the memtok design: that design reads the top 20 and applies logit_bias on the server side; 001/002 read the top 50 and let the driver choose the word. The context definition feeds either, which is why it is built first.

1. Problem & Resolution

Problem: no standard exists for measuring how much an outside weighting moves LLM1's next-word candidacy, nor for proving it stayed inside the subject, nor for knowing how much replies vary when nothing changes. Resolution (design): the three experiments above, with a recorded natural baseline, an automatically built and saved context definition, and stage-by-stage saved outputs.

2. Risk Analysis

1. Retrieval quality sets containment. Off-subject passages give off-subject weights even though every candidate is legitimate. Mitigation: similarity floor, fall back to natural.
2. Common words win by default. Mitigation: lift against background frequency, stopword filter.
3. Lean in the 001 blend: supported candidates move toward the weights, unsupported are untouched. Normalise first, compare with natural.
4. Phrase pointers can fire by chance. Drop when the next word does not match, and after a window.
5. Subword fragments versus whole words: influence only at word starts.
6. The automatic subject can be wrong and everything follows it. Logged per run, with scores. Inheritance is the riskiest case, flagged.
7. Weak effect expected in 001: one candidate usually dominates, so a halved blend rarely changes the word. Judge by candidacy shifts.
8. 002: grammar breaks if arbitrary words are injected (hence anchoring); copying and content drift at high α; in-subject is not the same as correct at that point.
9. Greedy decoding may not be repeatable on this server; 000 checks it.
10. One tokenizer must be used throughout, or pool entries are not comparable. Check the 4B and 0.5B vocabularies (ids 311, 728, 32671).
11. The files hold query excerpts. Permissions and caps as above.
12. Rarity-based subject choice can pick an incidental word such as a typo or a name. Recurrence term, cap, and visible scores limit it.
13. Attention-based "referenced words" record needs Hugging Face transformers (not llama-server) and is deferred.

3. Code Change

None delivered. First deliverable would be contextdef.py with commands build, show <id>, list. The recorder comes before it (build step 1). All deliveries follow the standing delivery rules, including the release card.

Pseudo code (001; 002 adds the lines marked):

SETTINGS (written into every run record)
  MODEL_FILE, PROMPT_TEMPLATE, N_CAND=50, BLEND=0.5
  SIM_FLOOR=?  MIN_OVERLAP=?  PHRASE_MAX_N=3  PHRASE_WINDOW=?  MAX_STEPS
  [002] ALPHA, N_NEW=5, MAX_BOOST=5.0

run_query(query, history):
  ctx   = build_context_definition(query, history)     # rules only, no model
  docs  = retrieve(ctx)                                # once per reply
  docs  = [d for d in docs if d.sim >= SIM_FLOOR]
  table = build_support_table(docs)                    # words + phrases -> lift, vecs[]
  normal = generate("natural");  infl = generate("influenced")
  fixed  = score_along_path(normal.ids)                # fixed-context comparison
  write_report(ctx, docs, table, normal, infl, fixed)

generate(mode):
  ids = prompt_ids(query, ctx);  active = []           # active = phrase pointers
  loop until end-of-reply or MAX_STEPS:
    cand = llm1_top50(ids)                             # raw: temp 1, no cuts
    p1   = normalise(cand.p)                           # over the 50
    p    = p1
    if mode == "influenced" and at_word_start(ids):
      [002] cand += anchored_injection(active, table, N_NEW)   # next words of active phrases only
      w = support_weights(cand, table, active)         # LLM2's job (counting first)
      if count(w > 0) >= MIN_OVERLAP:
        [001] p = normalise([(p1[c]+w[c])/2 if w[c]>0 else p1[c] for c in cand])
        [002] p = normalise([(1-ALPHA)*p1[c] + ALPHA*w[c] for c in cand], boost capped by MAX_BOOST)
    pick = argmax(p);  record(step, cand, p1, p, pick)
    ids += pick.id;  active = advance_pointers(active, pick, table)
  return ids

support_weights(cand, table, active):
  for c in cand:
    s[c] = max( table.word_lift(c.word),
                max over phrases in active whose next word matches c: phrase.lift )
  return normalise(s)                                  # 0 where unsupported

advance_pointers(active, pick, table):
  step forward pointers whose next word == pick.word; drop the rest
  start pointers for phrases whose first word == pick.word
  drop pointers older than PHRASE_WINDOW or finished

table entry: { text, words[], vecs[] (1 for a word, 2+ for a phrase), lift, n_docs }
lift = (freq in retrieved) / (freq in ordinary text), smoothed and capped

4. Build order

1. Natural-run recorder (raw top 50 per step, fixed context). This is the standard. Experiment 000 runs on it.
2. Context definition builder, saved files and index; retrieval once per reply with the similarity floor.
3. Support table, counting weights, blend, candidacy comparison. Experiment 001.
4. Injection and the α dial. Experiment 002.
5. Only then the LLM2 model pass in place of counting, judged against step 3.

5. Open items

• The prompt-retrieval arm (LLM1 with the retrieved passages in its prompt, no influence): proposed as the honest benchmark, costs one extra run per query. Not yet decided by the operator.
• Settings to fix: SIM_FLOOR, MIN_OVERLAP, PHRASE_WINDOW, lift smoothing and cap, retrieval top-k, scope floor, recency decay, ALPHA steps, the embedding model for retrieval and phrase arrays.
• Tokenizer check, embedding server choice (model, port, pooling), and whether the sampler's probabilities are returned before or after cuts.
• Where the background frequency table is built from (forum text or the attachments corpus).

6. Standing operator rules (carry forward)

• Reply header Claude: [Month DD, YYYY, HH:MM:SS AM/PM] (casual mode), real Sydney time from a code run immediately before each reply.
• Never suggest pkill/kill on services. Never rename his scripts. Deployment timing is his call.
• Code delivery: bump @version/@date; manual version-history entry; version shown once at start; zip with a dotted, never-reused version; xmllint --noout on XML; list unrequested changes BEFORE making them; end with a release card (include "legacy zipper used" for package zips).
• Critique and objections before building; statements over questions; do not mimic his tone; expand acronyms; no invented jargon; PDF never Word; checkpoint names get a new _N and are never reused (next is _3); prefers short conversations and a fresh chat per chapter.
• Terminology: "operator", "query", "reply".