Experiment 001 — Checkpoint

Date: 2026-10-27 (Sydney). Status: design agreed, pseudo code drafted, no program written yet. Resume: paste this file into a fresh conversation; the next step is build step 1 below.

Decisions (the operator's)

• LLM1 runs "au naturel" first to set the baseline; the influenced run is compared to it.
• LLM2 adds no words. It only reweights LLM1's own top 50 next-word candidates, strictly within the query's defined subject.
• Targeted value: candidacy for next-word selection (a word's standing in the pool at the moment of selection).
• Common candidates get their values added then halved (simple average). That is the crude baseline of influence.
• Two outputs per query: LLM1 normal and LLM1 influenced.
• Start with the simplest weighting (counting against retrieved passages). The LLM2 model pass comes only after it, and must beat it.
• Words and phrases are held as arrays of embeddings (a word = 1 vector, a phrase = 2 or more, kept in order, never averaged into one).

1. Problem & Resolution

Problem: no standard exists for measuring how much an outside weighting moves LLM1's next-word candidacy, nor for proving it stayed inside the subject.

Resolution (design):

1. Context definition built per query by plain rules (subject, intent, in-scope turns, standing rules), logged with the run.
2. Retrieval runs once per reply from the context definition. Passages below a similarity floor are discarded. If too few candidates are supported, LLM1 runs natural.
3. A support table is built from the retrieved passages: words and phrases, each with a lift (frequency in the retrieved passages relative to ordinary text) and its embedding array.
4. LLM1 is driven one word at a time through llama-server: prompt as token ids, n_predict 1, n_probs 50, prompt cache on, raw probabilities (temperature 1, no top-k, top-p 1, min-p 0).
5. Each step the 50 candidates are weighted from the support table and blended. The selected word is appended and LLM1 is re-driven.
6. Comparison: free-running (what a reader sees) and fixed-context (blended candidacy scored along the normal reply's path, so divergence cannot muddy it).

How RAG and phrases work (LLM2's role in 001):

• LLM2 here is the retrieval-and-weighting module, with a fixed interface: input = context definition + 50 candidates, output = one weight per candidate. A model pass can later replace counting without changing anything else.
• Retrieval: keyword plus embedding search over the forum corpus, from subject terms and in-scope excerpts; top-k chunks above the floor, cached for the whole reply.
• Phrases matter only through a pointer. A support phrase [serotonin, syndrome] is an ordered array. When the reply so far ends in "serotonin", a pointer to that phrase is active and the candidate "syndrome" inherits the phrase's lift. Without the pointer, phrases would contribute nothing beyond their single words. Averaging the array into one vector would lose the order, so it stays an array.
• Matching in version 1 is exact on words. Later, cosine similarity between the candidate's vector and the phrase's next vector gives soft matching.
• Influence is applied only at word starts. Mid-word, candidates are fragments, and candidacy is left unchanged.

2. Risk Analysis

1. Retrieval quality sets containment. Off-subject passages give off-subject weights even though every candidate is legitimate. Mitigation: similarity floor, and fall back to natural when nothing clears it.
2. Common words win by default. Mitigation: lift against background frequency, stopword filter.
3. Lean from the blend rule: supported candidates move toward the weights, unsupported ones are untouched, so supported words can be pushed down if the weights run low. Mitigation: normalise both over the 50 first; compare against natural; watch.
4. Phrase pointers can fire by chance on a first word. Mitigation: drop a pointer when the next word doesn't match, and after a window.
5. Subword fragments versus whole words. Mitigation: word-start-only rule above.
6. The automatic subject can be wrong, and everything follows it. Mitigation: the context definition is logged per run.
7. Weak effect expected. At most steps one candidate dominates, so a halved blend rarely changes the chosen word. Judge by candidacy shifts, not changed replies. A null on replies is a valid 001 result.
8. Copying risk: weighting toward retrieved wording can push toward verbatim echo. Watch in the comparison.
9. Tokenizer: pool entries are only comparable if one tokenizer is used throughout.
10. Attention-based "referenced words" record (needs Hugging Face transformers, not llama-server) is deferred, not part of 001.

3. Code Change

None delivered. Pseudo code only:

SETTINGS (written into every run record)
  MODEL_FILE, PROMPT_TEMPLATE, N_CAND=50, BLEND=0.5
  SIM_FLOOR=?  MIN_OVERLAP=?  PHRASE_MAX_N=3  PHRASE_WINDOW=?  MAX_STEPS

run_query(query, history):
  ctx   = build_context_definition(query, history)     # rules only, no model
  docs  = retrieve(ctx)                                # once per reply
  docs  = [d for d in docs if d.sim >= SIM_FLOOR]
  table = build_support_table(docs)                    # words + phrases -> lift, vecs[]
  normal = generate("natural");  infl = generate("influenced")
  fixed  = score_along_path(normal.ids)                # fixed-context comparison
  write_report(ctx, docs, table, normal, infl, fixed)

generate(mode):
  ids = prompt_ids(query, ctx);  active = []           # active = phrase pointers
  loop until end-of-reply or MAX_STEPS:
    cand = llm1_top50(ids)                             # raw: temp 1, no cuts
    p1   = normalise(cand.p)                           # over the 50
    p    = p1
    if mode == "influenced" and at_word_start(ids):
      w = support_weights(cand, table, active)         # LLM2's job (counting first)
      if count(w > 0) >= MIN_OVERLAP:
        p = normalise([(p1[c]+w[c])/2 if w[c]>0 else p1[c]  for c in cand])
    pick = argmax(p);  record(step, cand, p1, p, pick)
    ids += pick.id;  active = advance_pointers(active, pick, table)
  return ids

support_weights(cand, table, active):
  for c in cand:
    s[c] = max( table.word_lift(c.word),
                max over phrases in active whose next word matches c: phrase.lift )
  return normalise(s)                                  # 0 where unsupported

advance_pointers(active, pick, table):
  step forward pointers whose next word == pick.word; drop the rest
  start pointers for phrases whose first word == pick.word
  drop pointers older than PHRASE_WINDOW or finished

table entry: { text, words[], vecs[] (1 for a word, 2+ for a phrase), lift, n_docs }
lift = (freq in retrieved) / (freq in ordinary text), smoothed and capped

No program delivered, so the release card does not apply yet.

Build order

1. Natural-run recorder (raw top 50 per step, fixed context). This is the standard.
2. Context definition builder, retrieval once per reply, similarity floor.
3. Support table, counting weights, blend, candidacy comparison.
4. Only then the LLM2 model pass, judged against step 3.

Open settings to fix before the first run

SIM_FLOOR, MIN_OVERLAP, PHRASE_WINDOW, lift smoothing and cap, retrieval top-k, and the embedding model for retrieval and the phrase arrays.