Experiments 000, 001, 002 — Checkpoint (3)

Operator: Andrew. Written 2026-10-27 05:27 Sydney. Follows Experiment_001_checkpoint_2.md (kept, never overwritten; Experiment_001_checkpoint.md also kept). Status: two tools built and delivered (hotwords 2.1.0, contextdef_05b 1.0.0). Experiments themselves not started; the recorder (build step 1) is still not written. Resume: paste this file together with Experiment_001_checkpoint_2.md into a fresh conversation. Where the two differ, this file wins; the differences are listed in section 3. Related: MEMSTORE_CLONE_CHECKPOINT_2026-10-27_2.md (global parameters).

1. What happened since checkpoint 2

1. Hot word / hot phrase extractor built: Hotwords_v1.0.0.zip, v1.0.1 (non-ASCII crash fixed), then redesigned as Hotwords_v2.0.0.zip and Hotwords_v2.1.0.zip (current). All kept, never reused.
2. Context definer trial built: ContextDef05b_v1.0.0.zip (contextdef_05b.py).
3. Design moved from "documents" to small arrays of hot terms (see section 2).
4. The operator doubted the concept ("worried the whole concept is crap"). Claude's assessment is recorded in section 7 so the new chat starts from it.

2. Operator decisions and statements (design-significant, in order)

• No documents. Store only small arrays of tokens derived from LLM1's queries and replies. This should make RAG very fast and mostly cache-based.
• Array size is about relevance, not space.
• Position matters: early words usually carry more significance.
• The output of the hotwords tool is the input to LLM2, as the pseudo query, and it includes RAG's choices.
• LLM2's query is theoretical: its value is in what it deems important, coherent or not. Coherence is LLM1's job.
• Fuzziness is important (spelling variants must match: 2C-B, not 2-CB, but both must find each other).
• Influence is rarely overt: weak matches matter.
• Wants a few queries pushed through the 0.5B ("5b", for turnaround speed) as the context definer, to see what comes out.
• Wonders how all this works for coding (answer in section 7).

3. Differences from checkpoint 2

Topic in checkpoint 2	Now
Retrieval over passages / "documents", support table from retrieved passages	Retrieval over hot-term arrays (tool hotwords.py). No text kept. The support table is built from the arrays RAG chose.
Background frequency table from forum text or the attachments corpus	Dropped for now. Rarity comes from the counted texts themselves (word counter keyed by spelling key). Known weakness, risk 16 in section 5.
Context definition: rule-based only	Rule-based design stands (still not built). Plus a trial using the 0.5B to produce subject / intent / terms. Position weighting should be added to the rule-based subject score (early query words count more).
LLM2 input = context definition + 50 candidates	LLM2 input = the pseudo query (query's hot array + RAG's related terms, each with a weight) + the 50 candidates. How the pseudo query becomes 50 weights is not yet designed; proposal: a candidate's weight = the weight of the pseudo-query term it matches (spelling key, then fuzzy for plain words), 0 if none. Not agreed.
Words/phrases as arrays of embeddings, exact match in v1	Unchanged in intent, but v1 matching is now spelling key plus fuzzy (edit distance) on plain words. Still no embeddings.
Settings list (SIM_FLOOR etc.)	Several are now concrete (section 9).

Everything else in checkpoint 2 (experiments 000/001/002, decisions on stage separation, driving LLM1 one word at a time, pseudo code, risks 1 to 13, build order) stands.

4. Problem & Resolution

Problem: the retrieval side of Experiment 001 needs a way to turn LLM1's own history into a small, fast, relevance-ordered store that can be matched against a new query, tolerant of spelling variants, and that yields a query LLM2 can use. Resolution: hotwords.py 2.1.0 (details below) and a 0.5B context-definer trial. Both are tools, not yet wired into any experiment.

hotwords.py 2.1.0 (current)

• One ranked array per query or reply, no text kept. Score = (1 + ln count) x rarity x position weight. Position weight = 1 / (1 + first word index / 20). Rarity = ln((N+1)/(df+1)) + 1 over the texts counted so far, taken as if the new text were already counted.
• Array keeps terms scoring at least 40% of the top term, minimum 3, maximum 12. Phrases (2 to 3 words) qualify only if they occur at least twice inside their own text.
• Dedupe: same text (whitespace collapsed) changes nothing but times_seen; a different text giving the same array collapses into it (vocab untouched).
• Spelling key: accents, hyphens, apostrophes removed (2C-B = 2-CB = 2cb). Terms merge by key inside a text, are stored and counted by key, shown as first typed.
• Fuzzy matching (in pseudo only): plain words of 5+ letters match stored words within 1 edit (up to 8 letters) or 2 edits (9+), swap of neighbours = 1 edit, weight factor 1 - 0.1 per edit, best 3 variants. Words containing a digit and words under 5 letters never fuzzy-match (2C-B vs 2C-I and 25I-NBOMe vs 25B-NBOMe are different drugs).
• pseudo command builds the pseudo query and stores nothing. RAG match = share of the query's weight covered by a stored array (query rel x stored rel x factor, best match per query term). Defaults: 5 arrays, 12 extra terms, match floor 0.05. Formats: line (two lines query: / related:, phrases quoted, --weights adds term:0.42), table, json (full trace incl. fuzzy matches).
• Other commands: add, show, top, stats. Database schema 3; older databases are refused, not converted (HOTWORDS_DB points to a new file). Python 3.6 and SQLite 3.6.20 safe; UTF-8 handled explicitly. 98 checks pass on Python 3.11, 3.12, 3.13. Not yet run on the Clone's 3.6 / 3.8 / 3.15.
• A smoke test showed it works as designed: for "Is serotonin syndrome likely at a normal MDMA dose?" the related terms were "serotonin syndrome" | dangerous | risk | worries | combined | maois; the forum-slowness array was left out.

contextdef_05b.py 1.0.0 (trial)

• Sends queries to the 0.5B llama-server (default port 8081; /v1/chat/completions by default, --api completion for older builds with Qwen chat tags). Greedy, fixed seed, one worked example in the prompt. Asks for SUBJECT, INTENT, TERMS (most important first); parses forgivingly; marks each term as in the query or [+] added by the model; prints timing; saves JSON per query plus run.json under $MEMDIR/ctxdef_try/<run>/. Never touches the real context-definition store.
• Tested only against a stand-in server (38 checks). Not yet run against the real 0.5B. Unknown: whether the build has the chat route, how often the 0.5B breaks the three-line format.

5. Risk Analysis (new, adding to checkpoint 2's 13)

14. Echo. The archive is mostly LLM1's own replies, so boosting terms from it mostly amplifies what LLM1 already says. It helps only if the archive holds things LLM1 does not produce unaided (the operator's corrections, preferred wording). Highest-level risk to the whole idea.
15. Terms without context. A boosted term does not know whether the next word should be that term.
16. Rarity from the counted texts themselves. Once the archive is mostly about one subject, that subject's words become common and are marked down against generic words (seen already in a small test: "likely", "normal", "dose" outranked "serotonin", "syndrome" in the query's own array). Remedy would be an outside reference; the earlier seed-background was removed in 2.0.0.
17. Phrases almost never repeat inside a short query, so queries yield words; phrases come from replies.
18. Fuzzy matching is spelling only. It will not match short words (a misspelt MDMA or LSD) or any word with a digit, by design. Fuzzy search scans the whole word list per run: fine for thousands, to be revisited at hundreds of thousands.
19. Scores in stored arrays are snapshots taken when the array was made.
20. Code is not handled: fenced code is stripped, snake_case splits, and fuzzy would merge get_user with get_users. A code mode (keep code, whole identifiers plus parts, no fuzzy) is not built.
21. The 0.5B is the same family as LLM1 and weak at structure; its context definitions may be malformed or biased the same way (checkpoint 2, LLM2 weakness).

6. Code Change

Delivered this session (all kept): Hotwords_v1.0.0.zip, Hotwords_v1.0.1.zip, Hotwords_v2.0.0.zip, Hotwords_v2.1.0.zip, ContextDef05b_v1.0.0.zip, Experiment_001_checkpoint_2.md (previous). Release cards given with each; the open line on each is "run on the Clone's Python 3.6 / 3.8 / 3.15" and, for the context definer, "run against the real 0.5B".

7. Honest assessment given to the operator (so it is not lost)

• The plumbing carries little risk. The risky assumption is that boosting terms drawn from the archive improves LLM1's replies (risks 14 and 15).
• The mechanism is not new nonsense: pulling next-word weights toward a retrieved set is an established technique (kNN-LM, contextual biasing in speech recognition) and usually moves things modestly, which fits "influence is rarely overt".
• Experiments 000 and 001 can falsify it cheaply (about ten stored pairs; fluency gap and candidacy shift). Recommended, not yet agreed: write down a fail criterion before running (for example a large fluency gap with no shift toward the corrected wording), so a null result is an answer.
• For coding: the array and RAG structure carries over; the tokenizer and fuzzy rules do not (risk 20).

8. Build order (updated)

1. Run contextdef_05b.py on a few queries; report malformed rate, added terms, seconds per query. Run hotwords.py on the Clone's Python versions; feed it real query/reply pairs.
2. Natural-run recorder (raw top 50 per step, fixed context). Experiment 000 runs on it, and its query/reply pairs feed hotwords.py add.
3. Decide how the pseudo query becomes 50 weights (section 3, table), then the blend and candidacy comparison. Experiment 001.
4. Context definition builder (rule-based) with position weighting, saved files and index. Compare with the 0.5B trial output.
5. Injection and the α dial. Experiment 002.
6. LLM2 model pass in place of counting, judged against step 3.

9. Open items

• Prompt-retrieval arm (LLM1 with retrieved passages in its prompt, no influence) as the honest benchmark: still undecided. Note it needs passages, which the array design no longer stores; it would need the original replies kept somewhere else.
• Fail criterion for Experiment 001 (section 7).
• An outside reference for rarity (risk 16).
• How the pseudo query reaches LLM2 and becomes 50 weights.
• Settings now concrete in hotwords.py: H 20, keep 0.4, array 3 to 12, minimum phrase count 2, RAG 5 / 12 / 0.05, fuzzy 5 letters / 1 and 2 edits / 0.1 step / 3 variants. Still open: embedding model, tokenizer check (4B and 0.5B vocabularies), whether the sampler returns probabilities before or after cuts, lift smoothing, phrase window, ALPHA steps.
• Whether the 0.5B should do the context definition at all, or only the rule-based definer.

10. Standing operator rules (carry forward)

• Reply header Claude: [Month DD, YYYY, HH:MM:SS AM/PM] (casual mode), real Sydney time from a code run immediately before each reply.
• Never suggest pkill/kill on services. Never rename his scripts. Deployment timing is his call.
• Code delivery: bump @version/@date; manual with version history; version shown once at start; zip with a dotted, never-reused version; xmllint --noout on XML; list unrequested changes BEFORE making them; end with a release card (include "legacy zipper used" only for SMF package zips).
• Critique and objections before building; statements over questions; do not mimic his tone; expand acronyms; no invented jargon; PDF never Word.
• Checkpoint names get a new _N, never reused (next is _4). Prefers short conversations and a fresh chat per chapter.
• Terminology: "operator", "query", "reply".
• Environment: Clone is CentOS 6.10, Python 3.6 (also 3.8 and 3.15 per the operator), SQLite 3.6.20, llama.cpp: Qwen 4B on 8080 (LLM1), 0.5B on 8081. Python 3.6 on a POSIX/C locale does not switch to UTF-8, so non-ASCII must be handled explicitly (already done in both tools).