📝 NEXUS.mdTheme:

Project nexus: 3 linked posts

Links only. Made by nexus v2.2.2 for post 49866. Rebuilt automatically; edits are overwritten.

Every post below belongs to one project. This same overview is attached to each of them, so you can start from any post and see all the others. Step numbers give the order the posts were written (1 = oldest). Dates are UTC.

Master post: Project "Influence" Master (topic 8078, post #49866, step 3, 2026-10-26, 3 posts in the project, 2 of them under it) - chosen by hand
Posts in the project: 3
Began: 2026-10-11 (post #49761); latest: 2026-10-26 (post #49866); span: 15 days
Topics locked: 0 of 3

Hierarchy (a post is indented under the post it belongs to)

Project "Influence" Master (topic 8078, #49866, step 3, 2026-10-26) MASTER (this post)
- AI tools to Monitor the Site's local Qwen AI Systems (like "model inspector") (topic 7978, #49761, step 1, 2026-10-11)
  - Project 'Memstore-AI-Chatbot' (for Qwen2 and 3) (topic 8077, #49865, step 2, 2026-10-26)

Chronology, oldest first

Step	Topic no.	Post	Title	Posted	Topic status	Status	Under	Role
1	7978	#49761	AI tools to Monitor the Site's local Qwen AI Systems (like "model inspector")	2026-10-11	open	current	#49866	has contributor #49865
2	8077	#49865	Project 'Memstore-AI-Chatbot' (for Qwen2 and 3)	2026-10-26	open	current	#49761	contributes to #49761; has contributor #49866
3	8078	#49866 (this post)	Project "Influence" Master	2026-10-26	open	current	MASTER	contributes to #49865

Relations

#49865 (topic 8077) contributes to (sits under) #49761 (topic 7978)
#49866 (topic 8078) contributes to (sits under) #49865 (topic 8077)

📝 Experiment_001_checkpoint.mdTheme:

Experiment 001 — Checkpoint

Date: 2026-10-27 (Sydney). Status: design agreed, pseudo code drafted, no program written yet. Resume: paste this file into a fresh conversation; the next step is build step 1 below.

Decisions (the operator's)

LLM1 runs "au naturel" first to set the baseline; the influenced run is compared to it.
LLM2 adds no words. It only reweights LLM1's own top 50 next-word candidates, strictly within the query's defined subject.
Targeted value: candidacy for next-word selection (a word's standing in the pool at the moment of selection).
Common candidates get their values added then halved (simple average). That is the crude baseline of influence.
Two outputs per query: LLM1 normal and LLM1 influenced.
Start with the simplest weighting (counting against retrieved passages). The LLM2 model pass comes only after it, and must beat it.
Words and phrases are held as arrays of embeddings (a word = 1 vector, a phrase = 2 or more, kept in order, never averaged into one).

1. Problem & Resolution

Problem: no standard exists for measuring how much an outside weighting moves LLM1's next-word candidacy, nor for proving it stayed inside the subject.

Resolution (design):

Context definition built per query by plain rules (subject, intent, in-scope turns, standing rules), logged with the run.
Retrieval runs once per reply from the context definition. Passages below a similarity floor are discarded. If too few candidates are supported, LLM1 runs natural.
A support table is built from the retrieved passages: words and phrases, each with a lift (frequency in the retrieved passages relative to ordinary text) and its embedding array.
LLM1 is driven one word at a time through llama-server: prompt as token ids, n_predict 1, n_probs 50, prompt cache on, raw probabilities (temperature 1, no top-k, top-p 1, min-p 0).
Each step the 50 candidates are weighted from the support table and blended. The selected word is appended and LLM1 is re-driven.
Comparison: free-running (what a reader sees) and fixed-context (blended candidacy scored along the normal reply's path, so divergence cannot muddy it).

How RAG and phrases work (LLM2's role in 001):

LLM2 here is the retrieval-and-weighting module, with a fixed interface: input = context definition + 50 candidates, output = one weight per candidate. A model pass can later replace counting without changing anything else.
Retrieval: keyword plus embedding search over the forum corpus, from subject terms and in-scope excerpts; top-k chunks above the floor, cached for the whole reply.
Phrases matter only through a pointer. A support phrase [serotonin, syndrome] is an ordered array. When the reply so far ends in "serotonin", a pointer to that phrase is active and the candidate "syndrome" inherits the phrase's lift. Without the pointer, phrases would contribute nothing beyond their single words. Averaging the array into one vector would lose the order, so it stays an array.
Matching in version 1 is exact on words. Later, cosine similarity between the candidate's vector and the phrase's next vector gives soft matching.
Influence is applied only at word starts. Mid-word, candidates are fragments, and candidacy is left unchanged.

2. Risk Analysis

Retrieval quality sets containment. Off-subject passages give off-subject weights even though every candidate is legitimate. Mitigation: similarity floor, and fall back to natural when nothing clears it.
Common words win by default. Mitigation: lift against background frequency, stopword filter.
Lean from the blend rule: supported candidates move toward the weights, unsupported ones are untouched, so supported words can be pushed down if the weights run low. Mitigation: normalise both over the 50 first; compare against natural; watch.
Phrase pointers can fire by chance on a first word. Mitigation: drop a pointer when the next word doesn't match, and after a window.
Subword fragments versus whole words. Mitigation: word-start-only rule above.
The automatic subject can be wrong, and everything follows it. Mitigation: the context definition is logged per run.
Weak effect expected. At most steps one candidate dominates, so a halved blend rarely changes the chosen word. Judge by candidacy shifts, not changed replies. A null on replies is a valid 001 result.
Copying risk: weighting toward retrieved wording can push toward verbatim echo. Watch in the comparison.
Tokenizer: pool entries are only comparable if one tokenizer is used throughout.
Attention-based "referenced words" record (needs Hugging Face transformers, not llama-server) is deferred, not part of 001.

3. Code Change

None delivered. Pseudo code only:

SETTINGS (written into every run record)
  MODEL_FILE, PROMPT_TEMPLATE, N_CAND=50, BLEND=0.5
  SIM_FLOOR=?  MIN_OVERLAP=?  PHRASE_MAX_N=3  PHRASE_WINDOW=?  MAX_STEPS

run_query(query, history):
  ctx   = build_context_definition(query, history)     # rules only, no model
  docs  = retrieve(ctx)                                # once per reply
  docs  = [d for d in docs if d.sim >= SIM_FLOOR]
  table = build_support_table(docs)                    # words + phrases -> lift, vecs[]
  normal = generate("natural");  infl = generate("influenced")
  fixed  = score_along_path(normal.ids)                # fixed-context comparison
  write_report(ctx, docs, table, normal, infl, fixed)

generate(mode):
  ids = prompt_ids(query, ctx);  active = []           # active = phrase pointers
  loop until end-of-reply or MAX_STEPS:
    cand = llm1_top50(ids)                             # raw: temp 1, no cuts
    p1   = normalise(cand.p)                           # over the 50
    p    = p1
    if mode == "influenced" and at_word_start(ids):
      w = support_weights(cand, table, active)         # LLM2's job (counting first)
      if count(w > 0) >= MIN_OVERLAP:
        p = normalise([(p1[c]+w[c])/2 if w[c]>0 else p1[c]  for c in cand])
    pick = argmax(p);  record(step, cand, p1, p, pick)
    ids += pick.id;  active = advance_pointers(active, pick, table)
  return ids

support_weights(cand, table, active):
  for c in cand:
    s[c] = max( table.word_lift(c.word),
                max over phrases in active whose next word matches c: phrase.lift )
  return normalise(s)                                  # 0 where unsupported

advance_pointers(active, pick, table):
  step forward pointers whose next word == pick.word; drop the rest
  start pointers for phrases whose first word == pick.word
  drop pointers older than PHRASE_WINDOW or finished

table entry: { text, words[], vecs[] (1 for a word, 2+ for a phrase), lift, n_docs }
lift = (freq in retrieved) / (freq in ordinary text), smoothed and capped

No program delivered, so the release card does not apply yet.

Build order

Natural-run recorder (raw top 50 per step, fixed context). This is the standard.
Context definition builder, retrieval once per reply, similarity floor.
Support table, counting weights, blend, candidacy comparison.
Only then the LLM2 model pass, judged against step 3.

Open settings to fix before the first run

SIM_FLOOR, MIN_OVERLAP, PHRASE_WINDOW, lift smoothing and cap, retrieval top-k, and the embedding model for retrieval and the phrase arrays.

📝 MEMSTORE_CLONE_CHECKPOINT_2026-10-27_2.mdTheme:

MEMSTORE CLONE CHECKPOINT 2026-10-27 (2) — Memstore CLOSED, The AI Influence Project OPENS

Operator: Andrew. Written 2026-10-27 02:55 Sydney. Supersedes nothing; follows MEMSTORE_CLONE_CHECKPOINT_2026-10-26_1.md.

1. Decision

The operator closed the memstore chapter and opened The AI Influence Project. Memstore (facts layer) is finished for now. Everything below "Influence" is new work. Start the next conversation from this file.

2. The machine

The Clone VM: CentOS 6.10, i7-3520M, 2 real cores. Python 3.6 with SQLite 3.6.20 for memstore and memtok; python3.8 for the go-between.
Local Qwen models served by llama.cpp b10726: 4B on port 8080, 0.5B on port 8081.
Everything lives in /root/memstore (operator's standard): memstore.db, memtok.db, scripts, state.
Memstore contents: #1 "Operator runs CentOS 6.10 on the Clone"; #2 CORRECTION "Never suggest pkill on services"; #3 "my name is Andrew Egerszegi".

3. Memstore chapter — what was delivered (all zips delivered to the operator)

Zip	Content
v1.0.6	`memstore.py` 1.1.1 (`MEMDIR` default `/root/memstore`, `MEMSTORE_DB`, prints database path); `qwen_propose.py` 1.1.1
v1.0.7	`qwen_memproxy.py` 1.2.0, tool filtering (`--drop-tools`)
v1.0.8	`qwen_memproxy.py` 1.2.1, per-profile defaults: 4B drops `runtime` tools, 0.5B drops `all`
v1.0.9	`qwen_memproxy.py` 1.3.0, `remember:` in chat saves a fact through `memstore.py add`

Go-between: 8090 → 4B (8080), 8091 → 0.5B (8081). It injects memory, filters tools, handles remember:. Order in forward: filter tools, remember:, inject.

Findings worth keeping:

The "Windows" answers came from the web page's own client-side tool get_info, which returns the browser OS, not from the model.
The stray database was a root-cause path problem, fixed by the standard folder.
The 0.5B is distracted by tool lists; the 4B answers memory questions correctly and calls a single tool correctly.
4B placement default (system) confirmed good; 0.5B default user.

Known open items in the memstore chapter (not blocking):

4B on mixed memory+tool questions sometimes writes the placeholder $(get_datetime) as text instead of calling the tool. Option not built: a system-message line telling it to call the tool.
qwen_propose.py has never been run on real chats.
Optional, unbuilt: launcher option for --drop-tools; remember: rewriting "my" to "the operator's"; QwenStop.sh/PID file for ordered stop; removing stray /bin/memstore.db; the --limit 20 default.

4. The AI Influence Project — agreed design

Purpose: a second LLM-side store (the operator's "LLM2 token database") that influences LLM1's next-token probabilities using logit_bias, with saliency read from n_probs.

Layers. memstore = low-level approved FACTS (ground truth). memtok = derived, weighted influence items. memtok never writes to memstore.
Candidates. The top-20 n_probs list (top20.json: for the prompt "The best thing to do on a rainy weekend is", id 311 " to" at logprob -1.21, 728 " go", 32671 " ______" …) are "the tokens we will be INFLUENCING". The operator confirmed this. n_probs is output-side saliency and sits on the same axis logit_bias acts on. Attention weights are not exposed by the server.
Stored unit. Phrases, not single tokens (default, unless overruled).
Fact tie. Every seeded memtok records fact_id; sync removes memtoks whose fact is forgotten, superseded or rejected. CORRECTION entries are always-on and bypass scoring.
Embeddings. llama-server --embeddings; memtoks hold text + LLM1 token IDs + vector, so the embedding model need not share LLM1's vocabulary.
Bias rule (as built). bias aims at the next token of a phrase: start of phrase → gap to the leading candidate × --strength (0.5); mid-phrase (tokens already emitted) → full gap; target outside the top 20 → gap measured to the 20th candidate + margin 2.0; capped at --max-bias 5.0; shared tokens take the maximum. Output [[id,bias]] for /completion or {"id":bias} for chat requests.
Operator's latest requirement: LLM1's query is stored in the memtok DB. Built as retrieve --record.

5. Current state of the code — IMPORTANT

memtok.py v1.0.0 (2026-10-27) is written and tested but NOT packaged or delivered.

Location: scratchpad/b110/memtok.py (the build folder b110 is a copy of b109 plus memtok.py). Not yet done:

MEMTOK_MANUAL.md (with version history),
delivery zip Memstore_Clone_v1.1.0.zip with the legacy-zipper line on the release card ("Not needed" for non-SMF zips),
release card.

Commands: seed, add, prepare, retrieve (auto sync; --record, --top, --min-sim 0.25, --json, --no-sync), bias, hit, list, stats, sync, evict --max N (only fact_id IS NULL and value < 3.0; --dry-run), forget, queries, forget-query.

Tables in memtok.db: memtoks(id, text UNIQUE, token_ids JSON, vec JSON, hits, value, fact_id, origin, created, last_seen); queries(id, text UNIQUE, vec, hits, created, last_seen); query_memtoks(query_id, memtok_id, score).

Note: timestamps are stored in UTC. Weight = value × (1 + ln(hits)); score = max(cosine,0) × weight.

Tests run (stand-in embedding/tokenizer servers, scratch tmt.py, not shipped): seed skips corrections; prepare fills vectors and token IDs; --record stores a query and a repeat raises its count to 2; forget-query removes its links; rejecting a fact in memstore removes its memtok on sync; unreadable memstore refuses to delete anything; memstore.db hash unchanged by memtok commands; bias output. Tested on a newer Python than the Clone's 3.6 — run it on the Clone's Python 3.6 first (it only uses stdlib features that exist in 3.6; utcnow is fine there).

Unrequested decisions made and listed to the operator for the query store: separate queries and query_memtoks tables; storing only with --record; text cap 2000 characters; queries never touched by sync/evict. Still to list before delivery (from the original spec): log-damped count multiplier, --min-sim 0.25, eviction exemptions, one memtok per fact on seed, leading-space tokenisation, phrase-level default.

6. Open questions for the Influence Project

Tokenizer check: are the 4B (8080) and 0.5B (8081) vocabularies identical for these IDs? Run /completion with n_probs: 20 on both and compare ids 311, 728, 32671.
Where bias is applied: logit_bias is per request, not per token. Decide how the go-between feeds it (one pre-computed table at request start vs. step-by-step generation with re-biasing as phrases begin).
Measuring effect: repeat the same request with and without the table and compare n_probs.
Embedding server: which model, which port, pooling type (--pooling mean).
How a memtok's count and value rise in practice (hit after use; what counts as use).
Query store use: what the stored queries and their returned memtoks feed (value updates? evaluation?). Raw queries can hold secrets, so they never become memtoks automatically.

7. Standing operator rules (carry forward)

Reply header Claude: [Month DD, YYYY, HH:MM:SS AM/PM] (casual mode), actual Sydney time from code run immediately before each reply.
Never suggest pkill/kill on services — ordered shutdowns only. Never rename his scripts.
Delivery of code: bump @version/@date; manual version-history entry; version shown once at start; zip with dotted never-reused version; xmllint --noout on XML; list unrequested changes BEFORE making them; end with a release card (include "legacy zipper used").
Critique and objections before building; statements over questions; don't mimic his tone; expand acronyms; no invented jargon; deployment timing is his call; PDF never Word; checkpoint names get a new _N suffix and are never reused (next for 2026-10-27 is _3); he prefers short conversations and a fresh chat per chapter.
Terminology: "operator", "query", "reply".

8. First steps in the new conversation

Say the Memstore chapter is closed and the Influence Project is open.
Finish the memtok.py delivery: manual, listed decisions, Memstore_Clone_v1.1.0.zip (or a new Influence project zip name if the operator prefers), release card.
Operator runs on the Clone: memtok.py seed, then prepare with an embedding server, then retrieve --record.
Work through section 6, starting with the tokenizer check.

📝 HOTWORDS_MANUAL.mdTheme:

hotwords.py — manual

Version 2.1.0 — 2026-10-27 — Andrew.human & Claude.ai

Written for humans and for other AIs/machines. Everything a reader needs to run, check or extend the tool is here.

1. Purpose

hotwords.py does two jobs for the Influence experiments:

Reduce each query or reply of LLM1 to one small ranked array of its hottest words and phrases, and store the array. No text is kept.
Build the pseudo query for LLM2 (pseudo): the new query's own hot array, followed by the terms RAG chose from the stored arrays that match it.

The pseudo query is a theoretical query. Its value is in what it ranks as important. It is not meant to read as a sentence and nothing checks that it is coherent; making the final words coherent is LLM1's job. For that reason the tool has no coherence step: order and weights are the content.

2. Requirements and install

Python 3.6 or later, standard library only. Written to avoid anything newer than 3.6 and anything newer than SQLite 3.6.20 (no UPSERT, no JSON1, no window functions).
Copy hotwords.py to the folder you run your scripts from (e.g. /root/memstore). Nothing else to install.
The script prints its version once when it starts (to stderr), and every error message carries the version.
Output is always UTF-8, whatever the shell's locale (Python 3.6 on a POSIX/C locale would otherwise crash on non-ASCII text).

3. Where things live

Item	Default	Override
Folder	`/root/memstore`	env `MEMDIR`
Database	`$MEMDIR/hotwords.db`	env `HOTWORDS_DB`
Stopword list	`$MEMDIR/hotwords_stopwords.txt`	env `HOTWORDS_STOPWORDS`

The database path is printed at the start of every command. The database file is created with mode 600 and a missing folder with mode 700. The stopword file is created on first use from a built-in English list; edit it freely (one word per line, # starts a comment). Its hash is recorded with every array.

An older database is refused (version 1, or the 2.0.0 database), not converted, and is left untouched. Point HOTWORDS_DB at a new file.

4. Commands

add

Reduce a query or reply to its hot array and store it.

hotwords.py add --kind query --text "Why is the forum slow after the upgrade?"
hotwords.py add --kind reply --file reply1.txt reply2.txt --label run42
hotwords.py add --jsonl pairs.jsonl
cat reply.txt | hotwords.py add --kind reply --file -

--kind query|reply is required with --text and --file. With --jsonl each line may carry its own kind; --kind is the default.
--file takes one or more paths, one entry per file. - means standard input.
--jsonl lines look like {"kind": "reply", "text": "...", "label": "run42"}.
--label is free text stored with the array (a run id, a source). It is used for nothing else.
The stored array is printed.

Counted once. Two things stop repeats from skewing the counts:

The same text (whitespace collapsed) added again changes nothing except times_seen. Experiment 000 repeats one query about 20 times; it stays one entry.
A different text that reduces to the same array (same terms in the same order, same kind) is stored as that array, with times_seen increased and nothing else changed.

Exit code 0 on success, 1 if any text was skipped (no countable words), 2 on an error.

pseudo

Build the pseudo query for LLM2. Stores nothing.

hotwords.py pseudo --text "Is serotonin syndrome likely at a normal MDMA dose?"
hotwords.py pseudo --file query.txt --format json
echo "..." | hotwords.py pseudo --file - --format table

Options: --kind (what the input is; default query), --source query|reply|both (which stored arrays RAG may draw on; default both), --arrays N (default 5), --extra N (default 12), --min-match X (default 0.05), --format line|table|json (default line), --weights (with line: add each term's weight).

line (what goes to LLM2) is two lines. Phrases are quoted so their boundaries are clear:

query: likely | normal | dose | serotonin | syndrome | mdma
related: "serotonin syndrome" | dangerous | risk | worries | combined | maois

query: the new query's own hot array, most important first.
related: RAG's choices, most important first, or (none).

With --weights each term carries its weight, so faint terms stay visibly faint:

query: seretonin:1.00 | sindrome:0.95 | 2-cb:0.56
related: "serotonin syndrome":0.81 | risk:0.40 | combined:0.34 | "2c-b dose":0.22 | dose:0.21

A query term's weight is its relative score (1.0 for the top term). A related term's weight is array match × its relative score in that array, summed over the arrays used. Both are on a 0..1 scale, so they can be compared. Influence is rarely overt: a weak match is kept in proportion to its weight rather than cut, which is why the floor is low.

table shows the same with counts, scores, every fuzzy match used, which stored arrays were used and where each related term came from. json carries everything (every score and weight, the fuzzy matches, the arrays chosen with their match, the settings) and is the record to save with a run.

The query's own array is built exactly as add would build it, so pseudo followed by add of the same text gives the same array.

show

One stored array, with its settings: hotwords.py show 17.

top

Hottest terms across the stored arrays, ranked by the sum of their relative scores: hotwords.py top [--kind query|reply|both] [--n 20] [--words | --phrases].

stats

What the database holds, the array rules and RAG defaults in force, and how many arrays were made under a different stopword list or different rules.

5. How an array is made

5.1 Reading the text

Fenced code blocks and web addresses are removed.
The text is lower-cased and split into segments at punctuation (. , ; : ! ? ( ) [ ] { } " / \ |, dashes, line breaks). Phrases never cross a segment.
Words are runs of letters and digits, with an internal hyphen or apostrophe kept, so 2c-b, 5-ht2a and don't stay whole. A trailing 's is dropped.
Spelling key. Every term also gets a key: the term with accents, hyphens and apostrophes removed. 2C-B, 2-CB and 2cb all have the key 2cb. Terms whose keys are equal are the same term. Inside one text they merge (counts add up, the earliest position counts, the form typed first is shown). The key is also how terms are stored, counted for rarity and matched.
Position is the word index of a term's first occurrence, counting every word read (stopwords included): the first word is 0.

5.2 Which terms are candidates

A content word has at least 2 characters, is not all digits, and is not a stopword. Every content word is a candidate hot word.
A phrase is 2 or 3 consecutive words that start and end on a content word (a stopword may sit in the middle). To be a candidate it must occur at least twice inside its own text.
A two-word phrase that only ever appears inside an equally frequent three-word phrase is dropped in favour of the three-word phrase.

5.3 Score

score = (1 + ln count) × rarity × position weight

count: occurrences in the text.
rarity(word) = ln((N + 1) / (df + 1)) + 1. N is the number of texts counted, df the number of them containing the word. The text being processed is counted in as if already stored (N+1, df+1), so pseudo and add agree. A phrase's rarity is the average rarity of its content words. With nothing counted yet, rarity is 1.0 for everything.
position weight = 1 / (1 + first_index / 20): 1.0 for the first word, 0.5 at word 20, about 0.25 at word 60. Early words weigh more.

5.4 How many terms an array keeps — relevance, not space

Terms are ranked by score. The array keeps every term scoring at least 40% of the top term, but never fewer than 3 (if the text has them) and never more than 12. rel is a term's score divided by the top score, so it is 1.0 for the top term.

5.5 Stored per array

Kind, a hash of the term list, label, words read, stopword hash, tool version, the settings used, times_seen, UTC timestamps, and per term: rank, term, word or phrase, count, first index, score, rel. Scores and rel are a snapshot from the moment the array was made.

5.6 The word counter

One extra table, vocab, counts for every content word how many texts contained it. It exists only to compute rarity. It holds words, not texts.

6. How RAG chooses (`pseudo`)

Take the query's own array; each term's weight is its rel.
Matching terms. A query term matches a stored term when
- their spelling keys are equal (factor 1.0), or
- the query term is a plain word (letters only) of 5 or more letters and a word the database has counted differs by a few letters (a fuzzy match). Allowed: 1 edit for words of 5 to 8 letters, 2 edits for words of 9 or more. An edit is a letter added, dropped or changed, or two neighbouring letters swapped. The factor is 1.0 minus 0.1 per edit (0.9, 0.8), and at most the 3 closest near-spellings of each word are used. Phrases match when each of their words matches, with the factors multiplied.
- Anything containing a digit is never fuzzy-matched, and neither are words shorter than 5 letters. One character separates different drugs (2C-B/2C-I, 25I-NBOMe/25B-NBOMe), so those match only on the spelling key.
Find stored arrays (of the allowed kinds) holding matching terms. An array identical to the query's own (same kind, same terms) is skipped.
Match = the share of the query's total weight that the stored array covers, each matched term counted as (query rel) × (stored rel) × factor, using each query term's best match in that array. The score is 0..1. Because rel already includes rarity and position, a shared rare, early term counts for more than a shared common, late one.
The best --arrays stored arrays with match at least --min-match are used.
Their terms that do not match a query term are weighted match × stored rel, added up across arrays, and the best --extra become the related: terms. A stored term that fuzzy-matched a query word is not offered as related, since it is the same word.

All lookups use the small array_terms table (indexed by spelling key) and the list of counted words; no text and no embeddings are read. Every fuzzy match used is listed in table and json.

7. Privacy

The database holds hot terms, not text, but a secret that is a single rare word can appear in an array. The file is mode 600. There is no remove command in 2.0.0; delete the database and re-add what you want to keep.

8. Known limits in 2.0.0

Rarity comes from the counted texts themselves. Once the archive is heavily about one subject, that subject's own words become common and are marked down against generic words. Early on the effect is small; it grows with the archive. The earlier background-seeding command was removed, so there is no outside reference.
A phrase in a short query almost never repeats, so queries yield mostly words. Phrases enter through replies and through longer texts.
Fuzzy matching is about spelling only. It does not know that two different words mean the same thing; that needs embeddings, not built. Words with a digit and words under 5 letters never fuzzy-match, so a misspelt MDMA or LSD will not find its stored arrays. Token ids and vectors per term are not stored yet.
The fuzzy search reads the list of counted words on each pseudo run and compares it letter by letter against each long query word. That is quick for thousands of words; it should be revisited if the list grows to hundreds of thousands.
Scores in stored arrays are snapshots; they do not change as the archive grows.
English stopwords only. No remove command. No --json for add, show, top.

9. Version history

Version	Date	Change
2.1.0	2026-10-27	Fuzzy matching. Spelling key (accents, hyphens, apostrophes removed) so `2C-B`, `2-CB` and `2cb` are one term: merged inside a text, matched across texts, rarity counted per key. Near-spellings of plain words of 5+ letters match at a reduced weight (1 edit up to 8 letters, 2 from 9, a swap counts as one); words with a digit and short words never fuzzy-match. Faint matches count: RAG defaults changed to 5 arrays, 12 extra terms, match floor 0.05 (were 3, 8, 0.15). New `--weights` flag for the `line` format. Fuzzy matches listed in `table` and `json`. Database schema 3 (adds the spelling key); older databases are refused, not converted.
2.0.0	2026-10-27	Redesigned around small arrays instead of documents: one ranked hot-term array per query or reply, no text kept. Array length set by relevance (40% of the top score, 3 to 12 terms). Position weight added (early words score higher). Phrases must repeat within their own text. Dedupe by text hash and by array hash. New `pseudo` command builds the pseudo query for LLM2 (own array plus RAG's chosen terms), stores nothing. Removed `seed-background` and the document tables; rarity comes from a word counter. Settings recorded with every array. Version 1 databases are refused, not converted.
1.0.1	2026-10-27	Fix: non-ASCII text no longer crashes under a POSIX/C locale (Python 3.6): UTF-8 handled explicitly for arguments, standard input and output. Timestamps use timezone-aware UTC.
1.0.0	2026-10-27	First release: `add`, `top`, `show`, `stats`, `seed-background`; separate query and reply counts; counted-once rule; stopword file with recorded hash.

📝 CONTEXTDEF05B_MANUAL.mdTheme:

contextdef_05b.py — manual

Version 1.0.0 — 2026-10-27 — Andrew.human & Claude.ai

Written for humans and for other AIs/machines. Everything a reader needs to run, check or extend the tool is here.

1. Purpose

A trial: send a few queries through the small (0.5B) model on a llama-server and see what context definition comes out, how long it takes, and how it differs from the query's own words. It exists to answer, cheaply: can the 0.5B produce a usable subject, intent and list of important terms?

It is not the context definer. It does not read or write the real context-definition store. Everything it saves goes under ctxdef_try/.

2. Requirements and install

Python 3.6 or later, standard library only.
A llama-server with the 0.5B model running (default http://127.0.0.1:8081).
Copy contextdef_05b.py to the folder you run your scripts from. Nothing else to install.
It prints its version once when it starts (to stderr). Every error message carries the version.
Output is always UTF-8, whatever the shell's locale.

3. Running it

contextdef_05b.py --text "Why is serotonin syndrome so dangerous with MDMA?" --text "Is 2C-B safe with SSRIs?"
contextdef_05b.py --file queries.txt          # one query per line, blank lines skipped
cat queries.txt | contextdef_05b.py --file -

Option	Default	Meaning
`--text QUERY`		A query. Repeat for several.
`--file PATH`		Queries, one per line. `-` is standard input.
`--url`	`http://127.0.0.1:8081` (or env `CTXDEF_URL`)	Where the llama-server is.
`--api chat\|completion`	`chat`	`chat` uses `/v1/chat/completions` (the server applies the model's own chat template). `completion` uses `/completion` with Qwen's chat tags written into the prompt, for older builds that lack the chat route.
`--n-predict`	160	Most tokens the model may write.
`--temperature`	0	0 with `top_k` 1 and a fixed seed, so a repeat gives the same reply.
`--timeout`	180	Seconds to wait for one reply.
`--out`	`$MEMDIR/ctxdef_try`	Output folder (`MEMDIR` default `/root/memstore`).

Before the first query it asks the server for /health and stops with a clear message if the server cannot be reached. Exit code 0 when every reply parsed cleanly, 1 when any query had a note or failed, 2 on an error.

4. What it asks the model

A fixed system prompt plus one worked example (a forum-slowness query and its three-line answer), then your query. The model is asked for exactly three lines:

SUBJECT: the main thing the query is about, in 1 to 4 words
INTENT:  one of: question, how-to, troubleshoot, compare, explain, opinion, other
TERMS:   4 to 8 words or short phrases that matter most, most important first, comma separated

The prompt's SHA-256 is saved with every result, so a changed prompt is visible in the record. The worked example is there on purpose: a 0.5B model follows an example far better than an instruction.

5. What you see

[1/2] Why is serotonin syndrome so dangerous with MDMA?
  subject : serotonin syndrome
  intent  : explain
  terms   : serotonin syndrome | mdma | danger [+] | symptoms [+]
            ([+] = added by the model, not in the query)
  note    : no INTENT line
  time    : 1.84 s  (prompt 120 tok, 40 tok out, 33.3 tok/s)

[+] marks a term whose words are not all in the query. These are the model's own additions. Comparison ignores case, accents, hyphens and apostrophes, so 2c-b counts as in the query 2-CB.
note lines say what was wrong with the reply: a missing line, an intent not on the list, extra chatter outside the three lines. The reply is never silently repaired; the raw reply is saved untouched.
time is wall-clock seconds, plus the server's own figures when it reports them.

6. What it saves

Under $MEMDIR/ctxdef_try/<run>/ (run = UTC date-time; a second run in the same second gets -2, never overwrites):

File	Holds
`001.json`, `002.json` …	One per query: the query, the raw reply, the parsed subject / intent / terms (each term with `in_query`), notes, timing, settings, UTC time.
`run.json`	Settings, number of queries, number with a problem, total seconds.

Files are mode 600 and the folders mode 700.

7. Reading the results

What matters for Experiment 001 (where LLM2 adds no words):

Are the subject and the first terms the ones you would pick? The order is the model's idea of importance.
How many [+] terms appear, and are they sensible? Each is a word the model brought in.
How often is the reply malformed? A 0.5B model that breaks the format often needs a stricter prompt or a grammar-constrained reply, not a bigger model.
Is the time short enough to run on every query?

8. Known limits in 1.0.0

Tested here against a stand-in server, not a real 0.5B model. The chat route and the completion route are untried against your llama-server build.
Single query only: no conversation history, so no scope (which earlier turns count) and no standing rules from the memory store.
Three fields only. No comparison with the rule-based hot-term array yet.
The completion route assumes Qwen's chat tags (<|im_start|> … <|im_end|>). Another model family needs different tags.
No grammar-constrained output; the model can ignore the format.

9. Version history

Version	Date	Change
1.0.0	2026-10-27	First release: queries through the 0.5B llama-server, SUBJECT / INTENT / TERMS parsed forgivingly, terms marked as in the query or added, timing, one JSON file per query plus a run record under `ctxdef_try/`.

📝 Experiment_001_checkpoint_2.mdTheme:

Experiments 000, 001, 002 — Checkpoint (2)

Operator: Andrew. Written 2026-10-27 04:44 Sydney. Supersedes Experiment_001_checkpoint.md (that file is kept, never overwritten). Status: design only. No program written or delivered. Resume: paste this file into a fresh conversation. Next step is build step 1 (the recorder). Related file: MEMSTORE_CLONE_CHECKPOINT_2026-10-27_2.md (global parameters: /root/memstore, Python 3.6 on the Clone, Qwen 4B on 8080 and 0.5B on 8081, memstore corrections, memtok).

1. Aim

Find out whether a second module (LLM2 plus retrieval) can move LLM1's next-word candidacy, strictly inside the query's subject, and whether that is worth having compared with plain retrieval in the prompt. Candidacy = a word's standing in the pool at the moment of selection (eligibility after the sampler's cuts, then weight).

2. The three experiments

Experiment 000 — background variation (control and archive)

Part A, archive: 10 or more queries drawn at random from the operator's transcript archive, of different kinds (short fact, how-to, pharmacology, opinion, pronoun follow-up, off-subject). Each goes through LLM1 natural. Each pair is saved with the full record (raw top 50 per step, context definition, settings). Screen queries for secrets before saving.
Part B, repetition: pick one query (mid-length reply, some spread in its candidates) and run it about 20 times under three conditions:
1. fresh context, greedy decoding: expected identical. If not, the server is non-deterministic, and the "greedy is repeatable" assumption fails;
2. fresh context, fixed temperature, different seeds: the pool's natural spread;
3. same query repeated inside one growing conversation: repetition proper.
Measures: exact-match count, first divergence step, word-level difference between consecutive replies, movement of the top 50 at the same step.
Run LLM1 directly on port 8080 or with memory injection off. The go-between injects memstore facts, which would contaminate "au naturel".
Needs only the recorder, so it can start first. Purpose: the amount replies vary when nothing is changed, so later differences can be judged against it.

Experiment 001 — reweight only (no new words)

LLM2 adds no words. It reweights LLM1's own top-50 candidates, weights taken from what retrieval finds for the subject.
Blend: for candidates the support table supports, new value = (LLM1 + weight) / 2 on probabilities normalised over the 50; unsupported candidates unchanged; renormalise.
Two outputs per query: LLM1 normal and LLM1 influenced. Both free-running, plus a fixed-context comparison (blended candidacy scored along the normal reply's path, so divergence cannot muddy it).
Weighting version 1 is counting (a word's lift = frequency in the retrieved passages relative to ordinary text). No LLM2 model pass yet.
Influence applied only at word starts. Falls back to natural when fewer than MIN_OVERLAP candidates are supported or no passage clears the similarity floor.
Measures: candidacy shift per step, subject containment, where replies differ. A null on replies is a valid result.

Experiment 002 — adding words, heavier weights

Reverses the 001 rule "no new words". Words from outside LLM1's pool can be injected, with a raised weight: blend = (1 - α)·p1 + α·w, α above 0.5.
Injection rules: only the next word of an active phrase pointer (anchored by the word before it), only at word starts, at most N_NEW (start 5) per step, boost capped by MAX_BOOST (start 5.0 on the raw score, about 150 times the probability, as in memtok).
Runs per query: natural; 001 reweight at α 0.5; injection at α 0.5; injection at 0.7 and 0.9. Changing one thing at a time, so the first run where it breaks shows why.
Measures: injected-pick rate, picks outside the natural top 50, fluency gap (LLM1's average probability per word on its natural reply against each influenced reply), in-subject share of picked words, copying (longest verbatim run shared with the retrieved passages). The α at which fluency collapses is itself a result.
Depends on 001 (recorder, context definition, retrieval, support table with phrase arrays).

Order: 000, then 001, then 002.

3. Components and decisions

Context definition (rule-based, no model, Python 3.6 standard library only): inputs are the current query, the last six turns, and the standing rules (memstore corrections, read-only). Subject = top 3 to 5 words or 2-to-3-word runs, scored by rarity (how uncommon in the forum text) plus recurrence in recent turns. Pronoun-only queries inherit the previous subject and are marked inherited. Intent from a small table (question word, command verb, statement). Earlier turns scored in scope or background by overlap with the subject, discounted by age. Retrieval terms: subject phrases plus top terms from in-scope excerpts, capped at about 12, phrases kept as ordered word lists.
Saved to /root/memstore/ctxdef/<id>.json, one per run, plus an append-only index line per run. Fields: id, generator version, query (capped, with hash), subject (head, terms with scores and a why block, inherited, confidence), intent, in-scope turns with excerpts of at most 200 characters, background turn numbers, standing rules, retrieval terms, settings (including the rarity table hash), content hash. Mode 600 and folder 700 because queries can hold secrets. Nothing here is touched by sync or evict. A top score under a floor marks the definition low_confidence, and the run then goes natural.
Background frequency table: built once from the forum text, used for both the subject's rarity and the support table's lift.
Words and phrases are arrays of embeddings: a word is one vector, a phrase is an ordered array of two or more, never averaged into one. A phrase influences a next-word choice through a pointer: when the reply so far ends in the phrase's first word, its next word is "due" and inherits the phrase's lift. Version 1 matches words exactly. Soft matching by cosine similarity comes later.
LLM2 in 001 is code (retrieval plus counting) behind a fixed interface: input = context definition plus the 50 candidates, output = one weight per candidate. A model pass replaces counting at build step 4 and must beat it. The 0.5B (port 8081) is the first candidate, subject to the tokenizer check. Weakness to expect: it is weak at long lists, and it is the same family as LLM1, so its opinions are correlated. The 4B is the fallback.
Reading LLM2's scores: llama-server returns only the top N. Request n_probs set to a few hundred and look up the 50 candidate ids. Candidates outside get a floor value. (Corrects an earlier suggestion to read exactly the 50 ids.)
Driving LLM1: one request per word, prompt as token ids (so the chosen word is exactly the one fed back), n_predict 1, n_probs 50, prompt cache on, raw probabilities (temperature 1, no top-k, top-p 1, min-p 0). Check whether the build returns probabilities before or after the sampler.
Stages kept separate and each writes its own file under /root/memstore (context definition, retrieval passages with similarity, support table, per-step records for each run, comparison report), tied by one run id and a manifest listing each stage's output id, settings and versions. This allows walking a fault backwards, replaying a later stage from saved earlier output, and swapping stages.
Bundling into the go-between is deferred. Build the experiments as a standalone script reading queries from a file. The go-between would have to become a generator, which is a large change to working code. Move the modules in later if the results justify it.
Do not bundle a second model inside the first (adapter or extra head). That is a training project and is not needed.
Retrieval needs an embedding model plus vector search. Whether the 0.5B can serve chat and embeddings from one instance is untested. A small dedicated embedding model is the safer choice. Still open (memtok checkpoint, question 4).
Fits with the memtok design: that design reads the top 20 and applies logit_bias on the server side; 001/002 read the top 50 and let the driver choose the word. The context definition feeds either, which is why it is built first.

1. Problem & Resolution

Problem: no standard exists for measuring how much an outside weighting moves LLM1's next-word candidacy, nor for proving it stayed inside the subject, nor for knowing how much replies vary when nothing changes. Resolution (design): the three experiments above, with a recorded natural baseline, an automatically built and saved context definition, and stage-by-stage saved outputs.

2. Risk Analysis

Retrieval quality sets containment. Off-subject passages give off-subject weights even though every candidate is legitimate. Mitigation: similarity floor, fall back to natural.
Common words win by default. Mitigation: lift against background frequency, stopword filter.
Lean in the 001 blend: supported candidates move toward the weights, unsupported are untouched. Normalise first, compare with natural.
Phrase pointers can fire by chance. Drop when the next word does not match, and after a window.
Subword fragments versus whole words: influence only at word starts.
The automatic subject can be wrong and everything follows it. Logged per run, with scores. Inheritance is the riskiest case, flagged.
Weak effect expected in 001: one candidate usually dominates, so a halved blend rarely changes the word. Judge by candidacy shifts.
002: grammar breaks if arbitrary words are injected (hence anchoring); copying and content drift at high α; in-subject is not the same as correct at that point.
Greedy decoding may not be repeatable on this server; 000 checks it.
One tokenizer must be used throughout, or pool entries are not comparable. Check the 4B and 0.5B vocabularies (ids 311, 728, 32671).
The files hold query excerpts. Permissions and caps as above.
Rarity-based subject choice can pick an incidental word such as a typo or a name. Recurrence term, cap, and visible scores limit it.
Attention-based "referenced words" record needs Hugging Face transformers (not llama-server) and is deferred.

3. Code Change

None delivered. First deliverable would be contextdef.py with commands build, show <id>, list. The recorder comes before it (build step 1). All deliveries follow the standing delivery rules, including the release card.

Pseudo code (001; 002 adds the lines marked):

SETTINGS (written into every run record)
  MODEL_FILE, PROMPT_TEMPLATE, N_CAND=50, BLEND=0.5
  SIM_FLOOR=?  MIN_OVERLAP=?  PHRASE_MAX_N=3  PHRASE_WINDOW=?  MAX_STEPS
  [002] ALPHA, N_NEW=5, MAX_BOOST=5.0

run_query(query, history):
  ctx   = build_context_definition(query, history)     # rules only, no model
  docs  = retrieve(ctx)                                # once per reply
  docs  = [d for d in docs if d.sim >= SIM_FLOOR]
  table = build_support_table(docs)                    # words + phrases -> lift, vecs[]
  normal = generate("natural");  infl = generate("influenced")
  fixed  = score_along_path(normal.ids)                # fixed-context comparison
  write_report(ctx, docs, table, normal, infl, fixed)

generate(mode):
  ids = prompt_ids(query, ctx);  active = []           # active = phrase pointers
  loop until end-of-reply or MAX_STEPS:
    cand = llm1_top50(ids)                             # raw: temp 1, no cuts
    p1   = normalise(cand.p)                           # over the 50
    p    = p1
    if mode == "influenced" and at_word_start(ids):
      [002] cand += anchored_injection(active, table, N_NEW)   # next words of active phrases only
      w = support_weights(cand, table, active)         # LLM2's job (counting first)
      if count(w > 0) >= MIN_OVERLAP:
        [001] p = normalise([(p1[c]+w[c])/2 if w[c]>0 else p1[c] for c in cand])
        [002] p = normalise([(1-ALPHA)*p1[c] + ALPHA*w[c] for c in cand], boost capped by MAX_BOOST)
    pick = argmax(p);  record(step, cand, p1, p, pick)
    ids += pick.id;  active = advance_pointers(active, pick, table)
  return ids

support_weights(cand, table, active):
  for c in cand:
    s[c] = max( table.word_lift(c.word),
                max over phrases in active whose next word matches c: phrase.lift )
  return normalise(s)                                  # 0 where unsupported

advance_pointers(active, pick, table):
  step forward pointers whose next word == pick.word; drop the rest
  start pointers for phrases whose first word == pick.word
  drop pointers older than PHRASE_WINDOW or finished

table entry: { text, words[], vecs[] (1 for a word, 2+ for a phrase), lift, n_docs }
lift = (freq in retrieved) / (freq in ordinary text), smoothed and capped

4. Build order

Natural-run recorder (raw top 50 per step, fixed context). This is the standard. Experiment 000 runs on it.
Context definition builder, saved files and index; retrieval once per reply with the similarity floor.
Support table, counting weights, blend, candidacy comparison. Experiment 001.
Injection and the α dial. Experiment 002.
Only then the LLM2 model pass in place of counting, judged against step 3.

5. Open items

The prompt-retrieval arm (LLM1 with the retrieved passages in its prompt, no influence): proposed as the honest benchmark, costs one extra run per query. Not yet decided by the operator.
Settings to fix: SIM_FLOOR, MIN_OVERLAP, PHRASE_WINDOW, lift smoothing and cap, retrieval top-k, scope floor, recency decay, ALPHA steps, the embedding model for retrieval and phrase arrays.
Tokenizer check, embedding server choice (model, port, pooling), and whether the sampler's probabilities are returned before or after cuts.
Where the background frequency table is built from (forum text or the attachments corpus).

6. Standing operator rules (carry forward)

Reply header Claude: [Month DD, YYYY, HH:MM:SS AM/PM] (casual mode), real Sydney time from a code run immediately before each reply.
Never suggest pkill/kill on services. Never rename his scripts. Deployment timing is his call.
Code delivery: bump @version/@date; manual version-history entry; version shown once at start; zip with a dotted, never-reused version; xmllint --noout on XML; list unrequested changes BEFORE making them; end with a release card (include "legacy zipper used" for package zips).
Critique and objections before building; statements over questions; do not mimic his tone; expand acronyms; no invented jargon; PDF never Word; checkpoint names get a new _N and are never reused (next is _3); prefers short conversations and a fresh chat per chapter.
Terminology: "operator", "query", "reply".

📝 Experiment_001_checkpoint_3.mdTheme:

Experiments 000, 001, 002 — Checkpoint (3)

Operator: Andrew. Written 2026-10-27 05:27 Sydney. Follows Experiment_001_checkpoint_2.md (kept, never overwritten; Experiment_001_checkpoint.md also kept). Status: two tools built and delivered (hotwords 2.1.0, contextdef_05b 1.0.0). Experiments themselves not started; the recorder (build step 1) is still not written. Resume: paste this file together with Experiment_001_checkpoint_2.md into a fresh conversation. Where the two differ, this file wins; the differences are listed in section 3. Related: MEMSTORE_CLONE_CHECKPOINT_2026-10-27_2.md (global parameters).

1. What happened since checkpoint 2

Hot word / hot phrase extractor built: Hotwords_v1.0.0.zip, v1.0.1 (non-ASCII crash fixed), then redesigned as Hotwords_v2.0.0.zip and Hotwords_v2.1.0.zip (current). All kept, never reused.
Context definer trial built: ContextDef05b_v1.0.0.zip (contextdef_05b.py).
Design moved from "documents" to small arrays of hot terms (see section 2).
The operator doubted the concept ("worried the whole concept is crap"). Claude's assessment is recorded in section 7 so the new chat starts from it.

2. Operator decisions and statements (design-significant, in order)

No documents. Store only small arrays of tokens derived from LLM1's queries and replies. This should make RAG very fast and mostly cache-based.
Array size is about relevance, not space.
Position matters: early words usually carry more significance.
The output of the hotwords tool is the input to LLM2, as the pseudo query, and it includes RAG's choices.
LLM2's query is theoretical: its value is in what it deems important, coherent or not. Coherence is LLM1's job.
Fuzziness is important (spelling variants must match: 2C-B, not 2-CB, but both must find each other).
Influence is rarely overt: weak matches matter.
Wants a few queries pushed through the 0.5B ("5b", for turnaround speed) as the context definer, to see what comes out.
Wonders how all this works for coding (answer in section 7).

3. Differences from checkpoint 2

Topic in checkpoint 2	Now
Retrieval over passages / "documents", support table from retrieved passages	Retrieval over hot-term arrays (tool `hotwords.py`). No text kept. The support table is built from the arrays RAG chose.
Background frequency table from forum text or the attachments corpus	Dropped for now. Rarity comes from the counted texts themselves (word counter keyed by spelling key). Known weakness, risk 16 in section 5.
Context definition: rule-based only	Rule-based design stands (still not built). Plus a trial using the 0.5B to produce subject / intent / terms. Position weighting should be added to the rule-based subject score (early query words count more).
LLM2 input = context definition + 50 candidates	LLM2 input = the pseudo query (query's hot array + RAG's related terms, each with a weight) + the 50 candidates. How the pseudo query becomes 50 weights is not yet designed; proposal: a candidate's weight = the weight of the pseudo-query term it matches (spelling key, then fuzzy for plain words), 0 if none. Not agreed.
Words/phrases as arrays of embeddings, exact match in v1	Unchanged in intent, but v1 matching is now spelling key plus fuzzy (edit distance) on plain words. Still no embeddings.
Settings list (SIM_FLOOR etc.)	Several are now concrete (section 9).

Everything else in checkpoint 2 (experiments 000/001/002, decisions on stage separation, driving LLM1 one word at a time, pseudo code, risks 1 to 13, build order) stands.

4. Problem & Resolution

Problem: the retrieval side of Experiment 001 needs a way to turn LLM1's own history into a small, fast, relevance-ordered store that can be matched against a new query, tolerant of spelling variants, and that yields a query LLM2 can use. Resolution: hotwords.py 2.1.0 (details below) and a 0.5B context-definer trial. Both are tools, not yet wired into any experiment.

hotwords.py 2.1.0 (current)

One ranked array per query or reply, no text kept. Score = (1 + ln count) x rarity x position weight. Position weight = 1 / (1 + first word index / 20). Rarity = ln((N+1)/(df+1)) + 1 over the texts counted so far, taken as if the new text were already counted.
Array keeps terms scoring at least 40% of the top term, minimum 3, maximum 12. Phrases (2 to 3 words) qualify only if they occur at least twice inside their own text.
Dedupe: same text (whitespace collapsed) changes nothing but times_seen; a different text giving the same array collapses into it (vocab untouched).
Spelling key: accents, hyphens, apostrophes removed (2C-B = 2-CB = 2cb). Terms merge by key inside a text, are stored and counted by key, shown as first typed.
Fuzzy matching (in pseudo only): plain words of 5+ letters match stored words within 1 edit (up to 8 letters) or 2 edits (9+), swap of neighbours = 1 edit, weight factor 1 - 0.1 per edit, best 3 variants. Words containing a digit and words under 5 letters never fuzzy-match (2C-B vs 2C-I and 25I-NBOMe vs 25B-NBOMe are different drugs).
pseudo command builds the pseudo query and stores nothing. RAG match = share of the query's weight covered by a stored array (query rel x stored rel x factor, best match per query term). Defaults: 5 arrays, 12 extra terms, match floor 0.05. Formats: line (two lines query: / related:, phrases quoted, --weights adds term:0.42), table, json (full trace incl. fuzzy matches).
Other commands: add, show, top, stats. Database schema 3; older databases are refused, not converted (HOTWORDS_DB points to a new file). Python 3.6 and SQLite 3.6.20 safe; UTF-8 handled explicitly. 98 checks pass on Python 3.11, 3.12, 3.13. Not yet run on the Clone's 3.6 / 3.8 / 3.15.
A smoke test showed it works as designed: for "Is serotonin syndrome likely at a normal MDMA dose?" the related terms were "serotonin syndrome" | dangerous | risk | worries | combined | maois; the forum-slowness array was left out.

contextdef_05b.py 1.0.0 (trial)

Sends queries to the 0.5B llama-server (default port 8081; /v1/chat/completions by default, --api completion for older builds with Qwen chat tags). Greedy, fixed seed, one worked example in the prompt. Asks for SUBJECT, INTENT, TERMS (most important first); parses forgivingly; marks each term as in the query or [+] added by the model; prints timing; saves JSON per query plus run.json under $MEMDIR/ctxdef_try/<run>/. Never touches the real context-definition store.
Tested only against a stand-in server (38 checks). Not yet run against the real 0.5B. Unknown: whether the build has the chat route, how often the 0.5B breaks the three-line format.

5. Risk Analysis (new, adding to checkpoint 2's 13)

Echo. The archive is mostly LLM1's own replies, so boosting terms from it mostly amplifies what LLM1 already says. It helps only if the archive holds things LLM1 does not produce unaided (the operator's corrections, preferred wording). Highest-level risk to the whole idea.
Terms without context. A boosted term does not know whether the next word should be that term.
Rarity from the counted texts themselves. Once the archive is mostly about one subject, that subject's words become common and are marked down against generic words (seen already in a small test: "likely", "normal", "dose" outranked "serotonin", "syndrome" in the query's own array). Remedy would be an outside reference; the earlier seed-background was removed in 2.0.0.
Phrases almost never repeat inside a short query, so queries yield words; phrases come from replies.
Fuzzy matching is spelling only. It will not match short words (a misspelt MDMA or LSD) or any word with a digit, by design. Fuzzy search scans the whole word list per run: fine for thousands, to be revisited at hundreds of thousands.
Scores in stored arrays are snapshots taken when the array was made.
Code is not handled: fenced code is stripped, snake_case splits, and fuzzy would merge get_user with get_users. A code mode (keep code, whole identifiers plus parts, no fuzzy) is not built.
The 0.5B is the same family as LLM1 and weak at structure; its context definitions may be malformed or biased the same way (checkpoint 2, LLM2 weakness).

6. Code Change

Delivered this session (all kept): Hotwords_v1.0.0.zip, Hotwords_v1.0.1.zip, Hotwords_v2.0.0.zip, Hotwords_v2.1.0.zip, ContextDef05b_v1.0.0.zip, Experiment_001_checkpoint_2.md (previous). Release cards given with each; the open line on each is "run on the Clone's Python 3.6 / 3.8 / 3.15" and, for the context definer, "run against the real 0.5B".

7. Honest assessment given to the operator (so it is not lost)

The plumbing carries little risk. The risky assumption is that boosting terms drawn from the archive improves LLM1's replies (risks 14 and 15).
The mechanism is not new nonsense: pulling next-word weights toward a retrieved set is an established technique (kNN-LM, contextual biasing in speech recognition) and usually moves things modestly, which fits "influence is rarely overt".
Experiments 000 and 001 can falsify it cheaply (about ten stored pairs; fluency gap and candidacy shift). Recommended, not yet agreed: write down a fail criterion before running (for example a large fluency gap with no shift toward the corrected wording), so a null result is an answer.
For coding: the array and RAG structure carries over; the tokenizer and fuzzy rules do not (risk 20).

8. Build order (updated)

Run contextdef_05b.py on a few queries; report malformed rate, added terms, seconds per query. Run hotwords.py on the Clone's Python versions; feed it real query/reply pairs.
Natural-run recorder (raw top 50 per step, fixed context). Experiment 000 runs on it, and its query/reply pairs feed hotwords.py add.
Decide how the pseudo query becomes 50 weights (section 3, table), then the blend and candidacy comparison. Experiment 001.
Context definition builder (rule-based) with position weighting, saved files and index. Compare with the 0.5B trial output.
Injection and the α dial. Experiment 002.
LLM2 model pass in place of counting, judged against step 3.

9. Open items

Prompt-retrieval arm (LLM1 with retrieved passages in its prompt, no influence) as the honest benchmark: still undecided. Note it needs passages, which the array design no longer stores; it would need the original replies kept somewhere else.
Fail criterion for Experiment 001 (section 7).
An outside reference for rarity (risk 16).
How the pseudo query reaches LLM2 and becomes 50 weights.
Settings now concrete in hotwords.py: H 20, keep 0.4, array 3 to 12, minimum phrase count 2, RAG 5 / 12 / 0.05, fuzzy 5 letters / 1 and 2 edits / 0.1 step / 3 variants. Still open: embedding model, tokenizer check (4B and 0.5B vocabularies), whether the sampler returns probabilities before or after cuts, lift smoothing, phrase window, ALPHA steps.
Whether the 0.5B should do the context definition at all, or only the rule-based definer.

10. Standing operator rules (carry forward)

Reply header Claude: [Month DD, YYYY, HH:MM:SS AM/PM] (casual mode), real Sydney time from a code run immediately before each reply.
Never suggest pkill/kill on services. Never rename his scripts. Deployment timing is his call.
Code delivery: bump @version/@date; manual with version history; version shown once at start; zip with a dotted, never-reused version; xmllint --noout on XML; list unrequested changes BEFORE making them; end with a release card (include "legacy zipper used" only for SMF package zips).
Critique and objections before building; statements over questions; do not mimic his tone; expand acronyms; no invented jargon; PDF never Word.
Checkpoint names get a new _N, never reused (next is _4). Prefers short conversations and a fresh chat per chapter.
Terminology: "operator", "query", "reply".
Environment: Clone is CentOS 6.10, Python 3.6 (also 3.8 and 3.15 per the operator), SQLite 3.6.20, llama.cpp: Qwen 4B on 8080 (LLM1), 0.5B on 8081. Python 3.6 on a POSIX/C locale does not switch to UTF-8, so non-ASCII must be handled explicitly (already done in both tools).

📝 Experiment_001_checkpoint_5.mdTheme:

Experiments 000, 001, 002 — Checkpoint (5)

Operator: Andrew. Written 2026-10-27 07:28 Sydney (renamed _5 at 07:30). Follows Experiment_001_checkpoint_3.md (kept, never overwritten; Experiment_001_checkpoint_2.md and Experiment_001_checkpoint.md also kept). A different Experiment_001_checkpoint_4.md already exists in the operator's store; it was not seen when this file was written, so its content is not reflected here. Status: no new code delivered this session. contextdef_05b.py 1.0.0 was run for the first time against the real 0.5B on the Clone, and the 0.5B was asked one query directly. Experiments themselves not started; the recorder (build step 2 in checkpoint 3) is still not written. Resume: paste this file together with Experiment_001_checkpoint_3.md, Experiment_001_checkpoint_2.md and the existing Experiment_001_checkpoint_4.md into a fresh conversation. Where files differ, the newest wins. Checkpoint 2 holds the definitions of Experiments 000/001/002 and risks 1 to 13; checkpoint 3 holds hotwords.py 2.1.0 and risks 14 to 21; this file adds risks 22 to 24 (the existing _4 may number risks too; check for clashes).

1. What happened since checkpoint 3

All three checkpoints were supplied to this chat. Claude read them and proposed, not yet agreed: a fail criterion for Experiment 001 (section 3), and a change to the blend rule (section 3).
contextdef_05b.py v1.0.0 was run on the Clone against the real 0.5B (qwen2.5-0.5b-instruct-q8_0.gguf, llama.cpp build b10726, port 8081, chat route) with two queries. Results are in section 2.
The 0.5B was then asked one of the queries directly (plain chat request, temperature 0, 300 tokens). The reply is in section 2 and shows serious errors.
Wording note for future chats: when the operator says "the reply" he means the model's own answer to the query, not the parsed context-definition fields. Claude misread this twice in this session.

2. Results (real runs, Clone)

Context definer, 2 queries (run folder `/root/memstore/ctxdef_try/20261026-185424`, UTC date)

	Query 1: "Why is serotonin syndrome so dangerous with MDMA?"	Query 2: "Is 2C-B safe with SSRIs?"
Raw reply (exact)	`SUBJECT: serotonin syndrome` / `INTENT: how-to` / `TERMS: MDMA, dangerous, serotonin, syndrome`	`SUBJECT: 2C-B safety with SSRIs` / `INTENT: how-to` / `TERMS: 2C-B, SSRIs, safety, medication`
Terms added by the model (`in_query: false`)	none	`safety`, `medication`
Wall time	25.45 s (prompt of 168 tokens evaluated cold in 22.9 s)	3.54 s (only 15 prompt tokens evaluated; the worked example came from the prompt cache)
Generation speed	11.6 tokens/s	10.8 tokens/s

Chat route works on that build (--api completion not needed).
Format held on both queries: exactly three lines, nothing extra, 0 warnings. Two queries is too few for a malformed rate.
Parsed fields lowercase the terms; the raw reply keeps the model's own case (MDMA, 2C-B, SSRIs); 2c-b survived intact.
Intent is wrong on both (how-to for a why-question and a yes/no question). Same label twice suggests it is copied from the worked example in the prompt. Not confirmed (the script is not in the chat).
Subject is a paraphrase in query 2 ("safety" for "safe"). hotwords.py fuzzy matching would not link "safe" to "safety" (edit distance too large).
Term order: the model put MDMA first on query 1, not the subject words, so its "most important first" ranking is weak.
Only the terms would reach LLM2. Intent and subject are log labels. Filtering on in_query: false leaves 2c-b and ssris.

The 0.5B's direct answer to query 1 (300 tokens, cut off by the limit)

Speed 13.8 tokens/s, 21.7 s for 300 tokens. Errors: calls MDMA "mescaline"; vague "serotonin drugs"; points 1 and 3 repeat the same sentence; generic symptom list (fatigue, appetite, mood) instead of mental-state change, autonomic overactivity, clonus/hyperreflexia and hyperthermia; "risk of death is very high" overstated; never states the mechanism (serotonin release and reuptake block) or names MAOIs; point 4 ("cross-tolerance") off-topic. The same query on the 4B (port 8080) has not been run.

Note: the operator's curl used /v1/chat/completion (no final "s") and the server still answered.

3. Proposals made this session (not agreed)

Fail criterion for Experiment 001, relative to Experiment 000's noise. Write the rule before 000 runs, fix the numbers after 000.
- Pass: on the fixed-context comparison, candidacy shift toward the stored wording is clearly larger than the movement 000 shows between identical runs, and the in-subject share of picked words does not fall.
- Null: the shift is inside the 000 noise band, or it appears only with a noticeable fluency gap (LLM1's average probability per word drops against its natural reply).
- Not a fail: an unchanged free-running reply (already stated in checkpoint 2).
Blend rule. Averaging supported candidates toward small weights (fuzzy matches give 0.9 or 0.8 times small term weights) mostly lowers the supported words, the opposite of the intent. Alternative: additive boost in log space with a capped boost (the memtok approach), which cannot push a word down. Or rescale the weights so the top supported candidate is at least its own LLM1 probability before averaging. Claude prefers the additive boost.
0.5B role. Keep as a comparison trial only (checkpoint 3 build step 4); use the rule-based definer for the real path. Drop the intent field from the 0.5B's job (use the rule-based table: question word, command verb, statement). Require the subject to be a span copied from the query, or mark non-query subject words like terms. Filter terms with in_query: false before they reach the pseudo query.
Archive content. The ten stored pairs for Experiment 001 must include the operator's corrections or preferred wording. Replies from the 0.5B must not be stored; 4B replies should be checked before storing.
Next build step after the open check below: the natural-run recorder (queries from a file, port 8080 directly, per-step raw top 50, manifest as in checkpoint 2).

4. Problem & Resolution

Problem: unknown whether the 0.5B could serve as the context definer and what it produces on the real server; also unclear how reliable its content is. Resolution: run done (section 2). It follows the format and extracts query terms well, but mislabels intent, paraphrases, adds speculative terms, and its free answers are unreliable. Verdict (Claude's, not agreed): usable only for structured extraction, never as a source of stored content.

5. Risk Analysis (new, adding to checkpoints 2 and 3: risks 1 to 21)

Fabricated 0.5B text amplified through the archive. If a reply like the one in section 2 goes into hotwords.py add, rarity weighting ranks its unusual wrong terms ("mescaline", "cascade", "cross-tolerance") high, and boosting them later pushes LLM1 toward the 0.5B's errors. Sharpens risk 14 (echo).
Blend lowers supported words when weights are small (section 3, item 2). Related to checkpoint 2 risk 3.
Term mismatch from paraphrase. Model-produced subject or terms (safety for safe) will not match stored spellings under the key-plus-fuzzy rules, and added terms (medication) can pull in arrays about a broader subject (checkpoint 3 risk 21, now seen once).

6. Code Change

None delivered this session. No release card. Existing deliveries (all kept): Hotwords_v1.0.0.zip, Hotwords_v1.0.1.zip, Hotwords_v2.0.0.zip, Hotwords_v2.1.0.zip, ContextDef05b_v1.0.0.zip. If the operator wants contextdef_05b.py changed (drop intent, enforce subject span), the zip must be attached to the new chat; the script is not in these checkpoints. Next version would be 1.1.0.

7. Open items

Whether the Clone's llama-server returns probabilities before or after the sampler cuts. Test: one request with temperature 1, top-k 40, n_probs 50. If the returned list has 40 entries the probabilities come after the cuts; 50 entries means before. This decides the recorder's request flags.
Run the same direct-answer request on port 8080 (the 4B) and compare with the 0.5B answer.
Run contextdef_05b.py on about ten more queries, mixed kinds, including a pronoun follow-up ("what about with alcohol?"), where the script has no history. This will also give a malformed rate.
Run hotwords.py on the Clone's Python 3.6 / 3.8 / 3.15 (still not done).
Agree the fail criterion, the blend rule, and how the pseudo query becomes 50 weights.
Prompt-retrieval arm (needs original replies kept somewhere), outside reference for rarity, embedding model, tokenizer check, lift smoothing, phrase window, ALPHA steps (all unchanged from checkpoint 3).

8. Standing operator rules (carry forward)

Reply header Claude: [Month DD, YYYY, HH:MM:SS AM/PM] (casual mode), real Sydney time from a code run immediately before each reply.
Never suggest pkill/kill on services. Never rename his scripts. Deployment timing is his call.
Code delivery: bump @version/@date; manual with version history; version shown once at start; zip with a dotted, never-reused version; xmllint --noout on XML; list unrequested changes BEFORE making them; end with a release card (include "legacy zipper used" only for SMF package zips).
Critique and objections before building; statements over questions; do not mimic his tone; expand acronyms; no invented jargon; PDF never Word.
Checkpoint names get a new _N, never reused (next is _6). Prefers short conversations and a fresh chat per chapter.
Terminology: "operator", "query", "reply" (reply = the model's answer).
Environment: Clone is CentOS 6.10, Python 3.6 (also 3.8 and 3.15 per the operator), SQLite 3.6.20, llama.cpp: Qwen 4B on 8080 (LLM1), 0.5B (qwen2.5-0.5b-instruct-q8_0.gguf) on 8081, build b10726. Python 3.6 on a POSIX/C locale does not switch to UTF-8, so non-ASCII must be handled explicitly (already done in both tools).

📝 Experiment_001_checkpoint_4.mdTheme:

Experiments 000, 001, 002 — Checkpoint (4)

Operator: Andrew. Written 2026-10-27 07:28 Sydney. Follows Experiment_001_checkpoint_3.md (kept, never overwritten; Experiment_001_checkpoint_2.md and Experiment_001_checkpoint.md also kept). Status: no new code delivered this session. contextdef_05b.py 1.0.0 was run for the first time against the real 0.5B on the Clone, and the 0.5B was asked one query directly. Experiments themselves not started; the recorder (build step 2 in checkpoint 3) is still not written. Resume: paste this file together with Experiment_001_checkpoint_3.md and Experiment_001_checkpoint_2.md into a fresh conversation. Where files differ, the newest wins. Checkpoint 2 holds the definitions of Experiments 000/001/002 and risks 1 to 13; checkpoint 3 holds hotwords.py 2.1.0 and risks 14 to 21; this file adds risks 22 to 24.

1. What happened since checkpoint 3

All three checkpoints were supplied to this chat. Claude read them and proposed, not yet agreed: a fail criterion for Experiment 001 (section 3), and a change to the blend rule (section 3).
contextdef_05b.py v1.0.0 was run on the Clone against the real 0.5B (qwen2.5-0.5b-instruct-q8_0.gguf, llama.cpp build b10726, port 8081, chat route) with two queries. Results are in section 2.
The 0.5B was then asked one of the queries directly (plain chat request, temperature 0, 300 tokens). The reply is in section 2 and shows serious errors.
Wording note for future chats: when the operator says "the reply" he means the model's own answer to the query, not the parsed context-definition fields. Claude misread this twice in this session.

2. Results (real runs, Clone)

Context definer, 2 queries (run folder `/root/memstore/ctxdef_try/20261026-185424`, UTC date)

	Query 1: "Why is serotonin syndrome so dangerous with MDMA?"	Query 2: "Is 2C-B safe with SSRIs?"
Raw reply (exact)	`SUBJECT: serotonin syndrome` / `INTENT: how-to` / `TERMS: MDMA, dangerous, serotonin, syndrome`	`SUBJECT: 2C-B safety with SSRIs` / `INTENT: how-to` / `TERMS: 2C-B, SSRIs, safety, medication`
Terms added by the model (`in_query: false`)	none	`safety`, `medication`
Wall time	25.45 s (prompt of 168 tokens evaluated cold in 22.9 s)	3.54 s (only 15 prompt tokens evaluated; the worked example came from the prompt cache)
Generation speed	11.6 tokens/s	10.8 tokens/s

Chat route works on that build (--api completion not needed).
Format held on both queries: exactly three lines, nothing extra, 0 warnings. Two queries is too few for a malformed rate.
Parsed fields lowercase the terms; the raw reply keeps the model's own case (MDMA, 2C-B, SSRIs); 2c-b survived intact.
Intent is wrong on both (how-to for a why-question and a yes/no question). Same label twice suggests it is copied from the worked example in the prompt. Not confirmed (the script is not in the chat).
Subject is a paraphrase in query 2 ("safety" for "safe"). hotwords.py fuzzy matching would not link "safe" to "safety" (edit distance too large).
Term order: the model put MDMA first on query 1, not the subject words, so its "most important first" ranking is weak.
Only the terms would reach LLM2. Intent and subject are log labels. Filtering on in_query: false leaves 2c-b and ssris.

The 0.5B's direct answer to query 1 (300 tokens, cut off by the limit)

Speed 13.8 tokens/s, 21.7 s for 300 tokens. Errors: calls MDMA "mescaline"; vague "serotonin drugs"; points 1 and 3 repeat the same sentence; generic symptom list (fatigue, appetite, mood) instead of mental-state change, autonomic overactivity, clonus/hyperreflexia and hyperthermia; "risk of death is very high" overstated; never states the mechanism (serotonin release and reuptake block) or names MAOIs; point 4 ("cross-tolerance") off-topic. The same query on the 4B (port 8080) has not been run.

Note: the operator's curl used /v1/chat/completion (no final "s") and the server still answered.

3. Proposals made this session (not agreed)

Fail criterion for Experiment 001, relative to Experiment 000's noise. Write the rule before 000 runs, fix the numbers after 000.
- Pass: on the fixed-context comparison, candidacy shift toward the stored wording is clearly larger than the movement 000 shows between identical runs, and the in-subject share of picked words does not fall.
- Null: the shift is inside the 000 noise band, or it appears only with a noticeable fluency gap (LLM1's average probability per word drops against its natural reply).
- Not a fail: an unchanged free-running reply (already stated in checkpoint 2).
Blend rule. Averaging supported candidates toward small weights (fuzzy matches give 0.9 or 0.8 times small term weights) mostly lowers the supported words, the opposite of the intent. Alternative: additive boost in log space with a capped boost (the memtok approach), which cannot push a word down. Or rescale the weights so the top supported candidate is at least its own LLM1 probability before averaging. Claude prefers the additive boost.
0.5B role. Keep as a comparison trial only (checkpoint 3 build step 4); use the rule-based definer for the real path. Drop the intent field from the 0.5B's job (use the rule-based table: question word, command verb, statement). Require the subject to be a span copied from the query, or mark non-query subject words like terms. Filter terms with in_query: false before they reach the pseudo query.
Archive content. The ten stored pairs for Experiment 001 must include the operator's corrections or preferred wording. Replies from the 0.5B must not be stored; 4B replies should be checked before storing.
Next build step after the open check below: the natural-run recorder (queries from a file, port 8080 directly, per-step raw top 50, manifest as in checkpoint 2).

4. Problem & Resolution

Problem: unknown whether the 0.5B could serve as the context definer and what it produces on the real server; also unclear how reliable its content is. Resolution: run done (section 2). It follows the format and extracts query terms well, but mislabels intent, paraphrases, adds speculative terms, and its free answers are unreliable. Verdict (Claude's, not agreed): usable only for structured extraction, never as a source of stored content.

5. Risk Analysis (new, adding to checkpoints 2 and 3: risks 1 to 21)

Fabricated 0.5B text amplified through the archive. If a reply like the one in section 2 goes into hotwords.py add, rarity weighting ranks its unusual wrong terms ("mescaline", "cascade", "cross-tolerance") high, and boosting them later pushes LLM1 toward the 0.5B's errors. Sharpens risk 14 (echo).
Blend lowers supported words when weights are small (section 3, item 2). Related to checkpoint 2 risk 3.
Term mismatch from paraphrase. Model-produced subject or terms (safety for safe) will not match stored spellings under the key-plus-fuzzy rules, and added terms (medication) can pull in arrays about a broader subject (checkpoint 3 risk 21, now seen once).

6. Code Change

None delivered this session. No release card. Existing deliveries (all kept): Hotwords_v1.0.0.zip, Hotwords_v1.0.1.zip, Hotwords_v2.0.0.zip, Hotwords_v2.1.0.zip, ContextDef05b_v1.0.0.zip. If the operator wants contextdef_05b.py changed (drop intent, enforce subject span), the zip must be attached to the new chat; the script is not in these checkpoints. Next version would be 1.1.0.

7. Open items

Whether the Clone's llama-server returns probabilities before or after the sampler cuts. Test: one request with temperature 1, top-k 40, n_probs 50. If the returned list has 40 entries the probabilities come after the cuts; 50 entries means before. This decides the recorder's request flags.
Run the same direct-answer request on port 8080 (the 4B) and compare with the 0.5B answer.
Run contextdef_05b.py on about ten more queries, mixed kinds, including a pronoun follow-up ("what about with alcohol?"), where the script has no history. This will also give a malformed rate.
Run hotwords.py on the Clone's Python 3.6 / 3.8 / 3.15 (still not done).
Agree the fail criterion, the blend rule, and how the pseudo query becomes 50 weights.
Prompt-retrieval arm (needs original replies kept somewhere), outside reference for rarity, embedding model, tokenizer check, lift smoothing, phrase window, ALPHA steps (all unchanged from checkpoint 3).

8. Standing operator rules (carry forward)

Reply header Claude: [Month DD, YYYY, HH:MM:SS AM/PM] (casual mode), real Sydney time from a code run immediately before each reply.
Never suggest pkill/kill on services. Never rename his scripts. Deployment timing is his call.
Code delivery: bump @version/@date; manual with version history; version shown once at start; zip with a dotted, never-reused version; xmllint --noout on XML; list unrequested changes BEFORE making them; end with a release card (include "legacy zipper used" only for SMF package zips).
Critique and objections before building; statements over questions; do not mimic his tone; expand acronyms; no invented jargon; PDF never Word.
Checkpoint names get a new _N, never reused (next is _5). Prefers short conversations and a fresh chat per chapter.
Terminology: "operator", "query", "reply" (reply = the model's answer).
Environment: Clone is CentOS 6.10, Python 3.6 (also 3.8 and 3.15 per the operator), SQLite 3.6.20, llama.cpp: Qwen 4B on 8080 (LLM1), 0.5B (qwen2.5-0.5b-instruct-q8_0.gguf) on 8081, build b10726. Python 3.6 on a POSIX/C locale does not switch to UTF-8, so non-ASCII must be handled explicitly (already done in both tools).

📝 MEMSTORE_MANUAL.mdTheme:

memstore.py — Manual

Project: Project AI Chatbot Memstore Version: 1.1.0 Date: 2026-10-26 Another collaboration between Andrew.human and Claude.ai

Written to be read by people and by AI models.

What it is

A small personal memory store in one Python file, backed by one SQLite file. It holds short statements about the operator (one entry per statement), each with the quote it came from. It runs without any language model on the machine.

The rule it enforces: nothing becomes live memory unless the operator adds it or approves it. Only approved entries are ever printed by inject.

A separate program, qwen_propose.py (see QWEN_PROPOSE_MANUAL.md), can suggest entries using a local model. It never touches this database.

Requirements

Python 3.6 or later (written for Live's Python 3.6.3).
Python's built-in sqlite3 module. No other packages.
No FTS5 (SQLite's newer full-text search) needed: keyword matching is done in Python.

Upgrading from 1.0.0

Copy the new memstore.py over the old one and run any command. The first run adds three columns (kind, importance, superseded_by) to your existing database automatically. Existing entries become kind fact, importance 2. Back up memstore.db first if you want a way back; the 1.0.0 program cannot read a migrated database's new columns but will still open it.

Database file

Default /root/memstore/memstore.db (the folder comes from $MEMDIR if set), whatever folder you run the command from. Override with --db PATH or the environment variable MEMSTORE_DB. Created on first run. The path in use prints to stderr on every run.

Each entry holds:

Field	Meaning
`id`	Entry number
`text`	The statement, short, third person preferred ("Operator prefers X")
`quote`	The original words it came from
`source_id`	Which transcript or conversation it came from (optional)
`origin`	`stated` (said directly) or `inferred` (concluded, not said)
`kind`	`fact`, or `correction` (always injected)
`importance`	1 to 3, default 2; ranks `inject` output
`status`	`proposed`, `approved`, `rejected` or `superseded`
`superseded_by`	Id of the entry that replaced this one
`sensitive`	1 if a sensitive keyword matched the text or the quote
`created`, `decided`	UTC timestamps

Commands

Every run prints memstore 1.1.0 (2026-10-26) once to stderr at start, so inject output on stdout stays clean for piping.

Command	What it does
`add TEXT... [--quote Q] [--source S] [--kind fact\|correction] [--importance 1-3]`	Explicit entry. Approved immediately, origin `stated`. Refused only if the same text is already proposed or approved.
`supersede OLD_ID TEXT... [--quote Q] [--source S] [--kind K] [--importance N]`	Replace an approved entry. The new entry is approved at once. The old one is kept, marked `superseded`, and never injected. Kind and importance default to the old entry's.
`propose FILE.json`	Import candidates. Status `proposed`. Nothing is live yet.
`review`	Show everything awaiting approval, with quote and flags.
`approve ID... [--importance N]`	Approve specific entries, optionally setting importance.
`approve --all`	Approve all proposals except sensitive ones, which are listed as held back.
`reject ID...`	Reject proposals. Kept as rejected so they are not re-proposed.
`list [--status S]`	List entries. Each line shows id, status, importance, flags.
`inject [WORDS...] [--limit N]`	Print approved entries as `- text` lines (see below).
`forget --id N`	Delete one entry for real.
`forget --source S`	Delete every entry that came from source `S`.

How `inject` orders its output

Corrections first, as - CORRECTION: text. They are never filtered by keywords and never count against --limit.
Then facts. With keywords: only facts sharing a word with the keywords, ranked by shared words, then importance, then newest. Without keywords: ranked by importance, then newest.
--limit N caps the facts only.
Inferred entries carry an [inferred] suffix. Superseded, rejected and proposed entries are never printed.

Superseding versus forgetting

supersede keeps history: the old text stays in the database, visible in list. forget is real deletion (the row is removed, secure_delete is on, and the file is compacted with VACUUM). Use forget when text must not remain, including old superseded entries.

Proposal file format

A JSON list. Each item:

[
  {
    "text": "Operator prefers short replies",
    "quote": "keep it short please",
    "source_id": "chat-2026-10-25-01",
    "origin": "stated",
    "kind": "fact",
    "importance": 2
  }
]

text and quote are required. An item with no quote is rejected, because every entry must be checkable.
origin defaults to inferred. kind defaults to fact. importance defaults to 2 and must be 1 to 3.
A duplicate of any existing entry in any status (including rejected and superseded), ignoring case and spacing, is skipped. This is what stops a rejected suggestion returning on the next run.
propose exits with status 1 if any item was invalid, after importing the valid ones.

Sensitive entries

On import (and on add), an entry is flagged SENSITIVE if its text or its quote matches keywords for health, identity and belief, money and legal matters. Flagged entries are never approved by approve --all; each needs approve ID.

Limits, stated plainly:

Matching is by keyword only. It is a safety net, not a judge of meaning.
It does not detect personal names.
Entries added with add are flagged but approved anyway, because the operator typed them.

Typical session

python3 memstore.py add "Operator prefers Markdown" --quote "Markdown only"
python3 memstore.py add "Operator requires ordered shutdowns" --kind correction --importance 3
python3 memstore.py propose proposals.json
python3 memstore.py review
python3 memstore.py approve 4 7
python3 memstore.py approve --all
python3 memstore.py supersede 3 "Operator runs CentOS 6.10" --quote "6.10"
python3 memstore.py inject markdown --limit 20
python3 memstore.py forget --source chat-2026-10-25-01

Exit codes

0 success, 1 import finished with invalid items, 2 usage or input error.

Ordered shutdown

No service or background process is involved. The program starts, runs one command and exits.

Version history

Version	Date	Change
1.1.1	2026-10-27	Database default is now `/root/memstore/memstore.db` instead of `./memstore.db`; the path in use prints at start.
1.1.0	2026-10-26	Added correction kind, supersede, importance ranking, automatic migration. Fixed: rejected/superseded entries could be re-proposed; sensitive check ignored the quote; `inject --limit` kept the oldest entries.
1.0.0	2026-10-25	First release.

📝 QWEN_MEMCHAT_MANUAL.mdTheme:

qwen_memchat.py — Manual

Project: Project AI Chatbot Memstore Version: 1.0.2 Date: 2026-10-26 Another collaboration between Andrew.human and Claude.ai

Written to be read by people and by AI models.

What it is

A terminal chat with a local Qwen server. Qwen itself remembers nothing between chats. At the start of each chat this program asks memstore.py inject for the entries the operator has approved, and gives them to the model as background facts. It never changes the memory store.

Setting up on the Clone

Unzip the bundle into one folder, for example /root/memstore. It holds memstore.py, qwen_propose.py and qwen_memchat.py side by side.
Choose where the memory database lives and use the same path every time, for example export MEMSTORE_DB=/root/memstore/memstore.db (add it to your shell profile). From memstore 1.1.1 the default is /root/memstore/memstore.db whatever folder you run from, so the export is only needed to point somewhere else.
Start a Qwen server (Qwen34B.sh for port 8080, or Qwen25_05B.sh for port 8081).
Run python3.8 /root/memstore/qwen_memchat.py --python python3.6 (the Clone's python3.8 has no SQLite; run memstore.py itself with python3.6 too) (add --profile 05b for the small model).

The loop

Chat: qwen_memchat.py saves each chat in ~/qwen_logs as memchat_DATE_TIME.md, with Human: and Qwen: line labels.
Suggest: python3.8 qwen_propose.py ~/qwen_logs/memchat_X.md --operator-marker '^Human:' --assistant-marker '^Qwen:' --out p1.json --verbose
Review: python3.8 memstore.py propose p1.json, then memstore.py review, then approve ID.... Nothing is memory until you approve it.
Next chat: the approved entries are handed to Qwen automatically.

(The proposer's closing line says "on the VPS"; ignore that when everything runs on the Clone.)

You can also add entries by hand at any time: memstore.py add "Operator prefers X" --quote "...".

Options

Option	Meaning
`--profile 4b\|05b`	Model settings. Default `4b` (Qwen3-4B, port 8080). `05b` is Qwen2.5-0.5B on port 8081.
`--url`, `--model`, `--timeout`	Override the profile.
`--context-chars N`	Characters of system text plus history kept (profile: 16000 for 4b, 6000 for 05b). Older turns are dropped first when over.
`--max-tokens N`	Longest reply. Default 400.
`--memstore PATH`	Path to `memstore.py`. Default: next to this program.
`--python PATH`	Python used to run `memstore.py`. It needs SQLite. Default: the Python running this program. On the Clone `python3.8` has no SQLite but `python3.6` does, so use `--python python3.6`.
`--db FILE`	Database file; same as setting `MEMSTORE_DB` for this run.
`--limit N`	Most facts handed to the model. Default 20. Corrections do not count against it.
`--no-memory`	Chat without memory.
`--system TEXT`	Extra system text, placed before the memory.
`--print-system`	Print the system text with the memory in it, and exit. Used by the start scripts.
`--print-webui-config`	Same text as web page settings JSON, and exit.
`--logdir DIR`	Where chats are saved. Default `~/qwen_logs`.

Every run prints qwen_memchat 1.0.2 (2026-10-26) once to stderr at start.

Memory in the browser page

Qwen34B.sh (1.2.0) and Qwen25_05B.sh (1.1.0) build the memory text with --print-system / --print-webui-config when they start, save it in the memstore folder, and give it to llama-server using whichever flag that build has: --webui-config-file (web page settings) or --system-prompt-file. They print which one they used, or say that neither exists. Memory is read once at start, so after approving new entries stop the server in your usual ordered way and start it again. If the web page keeps its own saved system message from earlier use, that saved one can override the server's; clear it in the page's settings if the memory seems ignored.

In the chat

Type a line and press Enter. The reply appears as the model produces it.
/quit or /exit (or Ctrl-D) leaves.
Ctrl-C during a reply stops it. That exchange is not kept in the chat or the log.
If the model call fails or returns nothing, the exchange is not kept.

What the model is told

The system text says the listed facts were approved by the operator, to use them as background without reciting them, not to claim anything about the operator beyond them, and to follow lines marked CORRECTION. Small models can still ignore or misuse this; the facts are a nudge, not a guarantee. Check the Memory: N approved entries line at start to confirm they were passed.

Limits

The 4B reads a prompt at about 3 tokens per second on the Clone. The facts and chat history are re-read only when they change, as the server caches the start of the prompt, but the first message of a chat pays for the facts.
The server's own context limit (-c 8192 for the 4B) is the hard ceiling. --context-chars keeps the chat under it using a rough character count, not exact tokens.
If memstore.py is missing or fails, the chat continues without memory and says so on stderr.

Exit codes

0 normal exit, 2 usage error or log folder cannot be created.

Ordered shutdown

No service is involved. Leave the chat with /quit, then stop the Qwen server in your usual ordered way.

Version history

Version	Date	Change
1.0.2	2026-10-26	Added `--print-system` and `--print-webui-config` for the start scripts.
1.0.1	2026-10-26	Added `--python`, since the chat's own Python may lack SQLite.
1.0.0	2026-10-26	First release.

📝 QWEN_MEMPROXY_MANUAL.mdTheme:

qwen_memproxy.py and QwenMemProxy.sh — Manual

Project: Project AI Chatbot Memstore Version: qwen_memproxy.py 1.4.1, stages/contextdef_stage.py 1.0.1, stages/hotwords_rag_stage.py 1.0.0, stages/hotwords_store_stage.py 1.0.0, QwenMemProxy.sh 1.0.2 Date: 2026-10-27 Another collaboration between Andrew.human and Claude.ai

Written to be read by people and by AI models.

What it is

The llama-server web page ignores the system text the server offers it, so memory set at server start never reached the page. qwen_memproxy.py is a go-between that sits in front of a llama-server. It forwards every request unchanged (the web page, the API, streamed replies) except chat requests, to which it adds the operator-approved memstore entries as background facts. It never changes the memory store.

Ports

Profile	Open this	Forwards to
`4b`	`http://clone:8090/`	Qwen3-4B server on 8080
`05b`	`http://clone:8091/`	Qwen2.5-0.5B server on 8081

The Qwen servers and their ports are unchanged. Anything that talks to port 8080 or 8081 directly (such as qwen_memchat.py) works as before, without the go-between.

Using it

sh Qwen25_05B.sh            (start the Qwen server first)
sh QwenMemProxy.sh 05b      (start the go-between; open http://clone:8091/)
sh QwenMemProxy.sh stop 05b (ordered stop; do this before stopping the Qwen server)

QwenMemProxy.sh uses python3.8 for the go-between and python3.6 to read memstore (the Clone's python3.8 has no SQLite), and the database at $MEMSTORE_DB or else /root/memstore/memstore.db. Its log goes in /root/qwen_logs/memproxy_PROFILE_DATE.log and records start, stop and the entry count whenever it changes.

Where the facts go (`--place`)

Setting	What it does	Default for
`user`	Puts the facts in front of the latest question only, as "Facts about me ... My question: ...". Earlier turns are untouched, and the page's own chat view shows only what you typed.	`05b`
`system`	Adds the facts to the end of the system message.	`4b`

Why: a very small model (the 0.5B) gave little weight to system text and answered "Windows" despite the facts being present. Text next to the question is weighed more. For the 4B the system message is better because that part of the prompt is cached by the server, while facts in the question add about 100 tokens of reading (roughly 20 seconds at the 4B's speed) to every message. These defaults are untested guesses on real models; change with --place if the other works better.

How the memory is added

The entries are re-read every 5 seconds (--refresh), so a new approve or add applies to the next chat with no restart.
A memory block left in the page's system message by the start scripts is always removed first, so stale entries never reach the model.
With --place system the facts go at the end of the system message, or in a new one if the page sent none. With --place user they go in front of the latest user message.
Only POST /v1/chat/completions is changed. Everything else passes through byte for byte.
Each chat request writes one line to the log: chat request: memory added in front of the latest user message, ...to the page's system message, ...as a new system message, no memory to add ..., or not a normal chat body .... If the log shows no chat request line after you ask something, the page is not going through the go-between (check the port in the address bar).
If memstore cannot be read, requests are forwarded without memory and the log says so.

Stages: programs the proxy hooks in by itself

A stage is a small program in the stages folder (--stages DIR, default stages next to qwen_memproxy.py, normally /root/memstore/stages). The proxy scans the folder when it starts and again for every chat query, so a program added or removed later is hooked in or out without a restart. Each change is written to the log once.

The program says where it hooks in. Its header (first 80 lines) carries tag lines, in the same style as @version and @date:

 * @hook    before-model
 * @order   10

@hook is required. A .py or .sh program without a valid @hook line is listed in the log as not hooked in, and is not run.
@order is optional (default 50). Programs at the same hook point run lowest number first, then by file name.
@version is read for the log.
@needs FILE (optional, may be repeated) names a file that must sit in the same folder as qwen_memproxy.py. At start-up, and whenever the folder changes, the proxy warns WARNING: <stage> needs <file>, not found in <folder> instead of letting the stage fail on the first query. The stage is still hooked in.
Files whose names start with . or _, and files that are not .py or .sh, are ignored. .py programs run with --stage-python (default: the Python running the proxy), .sh programs with sh.

Hook points

Hook point	When it runs	What it may return
`before-model`	Before the query is forwarded to the model	`context` (text added after the question, under "[About this question, worked out automatically and possibly imperfect]"); `memory` (text that replaces the memory entries for later stages and for what is sent)
`after-reply`	After the whole reply has gone to the page, in a separate thread so the page is not kept waiting	Nothing is used; the output is only logged
`each-word`	Planned, for the work done one next-word candidate at a time. A program tagged with it is reported as not runnable yet

A query that starts with remember: runs no stages.

What a stage receives and returns

In: a work order as JSON on standard input. The query is never put on a command line. Keys: version (1), hook, proxy (its version), home (the folder holding qwen_memproxy.py), sql_python (the Python given with --python, the one that can read SQLite; the Clone's 3.8 cannot), profile (4b or 05b), query (the newest user message as typed), messages (the whole chat so far, as the page sent it), memory (the memory entries text), results (what earlier stages returned, by file name). after-reply stages also get reply.
Out: one JSON object on standard output. Keys other than context and memory are kept in results for later stages and shown in the log.
Standard error: shown in the log with the stage's name. Each stage prints its own version there once, as it starts.
Failure: a non-zero exit code, no answer within --stage-timeout seconds (the stage is asked to stop, and forced only if it ignores that), or output that is not a JSON object. The stage is logged as SKIPPED and the query goes on without it.

A minimal stage:

# @hook    before-model
import json, sys
order = json.load(sys.stdin)
print(json.dumps({"context": "The question has %d words." % len(order["query"].split())}))

The control file

stages.control in the stages folder (or any file given with --control FILE) is optional. One line per program; # starts a comment:

contextdef_stage.py  order=10  timeout=120
hotwords.py          enabled=no

Keys: enabled=yes|no, order=N, timeout=S. A line overrides the program's own header for those keys; the hook point can only be set by the program. A line that names a program not in the folder, or that the proxy cannot read, is reported in the log. The file is read again for every chat query. One folder with several control files (--control) is a convenient way to keep a set of experiments.

hotwords_rag_stage.py (the RAG step)

stages/hotwords_rag_stage.py hooks in at before-model, order 20, and needs hotwords.py next to the proxy.

It runs hotwords.py pseudo on the query: the query's own hot-term array plus the related terms RAG draws from the stored arrays. pseudo stores nothing.
It adds nothing to the prompt. The pseudo query is in the log and in results (query_terms, related_terms, rag_arrays, fuzzy_matches, and line, which is the pseudo query as hotwords.py prints it). Putting retrieved terms into the prompt would be a different experiment and would be a separate stage.
The query goes to hotwords.py on its standard input. hotwords.py runs with sql_python. The database is whichever hotwords.py uses ($HOTWORDS_DB, else $MEMDIR/hotwords.db).

hotwords_store_stage.py (feeds the RAG)

stages/hotwords_store_stage.py hooks in at after-reply, order 50, and needs hotwords.py next to the proxy.

It hands each finished query and reply to hotwords.py add, labelled proxy YYYYmmdd-HHMMSS, so the stored arrays grow as you chat.
It does not store replies from the 05b profile. The 0.5B's mistakes would be stored, and the rarity weighting would rank them high because they are unusual (risk 14 in the Experiment 001 checkpoints).
It stores the other models' replies unchecked. That is what Experiment 000 (the natural baseline) wants. Before Experiments 001 and 002, switch it off with hotwords_store_stage.py enabled=no in stages.control, and feed the store only what you approve or correct, with hotwords.py add by hand.
Result shown in the log: stored (true or false), why, and the lines hotwords.py printed about what it added.

contextdef_stage.py

stages/contextdef_stage.py hooks in at before-model, order 10, and needs contextdef_05b.py next to the proxy (the start-up warning says if it is missing). It runs contextdef_05b.py unchanged on the query and returns the subject, intent and terms.

contextdef_05b.py must sit in the same folder as qwen_memproxy.py, and the 0.5B server must be running on 8081, which is where contextdef_05b.py looks, whichever profile the proxy has.
Only the subject and the terms that appear in the query are added to what the chat model sees (Subject: ..., Key terms: ...). The intent and the terms the model added itself ([+]) are in the log and in results, and are not sent.
It finds the result by the run folder: line that contextdef_05b.py prints, then reads the JSON file there whose index is 1 and which has a parsed section. It was written from sample output, not from that script's source. If the output changes, the stage says what it could not find and the proxy skips it.
A query takes about 3.5 s once the 0.5B has its prompt cached, and about 25 s the first time after a cold start.

Options (`qwen_memproxy.py`)

Option	Meaning
`--profile 4b\|05b`	Ports as in the table. Default `4b`.
`--listen-port N`, `--upstream-port N`	Override the profile.
`--host ADDR`	Address to listen on. Default `0.0.0.0`.
`--memstore PATH`, `--python PATH`, `--db FILE`, `--limit N`	As for `qwen_memchat.py`.
`--place system\|user`	Where the facts go (see above).
`--drop-tools WORDS`	The llama-server web page offers the model its own tools. A tool whose name or description contains one of these comma-separated words is removed before the request is forwarded. Default `none` for both profiles (every tool passes through). `runtime` removes the page's `get_info` tool ("Get runtime info (OS name)"; the page's other tool is `get_datetime`): it reports the browser PC's operating system, and the 0.5B answered "Windows" from it instead of using the memory. On the 0.5B even the harmless `get_datetime` made some answers wander or start writing a tool call; `--drop-tools all` removes both if that happens again. `none` lets every tool through; `all` removes every tool. The log lists the tools the page offers (once per set) and what was dropped, so the real names can be checked.
`--refresh S`	Seconds between memory re-reads. Default 5.
`--stages DIR`	Folder of stage programs. Default `stages` next to `qwen_memproxy.py`. `none` switches stages off, which is the way to get a plain run for comparison.
`--control FILE`	Control file for the stages. Default `stages.control` inside the stages folder.
`--stage-timeout S`	Seconds a stage may take before it is stopped. Default 120.
`--stage-python PATH`	Python used to run `.py` stages. Default: the one running the proxy.
`--quiet`	One log line per chat query, instead of the full detail. Verbose is the default.

It prints qwen_memproxy 1.1.0 (2026-10-26) once at start, to its log.

Limits

It does not save conversations, but unless --quiet is given its log shows every query, the text actually sent to the model, each stage's output and the reply. The log files therefore hold whatever you type. Use qwen_memchat.py when you want transcripts for the proposer.
Anyone who can reach the listening port can chat with the model, as with the server itself. The stop address works only from the Clone itself.
A small model may still ignore the facts.

Exit codes

0 stopped normally, 2 the port could not be opened (often because one is already running).

Ordered shutdown

sh QwenMemProxy.sh stop 05b (or 4b) asks the program to stop itself and finish cleanly. Stop the go-between first, then the Qwen server in your usual way.

Saving a fact from the chat (`remember:`)

Start a message with remember: and the text after it is saved to the memory store.

Example: remember: my name is Andrew

The entry is approved straight away and recorded as stated. The whole line you typed is kept as its quote, and the source is memproxy.
It is already in the memory for that same question; no restart is needed.
The model is told to confirm in one short sentence. If the store refuses (or the fact is longer than 500 characters) it is told to say it was not saved; if the fact was already stored it is told to say so. The confirmation sentence comes from the model, so the proof is the log line chat request: remember: saved ... and memstore.py list.
Only the newest message is checked. Earlier remember: lines in the chat history never save again.
It does nothing about sensitive keywords: add approves immediately, as the manual for memstore.py says.
Nothing is saved from a chat unless you start the message this way.

Version history

Version	Date	Change
1.4.1	2026-10-27	`qwen_memproxy.py`: `@needs` header line with a start-up warning for missing files; work order gains `sql_python` and `profile`. New `stages/hotwords_rag_stage.py` 1.0.0 and `stages/hotwords_store_stage.py` 1.0.0 (around `hotwords.py` 2.1.0). `contextdef_stage.py` 1.0.1 (adds `@needs`). `stages.control` is no longer in the package, so an edited copy is not overwritten.
1.4.0	2026-10-27	`qwen_memproxy.py`: stages. The proxy scans a folder for `.py` and `.sh` programs and hooks each in at the point its `@hook` header line names (`before-model` or `after-reply`; `each-word` is reserved), ordered by `@order`. Optional control file. Verbose log by default (`--quiet` to reduce). New `stages/contextdef_stage.py` 1.0.0 runs `contextdef_05b.py`.
1.3.1	2026-10-27	`qwen_memproxy.py`: `--drop-tools` default is now `none` for both profiles, so every page tool passes through (was `all` for 05b, `runtime` for 4b).
1.3.0	2026-10-27	`qwen_memproxy.py`: a message starting with `remember:` saves the rest as an approved fact.
1.2.1	2026-10-27	`qwen_memproxy.py`: `--drop-tools` default is now per profile: `all` for 05b, `runtime` for 4b.
1.2.0	2026-10-27	`qwen_memproxy.py`: added `--drop-tools` (default `runtime`) to remove page tools before forwarding; the tools the page offers are logged once.
1.1.0	2026-10-26	`qwen_memproxy.py`: added `--place system\|user`. `QwenMemProxy.sh` 1.0.2: reports FAILED and exits 1 when the program never starts listening.
1.0.1	2026-10-26	Per-request log line. `QwenMemProxy.sh` waits up to 30 s for start-up.
1.0.0	2026-10-26	First release of `qwen_memproxy.py` and `QwenMemProxy.sh`.

📝 QWEN_PROPOSE_MANUAL.mdTheme:

qwen_propose.py — Manual

Project: Project AI Chatbot Memstore Version: 1.1.0 Date: 2026-10-26 Another collaboration between Andrew.human and Claude.ai

Written to be read by people and by AI models.

What it is

A program that reads conversation transcripts, asks a local language model to suggest memory entries, checks the suggestions with plain code, and writes a JSON file for memstore.py propose. The model only suggests. Nothing here writes to the memstore database, and nothing goes live until the operator approves it in memstore.py.

Where to run it

Not on Live. Run qwen_propose.py on the machine that hosts the model (the Clone, where the llama.cpp servers run), then copy the proposals file to the VPS and run memstore.py propose there. No transcript text leaves your own machines.

Requirements

Python 3.6 or later (the Clone's python3.8 works). Standard library only.
A local server speaking the OpenAI-style chat API at /v1/chat/completions. llama.cpp's llama-server does.
The two servers started by Qwen34B.sh (port 8080) and Qwen25_05B.sh (port 8081).
Tested only against stand-in servers, not against a real Qwen. Check your first real run with --verbose.

Profiles

Profile	Model	URL	Chunk size	Timeout per call
`4b` (default)	Qwen3-4B	`http://127.0.0.1:8080/v1/chat/completions`	2000 characters	3600 s
`05b`	Qwen2.5-0.5B	`http://127.0.0.1:8081/v1/chat/completions`	3000 characters	900 s

Any of --url, --model, --chunk-chars, --timeout you type overrides the profile.

Speed and quality: on the Clone the 4B generates about 1.3 to 1.7 tokens per second and reads a prompt at about 3 tokens per second, so a 2000-character chunk takes roughly 5 to 10 minutes. A long transcript means hours; run it in the background (nohup ... &) and watch the progress lines. The 0.5B is far faster but weak: expect it to suggest little, and expect the quote check to drop much of what it invents. Use it to test the plumbing, and the 4B for real work.

Usage

python3.8 qwen_propose.py chat1.md --out proposals_2026-10-26.json --verbose
python3.8 qwen_propose.py chat1.md --profile 05b --out proposals_05b_1.json --verbose
python3.8 qwen_propose.py chat1.md --operator-marker '^Human:' --assistant-marker '^Claude:' --out p3.json

Option	Meaning
`files`	One or more transcript files (`.md` or `.txt`).
`--profile 4b\|05b`	Settings for the model (table above). Default `4b`.
`--out FILE`	Proposals file to create. Default `/root/memstore/proposals.json`. Never overwritten: if it exists the program stops, so give each run a new name.
`--url`, `--model`, `--chunk-chars`, `--timeout`	Override the profile.
`--state FILE`	Remembers which transcripts were done, per model and file name, by content hash. Default `/root/memstore/qwen_propose_state.json`.
`--force`	Reprocess transcripts even if unchanged.
`--operator-marker RE`, `--assistant-marker RE`	Regular expressions for lines where each speaker starts. When given, only the operator's lines are shown to the model, so it cannot mistake the assistant's words for facts about the operator. Give both or neither.
`--verbose`	Show each dropped suggestion and why.

Every run prints qwen_propose 1.1.0 (2026-10-26) once to stderr, then one line naming the profile settings in use, then one progress line per chunk.

What the program checks itself (no model involved)

For every suggestion:

The quote must be at least 8 characters.
The quote's words must appear in order in the transcript chunk (spacing and letter case may differ). If not, the suggestion is dropped. This is the main defence against a small model inventing things.
The stored quote is the transcript's own wording, not the model's copy.
The text must be non-empty and at most 300 characters.
At most 10 suggestions are taken per chunk; repeats within a run are dropped.
origin is always set to inferred, whatever the model says. Only the operator can treat something as stated, by typing it with memstore.py add.
kind is fact unless the model says correction. A wrongly labelled correction would be always injected if approved, so check CORRECTION flags during review.
importance is always 2; set it when approving.

Model replies

The model is told to answer with JSON only, and the request asks the server to enforce JSON. If the server refuses that request (HTTP error), the program retries once without it. The program also accepts JSON wrapped in prose or code fences, and removes any <think>...</think> block first. A reply with no usable JSON counts as a chunk failure.

If any chunk of a transcript fails (model unreachable, timeout, or unusable reply), that transcript is not recorded as done, so the next run retries it. The exit status is 1.

Already-processed record

The state file is keyed by model|file name. A transcript done with the 4B is therefore still processed when you run the 0.5B, and the reverse. A changed transcript (different content hash) is processed again.

Untrusted text

Transcript text is treated as data, marked with delimiters, and the model is told not to follow instructions inside it. A small model can still be fooled, so the real protection is the quote check plus your own review.

Output file

A JSON list in the format memstore.py propose reads, with source_id set to the transcript's file name (use memstore.py forget --source NAME to remove everything that came from one transcript). If nothing survives the checks, no file is created.

Exit codes

0 success (including "nothing new"), 1 a transcript could not be read or a chunk failed, 2 usage error.

Ordered shutdown

No service or background process is involved. The program starts, runs and exits. The model servers are yours to start and stop in order; stop the one you finished with before starting the other, as the Clone has limited memory.

Version history

Version	Date	Change
1.1.1	2026-10-27	Default `--out` and `--state` now live in `/root/memstore` (or `$MEMDIR`), not the folder you ran from.
1.1.0	2026-10-26	Added `--profile 4b\|05b`. Fixed: the processed record ignored the model. Added JSON-only request with fallback, `<think>` removal, per-chunk progress lines.
1.0.0	2026-10-26	First release.

📝 Experiment_001_checkpoint_6.mdTheme:

Experiments 000, 001, 002 — Checkpoint (6)

Operator: Andrew. Written 2026-10-27 08:44 Sydney. Follows Experiment_001_checkpoint_5.md (kept, never overwritten; _4 has an identical body to _5; _3, _2 and Experiment_001_checkpoint.md also kept). Status: three tools delivered this session (contextdef 1.1.0 with its stage, the natural-run recorder 1.0.0). The recorder was run for real on the second 4B (port 8082): one probe and one reply. Experiment 000 itself is not started: it waits for the operator's ten queries. Operator is on a break. Resume: paste this file together with Experiment_001_checkpoint_5.md (or _4, same body), Experiment_001_checkpoint_3.md and Experiment_001_checkpoint_2.md into a fresh conversation, and attach Hotwords_v2.1.0.zip, Memstore_Clone_v1.0.13.zip, ContextDef05b_v1.1.0.zip and NaturalRecorder_v1.0.0.zip. Experiment_001_checkpoint.md is superseded by _2 and not needed. Where files differ, the newest wins. Checkpoint 2 holds the definitions of Experiments 000/001/002 and risks 1 to 13; checkpoint 3 holds hotwords.py 2.1.0 and risks 14 to 21; checkpoint 5 holds risks 22 to 24; this file adds risks 25 to 29. Terms: LLM1 = the chat model (the 4B). LLM2 = the second module that would weight or add words. RAG = choosing stored arrays that match the query (here hotwords.py pseudo). The pool = the top 50 candidates at a step.

1. What happened since checkpoint 5

All checkpoints and the zips (Hotwords_v2.1.0, Memstore_Clone_v1.0.12, ContextDef05b_v1.0.0) were read. Claude read contextdef_05b.py, contextdef_stage.py, hotwords_rag_stage.py, hotwords_store_stage.py and the hotwords.py manual. Not read: qwen_memproxy.py (beyond its profile table), memstore.py, the Qwen scripts.
Findings on the context definer (section 2). The operator consented to the listed changes and to Claude's stated defaults (section 3). ContextDef05b_v1.0.0 and the stage were replaced by new versions.
The natural-run recorder was built, tested against a stand-in server, then run by the operator on the real 4B (section 4).
The operator corrected what a "typical query" is (section 5), which changes the plan for Experiment 000 part A.

2. Context definer: findings and changes

Intent was not copied from the worked example (checkpoint 5 guessed so). The example's intent was troubleshoot; the 0.5B chose how-to on its own. It is simply weak at that field.
contextdef_stage.py 1.0.1 matched the script's output (run folder: line, index, parsed).
Bug found: contextdef_05b.py exits 1 on any note about a reply (extra line, missing field) and the stage treated any non-zero exit as failure, discarding a usable result.
The subject was never checked against the query (only terms were), so a paraphrase like "2C-B safety with SSRIs" reached the chat model.
Delivered: contextdef_05b.py 1.1.0 (intent removed from prompt, parser and JSON; subject_in_query added; prompt asks for the subject "copied from the query"; worked example subject now forum slow; prompt hash changed; exit codes unchanged). contextdef_stage.py 1.0.2 (needs 1.1.0; exit 1 accepted when the result file exists, notes passed on as warnings; subject sent only if in-query; intent removed; server errors reported with their text; refuses a result with no subject_in_query). Proxy manual updated (row "stage 1.0.2"); everything else in the Memstore tree unchanged.
Not yet run on the real 0.5B: the new prompt. Whether "copied from the query" stops the paraphrase on Is 2C-B safe with SSRIs? is unknown. Results from run 20261026-185424 used the old prompt and are not comparable.

3. Decisions and consents (operator, this session)

"you have my consent" covers: the contextdef/stage changes above; the additive boost in log space with a capped boost as the blend-rule default (not averaging); the checkpoint-3 proposal for turning the pseudo query into 50 weights (a candidate's weight = the weight of the pseudo-query term it matches by spelling key, then fuzzy for plain words, else 0); and writing the fail criterion from checkpoint 5 (section 3, item 1) into the recorder's report. The criterion's numbers stay open until Experiment 000 has run. None of the blend, weights or criterion is implemented yet.
Operator stated: for the MDMA query "not much would influence it except for SSRIs".
Operator stated: that query "isn't a typical user query ... that need memory" (section 5).
Operator asked about running a second 4B on Live. Claude advised against it (the Clone needed a glibc upgrade to run llama.cpp, so Live probably cannot start it as is; request load would slow the forum; second-machine results would break comparisons). Not pursued. The Clone is where it runs.

4. Results (real runs, Clone)

Environment as stated by the operator

Clone: 4 cores (nproc), 9864 MB RAM (7.2 GB available once cache counted), 2 GB swap. 4B file ~/models/Qwen3-4B-Q4_K_M.gguf, 2.5 GB.
Servers: 4B on 8080 (go-between profile 4b on 8090 forwards to it); a second 4B on 8082 (go-between profile 04b on 8092, added by the operator; not in the Memstore zip, whose PROFILES only has 4b and 05b); 0.5B on 8081 (profile 05b on 8091). Qwen34B.sh starts a 4B with -c 8192 -t 4 -np 1 --jinja --chat-template-kwargs '{"enable_thinking":false}'; Qwen25_05B.sh the 0.5B with -t 2 -tb 4.
Scripts live in /root/memstore; the operator runs the recorder from there through his own wrapper natural_recorder.sh (his name, keep it).
-np 1 means one slot and one prompt cache per server: any other request to the same copy during a recorder run evicts the cache. So the recorder gets one copy (8082) and nothing is sent through 8092 while it runs; chat goes through 8090.

Probe on 8082 (real)

/tokenize and /detokenize work and agree; the 13-token short prompt starts with ids 151644, 872, 198, so the special tags parse as single tokens.
Top-k 40 returned 50 candidates, so the probabilities come BEFORE the sampler's cuts. This closes the open item from checkpoints 3 to 5. The recorder asks for raw settings regardless.
Prompt cache works (second request evaluated 1 token, reused 12). A cached step takes about 0.3 s (about 3 tokens per second).

First real reply: run folder `/root/memstore/natural_runs/20261026-213201` (UTC date)

Query "Why is serotonin syndrome so dangerous with MDMA?", greedy, port 8082, 300 steps (hit the max_steps default, so the reply is cut off), 148 s, about 0.36 s per token with two slow patches (other load on the four cores).

Reply quality: far better than the 0.5B's (identifies MDMA and serotonin syndrome correctly, names 5-HT2A and 5-HT1A). Calls MDMA a serotonin reuptake inhibitor; never states serotonin release in the first 300 tokens; heavy markdown scaffolding, question repeated as a heading; cut before symptoms, hyperthermia, clonus or MAOIs.
Coverage: the 50 candidates cover essentially the whole distribution (norm_total median 1.0000, worst 0.9815). Mass beyond rank 10 is 0.14%, beyond rank 20 is 0.04%. Candidates with 1% or more: median 2, mean 2.1, max 12.
Decision points: top candidate at or above 0.99 in 47% of steps; below 0.5 in 16 of 300 (5%); below 0.8 in 74 (25%). Most are style choices ( potentially vs **, can vs is, excessive vs excess); about ten of 300 are content choices (e.g. step 212: cognition 0.49, perception 0.15, temperature 0.15).
Word starts: 196 of 300 steps flagged. The flag is too narrow: 20 of the 104 unflagged steps are word starts after punctuation or markup (after **, (, ,), 5 of them real decision points (e.g. step 184 over / ser / ex). Nothing is lost in the saved data (the previous token is in the steps file).
SSRIs and MAOIs in the pool: SSR appears in the top 50 at 19 steps, but its highest probability anywhere is 2e-5; antidepress is about 1e-5 to 1e-6. At steps 38 to 40 ("the use of certain medications or") the model takes or at 100% and such sits at rank 40 with probability 0. No candidate containing "MAO" appears at any of the 300 steps.

5. The typical query needs memory (operator correction)

A typical query for this project is one whose answer depends on something the 4B cannot know: the operator's systems, corrections, earlier decisions (e.g. "Which port does the 04b go-between listen on?"). The MDMA query was the wrong test case, and the statistics above describe that one reply, not the 4B. Consequences (Claude's, not agreed beyond consent):

Part A queries: about seven memory-dependent queries plus about three controls that need no memory (to check nothing shifts off-subject). The last three are the controls unless the operator says otherwise. Self-contained queries only until conversation support exists. The operator will supply them, screened for secrets.
The natural baseline on memory queries will be confidently wrong. Those replies must never be stored (sharpens risks 14 and 22).
Experiment 001 will very probably be null on such queries by construction: the needed words are not in the 4B's pool at usable probability, and 001 only reweights what is already there. A capped boost of about 150 times on a 1e-6 candidate reaches about 1e-4. Experiment 002 (adding words) becomes the main test.
The competitor already exists: the go-between injects memstore facts into the prompt. That is the prompt-retrieval arm (open item since checkpoint 2); the same queries through 8092 give a comparison reply (the recorder cannot drive a go-between token by token).
The archive should come from the operator's approved facts and corrections, not from LLM1's replies, which would reduce the echo risk (14) a good deal.
MDMA subject note: for MDMA the interaction that matters most is MAOIs; SSRIs mainly blunt the effect. Any stored correction on that subject should name both.

6. Problem & Resolution

Problem: no way yet to record LLM1's natural next-word candidates at every step on the real server, and an unreliable context definer (wrong intent, paraphrased subject, stage discarding results). Resolution: recorder 1.0.0 built, tested on a stand-in server and run for real (probe clean, one reply recorded); context definer and stage fixed (section 2). The first real numbers show the pool is almost fully covered by 50 candidates and that influence has little room to act where the needed fact is outside the pool; the choice of test queries is now settled as memory-dependent (section 5).

7. Risk Analysis (new; adding to risks 1 to 24)

Single-reply statistics mislead. The 5% decision-point rate and 47% of steps at 99% or more come from one long explanatory reply with heavy markdown. Opinion, planning and conversational replies will have many more genuine choices. Do not generalise from run 20261026-213201.
The natural baseline on memory queries is confabulated (ports, script names, decisions). Storing it or boosting its terms amplifies invention. Sharpens risks 14 and 22.
The needed words are outside the pool. SSRIs at about 1e-5, MAOIs absent. Reweighting (001, by averaging or by a capped boost) cannot reach them; only word injection (002) can, anchored at a slot like "certain medications or", with its own fluency risk (checkpoint 2 risk 8).
The word-start flag misses word starts after markup and punctuation (20 of 104 unflagged steps in the first reply). Influence applied only at flagged steps would skip about 8% of real decision points. Derive the flag from the previous token's last character (planned in 1.1.0; the compare command can compute it from saved files meanwhile).
Contention and the single cache slot. Two 4B copies, the go-betweens and the 0.5B share four cores. A busy neighbour distorts timing comparisons (seen: two patches at about double time), and any request to the recorder's copy evicts its prompt cache, making the next step a cold, slow one.

8. Code Change

Delivered this session (all kept, never overwritten):

ContextDef05b_v1.1.0.zip (contextdef_05b.py 1.1.0, CONTEXTDEF05B_MANUAL.md).
Memstore_Clone_v1.0.13.zip (whole tree; only stages/contextdef_stage.py 1.0.2 and QWEN_MEMPROXY_MANUAL.md changed from v1.0.12, which the operator supplied).
NaturalRecorder_v1.0.0.zip (natural_recorder.py 1.0.0, NATURAL_RECORDER_MANUAL.md): commands probe and run; the driver chooses each token (greedy, or seeded sampling over the top 50), feeds the token id back, records the raw top 50 per step (candidate id, token, raw probability, probability renormalised over the 50, chosen rank, word-start flag, sum of the 50 raw values); fixed Qwen3 non-thinking prompt (qwen3-nothink-v1); --repeat, --mode, --seed, --max-steps (default 300, use 600 for full replies), --ctx, --system-file; outputs per reply qNNN_rNN.json and .steps.jsonl, plus pairs.jsonl (not added to anything) and run.json under $MEMDIR/natural_runs/<run>/, mode 600/700. Part B condition 3 (the same query repeated inside one growing conversation) is not supported in 1.0.0.
Earlier deliveries still stand: Hotwords_v1.0.0.zip, v1.0.1, v2.0.0, v2.1.0, ContextDef05b_v1.0.0.zip, Memstore_Clone_v1.0.12.zip.
All three new deliveries were tested only against stand-in servers except the recorder's probe and one reply. Release cards were given with each (none missing; "legacy zipper" not needed, no SMF packages).

9. Open items

Operator to supply: about ten queries (seven memory-dependent, three controls), one per line, screened for secrets. Run with run --file queries.txt --max-steps 600 on 8082, nothing through 8092 meanwhile.
Before choosing the pairs to store: switch off hotwords_store_stage.py (enabled=no in stages.control).
Greedy repeat check: suggested --repeat 3 --max-steps 600 first (about 11 min), then 20; optionally compare the same query's reply hash on 8080 against 8082 with nothing else busy.
Recorder 1.1.0 (proposed, operator to confirm or cut): conversation support (history file into the prompt), word-start fix, a compare command (exact matches, first divergence step, word-level differences, top-50 movement) with a pool-presence check (for each stored term, the steps where it appears in the top 50 and its probability).
Run contextdef_05b.py 1.1.0 and the stage on the real 0.5B: the paraphrase fix, and a malformed rate on about ten queries including a pronoun follow-up.
Run hotwords.py on the Clone's Python 3.6 / 3.8 / 3.15 (still not done).
Implement the consented defaults (additive capped boost, pseudo-query-to-50-weights, fail criterion) and fix the criterion's numbers after Experiment 000.
Prompt-retrieval arm via the go-between (8092), outside reference for rarity, embedding model, tokenizer check for the 0.5B (the 4B's ids are consistent with Qwen), lift smoothing, phrase window, ALPHA steps (all unchanged).

10. Standing operator rules (carry forward)

Reply header Claude: [Month DD, YYYY, HH:MM:SS AM/PM] (casual mode), real Sydney time from a code run immediately before each reply.
Never suggest pkill/kill on services (ordered shutdowns only). Never rename his scripts or files (e.g. natural_recorder.sh). Deployment timing is his call.
Code delivery: bump @version/@date; manual with version history; version shown once at start; zip with a dotted, never-reused version; xmllint --noout on XML; list unrequested changes BEFORE making them; end with a release card (include "legacy zipper used" only for SMF package zips).
Critique and objections before building; statements over questions; do not mimic his tone; expand acronyms; no invented jargon; PDF never Word; unzip quietly and report findings.
Checkpoint names get a new _N, never reused (next is _7). Prefers short conversations and a fresh chat per chapter.
Terminology: "operator", "query", "reply" (reply = the model's answer).
Environment: see section 4. Python 3.6 on a POSIX/C locale does not switch to UTF-8, so non-ASCII must be handled explicitly (done in all tools). SQLite 3.6.20; the proxy's SQLite Python is passed to stages as sql_python (the Clone's 3.8 has no SQLite); the recorder needs no SQLite.

dopetalk

News:

dopetalk does not endorse any advertised product nor does it accept any liability for it's use or misuse

Author Topic: Project "Influence" Master (Read 19 times)

smfadmin (OP)

Project "Influence" Master

Experiments 000, 001, 002 — Checkpoint (2)

1. Aim

2. The three experiments

Experiment 000 — background variation (control and archive)

Experiment 001 — reweight only (no new words)

Experiment 002 — adding words, heavier weights

3. Components and decisions

1. Problem & Resolution

2. Risk Analysis

3. Code Change

4. Build order

5. Open items

6. Standing operator rules (carry forward)

Project nexus: 3 linked posts

Hierarchy (a post is indented under the post it belongs to)

Chronology, oldest first

Relations

Experiment 001 — Checkpoint

Decisions (the operator's)

1. Problem & Resolution

2. Risk Analysis

3. Code Change

Build order

Open settings to fix before the first run

MEMSTORE CLONE CHECKPOINT 2026-10-27 (2) — Memstore CLOSED, The AI Influence Project OPENS

1. Decision

2. The machine

3. Memstore chapter — what was delivered (all zips delivered to the operator)

4. The AI Influence Project — agreed design

5. Current state of the code — IMPORTANT

6. Open questions for the Influence Project

7. Standing operator rules (carry forward)

8. First steps in the new conversation

hotwords.py — manual

1. Purpose

2. Requirements and install

3. Where things live

4. Commands

add

pseudo

show

top

stats

5. How an array is made

5.1 Reading the text

5.2 Which terms are candidates

5.3 Score

5.4 How many terms an array keeps — relevance, not space

5.5 Stored per array

5.6 The word counter

6. How RAG chooses (pseudo)

7. Privacy

8. Known limits in 2.0.0

9. Version history

contextdef_05b.py — manual

1. Purpose

2. Requirements and install

3. Running it

4. What it asks the model

5. What you see

6. What it saves

7. Reading the results

8. Known limits in 1.0.0

9. Version history

Experiments 000, 001, 002 — Checkpoint (2)

1. Aim

2. The three experiments

Experiment 000 — background variation (control and archive)

Experiment 001 — reweight only (no new words)

Experiment 002 — adding words, heavier weights

3. Components and decisions

1. Problem & Resolution

2. Risk Analysis

3. Code Change

6. How RAG chooses (`pseudo`)

Context definer, 2 queries (run folder `/root/memstore/ctxdef_try/20261026-185424`, UTC date)

Context definer, 2 queries (run folder `/root/memstore/ctxdef_try/20261026-185424`, UTC date)

How `inject` orders its output

Where the facts go (`--place`)