Experiments 000, 001, 002 — Checkpoint (5)

Operator: Andrew. Written 2026-10-27 07:28 Sydney (renamed _5 at 07:30). Follows Experiment_001_checkpoint_3.md (kept, never overwritten; Experiment_001_checkpoint_2.md and Experiment_001_checkpoint.md also kept). A different Experiment_001_checkpoint_4.md already exists in the operator's store; it was not seen when this file was written, so its content is not reflected here. Status: no new code delivered this session. contextdef_05b.py 1.0.0 was run for the first time against the real 0.5B on the Clone, and the 0.5B was asked one query directly. Experiments themselves not started; the recorder (build step 2 in checkpoint 3) is still not written. Resume: paste this file together with Experiment_001_checkpoint_3.md, Experiment_001_checkpoint_2.md and the existing Experiment_001_checkpoint_4.md into a fresh conversation. Where files differ, the newest wins. Checkpoint 2 holds the definitions of Experiments 000/001/002 and risks 1 to 13; checkpoint 3 holds hotwords.py 2.1.0 and risks 14 to 21; this file adds risks 22 to 24 (the existing _4 may number risks too; check for clashes).

1. What happened since checkpoint 3

1. All three checkpoints were supplied to this chat. Claude read them and proposed, not yet agreed: a fail criterion for Experiment 001 (section 3), and a change to the blend rule (section 3).
2. contextdef_05b.py v1.0.0 was run on the Clone against the real 0.5B (qwen2.5-0.5b-instruct-q8_0.gguf, llama.cpp build b10726, port 8081, chat route) with two queries. Results are in section 2.
3. The 0.5B was then asked one of the queries directly (plain chat request, temperature 0, 300 tokens). The reply is in section 2 and shows serious errors.
4. Wording note for future chats: when the operator says "the reply" he means the model's own answer to the query, not the parsed context-definition fields. Claude misread this twice in this session.

2. Results (real runs, Clone)

Context definer, 2 queries (run folder /root/memstore/ctxdef_try/20261026-185424, UTC date)

	Query 1: "Why is serotonin syndrome so dangerous with MDMA?"	Query 2: "Is 2C-B safe with SSRIs?"
Raw reply (exact)	SUBJECT: serotonin syndrome / INTENT: how-to / TERMS: MDMA, dangerous, serotonin, syndrome	SUBJECT: 2C-B safety with SSRIs / INTENT: how-to / TERMS: 2C-B, SSRIs, safety, medication
Terms added by the model (in_query: false)	none	safety, medication
Wall time	25.45 s (prompt of 168 tokens evaluated cold in 22.9 s)	3.54 s (only 15 prompt tokens evaluated; the worked example came from the prompt cache)
Generation speed	11.6 tokens/s	10.8 tokens/s

• Chat route works on that build (--api completion not needed).
• Format held on both queries: exactly three lines, nothing extra, 0 warnings. Two queries is too few for a malformed rate.
• Parsed fields lowercase the terms; the raw reply keeps the model's own case (MDMA, 2C-B, SSRIs); 2c-b survived intact.
• Intent is wrong on both (how-to for a why-question and a yes/no question). Same label twice suggests it is copied from the worked example in the prompt. Not confirmed (the script is not in the chat).
• Subject is a paraphrase in query 2 ("safety" for "safe"). hotwords.py fuzzy matching would not link "safe" to "safety" (edit distance too large).
• Term order: the model put MDMA first on query 1, not the subject words, so its "most important first" ranking is weak.
• Only the terms would reach LLM2. Intent and subject are log labels. Filtering on in_query: false leaves 2c-b and ssris.

The 0.5B's direct answer to query 1 (300 tokens, cut off by the limit)

Speed 13.8 tokens/s, 21.7 s for 300 tokens. Errors: calls MDMA "mescaline"; vague "serotonin drugs"; points 1 and 3 repeat the same sentence; generic symptom list (fatigue, appetite, mood) instead of mental-state change, autonomic overactivity, clonus/hyperreflexia and hyperthermia; "risk of death is very high" overstated; never states the mechanism (serotonin release and reuptake block) or names MAOIs; point 4 ("cross-tolerance") off-topic. The same query on the 4B (port 8080) has not been run.

Note: the operator's curl used /v1/chat/completion (no final "s") and the server still answered.

3. Proposals made this session (not agreed)

1. Fail criterion for Experiment 001, relative to Experiment 000's noise. Write the rule before 000 runs, fix the numbers after 000.
   • Pass: on the fixed-context comparison, candidacy shift toward the stored wording is clearly larger than the movement 000 shows between identical runs, and the in-subject share of picked words does not fall.
   • Null: the shift is inside the 000 noise band, or it appears only with a noticeable fluency gap (LLM1's average probability per word drops against its natural reply).
   • Not a fail: an unchanged free-running reply (already stated in checkpoint 2).
2. Blend rule. Averaging supported candidates toward small weights (fuzzy matches give 0.9 or 0.8 times small term weights) mostly lowers the supported words, the opposite of the intent. Alternative: additive boost in log space with a capped boost (the memtok approach), which cannot push a word down. Or rescale the weights so the top supported candidate is at least its own LLM1 probability before averaging. Claude prefers the additive boost.
3. 0.5B role. Keep as a comparison trial only (checkpoint 3 build step 4); use the rule-based definer for the real path. Drop the intent field from the 0.5B's job (use the rule-based table: question word, command verb, statement). Require the subject to be a span copied from the query, or mark non-query subject words like terms. Filter terms with in_query: false before they reach the pseudo query.
4. Archive content. The ten stored pairs for Experiment 001 must include the operator's corrections or preferred wording. Replies from the 0.5B must not be stored; 4B replies should be checked before storing.
5. Next build step after the open check below: the natural-run recorder (queries from a file, port 8080 directly, per-step raw top 50, manifest as in checkpoint 2).

4. Problem & Resolution

Problem: unknown whether the 0.5B could serve as the context definer and what it produces on the real server; also unclear how reliable its content is. Resolution: run done (section 2). It follows the format and extracts query terms well, but mislabels intent, paraphrases, adds speculative terms, and its free answers are unreliable. Verdict (Claude's, not agreed): usable only for structured extraction, never as a source of stored content.

5. Risk Analysis (new, adding to checkpoints 2 and 3: risks 1 to 21)

22. Fabricated 0.5B text amplified through the archive. If a reply like the one in section 2 goes into hotwords.py add, rarity weighting ranks its unusual wrong terms ("mescaline", "cascade", "cross-tolerance") high, and boosting them later pushes LLM1 toward the 0.5B's errors. Sharpens risk 14 (echo).
23. Blend lowers supported words when weights are small (section 3, item 2). Related to checkpoint 2 risk 3.
24. Term mismatch from paraphrase. Model-produced subject or terms (safety for safe) will not match stored spellings under the key-plus-fuzzy rules, and added terms (medication) can pull in arrays about a broader subject (checkpoint 3 risk 21, now seen once).

6. Code Change

None delivered this session. No release card. Existing deliveries (all kept): Hotwords_v1.0.0.zip, Hotwords_v1.0.1.zip, Hotwords_v2.0.0.zip, Hotwords_v2.1.0.zip, ContextDef05b_v1.0.0.zip. If the operator wants contextdef_05b.py changed (drop intent, enforce subject span), the zip must be attached to the new chat; the script is not in these checkpoints. Next version would be 1.1.0.

7. Open items

• Whether the Clone's llama-server returns probabilities before or after the sampler cuts. Test: one request with temperature 1, top-k 40, n_probs 50. If the returned list has 40 entries the probabilities come after the cuts; 50 entries means before. This decides the recorder's request flags.
• Run the same direct-answer request on port 8080 (the 4B) and compare with the 0.5B answer.
• Run contextdef_05b.py on about ten more queries, mixed kinds, including a pronoun follow-up ("what about with alcohol?"), where the script has no history. This will also give a malformed rate.
• Run hotwords.py on the Clone's Python 3.6 / 3.8 / 3.15 (still not done).
• Agree the fail criterion, the blend rule, and how the pseudo query becomes 50 weights.
• Prompt-retrieval arm (needs original replies kept somewhere), outside reference for rarity, embedding model, tokenizer check, lift smoothing, phrase window, ALPHA steps (all unchanged from checkpoint 3).

8. Standing operator rules (carry forward)

• Reply header Claude: [Month DD, YYYY, HH:MM:SS AM/PM] (casual mode), real Sydney time from a code run immediately before each reply.
• Never suggest pkill/kill on services. Never rename his scripts. Deployment timing is his call.
• Code delivery: bump @version/@date; manual with version history; version shown once at start; zip with a dotted, never-reused version; xmllint --noout on XML; list unrequested changes BEFORE making them; end with a release card (include "legacy zipper used" only for SMF package zips).
• Critique and objections before building; statements over questions; do not mimic his tone; expand acronyms; no invented jargon; PDF never Word.
• Checkpoint names get a new _N, never reused (next is _6). Prefers short conversations and a fresh chat per chapter.
• Terminology: "operator", "query", "reply" (reply = the model's answer).
• Environment: Clone is CentOS 6.10, Python 3.6 (also 3.8 and 3.15 per the operator), SQLite 3.6.20, llama.cpp: Qwen 4B on 8080 (LLM1), 0.5B (qwen2.5-0.5b-instruct-q8_0.gguf) on 8081, build b10726. Python 3.6 on a POSIX/C locale does not switch to UTF-8, so non-ASCII must be handled explicitly (already done in both tools).