Sysplex Checkpoint — 2026-10-11 18:51 AEDT (Clone orchestrating; AI tools chapter)

Supersedes Sysplex_Checkpoint_20261010_1332.md. Paste/upload at the start of the next conversation.

Andrew = retired sysadmin, owner of drugs-and-users.org (SMF). Prefers to be called the operator; his input is a query/prompt, Claude's output a reply. Reply header: casual mode (Claude: [October 11, 2026, 06:51:00 PM], time from code execution every reply); he toggles with the words "casual" / "precise" (precise = Claude: [Sunday 11/10/2026--18:51:00:000 AEDT]). Markdown only. Few questions, more statements; keep simple things simple; critique before building ("arse-saver"). Don't mimic his tone. Delivery rules: bump @version/@date, version-history entry, print version at start, dotted versioned zip (never reused), xmllint on XML, list unrequested changes BEFORE making them, release card (including "tested on real torch" for AI tools); SMF package zips use his legacy zipper (Claude has none). Never rename his scripts. His grep is aliased --color=always: use grep --color=never when piping. Never overwrite an earlier checkpoint/transcript: number it.

1. The three boxes

Box	Short host / address	Role	Notes
Live	forum / forum.drugs-and-users.org, 128.199.200.202	production, 64-bit CentOS 6.10	tz +08; NO mirror scripts run here; use $RUN for glibc Python
Clone	clone / 192.168.233.140	64-bit CentOS 6.10, orchestrator	/bin is the master of all three; glibc 2.17 at /opt/glibc-2.17; i7-3520M, 2 cores/4 threads, 6 GB RAM, 2 GB swap; Python 3.8.18, torch 1.10.0a0, transformers 4.37.2
Dev	dev / 192.168.233.129	32-bit CentOS 6	no torch/transformers possible; tz +11; Python 3.8 built Oct 5 without --enable-optimizations, OpenSSL 1.1.1w

Each box: only /var/www/html (db smf209) used. Environment snapshot (box_snapshot/box_compare): Live and Clone match; Dev "good enough" (his words). No more PyMuPDF work (it works).

2. Done (carried over)

• Dev sync dev_sync_v1.0.2.zip; /bin tools bin_tools_v1.2.0.zip; Clone /bin master decision; Collaborative v6.0 cut by Andrew; mirror scripts (his own edits); --delete policy final; json_push_visual_Clone_to_Live / json_push_audio_Clone_to_Live with host guard; ISE_Data stores converge (only audio_store.json updated field differs). Details are in Sysplex_Checkpoint_20261010_1332.md sections 2-3.

3. Open items (carried over; parked while the AI chapter ran — "forget the script")

1. Credentials: copy Live's dbbacksmf, dbbacksm2, SQLhousekeeping into Clone's /bin; grep --color=never -ln -E -e '-u ?root -p' /bin/* for other scripts with the password on the command line; decide on rotation (treat as exposed if /bin is mirrored off-box); dbbacksm2 needs 1.0.4 (banner says 1.0.2).
2. Old json_push_media_Clone_to_Live still in Clone /bin; remove after grep -rn 'json_push_media' /bin /root /etc shows no callers.
3. Version bumps owed: json_push_visual_Clone_to_Live (1.0.4), json_push_audio_Clone_to_Live @edits entry.
4. Optional diff-script changes (ssh -q, tee, ls --time-style, hostname guard, wider glob).
5. 23 Live-only scripts: pull to Clone or declare Live-only.
6. Dev-only rsync_cpp_to_live / rsync_perl_to_live (may carry --delete); old mirror1.sh*, MIRRORo.sh on Dev.
7. Dev-newer files worth a diff.
8. AudioProcessor rewrites the store on a no-op run (false drift) — his decision.
9. Replace the two named excludes in mirrorISE_Clone with --exclude 'AudioProcessor*' (recommended).
10. processor_checkpoint.json absent on Clone (expected unless source says otherwise).
11. Collaborative <description> stale; re-zip with his legacy zipper; test 6.0 hooks on Clone's smf21.
12. Dry-run mirror1_Clone with -n once.
13. set -e in MIRROR_* stops on any non-zero step; consider || echo "step rc=$?".
14. Minor cosmetic fixes in mirror1_Dev, mirrorISE_Clone, json_push_audio_Clone_to_Live, mirrorISE_Dev_to_Live.
15. Dev: innodb_log_file_size; optional ImageMagick/evasive/reqtimeout; idle cups/samba/nfs/rpcbind.

4. This chapter: AI tools on Clone (Qwen2.5-0.5B)

4.1 Delivered (all Python 3.8 / torch 1.10 compatible)

Tool	Version	Zip	Purpose
orthogonal_probe.py	1.1.0	orthogonal_probe_v1.1.0.zip	Tests six ways of pushing a "memory" into a model: none, text, logit, steer, attn, keyscale; evidence rules (beats text, fluent, dose-response, decoy, >=10 pairs)
model_inspector.py	1.1.0	model_inspector_v1.1.0.zip	Read-only X-ray: identity, parameter census, attention anatomy, weights + per-tensor sha256, trace (logit lens, per-layer contributions, per-head attention, sink separated), --mode runs any probe push, --ablate, --verify-files, --compare a.json b.json
bundle	—	model__inspector_and_orthogonal_probe_v1.1.0.zip	Andrew's merged copy (folder name has a stray space: model_ inspector_...); contents byte-identical to the two zips
qwen_chat.py	1.0.0 -> 1.0.1	qwen_chat_v1.0.1.zip	Terminal chat with Qwen/Qwen2.5-0.5B-Instruct; 1.0.1 adds a status line every 10 s (--beat N, 0 = off) with phase, tokens/s, memory used/free, swap used, load. Needs pip3.8 install jinja2 (done on Clone). Chat works; first words can take a long time (looks frozen, isn't)
qwen_repeat.py	1.0.0	qwen_repeat_v1.0.0.zip	Repeatability tester (numerical noise by thread count, greedy repeats, sampled repeats with seeds, context-size effect, fingerprints). No jinja2 needed

All unit-tested without torch in the sandbox; "tested on real torch" is missing on every release card except as listed in 4.2.

4.2 What Clone has actually shown

• Inspector --selftest on Clone: PASS (7 checks incl. census 164,928 params = formula).
• Inspector --sections trace --mode steer --strength 0.5 --keywords jigsaw,puzzle on real Qwen2.5-0.5B: all 5 checks pass, zero error (hooks vs hidden states, exact layer decomposition, lens pipeline, steer push 8.8109 = strength x vector length). Took 163 s total, peak 2,843 MB.
• Findings: position 0 (attention sink) carries a vector about 1,640 long vs ~15 elsewhere (116x at layer 6); steer at layer 12 moved to 30.35% -> 52.21%, KL 0.457, 37% of probability moved; play/bike/ride/paddle rose 5-10x; jigsaw rank 1804 -> 726, puzzle 1967 -> 707 as next word; the difference in the running thought is 49% at layer 12 and 38% at the last layer; layers 0-11 identical in both runs. The memory pulls the output toward activities but not yet toward the word "jigsaw".
• Still to receive: orthogonal_probe.py --selftest output; qwen_repeat.py --selftest and the real report; full model_inspector report (sections 1-4) on real Qwen (census should show 494,032,768 unique params; 630,167,424 if the tied table were counted twice).
• Memory: the full inspector run died on memory while other models were loaded. Inspector alone peaks ~2.8 GB; qwen_chat holds ~2 GB. Run one model at a time; check free -m (want ~3.5 GB available); run the inspector in pieces (--sections identity,params,attention, then weights, then trace). If weights still dies alone, ship model_inspector v1.1.1 hashing the 545 MB word table in slices.

4.3 Technical facts learned (keep for the next build)

• transformers 4.37.2 _prepare_4d_causal_attention_mask reads a 4D mask as 1 = keep / 0 = block and inverts it; the probe's additive-mask attn mode needed a temporary converter swap plus a zero-boost equivalence guard.
• HF hidden_states[-1] is post-final-norm, so the steer layer must stay <= n_layers-2.
• Qwen's generation_config carries repetition_penalty 1.1, top_k 20, top_p 0.8, temperature 0.7; qwen_repeat switches these off on purpose, qwen_chat uses them (except temperature/top_p it sets itself).
• Qwen2.5 prompt format is ChatML; a 4.37 prefill computes logits for every position (about 0.6 GB transient per 1,000 context tokens).
• Greedy decoding is deterministic on one thread count; 1 vs 4 threads can change the last digits and flip only near-ties; sampling repeats exactly with a fixed seed.
• Weights are editable (named_parameters(), save_pretrained); personality options discussed: system prompt, steering vector (probe technique), LoRA fine-tune (slow on Clone's CPU). Web access would be a search-then-paste loop (not built).

4.4 The two-LLM "watcher" design — saved, NOT built, ABORTED 2026-10-11 12:20 (hardware)

Andrew's decision after the memory problems: "I don't have the hardware resources I need ... ABORT". The design, in case hardware changes (a few hours on a rented machine was mentioned once; not to be raised again):

• LLM1 = qwen_chat.py (small context). LLM2 = separate watcher process, same Qwen2.5-0.5B-Instruct weights, strictly read-only on qwen_shared_memory.md and the shared chatlog qwen_chat.log (one growing, timestamped log: [YYYY-MM-DD HH:MM:SS] you> ...); its own larger qwen_llm2_memory.md (one item per line importance 1-10 | words or phrase) maintained by Andrew by hand.
• LLM2 loops gently over memory + shared memory + a rolling window of the chatlog (your lines only by default), query "Tell me all you know about me"; scores each token by attention received from the query's last tokens divided by even-spread (sink excluded, middle-to-late layers), chunked (~450 tokens) for RAM; carries a table qwen_llm2_state.json (phrase weight starts at his importance, moves toward measured attention, decays if ignored) because the model itself holds nothing between passes; high-weight phrases recycled most; keywords file qwen_llm2_keywords.txt re-read each loop; repeat-count shown beside attention (repeated text lights up regardless of importance); LLM2 generates a reply only when a new LLM1 query appears in the log; side-by-side watch screen (LLM1 reply vs LLM2 reply, context tokens, seconds); control loop with equal weights first.
• Objections already raised: loops of unchanged input are deterministic (hence the chatlog as the varying input); attention is a proxy for value; chunking loses cross-chunk attention; two models ~4 GB on 6 GB RAM.
• Andrew's stated stage 1: see how LLM2 assigns value over a larger static pool, with LLM1 as the control; first measure how reliable and repeatable the replies are (qwen_repeat exists for that).

4.5 Pending / not built

qwen_chat v1.1.0 (growing qwen_chat.log, qwen_shared_memory.md with /remember /forget /memory, keyword recall from the log); LLM2 watcher; /web search; /persona steering; model_inspector v1.1.1 (memory-lean hashing, only if still needed); README for qwen_chat.

5. Andrew's stated next steps (his words)

"hold that thought" on all further coding; he is deciding what he can do with the hardware he has. Earlier: run the self-tests on Clone and send the outputs, "see what I have now".

6. Technical notes (carried over)

• SMF 2.1 hooks <hook hook=… function=… />; reverse="true" on uninstall; XML comments can't contain --. PclZip rejects streaming data-descriptor zips → legacy zipper.
• rsync: excludes protected from --delete (unless --delete-excluded); anchor with leading /; 3.0.6 no -H with --dry-run. CentOS 6 sh is POSIX bash (exec > >(tee …) fails); init start doesn't restart a running service.
• Running bin_compare from Clone as root: mkdir -p /root/m2 && cd /root/m2 && /root/bin_manifest; on Live and Dev the same via ssh -q; scp -q the manifests back; python3.8 /root/bin_compare bin_manifest_forum_bin_*.txt bin_manifest_clone_bin_*.txt bin_manifest_dev_bin_*.txt --out compare_bin.md. Report times are UTC.
• Live-vs-Clone diff of one script: ssh -q root@forum.drugs-and-users.org cat /bin/$f | diff - /bin/$f (< Live, > Clone).