Checkpoint 1: Clone AI readiness and Clone/Live comparison

Taken: 09/10/2026--06:04 AEDT For merge into: "The Sysplex (Clone, Live, Dev) Orchestrator" (to be driven from Clone) Merges with: "The Sysplex-wide full hardware and software comparison" checkpoint

1. Problem and resolution

Item	State
Clone could not import safetensors/tokenizers/transformers (needs manylinux_2_17, box has glibc 2.12)	RESOLVED. glibc 2.17 side prefix /opt/glibc-2.17, Python run through ld-2.17.so.
glibc 2.17 on Live and Clone	DONE. Bit-identical, checksum b1acd87b1650ac544ac8cb7e7dca043a.
transformers 4.37.2, tokenizers 0.15.2, safetensors 0.5.3	DONE on both. Live usable only through $RUN.
torch 1.10.0a0	Identical libtorch_cpu.so md5 5cce9fc213f9bbf6147a2838421141dd on both. Only the version label differs.
Clone python3.8	Wrapper script; real ELF is python3.8.real; backup python3.8.orig (md5 58b8c5e5a29a1ad3bc32568fe341a7ed).
Live python3.8	NOT wrapped (production). Use $RUN.
orthogonal_probe.py v1.0.0	Runs on Clone, slow (single-thread, no KV cache). Table rows complete slowly. v1.1.0 NOT approved or built.
model_inspector v1.0.0	Delivered, NEVER run on real torch. Run python3.8 model_inspector.py --selftest on Clone first.
box_snapshot / box_compare v1.0.0	Delivered, run on Clone and Live. Not yet run on Dev (32-bit CentOS 6).

2. Clone vs Live comparison (box_compare, 09/10/2026 05:12 AEDT)

Identical: glibc, PHP modules, llama.cpp, cron entries, python3.8 binary, all Python packages except 3.

Expected differences, no action: host/date/TZ/kernel (Clone 2.6.32-754, Live 2.6.32-431), CPU/RAM/swap/disk, processes, file-max/nproc, users claude/snattle on Live, Live mail stack (postfix, ports 25/465/587/2525/10025), fail2ban, haveged, reqtimeout, Clone devtoolset-7/cmake/ninja.

Real drift to decide (Andrew's call, nothing changed):

1. python = 3.8.18 on Clone (/root/bin/python), 2.6.6 on Live. Plain python scripts behave differently.
2. Tuning not mirrored. Live: memory_limit 256M, innodb_buffer_pool 256M, max_connections 60. Clone: 150M, 32M, 20/150.
3. 15 same-named /bin scripts differ in content: SQLhousekeeping, attachme, audio_run, backup, chkindeximages, clrcache, commands.txt, dbbacksm2, dbbacksmf, filecmd, filedircmd, json_push_audio_Clone_to_Live, pdfjslinks, sshLive.sh, sysmd. Look at json_push_audio_Clone_to_Live first.
4. crond not enabled at boot on Clone (crontab contents match).
5. httpd.conf, ssl.conf, mod_evasive.conf, php.ini, my.cnf, sshd_config, sysctl.conf, clock differ by checksum only. Content diff not yet done.
6. /opt on Live has Python-3.8.18 source and openssl-1.1.1w; Clone has ld.so.conf.d/openssl-1.1.1.conf, Live does not. Compare ssl.OPENSSL_VERSION on both.
7. Minor: Live yt-dlp is an older build; Live has idle3.8/pydoc3.8/2to3-3.8, Clone lacks them; patchelf and py-spy only on Clone (pip uninstall -y patchelf is optional cleanup).

Hardware note: Live reports avx2=yes, fma=yes (a E5-2630L v2 has neither), so the droplet is probably newer hardware than stated. Our torch is built with FMA off, so do not assume Live is faster until timed. Live has ~3.9 GB RAM; the model needs ~3.1 GB RSS next to Apache/MySQL.

3. Dev (32-bit) sync: next steps

32-bit ceiling: no modern PyTorch, so no torch/transformers/ISEmedia/probe on Dev. Practical sync is OS minor version, LAMP versions/settings, Python 3.8 from source, pure-Python ISE parts, and custom /bin scripts. Dev is planned for a rebuild to 64-bit CentOS 6.10 on a NUC vDisk (see memory: dev-64bit-rebuild). Dev also has an unresolved throughput collapse towards Live (dev-vm-network-slowdown).

Steps:

1. On Dev: chmod +x box_snapshot && ./box_snapshot, then scp snapshot_*.txt root@<clone-ip>:/root/.
2. On Clone: python3.8 box_compare snapshot_forum_*.txt snapshot_clone_*.txt snapshot_<devname>_*.txt --ignore-arch --out three_way.md.
3. Sort Dev's differences into: sync now / impossible on 32-bit / wait for the 64-bit rebuild.
4. box_snapshot has never run on 32-bit CentOS 6; if a section errors, paste the screen output.

4. Standing rules for the Orchestrator conversation

• Reply header Claude: [DD/MM/YYYY--HH:MM:SS:mmm TZ] from real Sydney time via code.
• Statements over questions. Simple things simple.
• Deployment timing is Andrew's call. No manual edits to SMF files (Package Manager only).
• Never rename his scripts. Never reuse an old name for an updated file (append _2, _3).
• Delivery rules for code: bump @version/@date, version history entry, print version at start, zip with never-reused dotted version, list unrequested changes BEFORE making them, release card.
• Do not retry patchelf on Python (it segfaulted). Avoid rsync -H with --dry-run on rsync 3.0.6.
• On Clone use plain python3.8 (wrapper). $RUN is for Live only.
• Do not spend time installing torch in the sandbox or run long silent tasks; say what is happening first.

5. Pending, in priority order

1. Dev snapshot and three-way comparison (section 3).
2. Run model_inspector.py --selftest on Clone, report result.
3. Decide on the drift items in section 2.
4. Probe v1.1.0 (interactive --repl, batch --file, KV cache, last-position lm_head, skip duplicate runs, --threads). Waiting for Andrew's "go". Changes must be listed before building.
5. Time the probe on Live ($RUN, OMP_NUM_THREADS=2) versus Clone (OMP_NUM_THREADS=4) to settle the speed question.

6. Key paths

• Clone freeze: /root/py38_clone_freeze.txt. Live freeze: /root/py38_forum.drugs-and-users.org_freeze.txt, /root/py38_live_after.txt.
• Snapshots and report: snapshot_<shorthost>_<YYYYMMDD-HHMMSS>.txt, box_compare ... --out report.md.
• Comparison report for this checkpoint: clone_vs_forum (box_compare v1.0.0, 2026-10-09).
• Delivered zips: orthogonal_probe_v1.0.0.zip (his), model_inspector_v1.0.0.zip, box_snapshot_v1.0.0.zip.