DMS/NX: Chapter 3 Deep Checkpoint

Session: 2026-09-25 17:00 → 2026-09-26 06:15 AEST | Andrew.human & Claude.ai Supersedes: DMS-NX__ISE_v9.6__Chapter3_Checkpoint_2026-09-25.md (mid-session; kept, not overwritten) Continued from: DMS-NX__ISE_v9.6__Checkpoint_2026-09-25.md (Chapter 2 close-out) Companion: DMS-NX_Chapter3_Changelog_2026-09-26.md (forum-ready summary)

This is a full hand-over. A new session should be able to resume from this file alone.

---

0. Standing policies (unchanged, apply to everything)

• No manual edits to SMF code or files, ever. Fixes go only through the Package Manager/install-script system. ISE Python scripts and ISE config files (Settings_ISE.json) are not SMF code.
• Deployment and hotfix timing on Live is Andrew's call. Claude presents fixes and options.
• No sycophancy. Claims are checked against real output before being agreed with, including Andrew's own.
• Never reuse a name for an updated transcript or checkpoint. Append _2, _3 and so on, and never overwrite.
• Every module gets a doc-block with @version/@date, a visible runtime version, a changelog in the doc-block, and a standalone Markdown manual.
• "Display error" rule: every failure prints unconditionally, and every run ends with a summary. Silence means clean.
• Every fix is verified by a real run (mock or live), not just reasoned about.

---

1. Machines: current state

	Live (forum, 128.199.200.202)	Clone (clone, 192.168.233.140)	Dev (dev, 192.168.233.129)
Arch / glibc	x86_64 / 2.12	x86_64 / 2.12	i686 / 2.12 (being retired)
RAM	2 GB	6 GB, 4 vCPU	n/a
Role	Production	64-bit Dev; runs heavy AI jobs	Old Dev, keyword-only ISE
Python 3.6	numpy lives in /root/.local (pip --user)	= Live (tarballs)	Synced; PyMuPDF HELD (own build)
Python 3.8	torch 1.10 source build in site-packages	= Live (see §4)	No _ssl; no torch; numpy 1.21.6
python3 →	3.6	3.6 (fixed tonight)	3.6
Settings_ISE.json	Live/main	Clone/main (fixed tonight)	Dev/main (fixed tonight)
ffprobe	MISSING	/usr/local/bin/ffprobe	/usr/bin/ffprobe (old 2.6.8)
tmux	yes	no (use nohup)	n/a
umask (root)	normal	077 (open question)	n/a

Link to Live from home: about 400 ms latency and about 10% packet loss (carrier NAT, unfixable). Single-stream transfers of new data crawl at about 90–130 kB/s. Hugging Face from Clone manages about 0.9–1.3 MB/s.

---

2. Code: versions delivered this session

File	Version	Where it runs	State
MediaSearch.py	1.5.2	Live / Clone / Dev (web bridge)	Confirmed working on Dev and Clone. Live not yet updated (Andrew's call)
MediaProcessor.py	ISE v9.21	Live	Delivered, mock-tested, not yet deployed
AudioProcessor.py	1.0.3	Clone only (host guard)	2-track test OK on v1.0.2. Full run about to start with v1.0.3
audio_vocab.txt	1.0.0	Clone	104 labels, 6 groups
AudioProcessor_manual.md	1.0.3	n/a	Standalone manual
clap_diag.py	1.1.0	Clone	Diagnostic, writes nothing
dev_py_sync.sh	1.0.2	Dev	Run, confirmed

Zips: AudioProcessor_v1.0.0/1/2/3.zip (v1.0.2 onward also contains clap_diag.py).

MediaSearch.py 1.4.0 → 1.5.2

• 1.5.0
  • The CLIP imports are optional: if they fail (or torch is half-installed, which raises something other than ImportError), search runs keyword-only.
  • The CLIP load is wrapped, so a failure falls back instead of erroring.
  • A bare /sort … never loads CLIP.
  • get_ranked_results() now returns a 3-tuple (results, order_mode, clip_note).
  • Batched DB lookup: get_topics_for_attachments() does one query per 1,000 IDs.
  • ise_trace timings: stores loaded, CLIP phase, batched DB lookup.
  • In keyword-only mode, a video's best frame is chosen by caption keyword hits.
• 1.5.1: keyword search is the default. --clip opts in. --keyword-only is kept as a no-op alias. Results page: "ranked by keyword hits".
• 1.5.2: DB_CONFIG["ssl_disabled"]=True only when Python lacks the ssl module (Dev 3.8), because mysql-connector 9 forces SSL otherwise.
• Why: Andrew's point that "this is NOT search". Since v1.2.0 the keyword gate decides the results; CLIP only re-ordered them, at about 1.2 GB per search.

MediaProcessor.py ISE v9.21

• Items that aren't IMAGE_EXTS/VIDEO_EXTS are set aside before counting. Audio (AUDIO_EXTS) gets one summary line; other unknown extensions get an [!] line.
• They're excluded from the checkpoint total, so remaining can reach 0.
• --attachid on audio exits with a clear message.
• Known: processor_checkpoint.json only refreshes when a new item is processed.
• Not included: the silent 60 s get_video_duration() fallback. That's for v9.22.

AudioProcessor.py 1.0.0 → 1.0.3

• Design.
  • Input is the audio items in media_store.json, located with Collabware.core_utils.locate_attachment_file().
  • Duration comes from ffprobe, with no fallback: a failure is an error.
  • A regular track (≤ --long-threshold 900 s) gets --samples 6 × 10 s windows, averaged into one item vector.
  • A long mix gets a window every --stride 30 s, each stored as windows["<id>_a<start>"], plus an item-level average. Keys never contain _t, which is the video-frame convention MediaSearch relies on.
  • Tags come from audio_vocab.txt: per group, a softmax over logit_scale_a.exp() × cosine. The top label is always kept, plus up to --top-k 2 total with probability ≥ 0.5 × the top. They're joined into a caption for the keyword gate.
  • Vectors are stored, so a vocabulary change is detected by sha1 and re-tagged automatically with no audio.
  • Settings (model, revision, window_s, threshold, stride, samples) live in _meta. A mismatch is refused unless --force, which rebuilds.
  • The store is written atomically after every item. ISE_Data/audio_checkpoint.json holds progress. --plan, --chunk-size, --attachid, --threads, --batch 4, --log, --monitor, --verbose.
• 1.0.1: ALLOWED_HOSTS={"clone"}; --any-host overrides. Added because the first real run was launched on Live; only the missing ffprobe stopped it.
• 1.0.2: model → laion/clap-htsat-unfused. TOKENIZERS_PARALLELISM=false is set before transformers is imported.
• 1.0.3: MODEL_REVISION="8fa0f1c6d0433df6e97c127f64b2a1d6c0dcda8a". A store from v1.0.2 (revision None, same model) is accepted, and its settings are upgraded in place.

---

3. Audio: the CLAP investigation (important history)

Test	larger_clap_music @ a0b4534a	clap-htsat-unfused @ 8fa0f1c6
Weights loaded	0 missing / 0 unexpected	0 / 0
logit_scale_a.exp()	1.03 (suspicious)	18.66
cos(mix 106, mix 108)	0.981	0.639
cos(sine, noise)	0.948	0.346
Best prompt, sine	flat ±0.01	"a sine wave tone" 0.757
Best prompt, mixes	flat ±0.005	"techno music" 0.388 / 0.501
Speed on Clone	19.4 s/window	8.3 s/window
Peak RSS	1628 MB	1320 MB

• Conclusion: larger_clap_music is silently broken under transformers 4.27.4, while clap-htsat-unfused works. The torch build is sound: all 6 op checks against numpy pass.
• Torch speed limit: USE_EIGEN_FOR_BLAS=ON, USE_MKL=OFF. OpenMP is present, but matrix maths is effectively single-core (htop shows about 106%). Accepted, because indexing is rare. A rebuild against OpenBLAS or multi-process workers are possible later.
• Real test (v1.0.2, 2 tracks):
  • ID 106 "The Sensuous Black Woman Part 1 vs." → hip hop, new wave, dark, happy, fast, spoken, bass, mu…
  • ID 108 "Breath vs. Track IV" → progressive house, dark, fast, instrumental, female, b…, musi…
  • Sensible (spoken on 106 is right). The quirk is pairs like "dark, happy", which --top-k 1 would remove; deferred until the full results are seen.
• Plan on Clone: 57 audio items (52 regular, 5 long), 971 windows. At 8.3 s/window, about 2 h 15 m for the remaining 959. The earlier "49" came from the post-attached manifest; media_store.json has 57.
• Dead weight: models--laion--larger_clap_music (744 MB) in hf_cache on Live and Clone is now unused.

---

4. Clone: what was done tonight (and gotchas found)

1. rsync abandoned for the first copy. The clone_py_sync 3.6 step spent more than an hour, much of it on libtorch.so from the dead torch 1.4 that had just been removed from Live. The tarball method is: tar on Live → rsync --partial the single file → md5 on both ends → move the old folder aside → unpack → import test → only then delete the old folder.
2. 3.6: py36.tgz (240 MB → 50 MB). The first attempt failed because Live's numpy is in /root/.local/lib/python3.6; it needed py36_userlocal.tgz (15.6 MB, after removing an orphaned caffe2 on Live).
3. 3.8: not copied. marshal data too short was a stale __pycache__, fixed by deleting the caches. Then 12 missing packages were installed, pinned to Live's versions. pycryptodomex needed CFLAGS="-std=gnu99", because its wheels need glibc 2.17 and gcc 4.4 rejects C99 loops. pymysql/mysql.connector had been bare leftover folders (namespace packages, __file__ = None).
4. 3.12 / yt-dlp venv / venv-audio-test: one tarball, py_rest.tgz, unpacked over folders that already existed. 3.12 has OpenSSL 1.1.1w.
5. python3 → 3.6 (ln -sfn).
6. Web search failure: three layers.
   • (a) torch was loaded through pytorch_custom.pth → /root/pytorch (/root is dr-xr-x---, so Apache can't read it), leaving Apache with an empty site-packages/torch folder. Fix: cp -a /root/pytorch/torch into site-packages.
   • (b) 10 symlinks in torch/lib pointed into /root/pytorch/build/lib. Replaced with real copies.
   • (c) umask 077: tonight's pip installs were root-only. Fixed with chmod -R go+rX site-packages.
   • /root/pytorch and pytorch_custom.pth are left in place (Live has the same .pth; it's harmless).
7. Four simultaneous CLIP searches (about 1.25 GB each) nearly exhausted Clone's memory. This prompted the keyword-default change.
8. Settings_ISE.json said Live, so every ISE link and thumbnail pointed to Live. Changed to {"environment": "Clone", "install": "main"}. The _SITE_ROOTS table in ise_settings.py already had the Clone rows. smf20//smf21/Settings_ISE.json don't exist on Clone.
9. CLAP models downloaded straight from Hugging Face on Clone: larger_clap_music in 14 min, blob-verified identical to Live; unfused 615 MB in 8 min.
10. venv-audio-test uses include-system-site-packages: it borrows torch, numpy and huggingface_hub from the system 3.8 and adds only transformers 4.27.4, tokenizers 0.13.0 and a torch stub. It can't be moved or renamed. Call /root/venv-audio-test/bin/python directly; no activate needed. py = system 3.8, which has no transformers.

5. Dev: what was done

• dev_py_sync v1.0.2 final run: 3 installed, 54 current, 1 held, 4 expected failures (3.6 cryptography 37.0.4; 3.8 numpy 1.24.4, PyYAML 6.0.3, brotli 1.2.0).
• mysql.connector on 3.6: 8.0.23 → 2.2.9. yum python-imaging installed OK.
• numpy 1.21.6 was installed on 3.8 later, through the 3.6-download route.
• PyMuPDF: pip reports 1.11.1 and fitz.VersionBind reports 1.11.2. It's Andrew's build and is never replaced.
• /root/random.py renamed to .bak; it caused false randint import failures in the inventory.
• Settings_ISE.json said Live, although it was supposed to be excluded from the mirror since v8.1. Changed to Dev. Something overwrote it, so the mirror exclude needs checking.
• Media search works on Dev (keyword-only, v1.5.2).

6. Live: what changed

• python3.6 -m pip uninstall -y clip torch (torch 1.4.0 couldn't import: GLIBC_2.14).
• Removed /root/.local/lib/python3.6/site-packages/caffe2 (105 MB, orphaned).
• x alias typo (*.html'l) fixed by Andrew.
• AudioProcessor v1.0.0 was installed and run on Live by mistake. It stopped only because ffprobe is missing. Delete it from Live: AudioProcessor.py, audio_vocab.txt and AudioProcessor_manual.md in The_ISE_Project.
• Transcription test on Live (by Andrew, not Claude): a Qwen-style ASR model through llama-mtmd-cli (language English<asr_text>…). Clean output: "They want to go back, back to the happy days of house. They say, 'I want to go…'". It then failed with failed to find a memory slot, meaning the context was full: the 5-minute track became 41 audio chunks. Fix: 30 s pieces (or -c 16384, but watch Live's 2 GB). Claude still needs the exact command (model and mmproj names) to write the loop.

---

7. Mirror excludes (must be in place)

Path	Clone mirror	Dev mirror	Why
Settings_ISE.json (+ smf2*/)	exclude	exclude	Per-install environment. Overwritten on both machines already
ISE_Data/hf_cache/	exclude	exclude	1.3 GB+; Clone downloads its own
ISE_Data/audio_store.json*, ISE_Data/audio_checkpoint.json	exclude	keep	Clone is the only writer
ISE_Data/frame_cache/, ISE_Data/caption_tmp/	exclude	exclude	MediaProcessor scratch (keep frame_cache on Live)
.cache/pip/	exclude	exclude	pip download cache

To delete on Dev: ISE_Data/hf_cache (never usable there) and caption_tmp.

---

8. Operating procedure (manual, after new uploads)

Live (images and video):

cd / && /usr/local/bin/python3.8 /var/www/html/The_ISE_Project/MediaIndexer.py
cd / && nohup /usr/local/bin/python3.8 /var/www/html/The_ISE_Project/MediaProcessor.py --log > /dev/null 2>&1 &
tail -f /var/www/html/ISE_Data/mediaprocessor.log

Clone (audio, only if audio was uploaded; run the mirror first):

cd / && /root/venv-audio-test/bin/python /var/www/html/The_ISE_Project/AudioProcessor.py --plan
cd / && nohup /root/venv-audio-test/bin/python /var/www/html/The_ISE_Project/AudioProcessor.py --log --monitor > /dev/null 2>&1 &
tail -f /var/www/html/ISE_Data/audioprocessor.log
rsync --partial -a /var/www/html/ISE_Data/audio_store.json root@128.199.200.202:/var/www/html/ISE_Data/

On Clone, after any pip install as root: chmod -R go+rX /usr/local/lib/python3.8/site-packages, until the umask is decided. CentOS 6 pgrep has no -a; use pgrep -fl. py_inventory.sh must run from / and must not be redirected; a redirect replaces the file with the "Wrote …" line, and that happened twice.

---

9. Experiments (in Claude's workspace, not on Andrew's machines)

• Claude's workspace can't reach Hugging Face, dl.fbaipublicfiles.com (Demucs) or openaipublic.azureedge.net (Whisper). GitHub releases and PyPI work.
• Spleeter 2stems (weights from GitHub) separated "Back In Time" (BackInTime.mp3, 5:12, 320 kbps). The whole track caused an OOM in 7 GB, so it was done in 45 s chunks. Sent BackInTime_vocals.mp3.
• DeepSpeech 0.9.3 (weights from GitHub) was poor on both the mix and the stem ("moreover sagacious", "anaesthesia banker"). The opening was recognisable. Finding: separation didn't improve the transcript; the recogniser is the bottleneck. So the Clone design should be a good ASR (the Qwen-style mtmd model or whisper.cpp) on 30 s pieces, and separation is optional.

---

10. Pending tasks (in order)

1. AudioProcessor full run on Clone, v1.0.3, nohup … --log --monitor (about 2 h 15 m). Then rsync audio_store.json to Live. The log didn't exist at 05:53, so the run hadn't started; check with pgrep -fl AudioProcessor.
2. Review the full tag results. Decide on --top-k 1 (it's a vocabulary/tagging change, so only a re-tag is needed, no audio) and whether the vocabulary needs editing.
3. MediaSearch audio support: read audio_store.json; keyword gate on caption; 🎵 in TYPE_EMOJI; collapse long-mix windows into one card with "best match at mm:ss"; mode=audio viewer URL (VIEWER_AUDIO_EXTS already exists).
4. Speech transcription for items tagged spoken/speech: 30 s pieces through the llama-mtmd ASR model (or whisper.cpp), with text and timestamps stored in audio_store.json. Needs Andrew's exact mtmd command.
5. MediaSearch speed: v1.5.3 stops loading embeddings_store.json in keyword mode (1.2–3.8 s on Dev). Later, a prebuilt media_search_index.json written at index time (caption, filename, subject, topic, board, date, avatar flag), so searches need no DB. First get a full trace from Live or Clone.
6. Bridge: a one-search-at-a-time lock (protects Live's 2 GB). Deploy MediaSearch 1.5.2 on Live (Andrew's call).
7. MediaIndexer: exclude id_msg=0 (avatars, 351) and attachment_type=3 (thumbnails, 349) at index time, plus --purge-excluded. That's 700 of the 925. Then CPR: a shared extension map in core_utils, plus webp/3gp.
8. ffprobe on Live (copy from Clone /usr/local/bin/ffprobe) and MediaProcessor v9.22 to report an ffprobe failure instead of silently guessing 60 s.
9. Mirror scripts (Dev and Clone): add the §7 excludes, especially Settings_ISE.json.
10. Clone umask 077: match Live (grep -n umask /root/.bashrc /root/.bash_profile /etc/profile /etc/bashrc). Andrew's call.
11. Clean-ups:
    • delete larger_clap_music (Live + Clone, 744 MB each)
    • delete Dev's hf_cache/caption_tmp
    • delete AudioProcessor from Live
    • /root/torch_sitepkg_leftover.bak on Clone can go
    • the Clone python → /root/bin/python (3.8) link is Andrew's call; /root/bin comes before /usr/bin on Clone
12. Clone VM restore point at this known-good state (full copy of the VM files; delete any stale VMware snapshot).
13. Refresh Live's inventory baseline (from /, not redirected) and do a final Clone diff.
14. Optional: clone_py_sync v1.0.2 (tarballs for the first copy, rsync for top-ups, /root/.local included, md5 checks); py_inventory.sh with cd / built in.
15. Conversation transcripts (.md + .pdf) for Chapters 2 and 3, following the PDF style rules. Not yet written.

---

11. Key paths

/var/www/html/The_ISE_Project/
    MediaSearch.py (1.5.2)  MediaProcessor.py (v9.21 pending)  MediaIndexer.py (v9.7)
    AudioProcessor.py (1.0.3, Clone only)  audio_vocab.txt  AudioProcessor_manual.md  clap_diag.py (1.1.0)
    clap_smoketest.py (0.1.0, hard-codes 2 threads)  qf_Mediasearch_bridge.php  ise_settings.py (has Clone rows)
/var/www/html/Collabware/core_utils.py (1.2.0)        /var/www/html/Settings_ISE.json (per install)
/var/www/html/ISE_Data/
    media_store.json  embeddings_store.json (2757 keys)  captions_store.json
    audio_store.json + audio_checkpoint.json   (Clone writes; pushed to Live)
    hf_cache/models--laion--clap-htsat-unfused/snapshots/8fa0f1c6...   (Clone; IN USE)
    hf_cache/models--laion--larger_clap_music/snapshots/a0b4534a...    (Live + Clone; UNUSED)
    hf_cache/hub/   (CLIP, Live)      frame_cache/ (keep on Live)      caption_tmp/
    mediaprocessor.log  audioprocessor.log
/root/venv-audio-test/   (3.8 venv, system site-packages)     /root/pytorch/ (Clone torch source tree)
/usr/local/lib/python3.8/site-packages/torch/   (Clone: real copy as of tonight)
Attachments: /var/www/html/attachments (id_folder 1), /var/www/html/attachments2 (id_folder 2)

---

DMS/NX Chapter 3 deep checkpoint, 2026-09-26 06:20 AEST