📝 DMS-NX__ISE_v9.6__Chapter3_Checkpoint_2026-09-25.mdTheme:

DMS/NX: Chapter 3 Session Checkpoint

Session: 2026-09-25 (evening) | Andrew.human & Claude.ai Continued from: DMS-NX__ISE_v9.6__Checkpoint_2026-09-25.md (Chapter 2 close-out)

What this session covered

Audio search design settled. CLAP runs on Clone and produces tags. Live never loads CLAP.
MediaProcessor.py ISE v9.21. Audio items no longer stall the run.
Dev (32-bit) Python sync finished. Uses dev_py_sync_v1.0.2.sh.
Clone (64-bit) Python rebuilt to match Live. Tarballs replaced rsync.
CLAP model on Clone. Downloaded from Hugging Face, blob-verified against Live.
CLAP smoke test on Clone. Timings below.
Live tidy-up. Dead torch 1.4.0, clip 1.0 and an orphaned caffe2 removed.

Standing policies (carried forward, unchanged)

No manual edits to SMF code or files, ever. Fixes go only through the Package Manager/install-script system.
Deployment and hotfix timing on production systems is Andrew's call. Claude doesn't direct it.
No sycophancy. Claims are checked against real source before being agreed with, including Andrew's own.
Never reuse an old name for an updated transcript or checkpoint. Each update gets a new name, and earlier files are never overwritten.

1. Audio search design (decided)

Decision	Detail
Where CLAP runs	Clone only. It peaked at 1.5–1.6 GB, which Live's 2 GB box can't afford for every search
How Live searches audio	Tags, not live CLAP. On Clone, CLAP scores each track or window against a fixed, editable vocabulary (genre, mood, instrument, vocals, tempo, sound type). The top tags are stored as caption text. Live's MediaSearch keyword gate matches against them, so no model loads on Live
Trade-off (accepted)	No free-text semantic queries for audio ("sounds like a rainy night"). This matches the v1.2.0 keyword gate already applied to images
Regular vs long	≤ 15 min = regular: 6 evenly spaced 10 s samples, averaged into one whole-track vector plus tags. > 15 min = long mix: one 10 s window every 30 s, each stored separately, plus a whole-track average
Store keys	Windows use `<id_attach>_a<start_sec>`. Never `_t`, which is the video-frame convention that MediaSearch's `is_video` check depends on
Stored vectors	The 512-dim CLAP vectors are kept, so changing the vocabulary only needs a cheap re-tag, not a new audio pass
Files	`ISE_Data/audio_store.json` and `ISE_Data/audio_checkpoint.json`. Written on Clone and pushed to Live
Model cache	AudioProcessor must pass `cache_dir=ISE_Data/hf_cache` explicitly. Otherwise it looks under `hub/`, misses the model, and re-downloads 744 MB

2. MediaProcessor.py ISE v9.21 (delivered)

Bug: MediaIndexer catalogs mp3/wav into media_store.json. MediaProcessor produced no embedding for them but still counted each one as processed. The ~49 audio items used up --chunk-size slots on every run, and the checkpoint's remaining could never reach 0.

Fix:

Non-image/video items are set aside before any other work.
Audio gets one summary line per run. Any other unknown extension gets an [!] line listing its IDs.
They're excluded from the checkpoint total.
--attachid on an audio ID exits with a clear message.

Mock-tested; not yet run on Live.

Known side effect: processor_checkpoint.json is only rewritten when a new item is processed, so an old on-disk copy keeps its stale remaining until then.

Not in v9.21: the silent 60 s get_video_duration() fallback when ffprobe is missing (task 8). Planned for v9.22.

3. The 925 unexplained IDs (index-time half identified)

MediaIndexer.py's query has no id_msg = 0 filter (avatars, 351) and no attachment_type = 3 filter (thumbnails, 349). So 700 of the 925 are embedded and captioned for nothing. MediaSearch v1.4.0 only hides avatars when searching.
Fix: exclude both in MediaIndexer, plus a --purge-excluded option to remove existing entries from the stores.
webp and 3gp are already handled correctly in both Media scripts. Those two fixes belong in CPR only.
Not yet implemented.

4. Dev (old 32-bit, i686): complete

dev_py_sync v1.0.2 (delivered)

HOLD list: PyMuPDF is Andrew's own 32-bit build and must never be replaced. pip reports it as 1.11.1; fitz.VersionBind reports 1.11.2. The mismatch is harmless.
--only-binary=:all: replaces --prefer-binary (v1.0.1 used --prefer-binary, which fell back to source builds).
Dev's Python 3.8 has no _ssl module. CentOS 6's OpenSSL 1.0.1e is older than the 1.0.2 that 3.8 needs. v1.0.2 therefore downloads 3.8 packages with 3.6's pip, targeting the real 3.8.18/i686, and installs them from local files.
Packages already at the target version are skipped.

Final run: 3 installed, 54 already current, 1 held, 4 expected failures (3.6 cryptography; 3.8 numpy, PyYAML, brotli). None of the four is needed on Dev.

Other changes on Dev:

mysql.connector on 3.6 replaced: 8.0.23 → Live's 2.2.9.
yum python-imaging installed OK.
/root/random.py renamed to .bak. It was shadowing Python's random module, which made numpy, requests and mysql.connector falsely fail in the inventory.
HTTPS from inside 3.8 on Dev still doesn't work. Accepted for a machine being retired.

To do on Dev:

Delete ISE_Data/hf_cache (1.3 GB; Dev can never use it) and caption_tmp.
Add --exclude='ISE_Data/hf_cache/', frame_cache/ and caption_tmp/ to Dev's mirror.
Dev does not need the audio_store.json exclude, because Dev never writes that file.

5. Clone (64-bit Dev): Python now matches Live

Why rsync was abandoned: Andrew's home link to Live has about 400 ms latency and about 10% loss. A single rsync stream of mostly-changed files crawled for more than an hour, and part of that was copying the dead torch just deleted on Live.

Method used instead: tar on Live, then pull the single file with rsync --partial, md5 both ends, move the old folder aside, unpack, import-test, and only then delete the old folder.

Part	How	Result
`/usr/local/lib/python3.6`	Tarball (240 MB → 50 MB)	✓
`/root/.local/lib/python3.6`	Tarball (15.6 MB after removing `caffe2`)	✓ Live's 3.6 numpy lives here (a `pip --user` install); the first attempt failed until this was included
`/usr/local/lib/python3.8`	Not copied. A stale `__pycache__` ("marshal data too short") was breaking torch; deleting the caches fixed it	✓
3.8 missing packages	pip install of 12 packages, pinned to Live's versions. `pycryptodomex` needed `CFLAGS="-std=gnu99"`, because gcc 4.4 rejects C99 loops and the manylinux2014 wheels need glibc 2.17	✓
`/opt/python-3.12.11`, `/opt/yt-dlp-venv`, `/root/venv-audio-test`	One tarball (unpacked over existing folders)	✓ 3.12 has OpenSSL 1.1.1w; yt-dlp 2026.08.19
`/usr/local/bin/python3`	`ln -sfn python3.6`	✓ now matches Live

Found along the way:

Clone's 3.8 pymysql and mysql.connector were bare leftover folders (namespace packages, __file__ = None). They "imported" but didn't work until reinstalled.
/root/bin/python → 3.8 exists only on Clone, and on Clone /root/bin comes before /usr/bin. Any #!/usr/bin/env python script gets 3.8 on Clone but 2.6 on Live. Andrew's call.
Clone has /usr/local/bin/ffprobe. That's the source copy for Live (task 8).
tmux isn't installed on Clone, so use nohup … > log 2>&1 &.

Live-side tidy (done):

python3.6 -m pip uninstall -y clip torch (torch 1.4.0 couldn't import: needs GLIBC_2.14).
Orphaned /root/.local/.../caffe2 (105 MB) removed.
Live's x alias had a stray l (*.html'l); Andrew fixed it.

6. CLAP on Clone

Model: laion/larger_clap_music @ a0b4534a14f58e20944452dff00a22a06ce629d1. Downloaded straight from Hugging Face to ISE_Data/hf_cache in 14 min (~926 kB/s, about 8× the Live link).

Verification: all 8 of Live's blobs match exactly, including pytorch_model.bin at 776,444,665 bytes. Clone has two extra small files (README.md, .gitattributes) because snapshot_download fetches the whole repository.

Smoke test on Clone (/var/www/html/The_ISE_Project/clap_smoketest.py):

	Live	Clone
Load	not recorded	29.4 s
One 10 s window	23.4 s	19.6 s
Peak RSS	1484 MB	1627 MB
Text embed (2 labels)	not recorded	0.69 s

Why Clone wasn't much faster: clap_smoketest.py hard-codes torch.set_num_threads(2), and Clone has nproc = 4. AudioProcessor will default to nproc threads.

venv-audio-test: it borrows torch, numpy and huggingface_hub from the system 3.8, and adds only transformers 4.27.4, tokenizers 0.13.0 and the torch stub. It can't be moved or renamed (absolute paths are baked in), and it breaks if the system 3.8 changes. Call /root/venv-audio-test/bin/python directly; no activate or deactivate needed.

Mirror excludes (reference)

Path	Clone	Dev	Why
`ISE_Data/hf_cache/`	exclude	exclude	1.3 GB; Clone downloads its own CLAP, Dev can't use it
`ISE_Data/audio_store.json*`, `ISE_Data/audio_checkpoint.json`	exclude	keep	Clone writes these; a Live→Clone pull would delete or overwrite work in progress
`ISE_Data/frame_cache/`, `ISE_Data/caption_tmp/`	exclude	exclude	MediaProcessor scratch on Live
`.cache/pip/`	exclude	exclude	pip download cache

Keep frame_cache/ on Live. MediaProcessor needs it to retry failed video-frame captions.

Pending tasks (in order)

Clone VM restore point. Full copy of the VM files at this known-good state. Then delete any stale VMware snapshot from before the Python sync.
AudioProcessor.py v1.0 + editable tag vocabulary. Claude is writing these now. Inputs received: core_utils.py v1.2.0, clap_smoketest.py v0.1.0, nproc=4.
Plan mode on Clone: window count and estimated run time for the 49 tracks.
Full AudioProcessor run on Clone under nohup, then push audio_store.json to Live.
MediaSearch audio support: read audio_store.json tags, 🎵 cards, collapse long-mix windows per track.
MediaIndexer: exclude avatars and thumbnails at index time, plus --purge-excluded (700 of the 925).
CPR: shared extension map in core_utils, and add webp/3gp there.
ffprobe on Live. Copy from Clone /usr/local/bin/ffprobe, then MediaProcessor v9.22 to report ffprobe failure unconditionally.
Refresh Live's baseline: cd / && bash /root/py_inventory.sh, not redirected, because a redirect replaces the real file with the "Wrote …" line. Then a final Clone diff.
Optional: v1.0.2 of clone_py_sync (tarballs for first copy, rsync for top-ups, /root/.local included, md5 checks) and add cd / to py_inventory.sh.
Dev: delete hf_cache and caption_tmp; add mirror excludes.
Conversation transcripts (.md + .pdf) for Chapters 2 and 3. Not yet written.

Scripts / files delivered this session

File	Status
`MediaProcessor.py` (ISE v9.21)	Delivered; mock-tested
`dev_py_sync_v1.0.2.sh`	Delivered; run on Dev and confirmed
`DMS-NX__ISE_v9.6__Chapter3_Checkpoint_2026-09-25.md`	This file

Key paths

Live 128.199.200.202 (hostname forum)          Clone (64-bit, NUC)          Dev (i686, retiring)

/var/www/html/ISE_Data/
    hf_cache/models--laion--larger_clap_music   (744 MB; Live + Clone)
    hf_cache/hub/                               (CLIP; Live)
    media_store.json / embeddings_store.json / captions_store.json
    audio_store.json + audio_checkpoint.json    (TO BE BUILT ON CLONE)
/var/www/html/The_ISE_Project/clap_smoketest.py
/var/www/html/Collabware/core_utils.py          (v1.2.0)
/root/venv-audio-test/                          (Python 3.8 venv, CLAP stack)
/root/.local/lib/python3.6/site-packages/       (Live 3.6 numpy, regex, ftfy, ...)
Attachment dirs: /var/www/html/attachments (id_folder 1), /var/www/html/attachments2 (id_folder 2)

DMS/NX Chapter 3 session checkpoint, 2026-09-25