📝 DMS-NX__ISE_v9.7__Chapter4_Checkpoint_2026-09-26_3.mdTheme:

DM2S/NX: Chapter 4 Checkpoint

Product: "DM2S/NX - The NextGen Document and Media Management System" (formerly DMS/NX, formerly ISE) Session: 2026-09-26 07:25 → 2026-09-27 05:40 AEST | Andrew.human & Claude Supersedes: DMS-NX__ISE_v9.6__Chapter4_Checkpoint_2026-09-26_2.md (kept, not overwritten). Chain: Chapter3_Checkpoint_2026-09-25_2 → Chapter4 → Chapter4_2 → this.

This is a full hand-over. §1 and §1b are what changed in Chapter 4 (§1b is new since _2). §4 onward carries forward everything from Chapter 3 that still applies.

0. Standing policies (Chapter 4 additions marked "New")

No manual edits to SMF code or files, ever. Fixes go through the Package Manager/install-script system only. ISE Python scripts and Settings_ISE.json are not SMF code.
Deployment and hotfix timing on Live is Andrew's call. Claude presents fixes and options.
No sycophancy. Claims are checked against real output, including Claude's own checkpoints (see §3).
Never reuse a name for an updated transcript or checkpoint. Append _2, _3 and so on.
Every module gets a doc-block (@version/@date/changelog), a visible runtime version, and a standalone Markdown manual.
"Display error" rule: every failure prints, and every run ends with a summary.
Every fix is verified by a real run (mock or live).
New (Chapter 4): command blocks meant for pasting contain only commands that run as written, with no <placeholder>s.
New (Chapter 4): keep simple things simple. Answer a simple question with the simplest command (e.g. one md5sum line per machine). Build a script only when Andrew asks for one. (Andrew: "u really complicate simple shit".)
New (Chapter 4): operational scripts live in /bin on each machine (Andrew drives everything from there). ISE's own code stays in The_ISE_Project. /bin is not mirrored, so copies are updated by hand; the version printed on each run shows drift.

1. Done in Chapter 4

AudioProcessor full run on Clone: COMPLETE and VERIFIED

Two processes ran at once. v1.0.2 was started as PID 14770 at 05:53:53, and v1.0.3 as PID 15101 at 05:57:19. Both processed the same items, which is where the doubled log lines come from. Andrew killed 14770 at about 08:20. After that the rate fell from 10–16 s/window to 7.4–8.9 s/window, which confirms the two were competing for CPU.
Result: 57 items and 668 long-mix windows (128+134+160+122+124). All 951 windows for this run were encoded, with 0 failures. The telemetry average was 11.01 s/window, inflated by the overlap. Peak RSS was 1309 MB and the store is 3.62 MB.
_meta verified after the run: revision 8fa0f1c6d0433df6e97c127f64b2a1d6c0dcda8a and model laion/clap-htsat-unfused. The overlap did no damage.
Pushed to Live and verified: md5 9f6c8cf62f7204051cdd7677e5ef6f43 on both machines, 3,800,480 bytes. audio_checkpoint.json stays on Clone only (§7).
Start-state mismatch, not explained: the log said 4 items done and 951 windows left, but the Chapter 3 checkpoint said 2 done and 959 left.

transcribe_pieces.sh v1.0.0: WORKS ON LIVE

Files: transcribe_pieces_v1.0.0.zip, containing the script and transcribe_pieces_manual.md. It's installed at /root/transcribe_pieces.sh on Live.
What it does: ffmpeg extracts 16 kHz mono PCM, ffmpeg -f segment cuts it into 30 s pieces (no ffprobe needed), and llama-mtmd-cli runs Qwen3-ASR once per piece. It writes /root/transcripts/<name>.transcript.txt ([mm:ss] text, tag stripped) and .raw.txt.
Changes from Andrew's transcribe.sh:
- The prompt is Transcribe this audio., not the 7-word summary. It can be overridden with ASR_PROMPT.
- --temp 0 makes output repeatable.
- Stdin comes from < /dev/null.
- The paths can be overridden with LLAMA_BIN, ASR_MODEL and ASR_MMPROJ.
- KEEP=1 keeps the work folder.
Mock test (Claude's workspace):
- A 95 s clip was cut into 4 pieces.
- An injected memory-slot failure printed [!], and the run carried on.
- A filename with a space worked.
- Pre-flight reported every missing item.
Real test on Live: instagram-1790350777642.mp4 gave 3 pieces, 3 ok. The transcript is accurate ("What is a tensor? A language model works with lists of numbers…"). The text arrives on stdout. The runs took 761 s and 715 s.
Timing (piece 0 log):
- The model loads in about 7 s, so loading is not the bottleneck.
- 30 s of audio becomes 6 mtmd chunks. Each takes 2–4 s to encode plus about 4.5 s of prefill.
- About 190 s of each piece's roughly 238 s goes on text generation, which is far too slow for a 0.6B model. The cause is unknown: swap, a single vCPU, or runaway generation. The check commands are in §2.
Cut sentences: pieces begin mid-sentence ("In each row.", "Itself does not…"). The fix options are overlap (about 2 s) or cutting at silence.

Machines

Production transcription host: Clone (Andrew: "either but Clone is best").
Clone already has everything, at the same paths as Live:
- /root/llama.cpp/build/bin/llama-mtmd-cli
- /root/models/qwen3-asr/Qwen3-ASR-0.6B-Q8_0.gguf (804,749,248 bytes)
- mmproj-Qwen3-ASR-0.6B-Q8_0.gguf (214,392,480 bytes)
- the SmolVLM files
No llama-server on either machine.
Clean-up candidate on Clone: /rsync-live-root/ (a duplicate copy of the models, about 1.5 GB). Andrew's call.
Live's clock reads 2 hours behind Clone and Sydney. Live 06:56 = Clone 08:56. Keep this in mind when comparing logs.

1b. Done since `_2` (2026-09-26 09:35 → 2026-09-27 05:40)

Mirror scripts: DONE

Clone mirrorISE.sh now has the Clone-only excludes, with its structure otherwise unchanged:
- First rsync (The_ISE_Project): AudioProcessor.py, AudioProcessor_manual.md, audio_vocab.txt, clap_diag.py, clap_smoketest.py.
- All three ISE_Data rsyncs: 'audio_store.json*' (also protects the .tmp) and audio_checkpoint.json.
- Mock-tested: protected files survived, normal updates and deletes still worked.
Dev mirror scripts already covered everything Dev needs.
Push + mirror in one script: Andrew runs it on Clone as /bin/json_push_audio_Clone_to_Live (v2.1.0, his rename of Claude's pushaudio_mirror.sh). It pushes audio_store.json to Live by IP, md5-checks both ends, refuses to push while AudioProcessor is running, then calls /bin/mirrorISE.sh.
Real run 2026-09-27 05:17 (Clone clock): push md5 match (9f6c8cf6…, unchanged). The mirror pulled 4 files Live → Clone: ise_trace.log, misc_indexer.log, misc_store.json, word_index_misc.json. Only 13,004 bytes actually crossed the link for 172.6 MB of files (rsync delta); the 30 MB/s figures are local rebuild speed, not network speed. Nothing except audio_store.json is ever written to Live.
Note: the ISE_Data rsync always pulled every non-excluded file from Live, including the misc stores. That was in Andrew's original script and is correct: Clone must hold Live's indexes for the data to agree.
Cosmetic: Live's SSH login banner prints on every connection. -e "ssh -q" on the rsync lines (and ssh -q for the md5 call) would silence it.

Coherence check: ALL 16 `.json` FILES AGREE (2026-09-27 ~05:30)

Method (the simple one, per the new policy), run on Live, Clone and Dev:
```
md5sum /var/www/html/ISE_Data/*.json
```
Result, compared line by line: Live = Dev = Clone for all 16 files. audio_checkpoint.json exists only on Clone, as designed.
Scope agreed: only the .json files at the root of the main ISE_Data. smf20/smf21 are separate "proof of compatibility" installs with their own databases and ISE, so they're out of scope. Logs, traces and subfolders don't matter.
Key values: audio_store.json 9f6c8cf62f7204051cdd7677e5ef6f43, misc_store.json 951dbcdcc0b56e689281125304fe739a, word_index_misc.json fba44685b7f1e6fc100197bd3665d775.
json_md5_push.sh / json_md5_diff.sh v1.2.0 also exist (Clone+Dev push lists to /root/json_md5/ on Live; Live diffs). Andrew is keeping them in /bin, but the one-line md5sum is the primary method. json_md5_check.sh is superseded.

Transcription policy (decided)

Everything gets indexed; only educational material gets transcribed.
Transcription is opt-in, not triggered by the spoken tag. Still undecided: opt-in by educational boards (named once, automatic) or by a hand-picked list of attachment IDs.
ID 106 ("The Sensuous Black Woman Part 1 vs.") is explicitly not to be transcribed. Andrew found its post via the SQL below; it stays an undocumented "gem".

Audio search findings

A MediaSearch query for *.mp3 returns nothing because MediaSearch 1.5.2 doesn't read audio_store.json yet (task 3). Wildcard support in MediaSearch is also unconfirmed.
audio_store.json layout (confirmed on Live): {"_meta": {...}, "items": {"<id_attach>": {"filename", "extension", "duration", "class", "windows", "vector", ...}}}. Keys are SMF attachment IDs; the store does not record the post.

Attachment → post lookup (worked on Live, and v1.6.0 needs the same join):

eval $(php -r 'include "/var/www/html/Settings.php"; echo "DB=$db_name P=$db_prefix";')
mysql -p "$DB" -e "SELECT a.id_attach, a.filename, a.id_msg, m.id_topic, m.subject FROM ${P}attachments a LEFT JOIN ${P}messages m ON m.id_msg = a.id_msg WHERE a.id_attach IN (106, 108);"

Open design question for audio results and ISEmega: when Andrew searches, does he remember the anchor (the post) or the content (what's in the file)? That decides whether results lead with post subject/board or with tags/transcripts.

Other

Search anomalies: Andrew sees quoted 3-word phrases matching on two of the words, or none. That's a real bug (a phrase should match as one unit). Trace it when he brings a query, the source (posts/PDF/Misc/media) and the module. For now he's putting effort into changelog.md and module doc-blocks instead of a full QCE chain. SMF native search remains his backup (finds what he wants ~99% of the time).
SimplePortal "time online" block repeats the top member (Chip, 126d 22h 23m) five times. That's an SP bug. Report it to SP's author, never patch it (requisite-mods policy).

2. Open, in order

Transcription speed diagnosis on Live. The work folder /tmp/transcribe.Aa1kQc is still there:

grep -iE 'eval time|tokens per second|n_threads|total time' /tmp/transcribe.Aa1kQc/piece_0000.log
wc -w /tmp/transcribe.Aa1kQc/piece_0000.out
nproc; free -m

Then run the same file on Clone for a direct comparison (pull it from Clone):

rsync --partial -a root@128.199.200.202:/root/instagram-1790350777642.mp4 /root/
KEEP=1 /root/transcribe_pieces.sh /root/instagram-1790350777642.mp4

transcribe_pieces v1.0.1, after item 1:
- an -n cap if the model runs on after the transcript
- piece overlap or silence cutting
- possibly a llama-server rebuild so the model stays loaded
- then an audio_store.json writer for opt-in educational items only (not every spoken item; see §1b transcription policy)
What are 2959 (1h00m50s) and 6240 (1h01m31s)? Both are tagged male, lecture, interview, slow. If they're talks, they're the first real transcription targets.
AudioProcessor v1.0.4:
- a single-instance lock (the PDFIndexer pattern)
- the version and PID on every log line
- a SIGTERM handler that prints an "interrupted" summary (the killed PID 14770 left none)
- sec_per_window default 19.6 → 8.3
Tag review (Chapter 3 task 2), now that full results exist. All of these can be fixed with a re-tag, no audio needed.
- fast is on almost every item.
- Opposites: slow, fast on 2846 and 2959, and dark, happy is common.
- country on dance tracks.
- music everywhere adds nothing.
- podcast/lecture false positives on music.
- Options: --top-k 1, top-1 only for opposite groups (tempo, mood, vocals), and dropping groups that don't discriminate.
Duplicate audio uploads. These have identical durations and tags:
- 6428/6433/6440 (6m29s)
- 6429/6434/6441 (6m56s)
- 5918/5920 (3m07s)
- 5915/5954 (1m29s)
The fix is the same SHA-256 "master copy" collapse planned for PDFs.
Chapter 3 tasks 3 and 5–15 carry forward unchanged:
- MediaSearch v1.6.0 audio support (needs the current MediaSearch.py uploaded; also check wildcard handling)
- MediaSearch v1.5.3 speed
- the bridge lock, and MediaSearch 1.5.2 onto Live
- MediaIndexer excludes for avatars and thumbnails
- ffprobe onto Live, and MediaProcessor v9.22
- Clone umask
- clean-ups
- the Clone VM restore point
- Live inventory baseline
- clone_py_sync v1.0.2
The Chapter 2/3 transcript item is dropped: Andrew now makes the transcript PDFs himself.

Deploying MediaProcessor v9.21 on Live is still pending too.
Unresolved conflict: the Chapter 3 checkpoint says glibc 2.12 on Live, Clone and Dev. Earlier notes say Live and Clone were upgraded to 2.17. Run ldd --version | head -1 on each machine to settle it.

3. Corrections to the Chapter 3 checkpoint

§6 (transcription test on Live): this was correct. The test was run on Live with /root/transcribe.sh and /root/transcribe3.sh, and /var/log/commands confirms runs on 21, 22 and 24 September. .bash_history had no llama-mtmd-cli lines because the command was inside those scripts. The scripts were edited after their last run: transcribe3.sh on 25 September at 23:48 and transcribe.sh on 26 September at 01:56. The exact command is:
```
/root/llama.cpp/build/bin/llama-mtmd-cli \
 -m /root/models/qwen3-asr/Qwen3-ASR-0.6B-Q8_0.gguf \
 --mmproj /root/models/qwen3-asr/mmproj-Qwen3-ASR-0.6B-Q8_0.gguf \
 --audio <16 kHz mono wav> \
 -p "Transcribe this audio then summarize it in 7 words or less"
```
§10 task 1 "hadn't started at 05:53": v1.0.2 started 2 seconds after that check (05:53:55), and v1.0.3 was then started on top of it, which caused the double run above.

4. Mirror excludes

Done. Dev's scripts were already complete, and Clone's mirrorISE.sh now has the Clone-only excludes (see §1b). Mirror runs on both machines are safe.

Low priority, both machines: in mirror1.sh, unanchored excludes (.htaccess, index.html, manifest.json, cache/) match at every depth, so new or changed files with those names deeper in the tree on Live (e.g. attachments/.htaccess) never reach the mirror. Existing copies are kept. To exclude only the top level, anchor the pattern with a leading slash, e.g. --exclude /.htaccess.

5. Carried forward from Chapter 3 (still current)

Machines

	Live (`forum`, 128.199.200.202)	Clone (`clone`, 192.168.233.140)	Dev (`dev`, 192.168.233.129)
Arch	x86_64	x86_64	i686 (being retired)
RAM	2 GB	6 GB, 4 vCPU	n/a
Role	Production	64-bit Dev; heavy AI jobs (audio, transcription)	Old Dev, keyword-only ISE
ffprobe	MISSING	`/usr/local/bin/ffprobe`	`/usr/bin/ffprobe` (2.6.8)
tmux	yes	no (use `nohup`)	n/a

Code versions

File	Version	State
`MediaSearch.py`	1.5.2	Dev and Clone. Live not updated (Andrew's call)
`MediaProcessor.py`	ISE v9.21	Mock-tested, not deployed
`AudioProcessor.py`	1.0.3	Clone only. Full run complete
`audio_vocab.txt`	1.0.0	104 labels, 6 groups, sha1 `54504f3cce`
`clap_diag.py`	1.1.0	Clone
`dev_py_sync.sh`	1.0.2	Dev
`transcribe_pieces.sh`	1.0.0	Live. Clone is the target
`/bin/mirrorISE.sh` (Clone)	Andrew's, + Clone-only excludes	In use
`/bin/json_push_audio_Clone_to_Live`	2.1.0 (Andrew's naming)	Clone. Real run OK 2026-09-27
`json_md5_push.sh` / `json_md5_diff.sh`	1.2.0	`/bin`. Main `ISE_Data` only by default

Operating procedure

Live (images and video):

cd / && /usr/local/bin/python3.8 /var/www/html/The_ISE_Project/MediaIndexer.py
cd / && nohup /usr/local/bin/python3.8 /var/www/html/The_ISE_Project/MediaProcessor.py --log > /dev/null 2>&1 &
tail -f /var/www/html/ISE_Data/mediaprocessor.log

Clone (audio). Check that nothing is already running first:

pgrep -fl AudioProcessor
cd / && /root/venv-audio-test/bin/python /var/www/html/The_ISE_Project/AudioProcessor.py --plan
cd / && nohup /root/venv-audio-test/bin/python /var/www/html/The_ISE_Project/AudioProcessor.py --log --monitor > /dev/null 2>&1 &
tail -f /var/www/html/ISE_Data/audioprocessor.log
rsync --partial -a /var/www/html/ISE_Data/audio_store.json root@128.199.200.202:/var/www/html/ISE_Data/

Other operating notes:

On Clone, after any pip install as root, run chmod -R go+rX /usr/local/lib/python3.8/site-packages.
CentOS 6 pgrep has no -a; use pgrep -fl, or ps -o pid,lstart,cmd.
py_inventory.sh must run from / and must not be redirected.

Key paths

/var/www/html/The_ISE_Project/   MediaSearch.py  MediaProcessor.py  MediaIndexer.py  AudioProcessor.py
                                 audio_vocab.txt  clap_diag.py  ise_settings.py  qf_Mediasearch_bridge.php
/var/www/html/ISE_Data/          media_store.json  embeddings_store.json  captions_store.json
                                 audio_store.json (Clone writes, pushed to Live)  audio_checkpoint.json (Clone only)
                                 audio_manifest.tsv  hf_cache/  frame_cache/ (keep on Live)  caption_tmp/
/root/transcribe_pieces.sh       /root/transcripts/     /root/models/qwen3-asr/   /root/models/smolvlm/
/root/llama.cpp/build/bin/llama-mtmd-cli               /root/venv-audio-test/ (Clone)

DM2S/NX Chapter 4 checkpoint _3, 2026-09-27 05:45 AEST

DM2S/NX: Chapter 4 Checkpoint

0. Standing policies (Chapter 4 additions marked "New")

1. Done in Chapter 4

AudioProcessor full run on Clone: COMPLETE and VERIFIED

transcribe_pieces.sh v1.0.0: WORKS ON LIVE

Machines

1b. Done since _2 (2026-09-26 09:35 → 2026-09-27 05:40)

Mirror scripts: DONE

Coherence check: ALL 16 .json FILES AGREE (2026-09-27 ~05:30)

Transcription policy (decided)

Audio search findings

Other

2. Open, in order

3. Corrections to the Chapter 3 checkpoint

4. Mirror excludes

5. Carried forward from Chapter 3 (still current)

Machines

Code versions

Operating procedure

Key paths

1b. Done since `_2` (2026-09-26 09:35 → 2026-09-27 05:40)

Coherence check: ALL 16 `.json` FILES AGREE (2026-09-27 ~05:30)