Date: Monday 12 October 2026, written 11:21 AM Sydney Covers: the chat "Processing new media concurrently", 08:36 to 11:02 AM, 12/10/2026 Source: the 46-page PDF printed 11:13 AM. The chat itself has since been deleted, so this file and that PDF are the only record. Status when the chat closed: processing finished and pushed to Live; you had just said "going to mirror now". The mirror's outcome is not recorded here.
| Item | State |
|---|---|
| Images/videos (MediaProcessor, ISE v9.27) | Run finished: "All 2 instances finished". Stores on Clone then held 2996 embeddings and 2993 captions |
| Audio (AudioProcessor v1.2.3, 2026-10-07) | 87 of 87 items done, 1705 long-mix windows. New track is ID 7440 |
| MediaIndexer | ISE v9.9.9 on both boxes. Clone catalogue is 2847 items after the deletions and the re-added track |
| Push to Live | Done about 10:58 with json_push_audio_Clone_to_Live and json_push_visual_Clone_to_Live |
| Dev | Behind Clone on the five processing stores (see section 5) |
| Mirror | You started it at 11:02. Result not seen |
py MediaProcessor.py --instances 2 --progress --log. Images and videos together, shared lock, cores split.--ft image in one window, --ft video in another). Both write embeddings_store.json.tmp, so one window's atomic replace takes the other's temp file and that one crashes. This happened at 09:13.--force. --attachid N always runs as a single copy.--plan (what's left and time estimate), --prune (drop stored entries whose attachment is gone; no backup made), --features-only. Don't run it alongside MediaProcessor.transformers falls back to a pure-Python tokenizer when it's absent. Always install with pip install --no-deps transformers==4.31.0. A plain pip install transformers or -U pulls tokenizers in and breaks both processors.rm. Live and Clone each have their own SMF database, so it must be done on both (or re-synced from Live).ls -t /var/www/html/attachments2 | head -3 on Live and Clone. They matched (7440 on both).open_clip imports transformers, which loads tokenizers; the compiled library needs glibc 2.14. This came from the earlier transformers/tokenizers install in system py38.pip uninstall -y transformers tokenizers, remove the fake safetensors-0.3.1.dist-info, import test open_clip. MediaProcessor then ran./root/.cache/huggingface. Confirmed at 09:11: blob and link both present, stamped 08:55.py MediaProcessor.py --attachid 6847; weights now cached./root was 92% used with 5.2GB free.zcat check was offered, not run). Deleted on Live and Clone.MediaIndexer, AudioProcessor --prune (removed stale 7331), then encoded 7440 (6 windows, 44s, 119.5–120 BPM, A# minor, −14.4 LUFS).transformers from system py38. AudioProcessor's CLAP encoder needs it./root/venv-audio-test was the working environment (it wasn't), then proposed a new venv, which you rejected.py -m pip install --no-deps transformers==4.31.0. AudioProcessor then ran: CLAP laion/clap-htsat-unfused loaded in 42.1s, log says "tokenizers (Rust) not available — using the pure-Python RoBERTa tokenizer". resume_download and "special tokens" warnings are harmless.py -c "import open_clip; print('ok')" after this reinstall was suggested but its result never came back. Run it before the next MediaProcessor run./var/www/html/ISE_Data/caption_failures.json. One retry: py MediaProcessor.py --attachid 735 --max-caption-retries 0. Not yet done as far as the transcript shows.| Change | Where | Status |
|---|---|---|
pip uninstall transformers tokenizers, then pip install --no-deps transformers==4.31.0 |
Clone, system py38 | Done |
Removed fake safetensors-0.3.1.dist-info |
Clone, /usr/local/lib/python3.8/site-packages |
Command given; you reported it working |
rm -f /var/www/html/ISE_Data/processor_checkpoint*.json |
Clone | Command given at 10:56 after you said to get rid of them. Not confirmed run. MediaProcessor decides skips from stored embeddings, and json_push_visual_Clone_to_Live skips the checkpoint if absent. Both from help text and your scripts, not from reading MediaProcessor.py |
cleanISE_Clone v1.0.0 (bash, for /bin on Clone) |
cleanISE_Clone_v1.0.0.zip (script + manual) |
Built and tested against a mock ISE_Data only. Not confirmed installed on Clone |
Cron line 23 * * * * /bin/cleanISE_Clone --apply >> /var/log/cleanISE_Clone.log 2>&1 |
not installed | Offered; you want frequent cleaning |
cleanISE_Clone rules (dry run by default, acts only with --apply; anything not listed is untouched):
*.tmp older than 60 minutes deleted.*.log over 2MB rotated to .log.1, then .log.2, original emptied. Only misc_indexer.log (4.3MB) qualifies today.processor_checkpoint_i*of*.json older than a day deleted.Risk: log rotation is copy-then-empty, so a line written in that instant can be lost. Every *_store.json, word_index*.json, chain_snapshot.json, orphan_state.json, caption_failures.json, audio_checkpoint.json and processor_checkpoint.json itself are left alone.
Release card (as delivered): version/date done (1.0.0, 2026-10-12); manual history done; version shown at start done; dotted zip name done; xmllint not needed; unrequested changes listed beforehand done; legacy zipper not needed (not an SMF package).
audio_checkpoint, audio_store, captions_store, embeddings_store, media_store). Different on six Live-owned indexes (misc_store, pdf_store, post_store, word_index, word_index_misc, word_index_pdf). Live's are newer, so Clone's copies are older, which is expected.word_index_media.json matches on all three boxes.ISE_Data to Live would overwrite them with older copies. Only the two push scripts should write the five processing stores.--delete on mirrorISE_*. Whichever box is the destination loses files that exist only there (processor_checkpoint*.json, pdf_dupe*). Check that matches what you want.dbbacksmf, dbbacksm2, SQLhousekeeping: if not already pulled from Live into Clone's /bin, the mirror would push Clone's older copies over Live's fixed ones. Not confirmed either way.mirror1_Clone -n has never been dry-run.json_diff_md5sums checks.py -c "import open_clip; print('ok')" on Clone before the next MediaProcessor run (see 3.5).caption_failures.json, decide whether to retry./root on Clone is 92% used (5.2GB free).media_store fell from 2853 to 2846 items between the 10:12 and 10:25 indexer runs, with 4 of those being audio (90 to 86). Deliberate deletions were 5 in total (6847, 7326, 7336, 7350, 7331), so 2 non-audio items beyond the expected ones went missing from Clone's catalogue. It may be nothing (e.g. other items on the same posts), but it was not checked. Live's media_store matches Clone's, so both agree.word_index_media.json: orphan (no .php, .py or .sh under the web root names it), only 127K. Deleting it means touching all three boxes, and json_diff_md5sums_All_Clone would report it missing until Live and Dev lose it too. Left in place.json_push_media_Clone_to_Live: old duplicate of json_push_visual_Clone_to_Live (same v1.0.2 block), still on your removal list, not removed.rw-rw-rw- files and logs: chmod on Clone alone would likely be undone by the next mirror; the fix belongs at the source. Not urgent.indeximages.log, mediawatchdog.log, orphan_check.log) are 0 bytes from Oct 7. Harmless.