Date: 2026-09-20 (evening) → 2026-09-21 (past midnight) Milestone: ISEmedia goes from 0/2706 real attachments resolved to a working, self-diagnosing, crash-safe, unattended-capable indexing/embedding pipeline, ready to mirror to Clone and Dev. Participants: Andrew (Architect & QA Lead), Claude (this session)
Files touched: Collabware/core_utils.py, The_ISE_Project/MediaIndexer.py, The_ISE_Project/MediaProcessor.py, The_ISE_Project/MediaSearch.py Final versions: core_utils.py 1.2.0 · MediaIndexer.py ISE v9.7 · MediaProcessor.py ISE v9.18 · MediaSearch.py 1.1.0
Problem & Resolution: locate_attachment_file() assumed the physical filename was {id_attach}_{original filename} and only checked three hardcoded directory names. Confirmed live: 0 of 2706 real attachments resolved. Real scheme (verified against id_attach 6731, "the race.png"): {directory for id_folder}/{id_attach}_{file_hash}, where both id_folder and file_hash come from the attachments table. Rewrote locate_attachment_file() to that signature; rewrote get_attachment_directories()/_parse_php_dir_array() to parse however many attachmentUploadDir entries actually exist (Live has 2, not the 3 previously hardcoded); updated MediaIndexer.py to select and store id_folder/file_hash in media_store.json.
Risk Analysis: Breaking change to locate_attachment_file()'s signature — intentional, since the old signature lacked the data needed to ever work. A media_store.json written before this fix lacks the new fields; detected and reported explicitly, not silently mishandled.
Code Change: core_utils.py (rewrite), MediaIndexer.py (query + storage), MediaProcessor.py (call-site signature update).
Problem & Resolution: After fix #1 deployed, a second full-miss run (0/2706 again) occurred — including id_attach 6731, already manually verified correct. Root cause: get_attachment_directories() connected using only Settings.php-derived credentials, while MediaIndexer.py had always allowed SMF_DB_HOST/SMF_DB_USER/SMF_DB_PASS/SMF_DB_NAME env var overrides. The two modules were silently talking to different databases. Fixed by applying the identical env var override in core_utils.py.
Risk Analysis: Silent failure mode — locate_attachment_file()'s except Exception: return None swallowed the connection error entirely. Any future code touching this DB config must apply the same override or risk repeating this exact class of bug (documented in the manual addendum).
Code Change: core_utils.py, get_attachment_directories().
Problem & Resolution: The env var bug (and everything like it) was invisible without per-item detail. Added debug= param to locate_attachment_file() (prints the specific reason for a miss); wired MediaProcessor.py --verbose through to it. Also added a pre-flight check — get_attachment_directories() called once before the main loop — so an environment-wide failure exits immediately with one clear message instead of silently burning a full run producing N copies of the same error.
Risk Analysis: None — additive, opt-in via --verbose; pre-flight failure is a hard exit(1), correct behavior for an unrecoverable environment problem.
Code Change: core_utils.py, MediaProcessor.py.
Problem & Resolution: Fix #3's debug output revealed the true next cause: pymysql not installed in whichever interpreter runs MediaProcessor.py (needs torch/open_clip/PIL too, apparently a separate venv from MediaIndexer.py's). Confirmed working interpreter: python3.8.
Risk Analysis: Operational/environment issue, not a code bug — no code change beyond the pre-flight check (#3) that now catches this class of problem fast. Documented in the manual addendum as a standing gotcha.
Code Change: None (environment fix on Live, outside this session's scope).
Problem & Resolution: embeddings_store.json/captions_store.json/checkpoint were only written once, after the entire run — fine for --chunk-size 10, unsafe for an unattended full-library run (CPU-only, up to 60s/item for captioning) where a crash partway lost everything with nothing on disk. Andrew's design: since the only real action per item is ADDING to an append-only store, persist immediately, not batched. All three stores now write after every single item, via a new _atomic_write_json() (temp file + os.replace()) for corruption-safety at the new write frequency.
Risk Analysis: None identified — atomic writes eliminate partial-file corruption risk; per-item write cost is negligible next to CLIP/SmolVLM inference time.
Code Change: MediaProcessor.py — _atomic_write_json(), write_checkpoint(), process_library().
--reindexall aliasProblem & Resolution: Andrew requested a flag literally named --reindexall for "wipe and redo everything." The existing --force flag already did exactly this — added as an alias on the same argparse dest rather than renaming, so nothing breaks for existing usage.
Risk Analysis: None — pure naming/documentation addition.
Code Change: MediaProcessor.py, --force/--reindexall argparse definition.
Problem & Resolution: An item with a saved embedding but a failed/timed-out caption was permanently skipped on every future run (already_processed_ids was embedding-only). Caught live: checkpoint said 20 done, only 16 had real captions (78, 95, 97, 99 confirmed missing). Fixed: a skipped item now gets its missing caption keys retried — no CLIP/ffmpeg re-run, reusing the cached video frame (frame_cache_dir, never deleted) or a freshly-resolved image path. Doesn't count against --chunk-size.
Risk Analysis: None — additive, reuses already-successful work rather than duplicating it.
Code Change: MediaProcessor.py, process_library()'s skip branch.
Problem & Resolution: Andrew's SSH session repeatedly stopped accepting any keystroke but Enter after runs. Root cause: llama-mtmd-cli/ffprobe/ffmpeg all inherited the real tty as stdin; the 60s caption timeout SIGKILLs the child on expiry, which never gets a chance to restore terminal settings — and a full-library run is close to guaranteed to hit at least one timeout. Fixed: stdin=subprocess.DEVNULL on all three subprocess calls, plus -nostdin on ffmpeg itself.
Risk Analysis: None — these processes never needed real stdin access; this only removes an unused, actively harmful capability.
Code Change: MediaProcessor.py, generate_caption(), get_video_duration(), extract_frames_at_intervals().
Problem & Resolution: All of the above failure modes (caption timeout/exception, file-not-found, CLIP encoding exceptions) printed only under --verbose, and there was no end-of-run signal when embeddings_store/captions_store didn't match — the 78/95/97/99 gap sat silent across two full runs. Fixed: every failure message now prints unconditionally (success messages stay --verbose-only); every run ends with unconditional summary lines (missing-caption IDs, files-not-found count, encoding-error count) whenever nonzero.
Risk Analysis: None — output-only change; slightly noisier stdout on a run with real problems, silent on a genuinely clean run (this is the intended behavior).
Code Change: MediaProcessor.py — generate_caption(), file-not-found branch, encoding-exception branch, end-of-run summary.
--logProblem & Resolution: Output only ever went to the terminal — a problem for an unattended tmux run, where scrollback isn't durable. Added --log [PATH] (bare form uses ISE_Data/mediaprocessor.log, append mode, timestamped run banners) via a small _TeeStream wrapping sys.stdout/sys.stderr at startup — captures every existing print() call site for free, no per-call-site edits needed.
Risk Analysis: None — opt-in, off by default, no behavior change without the flag.
Code Change: MediaProcessor.py — _TeeStream class, --log argparse wiring.
--help / no-args CLI convention (standing rule, applies to all future ISE CLI scripts)Problem & Resolution: Andrew's convention: show full help on no args or explicit -h/--help, unless a script is designed to run bare — then bare invocation runs normally and --help gives a one-line description. MediaSearch.py's manual argv[0]-as-query slice (a deliberate v7.5.1-style fix so a query starting with -/-- isn't misread as a flag) meant --help was silently swallowed as literal query text, and no-args showed only a terse error. Fixed by special-casing the exact tokens -h/--help before the manual slice runs — every other dash-prefixed query is completely unaffected. MediaIndexer.py/MediaProcessor.py (bare-run-by-design) got tightened one-line --help descriptions only.
Risk Analysis: None — preserves the existing dash-safe-query fix exactly; only changes behavior for the literal strings -h/--help and true zero-args.
Code Change: MediaSearch.py (main()), MediaIndexer.py/MediaProcessor.py (description strings).
MediaSearch.py has no filename/topic-subject match path — unlike PDFsearch.py/MiscSearch.py, results are pure CLIP-vector similarity; a deliberately-named file gets no search credit for its name./sort works (via shared SortEngine.py), but something like /type jpg filtering against media_store.json's stored extension field does not exist yet. Agreed approach: filter on the stored extension field directly, not fnmatch-over-filename, since the user's mental model is "a jpg file," not "a hashed attachment."MediaSearch.py already resolves and links the exact forum post each result hangs off (msg_url, board, subject) — this was mistaken for missing, it isn't.ISEmedia_Manual_Addendum.md delivered — covers core_utils.py, MediaIndexer.py, MediaProcessor.py, MediaSearch.py, and qf_Mediasearch_bridge.php in the same style as the existing ISE_Python_Scripts_Manual.md, for splicing in wherever Andrew keeps the canonical copy.
Status: ready to mirror to Clone and Dev.