📝 DM2S-NX_v10_0_Checkpoint_2026-10-05.mdTheme:

DM2S/NX v10.0 — Chapter 1 Checkpoint

Date: 2026-10-05, about 03:25 → 06:30 AEDT (written 11:57 AEDT) Session: v10.0 Chapter 1 — atomic store writes for the four in-place writers, manual update Participants: Andrew.human & Claude.ai Status: OPEN. Three of four writers delivered and tested in isolation; MiscIndexer.py must be redone against its current file. Nothing here has been deployed or run against the real DB.

1. Read this first — one thing must not be deployed

The MiscIndexer.py inside ISE_Atomic_Writers_v1.0.0.zip is built on a stale base. Do not deploy it.

The copy uploaded this session was 7.4.1#2 (dated 2026-08-20) and has no --single flag.
The manual (v5, written 2026-10-01) records MiscIndexer.py v7.5.0 with --single ID_ATTACH.
So the uploaded file was older than what runs in production. I patched it and called it 7.4.1#3, which would be a regression on Live or Clone: it drops --single and anything else added in 7.5.0, and its version number would go backwards.
If v1.0.0 was already deployed anywhere, restore the 7.5.0 file or a gzip_Live generation taken before the deploy, and check MiscIndexer.py --single N still works.
The other three scripts in v1.0.0 are fine and are byte-identical to the copies in v1.0.1.

Root cause on my side: I patched an uploaded file without first comparing its @version against the version the manual records for it. That check is now a proposed standing step (section 6).

2. What this session was about

Carry-over from the v9.9 closing checkpoint, item 10: convert the four scripts that still write their stores in place (IndexBuilder.py, MediaIndexer.py, MiscIndexer.py, orphan_check.py) to temp-file-then-rename. Reason (OVERSEER checkpoint 5.7): until that is done, a mirror, rsync or gzip during an index run can catch a half-written store, and the gated mirror can't be relied on.

Scope decision (Andrew): mirrorISE_Clone, mirrorISE_Dev and json_push_media_Clone_to_Live (which replaced json_push_audio_Clone_to_Live) are Andrew's to look after. Claude does not touch them. This removed v9.9 items 8 and 9 (gated mirror, ls diff) from this session.

3. Changes made (Problem & Resolution / Risk / Code Change)

3.1 Atomic store writes — `IndexBuilder.py` 7.4.3, `MediaIndexer.py` ISE v9.9.2, `orphan_check.py` 7.4.3

Problem & Resolution. Each script opened its store with open(path, "w") and dumped into it, so the real file is truncated and half-written for the duration of the dump. Fix: a helper, write_json_files_atomic(items, **dump_kwargs), copied inline into each script (no shared module, to avoid a new dependency across unrelated scripts). It writes every object to <path>.tmp, flushes and fsyncs, then renames all of them over their final names back to back with os.replace(). On any failure the .tmp files are removed and the live files are untouched. The old file's permissions are copied to the new one, and its owner too where permitted.

Risk Analysis.

Rename is atomic only within one filesystem; the .tmp sits in the same directory, so that holds.
For two-store writers (IndexBuilder: word_index.json + post_store.json) the pair can still disagree for the microseconds between the two renames. This cannot be closed without a different design and was judged not worth it.
If the second rename fails after the first succeeded, the first stays replaced. Rare, and not recoverable inside the helper.
A leftover .tmp (crash mid-write) is harmless, but an rsync pull could copy it before cleanup; the next --delete mirror removes it. Whether the mirror scripts should exclude *.tmp is Andrew's call.
A new file takes the writer's default umask and owner. Owner carry-over only works when the script runs as root.

Code Change. The write block in each script replaced by one helper call. orphan_check.py writes orphan_state.json with indent=2; MediaIndexer.py writes media_store.json with indent=4; IndexBuilder.py writes both stores with default json.dump settings. Python 3.6 compatible (Live runs 3.6 and 3.8).

3.2 `IndexBuilder.py` clean-ups (approved by Andrew: "fix all the anomalies")

DB_PASS removed from the DEBUG: print. It was going to the terminal and possibly to cron mail.
Dead _is_dev() removed, with its now-unused subprocess import. It referred to an undefined _DEV_IP; the NameError was swallowed by its own except, so it always returned False. Risk: if any other module imports IndexBuilder._is_dev, that import now fails. None is known.
Version mismatch fixed. Header said 7.4.2 while VERSION = "7.4.1", so the script printed the wrong version. Both are now 7.4.3.

3.3 Manual — `ISE_Python_Scripts_Manual_v6.md` (from v5)

New "Master v6 (2026-10-05)" entry at the top of the version history, covering the three converted scripts and the IndexBuilder.py clean-ups. It states that MiscIndexer.py is not covered yet.
Short "Atomic state-file write (v7.4.3)" note in the orphan_check.py section and "Atomic store write (ISE v9.9.2)" note in the MediaIndexer section.
Two quoted MediaIndexer version numbers updated (v9.7 → v9.9.2). Nothing else in the manual was changed; only those two original lines differ from v5.

4. Verification

Done:

All scripts compile (py_compile).
The write helper, extracted from each patched script and run in isolation, passes three cases: a normal two-file write (content correct, mode kept, no .tmp left); a planted failure on the second object (both originals untouched, no .tmp left); a brand-new file with indent=2.
MediaIndexer.py run in a sandbox with a stubbed Collabware.core_utils in --single mode: the new entry merged, existing entries survived, mode preserved, no .tmp left.
The manual diff against v5 was checked: 35 added lines, two changed lines.

Not done:

No script was run against the real DB, ise_settings.py or real stores. The first real run on Live is the end-to-end test.
MegaIndexer.sh was not run.

5. Findings that were not fixed

MediaIndexer.py can wipe media_store.json. If the DB query fails, or pymysql is missing, fetch_media_from_db() prints the error and returns {}; a plain run then writes {} over the store. Reproduced in the sandbox (0 items after the run). The atomic write does not prevent it. In --single mode a corrupt existing store is treated as {}, so the rewrite would leave only that one attachment. A proposed fix (abort with exit code 1 and leave the store untouched) was not approved and is not in any delivered file.
Unexplained file in the working folder. Near the end of the session the working folder contained patch_media2.py and a MediaIndexer.py marked ISE v9.9.3 implementing exactly that safeguard. Neither was created in this conversation, and I don't know their origin. I restored the approved v9.9.2 (verified byte-identical to v1.0.0) and left the v9.9.3 code out of every delivered zip. If it came from you or another session, say so; either way the safeguard stays unshipped until you approve it.
MegaIndexer.sh passes --monitor to IndexBuilder.py, which has no argument parsing, so the flag is silently ignored. Harmless; untouched. MegaIndexer.sh runs Posts, PDF (--fast) and Text, but not MediaIndexer.py.
PDFIndexer.py writes pdf_store.json but was not on the four-writer list and not examined. Unknown whether it is already atomic.
Manual gaps: no section for IndexBuilder.py (despite the opening saying it covers every script); MediaProcessor.py still quoted as v9.18, unconfirmed.

6. Standing rules — confirmed this session, plus one proposal

Confirmed in use:

Critique before building (the "arse-saver"): issues were listed before changes, and the one extra hazard (3.1 above, finding 1) was flagged rather than fixed.
Version bump in every changed module, doc-block history entry, version printed once at start, dotted never-reused zip version, release card.
Never touch the mirror and push scripts. Deployment timing is Andrew's.

Proposed (not yet agreed): before patching any uploaded file, compare its @version with the version the manual (or checkpoint version map) records for it, and stop if they differ. This would have caught the stale MiscIndexer.py before it was patched. It costs one grep per file.

7. Deliverables

File	State
`ISE_Atomic_Writers_v1.0.0.zip`	`IndexBuilder.py`, `MiscIndexer.py`, `MediaIndexer.py`, `orphan_check.py`. Do not deploy its `MiscIndexer.py`. Other three fine
`ISE_Atomic_Writers_v1.0.1.zip`	`IndexBuilder.py` 7.4.3, `MediaIndexer.py` ISE v9.9.2, `orphan_check.py` 7.4.3, `ISE_Python_Scripts_Manual_v6.md`. No `MiscIndexer.py`
`DM2S-NX_v10_0_Checkpoint_2026-10-05.md`	This file

Module versions at close:

Module	Version	Note
`IndexBuilder.py`	7.4.3	delivered, not deployed
`MediaIndexer.py`	ISE v9.9.2	delivered, not deployed
`orphan_check.py`	7.4.3	delivered, not deployed
`MiscIndexer.py`	7.5.0 (production, per manual)	not converted; v1.0.0 copy is stale
`ISE_Python_Scripts_Manual`	v6 (Master v6)	delivered

Version map entries from the v9.9 close (search.py 7.6.0, PDFsearch.py 7.7.0, MiscSearch.py 7.6.0, MediaSearch.py 1.8.0, SortEngine.py 8.1, COI_Chain.py 1.0.1) are unchanged by this session.

8. Open items — carry forward

Blocked on something from Andrew

Upload the current MiscIndexer.py from Live (should say 7.5.0). Then convert it, add its manual entry, ship as the next dotted zip.
Say whether ISE_Atomic_Writers_v1.0.0.zip was deployed anywhere. If so, restore MiscIndexer.py there.
Decide on the MediaIndexer.py store-wipe safeguard (finding 1), and say where patch_media2.py / the v9.9.3 file came from.
Decide whether the manual should gain an IndexBuilder.py section, and confirm the current MediaProcessor.py version.

Still open from the v9.9 list (unchanged unless noted)

v9.9 items 1–3: mirror Live → Clone then MediaProcessor.py --attachid=6882 on Clone; deploy ISE_AND_OR_v1.0.0.zip on Live and Clone; rebuild collabware_baseline.md5 after deploying new code (this session's files add to that deploy).
v9.9 item 10, partly done: three of four writers converted and delivered; MiscIndexer.py outstanding.
v9.9 items 4–7 (COI install, --snapshot, --check on all three machines, planted faults on Clone), 11–13 (COI in daily cron, OVERSEER RPM, .htaccess): unchanged. Item 11 should wait until the four writers are converted and deployed, otherwise the daily check can flag a half-written store.
v9.9 items 8–9 (gated mirror, ls diff): Andrew's, in his own mirror scripts.
v9.9 items 14–18 (ISE bugs: [listatt] disk check, bare /sort newest, ISEmega "Show all", 6982 re-measure, orphan entry 2419), 19–20 (QAT tools, clap_diag.py), 21–24 (audio quality), 25–27 (OVERSEER v2): unchanged.

New

Check whether PDFIndexer.py writes pdf_store.json atomically.
First real run of each converted script on Live, watching for .tmp files left behind and for permissions/owner on the new stores.
Decide whether the mirror scripts should exclude *.tmp (Andrew's call).
Open question from the gate discussion, still unanswered: when the COI check returns RC 12 or ssh to Live fails, should the mirror stop (proposed) or proceed. Only relevant when Andrew writes the gate in his mirror scripts.

9. Machine state

Not observed this session. No commands were run on Live, Clone or Dev; all work was on uploaded copies in the sandbox. The v9.9 close state (Live under swap pressure with media_store.json at 2723 items; Clone with free memory and no swap, mirror needed before MediaProcessor.py) is the last known state.

DM2S/NX v10.0 Chapter 1 checkpoint — 2026-10-05 11:57 AEDT