📝 COI_Chain_manual.mdv4.5 · 2026-09-28

COI_Chain.py: Manual

Version 1.0.1 · 2026-09-30 · The ISE Project, THE OVERSEER Another collaboration between Andrew.human and Claude.ai

What it is

The foundation layer of THE OVERSEER: the Chain-of-Integrity cross-checker. For every SMF attachment it tests each link of the chain, and reports every break with an IBM-style return code. It never fixes anything.

SMF DB row ──> owning post   (or owning member, for avatars)
           ──> file on disk  (folder from id_folder, name {id_attach}_{file_hash})
           ──> entry in every .json store that should hold it
and back:  every .json store entry ──> an SMF DB row
           every SMF-named file on disk ──> an SMF DB row

The two modes

Mode Runs on Touches the DB? Writes
--snapshot Live only Yes: SELECT only ISE_Data/chain_snapshot.json (temp file + rename)
--check Any machine Never Nothing. Report to stdout, RC as exit code

Why split them. ISE is half-blind without the DB, but DB independence is a virtue. The snapshot is the DB's word, frozen and dated: which attachments exist, which posts and members they point at, and where SMF keeps its attachment folders. Once the mirrors carry it down, Clone and Dev can run the full check without ever connecting to Live's MySQL.

The snapshot (chain_snapshot.json)

This is how the chain is represented in .json:

{
  "coi_snapshot_format": 1,
  "coi_version": "1.0.0",
  "created_utc": "2026-09-30T06:00:00+00:00",
  "created_epoch": 1790748000,
  "table_prefix": "smf209_",
  "folders": {"1": "/var/www/html/attachments", "2": "/var/www/html/attachments2"},
  "posts":   [1234, 1301, ...],
  "members": [1, 7, ...],
  "attachments": {
    "7006": {"msg": 1301, "member": 0, "folder": 2,
             "hash": "0c2b595f0eec4f86a150a3cca9af233e7a1e0970",
             "name": "COI_Blueprint.pdf", "ext": "pdf",
             "size": 1504964, "type": 0}
  }
}

The checks, and what each break means

Break Link Meaning
NO-POST row → post The attachment's post no longer exists (the old orphan_check.py case)
NO-MEMBER row → member An avatar whose member no longer exists
NO-OWNER row → owner Neither a post nor a member: should never happen
BAD-FOLDER row → disk id_folder isn't one of SMF's configured folders
FILE-MISSING row → disk The row exists, the file doesn't: rm -f, rsync, lost disk
SIZE-DIFFERS row → disk The file is there but isn't the size SMF recorded: replaced or truncated
FILE-NO-ROW disk → row An SMF-named file with no row (after a DB restore, or a failed upload)
ENTRY-NO-ROW store → row A store entry for an attachment SMF no longer has (the 2419 case)
NOT-INDEXED row → store SMF has it, the store that should hold it doesn't (the 6982/6998 case)
STALE-SNAPSHOT — Snapshot older than 26 h, so every other verdict may be out of date
STORE-UNREADABLE / FOLDER-UNREADABLE — Couldn't read a store or folder

INFO lines are not breaks: files ignored in attachment folders (index.php, .htaccess), attachments with no file_hash (old SMF naming, not checked), stores not present on this machine, and entries outside a store's rule.

Return codes

RC Meaning
0 Chain intact
4 Some breaks, up to --mass (default 25)
8 Mass break: more than --mass. Something happened (restore, rsync, rm -f). Investigate; fix nothing yet.
12 Couldn't run: no snapshot, wrong format, DB error

Which attachments each store should hold

These rules mirror what each indexer selects today. If an indexer's selection changes, change STORES at the top of COI_Chain.py too, or the coverage check will report false breaks or miss real ones.

Store File types Only if the post exists?
media_store.json mp4 webm 3gp jpg jpeg png webp gif mp3 wav No (MediaIndexer takes avatars and thumbnails too)
audio_store.json mp3 wav (under "items") No
pdf_store.json pdf No
misc_store.json md txt text json html py php js sh sql css c cpp java rb go rs pl xml yml yaml ini conf Yes (MiscIndexer joins messages)

PDFIndexer finds PDFs by magic bytes, not extension, so a PDF with a wrong extension shows up as an INFO line ("outside its rule"), not a break.

Running it

# Live: take the snapshot (needs DB), then check
python3.8 /var/www/html/Collabware/OVERSEER/COI_Chain.py --snapshot
python3.8 /var/www/html/Collabware/OVERSEER/COI_Chain.py --check

# Clone / Dev: check only, after the mirror has brought the snapshot down
python3 /var/www/html/Collabware/OVERSEER/COI_Chain.py --check

Options: --mass N sets the RC 8 threshold; --quiet omits INFO lines; --data-dir points at a different ISE_Data.

On Clone and Dev, the check uses Live's folder paths from the snapshot. If a machine doesn't hold a mirrored copy of the attachment folders at the same paths, every file will show as FILE-MISSING and the run ends in RC 8. On that machine, only the store checks mean anything.

Proving it works (planted faults)

test_COI_Chain.py builds a fake SMF (SQLite DB, two attachment folders, four stores), plants each fault, and checks for the right break and RC. It touches nothing outside /tmp/coitest. Run it after every change to COI:

python3 test_COI_Chain.py

Scenarios: clean chain; file deleted; fake store entry; post dropped; file with no row; untick (before and after the store catches up); file replaced; not indexed; avatar owner gone; stale snapshot; mass break; attachmentUploadDir parsing (plain string, keys kept, multi-byte path). 14/14 passing as delivered.

The real proof is yours: plant the same faults on Clone and confirm each is reported.

Not in this version

Version history

1.0.1 (2026-09-30): paths fixed for the move to Collabware/OVERSEER. ISE_Data and the DB settings (PDFsearch.DB_CONFIG) are now found at fixed paths (/var/www/html/ISE_Data, /var/www/html/The_ISE_Project) instead of relative to COI's own location.

1.0.0 (2026-09-30): first version. Snapshot and check modes; eleven break types; RC 0/4/8/12; store rules for media, audio, pdf and misc; planted-fault test suite.