Paste this whole file into a fresh chat to resume. It reflects state as of the end of the PDF-indexing/search work session (PDFIndexer.py two-engine union + resume/locking, PDFsearch.py filename/topic search, orphan_check.py, cron cleanup). Older ISE post-search work (search.py/IndexBuilder.py/ RankingEngine.py) is included for completeness but was not touched this session — treat those as stable/working unless told otherwise.
Environment: live box is CentOS 6.10, Python 3.6.3, 2GB RAM. This OS's age is the root cause of most dependency fights below — expect the same pattern (no wheel, needs Rust, needs Python 3.8+) with any new tool.
| File | Location | Status | Use it? |
|---|---|---|---|
PDFIndexer.py |
/var/www/html/The_ISE_Project/ |
Rebuilt this session: two-engine (PyMuPDF+pypdf) union extraction, --fast mode, checkpointing, resume-by-default, --start-at, single-instance file lock |
Yes — current, deployed, working |
PDFsearch.py |
/var/www/html/The_ISE_Project/ |
Rebuilt this session: adds filename/topic search merged with content search, live progress | Yes — current, deployed, working |
orphan_check.py |
/var/www/html/The_ISE_Project/ |
New this session. Weekly cron job, avatar-excluded, diffs against orphan_state.json |
Yes — deployed, in crontab |
word_index_pdf.json / pdf_store.json |
ISE_Data/ |
Rebuilt from scratch with the new two-engine extractor; 497/676 PDFs indexed, 179 skipped (genuinely image-only, confirmed) | Yes — current on disk |
orphan_state.json |
ISE_Data/ |
State file for orphan_check.py, auto-managed |
Don't touch by hand |
all_orphans.py |
/var/www/html/The_ISE_Project/ |
Old one-off diagnostic (no avatar filter, no diffing) | No — superseded by orphan_check.py, removed from crontab, left on disk unused |
search.py, IndexBuilder.py, IndexLookup.py, Tokeniser.py, RankingEngine.py, ResultFormatter.py, ise_trace.py |
/var/www/html/The_ISE_Project/ |
Older ISE post-search engine (ISE 5.1). Not touched this session. | Yes — assumed stable; re-verify only if something seems broken |
qf_search_bridge.php, qf_PDFsearch_bridge.php |
SMF theme/Sources | PHP bridges wiring the dropdown search-engine options to the Python scripts | Yes — working as of ISE 5.1 |
Search.php, index_template.php (SMF core, patched) |
SMF | Dropdown values 'ISE' / 'ISEpdf' wired in |
Yes — working |
| pdfplumber | tried, rejected | Needs Rust to build cryptography from source on this box; no working path found |
No |
| pypdfium2 | tried, rejected | No compatible wheel at any version for this platform | No |
| tesseract 3.04 (EPEL) / ocrmypdf 12.7.2 | tried, parked | ocrmypdf requires tesseract ≥4.0 (dead end); tesseract 3.04 direct-CLI use never actually tested | Parked — user said "I don't need OCR", not pursuing |
dmesg confirmed an actual OOM kill of a PDFIndexer.py run (913MB resident) — this is a real, hit-once constraint, not theoretical.pymupdf==1.19.2 (no newer wheel for this Python/platform)pypdf==3.1.0 (5.x requires typing.Protocol, Python 3.8+, crashes on import here)poppler-utils on this box is 0.12.4 (2010) and silently mis-decodes PDFs with malformed ToUnicode CMaps into garbage tokens instead of erroring — this was the original root cause of the very first bug in this session (search for "listatt" finding nothing despite the word being visibly present).PDFIndexer.py --monitor output, not just assumed.--fast (PyMuPDF only) is real and works — confirmed correct index output, ~3-20x faster depending on file, meaningfully lighter on memory.--fast every 6h, full union every 2 days) write to the same word_index_pdf.json/pdf_store.json — the single-instance lock (/tmp/pdfindexer.lock, fcntl) was added and tested (confirmed: a second instance exits immediately with a clear message rather than racing) specifically to stop one overwriting the other's progress if they ever overlap.id_msg (58213) doesn't resolve to any row in smf209_messages — the post was deleted. SMF's dlattach action requires both topic= and attach= to serve a file; confirmed (user tested directly) there is no working fallback without a valid topic. This is a genuine, currently-unfixable-in-code data gap, not a PDFsearch.py bug.id_msg=0 — confirmed this is SMF's normal avatar convention (profile pictures), not orphaning. Only ~6 of the original 360-ish flagged rows were real orphaned content PDFs.all_orphans.py (old diagnostic) has no avatar filter and no diff logic — confirmed strictly redundant with orphan_check.py. User agreed to remove it from cron after initially wanting to keep it.0 */2 * * * python3 IndexBuilder.py (post index, every 2h)
5 */6 * * * python3 PDFIndexer.py --fast --monitor (PDF index, fast, every 6h)
10 0 */2 * * python3 PDFIndexer.py --monitor (PDF index, full union, every 2 days)
15 */12 * * * attach_filetypes.sh /var/www/html/attachments (every 12h)
0 6 * * 1 python3 orphan_check.py --html ... --quiet-if-empty (weekly Monday 06:00)all_orphans.py's 07:00 Monday line was removed.
ocrmypdf, which needs tesseract ≥4.0) was never actually tested end-to-end. Revisit only if/when wanted..split() in PDFIndexer.py — not punctuation-aware ([listatt] and listatt index as separate tokens). Known limitation, never prioritized for a fix.pdf_store.json's filename field is always None — resolve_filename() exists but isn't wired into PDFIndexer.py's main(). Doesn't block search (PDFsearch.py resolves real filenames itself via direct DB query) but is a known gap in the index file's own completeness.ISE_Python_Scripts_Manual.md, covering PDFIndexer.py/PDFsearch.py/orphan_check.py CLI options) — exists as an output file; not saved into project memory/repo, worth confirming it's actually been kept somewhere durable if it's meant to be a lasting reference./var/www/html/smf20/The_ISE_Project). Not started.None queued — user explicitly paused ISE work this session ("no more ISE!"). When resumed, the next actual action is: replicate the PDFIndexer.py/PDFsearch.py two-engine-union + locking/resume work on the dev box, using this session's saved transcript as the reference.