Covers every Python script currently in /var/www/html/The_ISE_Project/. The five search/index scripts share the same settings-loading approach (reading DB credentials and the board URL from Settings.php via ISE_ROOT), so none of them need separate configuration — they auto-detect the forum's DB and paths. pdf_duplicate_spotter.py reuses that same infrastructure via direct imports rather than duplicating it.
Builds the PDF search index: discovers every real PDF attachment on disk (via magic-byte detection, not the DB's stored file extension, which can be wrong), extracts text from each, tokenises it, and writes two files:
ISE_Data/word_index_pdf.json — word → list of [id_attach, page] hitsISE_Data/pdf_store.json — id_attach → {filename, pages}Runs two independent extractors on every page and takes the union of whatever words each one finds — a word either engine catches makes it into the index:
3.1.0 — later versions require Python 3.8+ and will crash on this box's Python 3.6)A third engine, pdfplumber, was tried and rejected — its dependency chain requires compiling cryptography, which needs a Rust toolchain not available on this CentOS 6 box. pypdfium2 was also tried and rejected — no compatible wheel exists for this platform at any version.
python3 PDFIndexer.py [options]| Flag | Default | Description |
|---|---|---|
--attach-dir DIR |
auto-detected (<board root>/attachments) |
Directory to scan for PDF attachments |
--dry-run |
off | Discover + extract only — don't write anything to disk |
--limit N |
none | Process only the first N discovered PDFs (for testing) |
--monitor |
off | Print live progress — which engine is running per file, new words added per file, and a running summary every 5% |
--verbose |
off | Also stream log output to stdout, not just the log file |
--fresh |
off | Ignore any existing index on disk and rebuild everything from scratch. Default behaviour without this flag is to resume: load the existing index and skip attachments already indexed |
--checkpoint-every N |
25 |
Save progress to disk every N processed PDFs, so a crash (e.g. an OOM kill) only loses recent work rather than the whole run. Set to 0 to disable and only save once at the end |
--start-at ID_ATTACH |
none | Skip every attachment with id_attach below this value — for manually restarting a run from a known point, independent of the resume/skip logic |
--fast |
off | PyMuPDF only — skips pypdf entirely. ~20x faster per file and noticeably lighter on memory, at the cost of losing pypdf's independent-parser coverage from the union. Good for quick reindexes or spot-checks |
# Standard run — thorough (both engines), resumes automatically if re-run
python3 PDFIndexer.py --monitor
# Quick single-engine pass
python3 PDFIndexer.py --fast --monitor
# Full rebuild from scratch, ignoring whatever's already indexed
python3 PDFIndexer.py --fresh --monitor
# Resume a run that was killed partway, starting from a known attachment ID
python3 PDFIndexer.py --start-at 5235 --monitor
# Test on a handful of files without writing anything
python3 PDFIndexer.py --limit 5 --dry-run --monitor--checkpoint-every bounds how much work is lost if that happens; --fast reduces peak memory substantially by skipping the second extraction pass. Watch free -h during a full --monitor run if concerned..split() (tokenise() in the script) — not punctuation-aware. [listatt] and listatt are indexed as separate tokens. Flagged as a known limitation, not yet replaced.pdf_store.json's filename field is currently always None — resolve_filename() exists in the script but isn't wired into main(). PDFsearch.py resolves real filenames itself via a direct DB query instead, so this doesn't block search, but it's a known gap in the index file itself.Searches the index PDFIndexer.py builds, plus (as of the filename/topic feature) scans every PDF attachment's own filename and its linked forum topic's subject line — so a document titled with the right words surfaces even if those words never appear in the extracted PDF text itself.
python3 PDFsearch.py QUERY [options]QUERY is a required positional argument — the search terms.
| Flag | Default | Description |
|---|---|---|
--limit N |
none (no cap) | Cap the number of results returned. Omit for every match — useful to pass e.g. --limit 20 for a quick SSH sanity check |
--html FILE |
none | Write an HTML results report to this path, in addition to (or instead of) console output |
--source {ui,ssh} |
ssh |
Present so argparse accepts the literal --source=ui flag the web UI bridge passes. ise_trace.py detects UI vs SSH itself by checking sys.argv, independent of this flag |
# Console search, every match
python3 PDFsearch.py listatt
# Capped, for a quick check over SSH
python3 PDFsearch.py heroin --limit 20
# Full HTML report
python3 PDFsearch.py heroin --html /var/www/html/ISE_Data/results.htmlid_msg doesn't resolve to a real row in the messages table (the post it was attached to is gone), the result shows "No topic found" instead of an Open PDF link, since SMF's dlattach action requires a valid topic to serve the file at all — confirmed there's no working fallback for this case.As of this session, query terms containing * or ? are matched via fnmatch (case-sensitive, matching this script's existing case-sensitive exact-term lookup) against every key in word_index_pdf and every filename/topic subject, instead of the normal exact-match/ substring lookup. A term with no wildcard characters keeps the original fast path. Query terms come through tokenise()'s plain text.split() here — wildcard characters survive that unchanged since .split() doesn't strip anything, unlike MiscSearch.py's regex tokenizer, which needed its pattern extended to preserve them.
UI-triggered searches (via qf_PDFsearch_bridge.php) are syslogged by the PHP bridge before this script even runs. SSH-triggered runs bypass that PHP layer entirely — without a fix, they'd never reach syslog at all, only ise_trace.py's separate log. PDFsearch.py now syslogs the query itself (tag ise-pdf-search-ssh, message prefixed [SSH]) when it detects it's running via SSH, so both paths end up logged consistently — this same gap existed in search.py and was fixed there too.
Finds byte-identical PDF duplicates in the attachments directory — the same file content re-uploaded under a different id_attach, usually with a different filename and/or topic. Not near-duplicate detection (a different scan of the same book, an edited copy, etc.) — that's a separate, harder problem this doesn't attempt.
Reuses existing infrastructure rather than reimplementing it: discover_pdfs()/ATTACH_DIR from PDFIndexer.py for the same magic-byte PDF scan already used for indexing, and DB_CONFIG/get_topic_for_attachment()/build_download_url() from PDFsearch.py for the same DB lookups/link-building already used there.
Method: SHA-256 over each file's actual bytes, streamed in 1MB chunks (2GB-RAM box constraint — never loads a whole PDF into memory). Files sharing a hash are byte-identical, full stop — no false positives possible from this method, and correspondingly no way to catch a duplicate that isn't byte-identical (see PIHKAL/TIHKAL note below).
python3 pdf_duplicate_spotter.py [options]| Flag | Default | Description |
|---|---|---|
--attach-dir DIR |
ATTACH_DIR (from PDFIndexer.py) |
Directory to scan for PDFs |
--json PATH |
ISE_Data/pdf_duplicates.json (via ISE_ROOT, same sibling-of-The_ISE_Project convention as every other script's data files) |
Write the full report as JSON here — this is what pdf_dupe_review.php reads, so leave it at the default unless you're also pointing that page somewhere else |
# Standard run — writes to the default path pdf_dupe_review.php expects
python3 pdf_duplicate_spotter.py
# Custom output location
python3 pdf_duplicate_spotter.py --json /tmp/dupes.jsonNot a Python script (out of this manual's stated scope, noted here for findability) — an admin-only page in The_ISE_Project/ that reads this script's --json output and lists every duplicate copy as two plain links: View post (opens the actual forum post) and Delete post (single click, JS-confirmed, SMF session-token guarded, deletes the whole post via SMF's own removeMessage() — which also cleans up that post's attachment as part of the same call). Backs up the message row to ise_pdf_dupe_backup_v2 before deleting (metadata only, not the PDF bytes — a forensic record, not a one-click undo, since SMF reassigns ids on delete) and remembers deleted ids in ISE_Data/pdf_dupe_deleted.json so they stop appearing without needing to rerun the spotter.
id_topic/id_msg — the post they were attached to is already gone) still appear in a duplicate group if their bytes match, but pdf_dupe_review.php has no post to link to or delete for those — they show with no View/Delete links.Builds the search index for text-based attachments: .md, .txt, .text, .json, and .html (full content, tokenised and indexed). Simpler than PDFIndexer.py in a few key ways — these are already plain text, so there's no extraction step, no multi-engine union, and no resume/checkpoint logic (every run is a full fresh rebuild, since the corpus is expected to stay small).
.html gets one extra step before tokenising: strip_html_for_indexing() removes <script>/<style> blocks (tag and content together), strips every remaining tag, and decodes HTML entities (& → &, etc.) via re-based regexes — deliberately simple, not a full HTML parser, good enough for indexing purposes only. This keeps markup (div, class, href, inline JS/CSS) out of word_index_misc.json — only visible page text is indexed. misc_store.json's content field for .html entries holds the stripped text (so search snippets are readable), but the checksum field is still computed on the original, unstripped file.
MiscIndexer.py also discovers a much wider set of code/config attachment types — .py, .php, .js, .sh, .sql, .css, .c, .cpp, .java, .rb, .go, .rs, .pl, .xml, .yml, .yaml, .ini, .conf — but treats them as filename/type-only: FILENAME_ONLY_EXTENSIONS skips read_text_file() and tokenise() entirely for these, so nothing they contain enters word_index_misc.json. Their misc_store.json entry has content: "", zero word counts, and filename_only: true — they're viewable via the "Open file" download link and filename/topic searchable, but never full-text searched. .html is deliberately not in this set — it's mostly prose with markup wrapped around it, so it gets full content indexing (above) instead.
ISE_Data/word_index_misc.json — word → list of {id_attach, count} hitsISE_Data/misc_store.json — id_attach → full metadata + raw content (filename, file_type, board/topic/message context, headings, word count, checksum, filename_only flag, etc. — resolved once at index time via a DB join through messages/topics/boards, so MiscSearch.py never needs a separate lookup)Unlike PDFIndexer.py's magic-byte scan, plain-text files have no reliable magic number to sniff — and SMF strips real file extensions from attachments' on-disk filenames anyway. Discovery is therefore DB-driven by necessity: MiscIndexer.py cross-references smf_attachments.fileext against what's actually present on disk (matched by numeric id_attach prefix), and only indexes attachments that exist in both places.
python3 MiscIndexer.py [options]| Flag | Default | Description |
|---|---|---|
--attach-dir DIR |
auto-detected (<board root>/attachments) |
Directory to scan for text attachments |
--dry-run |
off | Discover + read only — don't write anything to disk |
--limit N |
none | Process only the first N discovered files (for testing) |
--monitor |
off | Print live progress and running stats (unique words so far, in-memory KB) every ~5% |
--verbose |
off | Also stream log output to stdout, not just the log file |
# Standard run — full rebuild, verbose progress
python3 MiscIndexer.py --monitor
# Test on a handful of files without writing anything
python3 MiscIndexer.py --limit 5 --dry-run --monitorAlways prints (regardless of --monitor): attachments matched vs indexed, counts skipped for non-UTF-8 decoding or being empty, a breakdown by file type, total unique words in the index, average word count per file, on-disk size of the written index files, and the log file path. Non-UTF-8 files are skipped and logged, not decoded with a fallback (no latin-1 attempt) — check the log for exact filenames if any show up as skipped.
PDFIndexer.py's resume/checkpoint pattern) if the .md/.txt/ .text/.json attachment count grows substantially.Tokeniser.py module used by post/PDF search — a known simplification, not yet reconciled for exact behavioral parity.#/##... markdown syntax, only for file_type == "md" — always empty for .txt/.text/.json.Searches the index MiscIndexer.py builds. Also searches filename and topic subject (same rationale as PDFsearch.py's title/topic matching) — merged into one result set per attachment, with title-only matches shown alongside content matches rather than silently dropped.
python3 MiscSearch.py QUERY [options]| Flag | Default | Description |
|---|---|---|
--limit N |
none (no cap) | Cap the number of results returned |
--html FILE |
none | Write an HTML results report to this path |
--source {ui,ssh} |
ssh |
Same purpose as PDFsearch.py's flag — argparse compatibility for the bridge's --source=ui; ise_trace.py detects UI vs SSH by checking sys.argv directly |
# Console search
python3 MiscSearch.py listatt
# Full HTML report
python3 MiscSearch.py heroin --html /var/www/html/ISE_Data/misc_results.htmlAs of this session, query terms containing * (any run of characters) or ? (exactly one character) are matched via fnmatch against every key in word_index_misc (content) and every filename/topic subject (title search), instead of the normal exact-match/substring lookup. A term with no wildcard characters still takes the original fast path — wildcards only trigger the full-vocabulary scan when present. The query tokenizer regex (_TOKEN_RE) was extended to preserve */? so they survive tokenisation — the content tokenizer in MiscIndexer.py was left unchanged, since indexed file content never legitimately contains literal wildcard characters worth preserving.
Entries with filename_only: true in misc_store.json (see MiscIndexer.py above) render with a "📁 Filename/type match only" badge in place of a content snippet — they can only ever appear via search_filename_topic(), never via content search, since nothing of theirs is in the word index.
.md/.txt file is one document, not a multi-page PDF, so each result is a single card with a content snippet around the matched term, not a list of matching page numbers.MiscIndexer.py already resolved and stored the owning topic/message/board context at index time, so "Jump to post" links are built straight from misc_store.json with no extra query.0 in that summary rather than a standalone early "No matches." message, matching the pattern MiscIndexer.py already used.Colors are isolated into a single :root { --var: ... } CSS block at the top of the generated HTML, matching the pattern used for PDF search results — intended so a theme can be swapped in (e.g. by reskinning the :root block) without touching the card markup/logic underneath it.
Same gap and fix as PDFsearch.py/search.py: UI-triggered searches are syslogged by qf_Miscsearch_bridge.php before this script runs; SSH-triggered runs syslog the query themselves (tag ise-misc-search-ssh, message prefixed [SSH]) so both paths are consistently logged.
For the user-facing summary of what each quoting style does per source, see the shared help page rather than duplicating it here: <boardurl>/The_ISE_Project/ISE_help.html (or type /help in any search box). What follows is the code-level "why" — the actual functions and matching paths involved, for anyone debugging or extending this.
search.py (ISE posts) is the only one of the three search sources with a real quoting/phrase concept — PDFsearch.py and MiscSearch.py have no equivalent at all (see each script's Wildcard search note above for how they actually handle a literal quote character passed to them: PDFsearch.py's .split() glues quotes onto the token, breaking the match; MiscSearch.py's tokenizer regex silently drops quote characters entirely).
For ISE (posts), the pipeline is QueryParser.parse() → Tokeniser.tokenise() → ANDMatcher.match() → IndexLookup.py's lookup_word()/lookup_phrase():
WORD token. Lowercased, looked up directly against word_index.json (fast path) if it's a plain \w+ string."double quotes" — parsed as a real PHRASE token. Matched via lookup_phrase(): an exact, case-sensitive, literal substring search against the raw post text (phrase in post["text"]) — whitespace and punctuation inside the quotes must match exactly.'single quotes' — not special-cased anywhere in this pipeline. QueryParser.py's regex only recognizes "..." for phrases; a single-quoted word is caught by the generic \S+ word pattern including its quote characters. Tokeniser.py's docstring claims WORD terms get "punctuation stripped," but the actual code doesn't strip anything — it only lowercases. So 'test' becomes the literal token 'test', which isn't \w+, so IndexLookup.lookup_word() falls through to its punctuation-aware path: a case-insensitive literal substring search for 'test', apostrophes included. In practice this means single-quoting a word in ISE search will almost always return nothing, since real post text essentially never contains that exact quoted form. This is a real, confirmed gap, not a design choice — worth fixing (either treat single quotes like double quotes, or strip them) if/when the /help page's quote-behavior explanation needs to describe intended rather than actual behavior.claude.ai) — same lookup_word() fallback as above: case-insensitive literal substring match against post text, requiring that exact adjacent string — not claude and ai matched as separate words elsewhere in the post./help and the footer link)ISE_help.html is a single static file living in The_ISE_Project/ (same directory as all three search scripts, web-accessible via the .htaccess <Files "*.html"> rule) — the single shared source that all three search UIs point to, covering exactly the quote/wildcard behavior documented above.
Each of search.py, PDFsearch.py, and MiscSearch.py defines the same pair of names near its top:
HELP_TRIGGER = "/help"
HELP_HTML_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), "ISE_help.html")
def _serve_help(html_path):
with open(HELP_HTML_PATH, "r", encoding="utf-8") as f:
content = f.read()
if html_path:
with open(html_path, "w", encoding="utf-8") as f:
f.write(content)
return TrueAt the very top of each script's argument handling, before any real search logic runs, the query is checked against HELP_TRIGGER case/whitespace-insensitively (query.strip().lower() == HELP_TRIGGER). On a match, _serve_help() copies ISE_help.html's content directly into whatever --html path the PHP bridge passed — since every bridge already just readfile()s that path (see qf_search_bridge.php/ qf_PDFsearch_bridge.php/qf_Miscsearch_bridge.php), no PHP changes were needed at all. The console/SSH path (no --html) just prints the shared file's own path instead of writing anywhere. Both paths log via ise_trace.log() and, for SSH runs, call ise_trace.new_trace() same as a normal search would.
ISE_help.html carries its own copy of ResultFormatter.py's 13-theme CSS block and switcher, and on load reads whichever of ise_search_theme/ise_pdf_theme/ise_misc_theme is set in localStorage, so it opens already matching whichever theme was last picked on any results page.
Separately, ResultFormatter.py's format_html(), PDFsearch.py's write_html(), and MiscSearch.py's write_html() each got a small permanent footer link — <a href="{BASE_URL}/The_ISE_Project/ISE_help.html"> — added just before </body>, so the help page is reachable from every normal results page too, not only via the /help keyword.
*/?) support for ISE (posts) — PDFsearch.py and MiscSearch.py have it (see their Wildcard search sections above); search.py doesn't yet, since its matching runs through the modules documented above rather than a direct dict lookup.pip install --upgrade move past these without re-testing:
pymupdf==1.19.2 (later versions have no cp36 wheel)pypdf==3.1.0 (5.x uses typing.Protocol, unavailable before Python 3.8, and will crash immediately on import)mysql-connector-python — used by MiscIndexer.py/MiscSearch.py for the DB-driven discovery/context-resolution join$boardurl) from Settings.php via ISE_ROOT — no separate config file or hardcoded credentials anywhere.ise_trace.py: shared logging/tracing module imported by all three search scripts (search.py, PDFsearch.py, MiscSearch.py) and both the post/Misc indexers. On an SSH-triggered run, new_trace() starts a fresh block in the shared ISE_Data/ise_trace.log; a UI-triggered run (detected via the bridge's --source=ui) continues the existing request's trace instead. MiscIndexer.py always starts a fresh trace block regardless, since it's manual/SSH-only — no bridge ever invokes it. All scripts wrap main() in a top-level exception handler that logs the full traceback via ise_trace.log_exception() before exiting non-zero.ise_trace.py's log is detailed pipeline tracing for debugging; syslog (facility LOCAL0) is the actual search-query audit trail reviewed via daily_queries.py/ query_log_reader.py. Historically only UI-triggered searches reached syslog (via the PHP bridge/Search.php) — SSH-triggered runs of any of the three search scripts silently never syslogged at all until this was caught and fixed across all three.