📝 ISE_Python_Scripts_Manual.mdv4.5 · 2026-09-28

ISE Project — Python Scripts Manual

Covers every Python script currently in /var/www/html/The_ISE_Project/. There are now five search sources (ISE/posts, ISEpdf, ISEmisc, ISEmega, ISEmedia) built on two different foundations — worth knowing before debugging any of them:

PDFIndexer.py

Builds the PDF search index: discovers every real PDF attachment on disk (via magic-byte detection, not the DB's stored file extension, which can be wrong), extracts text from each, tokenises it, and writes two files:

Extraction engines

Runs two independent extractors on every page and takes the union of whatever words each one finds — a word either engine catches makes it into the index:

A third engine, pdfplumber, was tried and rejected — its dependency chain requires compiling cryptography, which needs a Rust toolchain not available on this CentOS 6 box. pypdfium2 was also tried and rejected — no compatible wheel exists for this platform at any version.

Usage

python3 PDFIndexer.py [options]

Options

Flag Default Description
--attach-dir DIR auto-detected (<board root>/attachments) Directory to scan for PDF attachments
--dry-run off Discover + extract only — don't write anything to disk
--limit N none Process only the first N discovered PDFs (for testing)
--monitor off Print live progress — which engine is running per file, new words added per file, and a running summary every 5%
--verbose off Also stream log output to stdout, not just the log file
--fresh off Ignore any existing index on disk and rebuild everything from scratch. Default behaviour without this flag is to resume: load the existing index and skip attachments already indexed
--checkpoint-every N 25 Save progress to disk every N processed PDFs, so a crash (e.g. an OOM kill) only loses recent work rather than the whole run. Set to 0 to disable and only save once at the end
--start-at ID_ATTACH none Skip every attachment with id_attach below this value — for manually restarting a run from a known point, independent of the resume/skip logic
--fast off PyMuPDF only — skips pypdf entirely. ~20x faster per file and noticeably lighter on memory, at the cost of losing pypdf's independent-parser coverage from the union. Good for quick reindexes or spot-checks

Examples

# Standard run — thorough (both engines), resumes automatically if re-run
python3 PDFIndexer.py --monitor

# Quick single-engine pass
python3 PDFIndexer.py --fast --monitor

# Full rebuild from scratch, ignoring whatever's already indexed
python3 PDFIndexer.py --fresh --monitor

# Resume a run that was killed partway, starting from a known attachment ID
python3 PDFIndexer.py --start-at 5235 --monitor

# Test on a handful of files without writing anything
python3 PDFIndexer.py --limit 5 --dry-run --monitor

Notes

PDFsearch.py

Searches the index PDFIndexer.py builds, plus (as of the filename/topic feature) scans every PDF attachment's own filename and its linked forum topic's subject line — so a document titled with the right words surfaces even if those words never appear in the extracted PDF text itself.

Usage

python3 PDFsearch.py QUERY [options]

QUERY is a required positional argument — the search terms.

Options

Flag Default Description
--limit N none (no cap) Cap the number of results returned. Omit for every match — useful to pass e.g. --limit 20 for a quick SSH sanity check
--html FILE none Write an HTML results report to this path, in addition to (or instead of) console output
--source {ui,ssh} ssh Present so argparse accepts the literal --source=ui flag the web UI bridge passes. ise_trace.py detects UI vs SSH itself by checking sys.argv, independent of this flag

Examples

# Console search, every match
python3 PDFsearch.py listatt

# Capped, for a quick check over SSH
python3 PDFsearch.py heroin --limit 20

# Full HTML report
python3 PDFsearch.py heroin --html /var/www/html/ISE_Data/results.html

What a result can look like

Wildcard search

As of this session, query terms containing * or ? are matched via fnmatch (case-sensitive, matching this script's existing case-sensitive exact-term lookup) against every key in word_index_pdf and every filename/topic subject, instead of the normal exact-match/ substring lookup. A term with no wildcard characters keeps the original fast path. Query terms come through tokenise()'s plain text.split() here — wildcard characters survive that unchanged since .split() doesn't strip anything, unlike MiscSearch.py's regex tokenizer, which needed its pattern extended to preserve them.

syslog note

UI-triggered searches (via qf_PDFsearch_bridge.php) are syslogged by the PHP bridge before this script even runs. SSH-triggered runs bypass that PHP layer entirely — without a fix, they'd never reach syslog at all, only ise_trace.py's separate log. PDFsearch.py now syslogs the query itself (tag ise-pdf-search-ssh, message prefixed [SSH]) when it detects it's running via SSH, so both paths end up logged consistently — this same gap existed in search.py and was fixed there too.

Fixed bug (confirmed 2026-09-23, fixed in v7.5.2): zero-match return shape

get_ranked_results() returns a bare [] on a genuine zero-match search (when both content_results and title_hits are empty) — but the function's own documented contract, and every real caller (main() here, and MegaSearch.py's PDFsearch.get_ranked_results(...) call), unpacks it as a 2-tuple: results, order_mode = get_ranked_results(...). A genuine zero-match PDF search therefore raises ValueError: not enough values to unpack (expected 2, got 0), not the clean "No matches." path MiscSearch.py/search.py take for the same situation.

Real-world impact, traced end-to-end:

Fix applied in v7.5.2: the zero-match early-return in get_ranked_results() now returns return [], order_mode instead of a bare return [], restoring the documented 2-tuple contract. Delivered as a corrected file; deployment timing to Live is Andrew's call, per standing policy.

orphan_check.py — attachment/post integrity report

A standalone diagnostic script, not part of the search UI or any of the five search sources' dropdowns — an admin/maintenance tool meant to run on a schedule (cron) rather than by hand each time.

Finds attachments whose id_msg no longer resolves to a real row in smf209_messages — i.e. the post an attachment was uploaded against is gone (deleted, moved, or the upload never got attached to a finished post). Deliberately excludes avatars: SMF stores profile avatars as attachments with id_msg=0 by design, which is normal, not orphaning, and would otherwise swamp every report with roughly 360 harmless entries (confirmed on this install).

Keeps a small state file, ISE_Data/orphan_state.json, so each run reports what's new since the last run instead of re-listing the same known orphans every week.

Usage:

python3 orphan_check.py                  # text report to stdout
python3 orphan_check.py --html out.html   # HTML report
python3 orphan_check.py --quiet-if-empty  # print nothing when no new orphans (good for cron)

Fixed bug (confirmed 2026-09-24, fixed in v7.4.2): broken import

Every invocation of this script — CLI, --html, and cron — failed immediately with:

ImportError: cannot import name 'BOARD_URL' from 'PDFsearch'

Root cause: PDFsearch.py's own settings-loading was refactored onto ise_settings.py (v4.0, 2026-08-21) two days after orphan_check.py's last edit (2026-08-19). That refactor renamed PDFsearch.py's public base-URL constant from BOARD_URL to BASE_URL (now sourced from ise_settings.get_base_url()); orphan_check.py was never updated for the rename and kept importing the old name.

Fix applied in v7.4.2: BOARD_URL was never actually referenced anywhere else in this file (confirmed — it builds no links and does nothing with a base URL), so rather than re-import it under its new name only to leave it unused, it was simply dropped from the import line, leaving from PDFsearch import DB_CONFIG, ISE_ROOT — both of which this file does use. Delivered as a corrected file; deployment timing to Live is Andrew's call, per standing policy.

pdf_duplicate_spotter.py

Finds byte-identical PDF duplicates in the attachments directory — the same file content re-uploaded under a different id_attach, usually with a different filename and/or topic. Not near-duplicate detection (a different scan of the same book, an edited copy, etc.) — that's a separate, harder problem this doesn't attempt.

Reuses existing infrastructure rather than reimplementing it: discover_pdfs()/ATTACH_DIR from PDFIndexer.py for the same magic-byte PDF scan already used for indexing, and DB_CONFIG/get_topic_for_attachment()/build_download_url() from PDFsearch.py for the same DB lookups/link-building already used there.

Method: SHA-256 over each file's actual bytes, streamed in 1MB chunks (2GB-RAM box constraint — never loads a whole PDF into memory). Files sharing a hash are byte-identical, full stop — no false positives possible from this method, and correspondingly no way to catch a duplicate that isn't byte-identical (see PIHKAL/TIHKAL note below).

Usage

python3 pdf_duplicate_spotter.py [options]

Options

Flag Default Description
--attach-dir DIR ATTACH_DIR (from PDFIndexer.py) Directory to scan for PDFs
--json PATH ISE_Data/pdf_duplicates.json (via ISE_ROOT, same sibling-of-The_ISE_Project convention as every other script's data files) Write the full report as JSON here — this is what pdf_dupe_review.php reads, so leave it at the default unless you're also pointing that page somewhere else

Examples

# Standard run — writes to the default path pdf_dupe_review.php expects
python3 pdf_duplicate_spotter.py

# Custom output location
python3 pdf_duplicate_spotter.py --json /tmp/dupes.json

Companion page: pdf_dupe_review.php

Not a Python script (out of this manual's stated scope, noted here for findability) — an admin-only page in The_ISE_Project/ that reads this script's --json output and lists every duplicate copy as two plain links: View post (opens the actual forum post) and Delete post (single click, JS-confirmed, SMF session-token guarded, deletes the whole post via SMF's own removeMessage() — which also cleans up that post's attachment as part of the same call). Backs up the message row to ise_pdf_dupe_backup_v2 before deleting (metadata only, not the PDF bytes — a forensic record, not a one-click undo, since SMF reassigns ids on delete) and remembers deleted ids in ISE_Data/pdf_dupe_deleted.json so they stop appearing without needing to rerun the spotter.

Notes

MiscIndexer.py

Builds the search index for text-based attachments: .md, .txt, .text, .json, and .html (full content, tokenised and indexed). Simpler than PDFIndexer.py in a few key ways — these are already plain text, so there's no extraction step, no multi-engine union, and no resume/checkpoint logic (every run is a full fresh rebuild, since the corpus is expected to stay small).

.html gets one extra step before tokenising: strip_html_for_indexing() removes <script>/<style> blocks (tag and content together), strips every remaining tag, and decodes HTML entities (&amp; → &, etc.) via re-based regexes — deliberately simple, not a full HTML parser, good enough for indexing purposes only. This keeps markup (div, class, href, inline JS/CSS) out of word_index_misc.json — only visible page text is indexed. misc_store.json's content field for .html entries holds the stripped text (so search snippets are readable), but the checksum field is still computed on the original, unstripped file.

MiscIndexer.py also discovers a much wider set of code/config attachment types — .py, .php, .js, .sh, .sql, .css, .c, .cpp, .java, .rb, .go, .rs, .pl, .xml, .yml, .yaml, .ini, .conf — but treats them as filename/type-only: FILENAME_ONLY_EXTENSIONS skips read_text_file() and tokenise() entirely for these, so nothing they contain enters word_index_misc.json. Their misc_store.json entry has content: "", zero word counts, and filename_only: true — they're viewable via the "Open file" download link and filename/topic searchable, but never full-text searched. .html is deliberately not in this set — it's mostly prose with markup wrapped around it, so it gets full content indexing (above) instead.

Discovery

Unlike PDFIndexer.py's magic-byte scan, plain-text files have no reliable magic number to sniff — and SMF strips real file extensions from attachments' on-disk filenames anyway. Discovery is therefore DB-driven by necessity: MiscIndexer.py cross-references smf_attachments.fileext against what's actually present on disk (matched by numeric id_attach prefix), and only indexes attachments that exist in both places.

Usage

python3 MiscIndexer.py [options]

Options

Flag Default Description
--attach-dir DIR auto-detected (<board root>/attachments) Directory to scan for text attachments
--dry-run off Discover + read only — don't write anything to disk
--limit N none Process only the first N discovered files (for testing)
--monitor off Print live progress and running stats (unique words so far, in-memory KB) every ~5%
--verbose off Also stream log output to stdout, not just the log file

Examples

# Standard run — full rebuild, verbose progress
python3 MiscIndexer.py --monitor

# Test on a handful of files without writing anything
python3 MiscIndexer.py --limit 5 --dry-run --monitor

End-of-run summary

Always prints (regardless of --monitor): attachments matched vs indexed, counts skipped for non-UTF-8 decoding or being empty, a breakdown by file type, total unique words in the index, average word count per file, on-disk size of the written index files, and the log file path. Non-UTF-8 files are skipped and logged, not decoded with a fallback (no latin-1 attempt) — check the log for exact filenames if any show up as skipped.

Notes

MiscSearch.py

Searches the index MiscIndexer.py builds. Also searches filename and topic subject (same rationale as PDFsearch.py's title/topic matching) — merged into one result set per attachment, with title-only matches shown alongside content matches rather than silently dropped.

Usage

python3 MiscSearch.py QUERY [options]

Options

Flag Default Description
--limit N none (no cap) Cap the number of results returned
--html FILE none Write an HTML results report to this path
--source {ui,ssh} ssh Same purpose as PDFsearch.py's flag — argparse compatibility for the bridge's --source=ui; ise_trace.py detects UI vs SSH by checking sys.argv directly

Examples

# Console search
python3 MiscSearch.py listatt

# Full HTML report
python3 MiscSearch.py heroin --html /var/www/html/ISE_Data/misc_results.html

Wildcard search

As of this session, query terms containing * (any run of characters) or ? (exactly one character) are matched via fnmatch against every key in word_index_misc (content) and every filename/topic subject (title search), instead of the normal exact-match/substring lookup. A term with no wildcard characters still takes the original fast path — wildcards only trigger the full-vocabulary scan when present. The query tokenizer regex (_TOKEN_RE) was extended to preserve */? so they survive tokenisation — the content tokenizer in MiscIndexer.py was left unchanged, since indexed file content never legitimately contains literal wildcard characters worth preserving.

Filename-only results

Entries with filename_only: true in misc_store.json (see MiscIndexer.py above) render with a "📁 Filename/type match only" badge in place of a content snippet — they can only ever appear via search_filename_topic(), never via content search, since nothing of theirs is in the word index.

Differences from PDFsearch.py

HTML output / theming

Colors are isolated into a single :root { --var: ... } CSS block at the top of the generated HTML, matching the pattern used for PDF search results — intended so a theme can be swapped in (e.g. by reskinning the :root block) without touching the card markup/logic underneath it.

syslog note

Same gap and fix as PDFsearch.py/search.py: UI-triggered searches are syslogged by qf_Miscsearch_bridge.php before this script runs; SSH-triggered runs syslog the query themselves (tag ise-misc-search-ssh, message prefixed [SSH]) so both paths are consistently logged.

MegaSearch.py

Queries search.py, PDFsearch.py, and MiscSearch.py's get_ranked_results() directly (no subprocess, no re-searching through their CLI layers) and merges the three result lists into one page. Does not include ISEmedia — despite the name, ISEmega is "Posts + PDF + Misc, merged," not "everything." ISEmedia is a fifth, separate, standalone search source with its own dropdown option; nothing currently feeds Media results into a Mega search. Worth knowing plainly since the name invites the assumption otherwise.

Each of the three source calls is individually wrapped in its own try/except Exception, logged via ise_trace.log() as e.g. "MegaSearch.py: pdf source failed" and left as an empty list rather than aborting the whole search — so one source being unavailable (a DB connection failure, an index file missing) degrades gracefully instead of taking out the other two. Known caveat, not yet fixed: that same try/except also catches a completely normal zero-match PDF result (see PDFsearch.py's "Known bug" note below) — a genuine "PDF just didn't match anything" case gets logged identically to a real failure ("pdf source failed"), which is misleading in the trace log even though the merged Mega result set itself still comes out correct (posts/misc results are unaffected). Fixing the root cause in PDFsearch.py would also clean up this log noise.

Merge strategy

Two different merge strategies depending on order_mode, not one merge with a sort bolted on afterward:

The /sort/#ise-order- directive is stripped from the query once, in get_mega_results(), before any of the three sources sees it — so all three always search on an already-clean query and apply their own default (relevance) sort internally; the redundant per-source sort when a date directive is active is intentional, it guarantees the three sources can never land on inconsistent per-source order_modes.

Result cap — the one source that actually has a default cap

Unlike search.py/PDFsearch.py/MiscSearch.py on their own (all effectively unlimited via the UI — see each script's own --limit note above; none of their bridges pass --limit), MegaSearch.py is the one source with a real default result cap:

MEGA_SOFT_CAP = 180
MEGA_PAGE_SIZE = None   # argparse default for --limit

run()'s cap precedence: an explicit --limit N always wins; else --all (or &all=1 on the page) bypasses the cap entirely; else MEGA_SOFT_CAP (180) applies. Since qf_MegaSearch_bridge.php passes neither --limit nor --all, every UI-triggered Mega search is capped at 180 displayed results by default — everything is still searched and ranked first, the cap only limits how many result cards get drawn, and the on-page notice says so with a "Show all" link.

Usage

python3 MegaSearch.py "your query here" [--html FILE] [--limit N] [--all] [--source {ui,ssh}]

Card rendering — delegates rather than duplicates

_render_posts_card() delegates to ResultFormatter.py's real _render_cluster_item() (wrapping each flat post result as a single-item pseudo-cluster) rather than maintaining a hand-copied version — this is a deliberate fix for how the Formatted/Raw buttons and ranking explainer drifted out of sync the first time this was tried as a separate implementation. PDF/Misc cards are Mega's own rendering.

MediaIndexer.py / MediaProcessor.py — building the ISEmedia index

Two scripts, not one — indexing ISEmedia is a two-stage pipeline, unlike the single-script indexers for the other three sources:

Both scripts, shared traits

Versioning — a different convention from the rest of ISE

Both scripts' docblocks carry a version tag that looks like an overall project version ("ISE v9.7" for MediaIndexer.py, "ISE v9.18" for MediaProcessor.py" as of this writing) plus a RUNTIME_VERSION string kept manually in sync with it — this is a **separate, Collabware-native convention**, not the same thing as every other ISE module's own per-file semver (search.py v7.5.2, MediaSearch.py v1.2.0, etc.). Don't read "ISE v9.7" here as meaning the same thing as the package's own "Version 9.5" (install_ise.php`) — they're two different counters from two different codebases that happen to share a project.

MediaSearch.py — ISEmedia search

v1.7.1 (2026-09-28, DM2S/NX v9.8). Keyword search over images, video and audio. No model is loaded at search time. The AI work (CLIP embeddings, SmolVLM captions, CLAP audio tags) is done at index time by MediaProcessor.py / AudioProcessor.py. CLIP re-ordering stays available as an opt-in (--clip).

What it searches

Candidates are every attachment in media_store.json (MediaIndexer's catalogue, including .mp3/.wav), plus anything in captions_store.json and audio_store.json. A search term is checked against five fields:

Field Source Match rule
Filename DB a.filename (else media_store) Whole string. A plain word is a substring; a wildcard must fit the whole name
Internal filename <id_attach>_<file_hash> from media_store Same as filename. Omitted when file_hash is empty
Post subject DB m.subject, html-unescaped Same as filename
Caption captions_store (image, each video frame <id>_t<sec>), audio_store item + windows Whole words. The best-matching frame or window is shown
Tags audio_store item tags ([[keyword, prob, group], ...], keyword used) Whole words

The matching rules are the same as MiscSearch.py's:

Score = the sum of field weights per distinct term hit: caption 40, tags 40, filename 25, internal 25, subject 15. Each result carries match_evidence (term → field → value), which is shown on every card. Avatars are dropped (v1.4.0 rule).

Internal filename (v1.7.1)

The internal name <id_attach>_<file_hash> uses file_hash from media_store.json when it's there. Otherwise it uses a.file_hash from the same DB query that fetches topics. An item known only to audio_store.json or captions_store.json is still found by 6998_*.

Tempo and key filters (v1.7.0)

bpm:124 (±1.5), bpm:122-126, and key:Am / key:A-minor / key:F# / key:Gb-major are pulled out of the query before tokenising (parse_filters()).

Features come from audio_store.json (AudioProcessor ≥ 1.1.0): bpm, bpm_range, key, lufs, lra and changes per item, plus bpm and key per window. Cards show 118-128 BPM (124), A minor, -9.8 LUFS, ~14 tracks and the approximate track changes.

Usage

python3.8 MediaSearch.py QUERY [--limit N] [--all] [--html FILE] [--clip] [--source {ui,ssh}]
python3.8 MediaSearch.py '*.png'
python3.8 MediaSearch.py '6204_*'
python3.8 MediaSearch.py '"brown horse"'
python3.8 MediaSearch.py '/sort newest *.mp3'

QUERY is taken from argv[0] before argparse sees it, the same as in the other search scripts.

Flag Default Description
--limit N none Show exactly the best N
--all off Draw every match (bypass the soft cap). The bridge passes it for &all=1
--html FILE none Write the results page. Draws at most MEDIA_SOFT_CAP = 180 cards unless --all/--limit is given. When it caps, a red "Showing 180 of N — Show all" notice appears (same as ISEmega)
--clip off Load CLIP and add similarity×150 to image/video matches. Doesn't change which items match. Reads embeddings_store.json, which nothing else here reads
--keyword-only — No-op, kept for compatibility
--source {ui,ssh} ssh Same as the other search scripts

The CLI output ends with a Summary: line. An optional store that exists but can't be read is reported with [!] on stderr and as a red line on the results page, and the search carries on without it.

API (for MegaSearch.py)

get_ranked_results(query, limit=None, keyword_only=True) returns (results, order_mode, clip_note, warnings). It was a 3-tuple before v1.6.0. Importing the module no longer imports torch.

Bridge

qf_Mediasearch_bridge.php v1.1.0 no longer passes --limit 180, which silently truncated results. It passes --all when &all=1 is in the URL. It still resolves python3.8 and sets HF_HOME; that's only needed for --clip.

Ranking engine

RankingEngine.py has no media_ranking_engine() (confirmed again against the v9.8 source), so v1.6.0 removed the dead import. Media scoring is the field-weight sum above, which is simple and inspectable.

syslog / help / theming

These are unchanged: /help serves the shared ISE_help.html (its ISEmedia section is updated for v1.6.0), ise_trace logging runs throughout, and the theme key is ise_media_theme.

Query parsing internals (search.py / QueryParser.py / Tokeniser.py / ANDMatcher.py / IndexLookup.py)

For the user-facing summary of what each quoting style does per source, see the shared help page rather than duplicating it here: <boardurl>/The_ISE_Project/ISE_help.html (or type /help in any search box). What follows is the code-level "why" — the actual functions and matching paths involved, for anyone debugging or extending this.

search.py (ISE posts) is the only one of the three search sources with a real quoting/phrase concept — PDFsearch.py and MiscSearch.py have no equivalent at all (see each script's Wildcard search note above for how they actually handle a literal quote character passed to them: PDFsearch.py's .split() glues quotes onto the token, breaking the match; MiscSearch.py's tokenizer regex silently drops quote characters entirely).

For ISE (posts), the pipeline is QueryParser.parse() → Tokeniser.tokenise() → ANDMatcher.match() → IndexLookup.py's lookup_word()/lookup_phrase():

SortEngine.py (shared /sort directive parsing)

Built in v9.3.0. A small shared module — extract_order_mode(query, valid_modes, default) — used by search.py, PDFsearch.py, MiscSearch.py, and MegaSearch.py so the /sort <mode> and #ise-order-<mode> query-string syntax is recognised identically everywhere, instead of each module hand-rolling its own copy of the same regex (the old version of this file predated RankingEngine.py/ ISEmega entirely and was dead code — nothing in the current pipeline called it).

def extract_order_mode(query: str, valid_modes: set, default: str) -> tuple:
    # strips a /sort <mode> or #ise-order-<mode> directive out of query,
    # resolving it against the caller's own valid_modes set.
    # Returns (cleaned_query, order_mode).

/sort is checked first (the documented, user-facing syntax); #ise-order- is the older internal syntax, still recognised for back-compat. If both are present in one query, /sort wins. A mode name not in the caller's valid_modes is left in the query untouched — it just fails to match anything as a search term, same as any other typo, rather than silently vanishing with no explanation.

Where it's called, and with what valid_modes:

Caller valid_modes Notes
search.py (ISE/posts) relevance, score, date, newest, oldest, newmod, oldmod, subject, board, author, occurrences Full ORDER_KEYS dict; date is a legacy alias for newest. Called from inside get_ranked_results() itself (not just run()) so MegaSearch.py's direct call gets directive parsing too — before this, a /sort directive routed through ISEmega leaked into the query as literal search terms.
PDFsearch.py relevance, score, newest, oldest No newmod/oldmod — see below.
MiscSearch.py relevance, score, newest, oldest Same.
MegaSearch.py relevance, score, newest, oldest Stripped once in get_mega_results(), before any of the three sources is queried, so all three see an already-clean query and never re-derive the directive independently. Does not cover ISEmedia — see MegaSearch.py section above.
MediaSearch.py relevance, score, newest, oldest Same shape as PDF/Misc — no newmod/oldmod (attachments carry poster_time only). The real difference for Media isn't in this parsing step at all — it's that a non-empty query also gates results by literal keyword match before any ORDER_KEYS sort runs (v1.2.0), which none of the other four sources need to do here since their own word-index lookup already did that filtering upstream. Note: SortEngine.py's own module docblock (v8.0, 2026-09-14) only names search.py, PDFsearch.py, and MiscSearch.py as callers in its "Used by" line — stale as of MediaSearch.py's build (2026-09-23); the real import (from SortEngine import extract_order_mode) confirms Media uses it too, the docblock just predates that.

Why newmod/oldmod is ISE (posts)-only: PDFsearch.py/ MiscSearch.py results carry poster_time only — PDF/Misc attachments aren't edited the way a forum post is, so there's no modified_time field in pdf_store.json/misc_store.json to sort by. search.py's ORDER_KEYS["newmod"]/["oldmod"] fall back to poster_time for any post that's never been edited (_effective_modified()), matching ISE_help.html's documented behavior exactly.

ISEmega's date sort is a different merge, not a re-sort: ise_render.py gained merge_by_date() alongside its existing interlace_results(). MegaSearch.get_mega_results() picks one or the other based on order_mode — interlace_results() (round-robin, unchanged) for relevance/score, merge_by_date() (flat poster_time sort across all three tagged lists) for newest/oldest. These are genuinely different strategies, not the same merge with a final re-sort bolted on: interlacing exists specifically to stop one source dominating a relevance-score merge, which isn't a concern once you're sorting by date — so a date sort bypasses interlacing entirely.

In-app help (/help and the footer link)

ISE_help.html is a single static file living in The_ISE_Project/ (same directory as all five search scripts, web-accessible via the .htaccess <Files "*.html"> rule) — the single shared source all five search UIs point to. It now also documents ISEmega and ISEmedia (added this audit — both were previously undocumented there despite ISEmega having existed for some time and being partially wired into the page's /sort table already).

search.py, PDFsearch.py, and MiscSearch.py each define the same pair of names near its top (MegaSearch.py reuses search.py's copy directly rather than redefining it — see its own /help short-circuit; MediaSearch.py defines its own, same shape):

HELP_TRIGGER = "/help"
HELP_HTML_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), "ISE_help.html")

def _serve_help(html_path):
    with open(HELP_HTML_PATH, "r", encoding="utf-8") as f:
        content = f.read()
    if html_path:
        with open(html_path, "w", encoding="utf-8") as f:
            f.write(content)
    return True

At the very top of each script's argument handling, before any real search logic runs, the query is checked against HELP_TRIGGER case/whitespace-insensitively (query.strip().lower() == HELP_TRIGGER). On a match, _serve_help() copies ISE_help.html's content directly into whatever --html path the PHP bridge passed — since every bridge already just readfile()s that path (see qf_search_bridge.php/ qf_PDFsearch_bridge.php/qf_Miscsearch_bridge.php), no PHP changes were needed at all. The console/SSH path (no --html) just prints the shared file's own path instead of writing anywhere. Both paths log via ise_trace.log() and, for SSH runs, call ise_trace.new_trace() same as a normal search would.

ISE_help.html carries its own copy of ResultFormatter.py's 13-theme CSS block and switcher, and on load reads whichever of ise_search_theme/ise_pdf_theme/ise_misc_theme is set in localStorage, so it opens already matching whichever theme was last picked on any results page.

Separately, ResultFormatter.py's format_html(), PDFsearch.py's write_html(), and MiscSearch.py's write_html() each got a small permanent footer link — <a href="{BASE_URL}/The_ISE_Project/ISE_help.html"> — added just before </body>, so the help page is reachable from every normal results page too, not only via the /help keyword.

Planned work (not yet built)

Shared dependency notes

No longer one shared runtime — two, as of ISEmedia. Everything below the line was true, and mostly still is, for the original four posts/PDF/misc/mega scripts. ISEmedia's two build-side scripts (MediaIndexer.py, MediaProcessor.py) and its search script (MediaSearch.py) need Python 3.8 specifically — torch/open_clip are only installed under python3.8's site-packages on Live, and will fail to import under the system default. Confirmed live (2026-09-23): running any of the three Media scripts under bare python3 fails on import numpy/import torch before ever reaching real logic. qf_Mediasearch_bridge.php resolves python3.8 explicitly for this reason (see MediaSearch.py section above); install_ise.php v9.5 does the same for MediaIndexer.py's background install-time job. IndexBuilder.py/PDFIndexer.py/MiscIndexer.py are deliberately not touched to require 3.8 — no evidence they need it, and changing their interpreter risks three scripts that already work.