📝 ISE_Python_Scripts_Manual.mdv4.4 · 2026-09-05

ISE Project — Python Scripts Manual

Covers every Python script currently in /var/www/html/The_ISE_Project/. There are now five search sources (ISE/posts, ISEpdf, ISEmisc, ISEmega, ISEmedia) built on two different foundations — worth knowing before debugging any of them:

PDFIndexer.py

Builds the PDF search index: discovers every real PDF attachment on disk (via magic-byte detection, not the DB's stored file extension, which can be wrong), extracts text from each, tokenises it, and writes two files:

Extraction engines

Runs two independent extractors on every page and takes the union of whatever words each one finds — a word either engine catches makes it into the index:

A third engine, pdfplumber, was tried and rejected — its dependency chain requires compiling cryptography, which needs a Rust toolchain not available on this CentOS 6 box. pypdfium2 was also tried and rejected — no compatible wheel exists for this platform at any version.

Usage

python3 PDFIndexer.py [options]

Options

Flag Default Description
--attach-dir DIR auto-detected (<board root>/attachments) Directory to scan for PDF attachments
--dry-run off Discover + extract only — don't write anything to disk
--limit N none Process only the first N discovered PDFs (for testing)
--monitor off Print live progress — which engine is running per file, new words added per file, and a running summary every 5%
--verbose off Also stream log output to stdout, not just the log file
--fresh off Ignore any existing index on disk and rebuild everything from scratch. Default behaviour without this flag is to resume: load the existing index and skip attachments already indexed
--checkpoint-every N 25 Save progress to disk every N processed PDFs, so a crash (e.g. an OOM kill) only loses recent work rather than the whole run. Set to 0 to disable and only save once at the end
--start-at ID_ATTACH none Skip every attachment with id_attach below this value — for manually restarting a run from a known point, independent of the resume/skip logic
--fast off PyMuPDF only — skips pypdf entirely. ~20x faster per file and noticeably lighter on memory, at the cost of losing pypdf's independent-parser coverage from the union. Good for quick reindexes or spot-checks

Examples

# Standard run — thorough (both engines), resumes automatically if re-run
python3 PDFIndexer.py --monitor

# Quick single-engine pass
python3 PDFIndexer.py --fast --monitor

# Full rebuild from scratch, ignoring whatever's already indexed
python3 PDFIndexer.py --fresh --monitor

# Resume a run that was killed partway, starting from a known attachment ID
python3 PDFIndexer.py --start-at 5235 --monitor

# Test on a handful of files without writing anything
python3 PDFIndexer.py --limit 5 --dry-run --monitor

Notes

PDFsearch.py

Searches the index PDFIndexer.py builds, plus (as of the filename/topic feature) scans every PDF attachment's own filename and its linked forum topic's subject line — so a document titled with the right words surfaces even if those words never appear in the extracted PDF text itself.

Usage

python3 PDFsearch.py QUERY [options]

QUERY is a required positional argument — the search terms.

Options

Flag Default Description
--limit N none (no cap) Cap the number of results returned. Omit for every match — useful to pass e.g. --limit 20 for a quick SSH sanity check
--html FILE none Write an HTML results report to this path, in addition to (or instead of) console output
--source {ui,ssh} ssh Present so argparse accepts the literal --source=ui flag the web UI bridge passes. ise_trace.py detects UI vs SSH itself by checking sys.argv, independent of this flag

Examples

# Console search, every match
python3 PDFsearch.py listatt

# Capped, for a quick check over SSH
python3 PDFsearch.py heroin --limit 20

# Full HTML report
python3 PDFsearch.py heroin --html /var/www/html/ISE_Data/results.html

What a result can look like

Wildcard search

As of this session, query terms containing * or ? are matched via fnmatch (case-sensitive, matching this script's existing case-sensitive exact-term lookup) against every key in word_index_pdf and every filename/topic subject, instead of the normal exact-match/ substring lookup. A term with no wildcard characters keeps the original fast path. Query terms come through tokenise()'s plain text.split() here — wildcard characters survive that unchanged since .split() doesn't strip anything, unlike MiscSearch.py's regex tokenizer, which needed its pattern extended to preserve them.

syslog note

UI-triggered searches (via qf_PDFsearch_bridge.php) are syslogged by the PHP bridge before this script even runs. SSH-triggered runs bypass that PHP layer entirely — without a fix, they'd never reach syslog at all, only ise_trace.py's separate log. PDFsearch.py now syslogs the query itself (tag ise-pdf-search-ssh, message prefixed [SSH]) when it detects it's running via SSH, so both paths end up logged consistently — this same gap existed in search.py and was fixed there too.

Fixed bug (confirmed 2026-09-23, fixed in v7.5.2): zero-match return shape

get_ranked_results() returns a bare [] on a genuine zero-match search (when both content_results and title_hits are empty) — but the function's own documented contract, and every real caller (main() here, and MegaSearch.py's PDFsearch.get_ranked_results(...) call), unpacks it as a 2-tuple: results, order_mode = get_ranked_results(...). A genuine zero-match PDF search therefore raises ValueError: not enough values to unpack (expected 2, got 0), not the clean "No matches." path MiscSearch.py/search.py take for the same situation.

Real-world impact, traced end-to-end:

Fix applied in v7.5.2: the zero-match early-return in get_ranked_results() now returns return [], order_mode instead of a bare return [], restoring the documented 2-tuple contract. Delivered as a corrected file; deployment timing to Live is Andrew's call, per standing policy.

orphan_check.py — attachment/post integrity report

A standalone diagnostic script, not part of the search UI or any of the five search sources' dropdowns — an admin/maintenance tool meant to run on a schedule (cron) rather than by hand each time.

Finds attachments whose id_msg no longer resolves to a real row in smf209_messages — i.e. the post an attachment was uploaded against is gone (deleted, moved, or the upload never got attached to a finished post). Deliberately excludes avatars: SMF stores profile avatars as attachments with id_msg=0 by design, which is normal, not orphaning, and would otherwise swamp every report with roughly 360 harmless entries (confirmed on this install).

Keeps a small state file, ISE_Data/orphan_state.json, so each run reports what's new since the last run instead of re-listing the same known orphans every week.

Usage:

python3 orphan_check.py                  # text report to stdout
python3 orphan_check.py --html out.html   # HTML report
python3 orphan_check.py --quiet-if-empty  # print nothing when no new orphans (good for cron)

Fixed bug (confirmed 2026-09-24, fixed in v7.4.2): broken import

Every invocation of this script — CLI, --html, and cron — failed immediately with:

ImportError: cannot import name 'BOARD_URL' from 'PDFsearch'

Root cause: PDFsearch.py's own settings-loading was refactored onto ise_settings.py (v4.0, 2026-08-21) two days after orphan_check.py's last edit (2026-08-19). That refactor renamed PDFsearch.py's public base-URL constant from BOARD_URL to BASE_URL (now sourced from ise_settings.get_base_url()); orphan_check.py was never updated for the rename and kept importing the old name.

Fix applied in v7.4.2: BOARD_URL was never actually referenced anywhere else in this file (confirmed — it builds no links and does nothing with a base URL), so rather than re-import it under its new name only to leave it unused, it was simply dropped from the import line, leaving from PDFsearch import DB_CONFIG, ISE_ROOT — both of which this file does use. Delivered as a corrected file; deployment timing to Live is Andrew's call, per standing policy.

pdf_duplicate_spotter.py

Finds byte-identical PDF duplicates in the attachments directory — the same file content re-uploaded under a different id_attach, usually with a different filename and/or topic. Not near-duplicate detection (a different scan of the same book, an edited copy, etc.) — that's a separate, harder problem this doesn't attempt.

Reuses existing infrastructure rather than reimplementing it: discover_pdfs()/ATTACH_DIR from PDFIndexer.py for the same magic-byte PDF scan already used for indexing, and DB_CONFIG/get_topic_for_attachment()/build_download_url() from PDFsearch.py for the same DB lookups/link-building already used there.

Method: SHA-256 over each file's actual bytes, streamed in 1MB chunks (2GB-RAM box constraint — never loads a whole PDF into memory). Files sharing a hash are byte-identical, full stop — no false positives possible from this method, and correspondingly no way to catch a duplicate that isn't byte-identical (see PIHKAL/TIHKAL note below).

Usage

python3 pdf_duplicate_spotter.py [options]

Options

Flag Default Description
--attach-dir DIR ATTACH_DIR (from PDFIndexer.py) Directory to scan for PDFs
--json PATH ISE_Data/pdf_duplicates.json (via ISE_ROOT, same sibling-of-The_ISE_Project convention as every other script's data files) Write the full report as JSON here — this is what pdf_dupe_review.php reads, so leave it at the default unless you're also pointing that page somewhere else

Examples

# Standard run — writes to the default path pdf_dupe_review.php expects
python3 pdf_duplicate_spotter.py

# Custom output location
python3 pdf_duplicate_spotter.py --json /tmp/dupes.json

Companion page: pdf_dupe_review.php

Not a Python script (out of this manual's stated scope, noted here for findability) — an admin-only page in The_ISE_Project/ that reads this script's --json output and lists every duplicate copy as two plain links: View post (opens the actual forum post) and Delete post (single click, JS-confirmed, SMF session-token guarded, deletes the whole post via SMF's own removeMessage() — which also cleans up that post's attachment as part of the same call). Backs up the message row to ise_pdf_dupe_backup_v2 before deleting (metadata only, not the PDF bytes — a forensic record, not a one-click undo, since SMF reassigns ids on delete) and remembers deleted ids in ISE_Data/pdf_dupe_deleted.json so they stop appearing without needing to rerun the spotter.

Notes

MiscIndexer.py

Builds the search index for text-based attachments: .md, .txt, .text, .json, and .html (full content, tokenised and indexed). Simpler than PDFIndexer.py in a few key ways — these are already plain text, so there's no extraction step, no multi-engine union, and no resume/checkpoint logic (every run is a full fresh rebuild, since the corpus is expected to stay small).

.html gets one extra step before tokenising: strip_html_for_indexing() removes <script>/<style> blocks (tag and content together), strips every remaining tag, and decodes HTML entities (&amp; → &, etc.) via re-based regexes — deliberately simple, not a full HTML parser, good enough for indexing purposes only. This keeps markup (div, class, href, inline JS/CSS) out of word_index_misc.json — only visible page text is indexed. misc_store.json's content field for .html entries holds the stripped text (so search snippets are readable), but the checksum field is still computed on the original, unstripped file.

MiscIndexer.py also discovers a much wider set of code/config attachment types — .py, .php, .js, .sh, .sql, .css, .c, .cpp, .java, .rb, .go, .rs, .pl, .xml, .yml, .yaml, .ini, .conf — but treats them as filename/type-only: FILENAME_ONLY_EXTENSIONS skips read_text_file() and tokenise() entirely for these, so nothing they contain enters word_index_misc.json. Their misc_store.json entry has content: "", zero word counts, and filename_only: true — they're viewable via the "Open file" download link and filename/topic searchable, but never full-text searched. .html is deliberately not in this set — it's mostly prose with markup wrapped around it, so it gets full content indexing (above) instead.

Discovery

Unlike PDFIndexer.py's magic-byte scan, plain-text files have no reliable magic number to sniff — and SMF strips real file extensions from attachments' on-disk filenames anyway. Discovery is therefore DB-driven by necessity: MiscIndexer.py cross-references smf_attachments.fileext against what's actually present on disk (matched by numeric id_attach prefix), and only indexes attachments that exist in both places.

Usage

python3 MiscIndexer.py [options]

Options

Flag Default Description
--attach-dir DIR auto-detected (<board root>/attachments) Directory to scan for text attachments
--dry-run off Discover + read only — don't write anything to disk
--limit N none Process only the first N discovered files (for testing)
--monitor off Print live progress and running stats (unique words so far, in-memory KB) every ~5%
--verbose off Also stream log output to stdout, not just the log file

Examples

# Standard run — full rebuild, verbose progress
python3 MiscIndexer.py --monitor

# Test on a handful of files without writing anything
python3 MiscIndexer.py --limit 5 --dry-run --monitor

End-of-run summary

Always prints (regardless of --monitor): attachments matched vs indexed, counts skipped for non-UTF-8 decoding or being empty, a breakdown by file type, total unique words in the index, average word count per file, on-disk size of the written index files, and the log file path. Non-UTF-8 files are skipped and logged, not decoded with a fallback (no latin-1 attempt) — check the log for exact filenames if any show up as skipped.

Notes

MiscSearch.py

Searches the index MiscIndexer.py builds. Also searches filename and topic subject (same rationale as PDFsearch.py's title/topic matching) — merged into one result set per attachment, with title-only matches shown alongside content matches rather than silently dropped.

Usage

python3 MiscSearch.py QUERY [options]

Options

Flag Default Description
--limit N none (no cap) Cap the number of results returned
--html FILE none Write an HTML results report to this path
--source {ui,ssh} ssh Same purpose as PDFsearch.py's flag — argparse compatibility for the bridge's --source=ui; ise_trace.py detects UI vs SSH by checking sys.argv directly

Examples

# Console search
python3 MiscSearch.py listatt

# Full HTML report
python3 MiscSearch.py heroin --html /var/www/html/ISE_Data/misc_results.html

Wildcard search

As of this session, query terms containing * (any run of characters) or ? (exactly one character) are matched via fnmatch against every key in word_index_misc (content) and every filename/topic subject (title search), instead of the normal exact-match/substring lookup. A term with no wildcard characters still takes the original fast path — wildcards only trigger the full-vocabulary scan when present. The query tokenizer regex (_TOKEN_RE) was extended to preserve */? so they survive tokenisation — the content tokenizer in MiscIndexer.py was left unchanged, since indexed file content never legitimately contains literal wildcard characters worth preserving.

Filename-only results

Entries with filename_only: true in misc_store.json (see MiscIndexer.py above) render with a "📁 Filename/type match only" badge in place of a content snippet — they can only ever appear via search_filename_topic(), never via content search, since nothing of theirs is in the word index.

Differences from PDFsearch.py

HTML output / theming

Colors are isolated into a single :root { --var: ... } CSS block at the top of the generated HTML, matching the pattern used for PDF search results — intended so a theme can be swapped in (e.g. by reskinning the :root block) without touching the card markup/logic underneath it.

syslog note

Same gap and fix as PDFsearch.py/search.py: UI-triggered searches are syslogged by qf_Miscsearch_bridge.php before this script runs; SSH-triggered runs syslog the query themselves (tag ise-misc-search-ssh, message prefixed [SSH]) so both paths are consistently logged.

MegaSearch.py

Queries search.py, PDFsearch.py, and MiscSearch.py's get_ranked_results() directly (no subprocess, no re-searching through their CLI layers) and merges the three result lists into one page. Does not include ISEmedia — despite the name, ISEmega is "Posts + PDF + Misc, merged," not "everything." ISEmedia is a fifth, separate, standalone search source with its own dropdown option; nothing currently feeds Media results into a Mega search. Worth knowing plainly since the name invites the assumption otherwise.

Each of the three source calls is individually wrapped in its own try/except Exception, logged via ise_trace.log() as e.g. "MegaSearch.py: pdf source failed" and left as an empty list rather than aborting the whole search — so one source being unavailable (a DB connection failure, an index file missing) degrades gracefully instead of taking out the other two. Known caveat, not yet fixed: that same try/except also catches a completely normal zero-match PDF result (see PDFsearch.py's "Known bug" note below) — a genuine "PDF just didn't match anything" case gets logged identically to a real failure ("pdf source failed"), which is misleading in the trace log even though the merged Mega result set itself still comes out correct (posts/misc results are unaffected). Fixing the root cause in PDFsearch.py would also clean up this log noise.

Merge strategy

Two different merge strategies depending on order_mode, not one merge with a sort bolted on afterward:

The /sort/#ise-order- directive is stripped from the query once, in get_mega_results(), before any of the three sources sees it — so all three always search on an already-clean query and apply their own default (relevance) sort internally; the redundant per-source sort when a date directive is active is intentional, it guarantees the three sources can never land on inconsistent per-source order_modes.

Result cap — the one source that actually has a default cap

Unlike search.py/PDFsearch.py/MiscSearch.py on their own (all effectively unlimited via the UI — see each script's own --limit note above; none of their bridges pass --limit), MegaSearch.py is the one source with a real default result cap:

MEGA_SOFT_CAP = 180
MEGA_PAGE_SIZE = None   # argparse default for --limit

run()'s cap precedence: an explicit --limit N always wins; else --all (or &all=1 on the page) bypasses the cap entirely; else MEGA_SOFT_CAP (180) applies. Since qf_MegaSearch_bridge.php passes neither --limit nor --all, every UI-triggered Mega search is capped at 180 displayed results by default — everything is still searched and ranked first, the cap only limits how many result cards get drawn, and the on-page notice says so with a "Show all" link.

Usage

python3 MegaSearch.py "your query here" [--html FILE] [--limit N] [--all] [--source {ui,ssh}]

Card rendering — delegates rather than duplicates

_render_posts_card() delegates to ResultFormatter.py's real _render_cluster_item() (wrapping each flat post result as a single-item pseudo-cluster) rather than maintaining a hand-copied version — this is a deliberate fix for how the Formatted/Raw buttons and ranking explainer drifted out of sync the first time this was tried as a separate implementation. PDF/Misc cards are Mega's own rendering.

MediaIndexer.py / MediaProcessor.py — building the ISEmedia index

Two scripts, not one — indexing ISEmedia is a two-stage pipeline, unlike the single-script indexers for the other three sources:

Both scripts, shared traits

Versioning — a different convention from the rest of ISE

Both scripts' docblocks carry a version tag that looks like an overall project version ("ISE v9.7" for MediaIndexer.py, "ISE v9.18" for MediaProcessor.py" as of this writing) plus a RUNTIME_VERSION string kept manually in sync with it — this is a **separate, Collabware-native convention**, not the same thing as every other ISE module's own per-file semver (search.py v7.5.2, MediaSearch.py v1.2.0, etc.). Don't read "ISE v9.7" here as meaning the same thing as the package's own "Version 9.5" (install_ise.php`) — they're two different counters from two different codebases that happen to share a project.

MediaSearch.py — ISEmedia search

Searches the embeddings MediaProcessor.py builds. Architecturally different from the other four search sources — no word index, no QueryParser.py/Tokeniser.py/ANDMatcher.py pipeline. Every search encodes the query text with the same CLIP model used to encode every stored image/video, then ranks by cosine similarity — a meaning-based match, not a literal one, to start with.

Usage

python3.8 MediaSearch.py QUERY [--limit N] [--html FILE] [--source {ui,ssh}]

QUERY is pulled from argv[0] before argparse ever sees it (same reason as PDFsearch.py/MegaSearch.py — a query starting with -/-- must not be swallowed as an unrecognized flag).

Options

Flag Default Description
--limit N None (no cap in the script itself) Cap the number of results returned. qf_Mediasearch_bridge.php always passes --limit 180 (added v1.0.2), matching MegaSearch.py's soft-cap number — but this is MediaSearch.py's own separate hardcoded bridge argument, not a shared constant with MegaSearch.py's MEGA_SOFT_CAP.
--html FILE none Write an HTML results report to this path
--source {ui,ssh} ssh Same convention as the other four search scripts

The keyword gate (v1.2.0) — why Media differs from the other four sources

For ISE/PDFsearch/MiscSearch/MegaSearch, a /sort newest/oldest is always safe to apply with no relevance filtering, because their word-index lookup already reduced the candidate set to genuine matches before sort ever runs. MediaSearch.py has no equivalent lookup step — CLIP similarity treats every indexed item as a candidate regardless of query — so as of v1.2.0, a query with real search terms now gates results (a literal match required in caption, filename, or the post subject) before any sort mode is applied, not just relevance. A bare /sort newest with nothing left after the directive is stripped is untouched (nothing to gate by). See MediaSearch.py's own docblock for the full incident that led to this (a "brown" search surfacing an unrelated close-up above genuine "brown" matches, then still surfacing zero real matches under /sort newest once the first fix — a score boost, not a gate — turned out to not matter under date-sort at all). Known, accepted trade-off: a genuinely relevant image whose caption never uses the query word (the original reason CLIP semantic search was built at all) no longer surfaces for a worded query — flagged to Andrew explicitly before shipping, not discovered after the fact.

Ranking — real engine vs. fallback

MediaSearch.py defensively imports RankingEngine.media_ranking_engine (try/except ImportError) and, if present, calls .rank_media(results) inside its own separate try/except — falling back to a plain similarity * 150 score on any failure or if the import itself fails, rather than crashing. Confirmed against the real RankingEngine.py source (this audit, 2026-09-23): that function does not exist. The real RankingEngine.py is v7.4.1#5 and is entirely post/PDF/Misc- oriented (QueryMatchRule, LocationRule, OccurrenceRule, etc., all keyed on fields like text/id_msg/board_name) — no MEDIA_RULES, no SimilarityRule, no rank_media()/media_ranking_engine() factory anywhere in it. MediaSearch.py's own docblock already flagged this as unverified ("I have never seen RankingEngine.py's actual source, only that one changelog sentence") — this audit confirms the changelog claim it was built from was never actually implemented. Not a live bug: the defensive coding means _HAS_MEDIA_RANKING is simply always False in production, and every Media search runs on the documented similarity-only fallback path — results still come back correctly ranked, just not through a dedicated rules engine the way the other four sources are. Two honest ways to close this gap, Andrew's call: (a) build the real media_ranking_engine()/MEDIA_RULES in RankingEngine.py so Media gets the same explainable, rule-by-rule scoring the other sources have, or (b) leave the fallback as the real, permanent design and update MediaSearch.py's docblock/comments to stop describing it as a fallback for a "real engine" that exists.

Interpreter + HF cache — the one source with extra runtime dependencies

MediaSearch.py needs python3.8 specifically (torch/open_clip are only installed under python3.8's site-packages on Live) and a readable, pre-populated Hugging Face cache (ISE_Data/hf_cache, pointed to via HF_HOME) for the CLIP checkpoint — neither requirement applies to any other ISE search script. qf_Mediasearch_bridge.php resolves the interpreter itself (resolve_media_python(): two candidate absolute paths, then command -v python3.8, then a bare python3 fallback of last resort) rather than trusting a bare python3 to land on the right interpreter the way the other four bridges safely do. See this manual's install_ise.php-adjacent notes / the 2026-09-23 checkpoint for the full diagnostic chain (interpreter → torch install location → HF cache permissions) and install_ise.php v9.5's ise_ensure_hf_cache() for how a fresh environment now self-provisions this cache automatically.

syslog / help / theming

Same conventions as the other four: /help short-circuits to the shared ISE_help.html, ise_trace.py logging throughout (ise_trace.log("MediaSearch.py: keyword gate applied", ...) etc.), results-page footer links to the shared help page. No wildcard or quoting support — see ISE_help.html's dedicated ISEmedia section.

Query parsing internals (search.py / QueryParser.py / Tokeniser.py / ANDMatcher.py / IndexLookup.py)

For the user-facing summary of what each quoting style does per source, see the shared help page rather than duplicating it here: <boardurl>/The_ISE_Project/ISE_help.html (or type /help in any search box). What follows is the code-level "why" — the actual functions and matching paths involved, for anyone debugging or extending this.

search.py (ISE posts) is the only one of the three search sources with a real quoting/phrase concept — PDFsearch.py and MiscSearch.py have no equivalent at all (see each script's Wildcard search note above for how they actually handle a literal quote character passed to them: PDFsearch.py's .split() glues quotes onto the token, breaking the match; MiscSearch.py's tokenizer regex silently drops quote characters entirely).

For ISE (posts), the pipeline is QueryParser.parse() → Tokeniser.tokenise() → ANDMatcher.match() → IndexLookup.py's lookup_word()/lookup_phrase():

SortEngine.py (shared /sort directive parsing)

Built in v9.3.0. A small shared module — extract_order_mode(query, valid_modes, default) — used by search.py, PDFsearch.py, MiscSearch.py, and MegaSearch.py so the /sort <mode> and #ise-order-<mode> query-string syntax is recognised identically everywhere, instead of each module hand-rolling its own copy of the same regex (the old version of this file predated RankingEngine.py/ ISEmega entirely and was dead code — nothing in the current pipeline called it).

def extract_order_mode(query: str, valid_modes: set, default: str) -> tuple:
    # strips a /sort <mode> or #ise-order-<mode> directive out of query,
    # resolving it against the caller's own valid_modes set.
    # Returns (cleaned_query, order_mode).

/sort is checked first (the documented, user-facing syntax); #ise-order- is the older internal syntax, still recognised for back-compat. If both are present in one query, /sort wins. A mode name not in the caller's valid_modes is left in the query untouched — it just fails to match anything as a search term, same as any other typo, rather than silently vanishing with no explanation.

Where it's called, and with what valid_modes:

Caller valid_modes Notes
search.py (ISE/posts) relevance, score, date, newest, oldest, newmod, oldmod, subject, board, author, occurrences Full ORDER_KEYS dict; date is a legacy alias for newest. Called from inside get_ranked_results() itself (not just run()) so MegaSearch.py's direct call gets directive parsing too — before this, a /sort directive routed through ISEmega leaked into the query as literal search terms.
PDFsearch.py relevance, score, newest, oldest No newmod/oldmod — see below.
MiscSearch.py relevance, score, newest, oldest Same.
MegaSearch.py relevance, score, newest, oldest Stripped once in get_mega_results(), before any of the three sources is queried, so all three see an already-clean query and never re-derive the directive independently. Does not cover ISEmedia — see MegaSearch.py section above.
MediaSearch.py relevance, score, newest, oldest Same shape as PDF/Misc — no newmod/oldmod (attachments carry poster_time only). The real difference for Media isn't in this parsing step at all — it's that a non-empty query also gates results by literal keyword match before any ORDER_KEYS sort runs (v1.2.0), which none of the other four sources need to do here since their own word-index lookup already did that filtering upstream. Note: SortEngine.py's own module docblock (v8.0, 2026-09-14) only names search.py, PDFsearch.py, and MiscSearch.py as callers in its "Used by" line — stale as of MediaSearch.py's build (2026-09-23); the real import (from SortEngine import extract_order_mode) confirms Media uses it too, the docblock just predates that.

Why newmod/oldmod is ISE (posts)-only: PDFsearch.py/ MiscSearch.py results carry poster_time only — PDF/Misc attachments aren't edited the way a forum post is, so there's no modified_time field in pdf_store.json/misc_store.json to sort by. search.py's ORDER_KEYS["newmod"]/["oldmod"] fall back to poster_time for any post that's never been edited (_effective_modified()), matching ISE_help.html's documented behavior exactly.

ISEmega's date sort is a different merge, not a re-sort: ise_render.py gained merge_by_date() alongside its existing interlace_results(). MegaSearch.get_mega_results() picks one or the other based on order_mode — interlace_results() (round-robin, unchanged) for relevance/score, merge_by_date() (flat poster_time sort across all three tagged lists) for newest/oldest. These are genuinely different strategies, not the same merge with a final re-sort bolted on: interlacing exists specifically to stop one source dominating a relevance-score merge, which isn't a concern once you're sorting by date — so a date sort bypasses interlacing entirely.

In-app help (/help and the footer link)

ISE_help.html is a single static file living in The_ISE_Project/ (same directory as all five search scripts, web-accessible via the .htaccess <Files "*.html"> rule) — the single shared source all five search UIs point to. It now also documents ISEmega and ISEmedia (added this audit — both were previously undocumented there despite ISEmega having existed for some time and being partially wired into the page's /sort table already).

search.py, PDFsearch.py, and MiscSearch.py each define the same pair of names near its top (MegaSearch.py reuses search.py's copy directly rather than redefining it — see its own /help short-circuit; MediaSearch.py defines its own, same shape):

HELP_TRIGGER = "/help"
HELP_HTML_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), "ISE_help.html")

def _serve_help(html_path):
    with open(HELP_HTML_PATH, "r", encoding="utf-8") as f:
        content = f.read()
    if html_path:
        with open(html_path, "w", encoding="utf-8") as f:
            f.write(content)
    return True

At the very top of each script's argument handling, before any real search logic runs, the query is checked against HELP_TRIGGER case/whitespace-insensitively (query.strip().lower() == HELP_TRIGGER). On a match, _serve_help() copies ISE_help.html's content directly into whatever --html path the PHP bridge passed — since every bridge already just readfile()s that path (see qf_search_bridge.php/ qf_PDFsearch_bridge.php/qf_Miscsearch_bridge.php), no PHP changes were needed at all. The console/SSH path (no --html) just prints the shared file's own path instead of writing anywhere. Both paths log via ise_trace.log() and, for SSH runs, call ise_trace.new_trace() same as a normal search would.

ISE_help.html carries its own copy of ResultFormatter.py's 13-theme CSS block and switcher, and on load reads whichever of ise_search_theme/ise_pdf_theme/ise_misc_theme is set in localStorage, so it opens already matching whichever theme was last picked on any results page.

Separately, ResultFormatter.py's format_html(), PDFsearch.py's write_html(), and MiscSearch.py's write_html() each got a small permanent footer link — <a href="{BASE_URL}/The_ISE_Project/ISE_help.html"> — added just before </body>, so the help page is reachable from every normal results page too, not only via the /help keyword.

Planned work (not yet built)

Shared dependency notes

No longer one shared runtime — two, as of ISEmedia. Everything below the line was true, and mostly still is, for the original four posts/PDF/misc/mega scripts. ISEmedia's two build-side scripts (MediaIndexer.py, MediaProcessor.py) and its search script (MediaSearch.py) need Python 3.8 specifically — torch/open_clip are only installed under python3.8's site-packages on Live, and will fail to import under the system default. Confirmed live (2026-09-23): running any of the three Media scripts under bare python3 fails on import numpy/import torch before ever reaching real logic. qf_Mediasearch_bridge.php resolves python3.8 explicitly for this reason (see MediaSearch.py section above); install_ise.php v9.5 does the same for MediaIndexer.py's background install-time job. IndexBuilder.py/PDFIndexer.py/MiscIndexer.py are deliberately not touched to require 3.8 — no evidence they need it, and changing their interpreter risks three scripts that already work.