A custom search mod for SMF. Started as forum-wide post search ("ISE"), PDF-attachment search ("ISEpdf"), and text-attachment search ("ISEmisc") as options in the existing CustomSearch dropdown; now five search sources — this update (2026-09-23) adds full coverage of the two that had grown past what this manual documented: ISEmega (a combined view across the first three) and ISEmedia (image/video search by meaning and keyword). This is the single consolidated operator manual for all five; ISE_Python_Scripts_Manual.md is the companion code-level reference (per-script options/internals), and ISE_help.html is the in-app, end-user-facing help page — all three were audited together against the real source (v9.5) for this update, not just cross-checked against each other's claims.
.md/.txt/.text/.json/ .html — both content and filename/topic subject; .html has its tags/scripts/styles stripped before indexing so only visible text is searched) plus a much wider set of code/config attachment types by filename and file type only (.py/.php/.js/.sh/.sql/.css/ .c/.cpp/.java/.rb/.go/.rs/.pl/.xml/.yml/.yaml/.ini/ .conf — no content read, no full-text search, just viewable via the "Open file" link and filename/topic searchable), with results linking straight to the owning forum post./sort newest/oldest) — a convenience view across the first three, not a sixth independent index. Does not include ISEmedia — despite the name, "mega" stops at the original three; worth stating plainly since it's easy to assume otherwise. Shows up to 180 results by default (a soft cap, bypassable — see §7).ISE/ISEpdf/ISEmisc/ISEmega bypass the database at query time — they read pre-built JSON indexes for speed, so those indexes must be rebuilt manually after posts or attachments change (see §4). ISEmedia works the same way in spirit (pre-built embeddings in media_store.json, not a live DB/CLIP-encode-everything pass per search) but is built via its own two-stage pipeline (MediaIndexer.py then MediaProcessor.py), covered in ISE_Python_Scripts_Manual.md rather than duplicated here.
<SMF root>/
├── Sources/Search.php — dispatcher: case 'ISE' / 'ISEpdf' / 'ISEmisc'
├── Sources/ManageSettings.php — CustomSearch_use_ISE / _ISEpdf / _ISEmisc
│ checkboxes, plus ise_reset_pdfjs /
│ ise_reset_isedata reset checkboxes
├── Sources/Admin.php — registers CustomSearch in admin menu
├── Themes/default/index.template.php — dropdown options
├── Themes/default/languages/Modifications.english.php — label strings
│ (also patched at Themes/default/language/ — singular — for installs
│ that use that spelling instead)
├── ISE_Data/ — generated indexes + logs (777, apache-owned)
│ ├── word_index.json / post_store.json — forum post index
│ ├── word_index_pdf.json / pdf_store.json — PDF index
│ ├── word_index_misc.json / misc_store.json — Misc (.md/.txt/.text/.json/.html content, plus filename-only code/config types) index
│ ├── media_store.json — ISEmedia catalog (MediaIndexer.py) + CLIP embeddings/captions (MediaProcessor.py)
│ ├── hf_cache/ — ISEmedia only: shared, apache-readable Hugging Face model cache
│ │ (CLIP checkpoint) — see §3/§11; nothing else in ISE uses this
│ ├── Settings_ISE.json — per-install {"environment", "install"} selector read by ise_settings.py
│ ├── ise_trace.log — shared tracing log, ISE/ISEpdf/ISEmisc/ISEmega/ISEmedia
│ └── *.log — other trace/debug logs
├── The_ISE_Project/ — all scripts
│ ├── qf_search_bridge.php — web entry point for ISE
│ ├── qf_PDFsearch_bridge.php — web entry point for ISEpdf
│ ├── qf_Miscsearch_bridge.php — web entry point for ISEmisc
│ ├── qf_MegaSearch_bridge.php — web entry point for ISEmega
│ ├── qf_Mediasearch_bridge.php — web entry point for ISEmedia (the only bridge that
│ │ resolves python3.8 explicitly — see §3/§11)
│ ├── search.py — forum post search (called by bridge)
│ ├── PDFsearch.py — PDF search (called by bridge)
│ ├── MiscSearch.py — Misc search (called by bridge)
│ ├── MegaSearch.py — combined Posts+PDF+Misc search (called by bridge)
│ ├── MediaSearch.py — image/video search (called by bridge; needs python3.8)
│ ├── ISE_help.html — shared help page, all five sources
│ │ (see §9) — served on "/help" query,
│ │ linked from every results page footer
│ ├── IndexBuilder.py — builds word_index.json / post_store.json
│ ├── PDFIndexer.py — builds word_index_pdf.json / pdf_store.json
│ ├── MiscIndexer.py — builds word_index_misc.json / misc_store.json
│ ├── MediaIndexer.py — catalogs image/video attachments into media_store.json
│ │ (needs python3.8; settings via Collabware.core_utils,
│ │ not ise_settings.py — a different codebase glued in,
│ │ see ISE_Python_Scripts_Manual.md)
│ ├── MediaProcessor.py — computes CLIP embeddings + captions into media_store.json
│ │ (needs python3.8; same Collabware.core_utils note)
│ ├── IndexLookup.py — shared index-reading helper (posts)
│ ├── ise_trace.py — shared tracing module (all five search scripts;
│ │ NOT used by MediaIndexer.py/MediaProcessor.py, which
│ │ log via their own --monitor/--verbose telemetry instead)
│ ├── ise_settings.py — shared base-URL/project-URL resolver (Settings_ISE.json
│ │ lookup) for search.py/PDFsearch.py/MiscSearch.py/
│ │ MegaSearch.py/MediaSearch.py
│ ├── SortEngine.py — shared /sort directive parser, all five search scripts
│ ├── reset_ise_data.php — install-time code hook, see §3
│ ├── orphan_check.py — standalone admin/maintenance tool (cron), not part
│ │ of the search UI/dropdowns; finds attachments whose
│ │ id_msg no longer resolves to a real post
│ └── RankingEngine.py, QueryParser.py, etc. — scoring/parsing internals (posts/PDF/misc;
│ ISEmedia currently falls back to its own plain similarity score rather than a
│ dedicated ranking-engine interface — see ISE_Python_Scripts_Manual.md)
└── pdfjs/ — bundled PDF.js viewerEach SMF instance (live, smf20, smf21, ...) has its own complete, independent copy of The_ISE_Project/ and ISE_Data/ — nothing is shared between instances by design.
readme.txt in the package): Board-color-and-icons, Automatic Package Version Emulation, CustomSearch.ISE_V<version>.zip via SMF's package manager as normal.install_ise.php runs automatically and creates The_ISE_Project/ and ISE_Data/ with 777 permissions, then kicks off IndexBuilder.py, PDFIndexer.py, MiscIndexer.py, and MediaIndexer.py in the background (non-blocking — the install request itself returns immediately). Progress/output for each lands in its own log file under ISE_Data/. As of v9.5, this step also self-provisions ISE_Data/hf_cache (ISEmedia's CLIP model cache) if it's not already populated — a cached copy from root's own Hugging Face cache if the install is running as root, otherwise a backgrounded fresh download straight into hf_cache; either way this doesn't block the install request finishing. See §11 if a fresh environment's ISEmedia search still fails after install with a cache-related error.python3.8 and the HF cache are both actually reachable by the web server user (apache), not just installed; a working ISE/ISEpdf/ISEmisc/ISEmega search says nothing about whether ISEmedia will work, since it runs on a different Python interpreter and has its own model-cache dependency (see §2, §11).Uninstalling reverses the install.xml file edits and deletes The_ISE_Project/ only. ISE_Data/ and pdfjs/ are deliberately left untouched on uninstall — both hold real generated data (indexes, trace logs, PDF.js viewer assets, hf_cache) that shouldn't vanish just because the mod entry is removed from Package Manager's list.
cd <SMF root>/The_ISE_Project
python3 IndexBuilder.py --monitor # forum posts → word_index.json / post_store.json
python3 PDFIndexer.py --monitor # PDF attachments → word_index_pdf.json / pdf_store.json
python3 MiscIndexer.py --monitor # .md/.txt/.text/.json → word_index_misc.json / misc_store.jsonIndexBuilder.py and MiscIndexer.py are both full-rebuild-only — no resume/incremental mode, safe to re-run anytime. PDFIndexer.py has its own resume-by-default behavior; see the Python Scripts Manual for details.
Suggested cron schedule (adjust to your forum's actual posting/upload volume):
# Every 2 hours (on the hour) — rebuild the Post index
0 */2 * * * python3 /var/www/html/The_ISE_Project/IndexBuilder.py >> /var/www/html/ISE_Data/indexbuilder.log 2>&1
# Every 2 hours (on the half-hour) — rebuild the Misc/text index
30 */2 * * * python3 /var/www/html/The_ISE_Project/MiscIndexer.py --monitor >> /var/www/html/ISE_Data/misc_indexer.log 2>&1
# Every 6 hours — fast rebuild of the PDF index
5 */6 * * * python3 /var/www/html/The_ISE_Project/PDFIndexer.py --fast --monitor >> /var/www/html/ISE_Data/pdfindexer_fast.log 2>&1
# Every 2 days — full union rebuild (both extractors) of the PDF index
10 0 */2 * * python3 /var/www/html/The_ISE_Project/PDFIndexer.py --monitor >> /var/www/html/ISE_Data/pdfindexer_union.log 2>&1Two checkboxes on the CustomSearch settings page control a one-shot, self-clearing reset mechanism for the next install/reinstall:
pdfjs/ on the next install. Since the library refresh itself already happens on every normal install (see §3), this checkbox only matters when you want a genuine clean slate — including clearing out anything not part of the vendored library, such as custom "ln" symlinks. Ticking this means you'll need to re-establish those links afterward — the checkbox label says so directly.ISE_Data/ on the next install, wiping all three indexes and the trace log. You'll need to rebuild all three indexes from scratch afterward (§4).Both default to unchecked. Each is a one-shot flag: tick it, save, run the install/reinstall, and it clears itself back to unchecked automatically once it fires — so it can't accidentally wipe data again on a later reinstall you didn't intend to reset.
Supported for ISEpdf and ISEmisc; not yet for ISE (forum posts) — see the in-app help page for the user-facing explanation of */? syntax and per-source behavior: <boardurl>/The_ISE_Project/ISE_help.html, or type /help in any of the three search boxes (§9).
Implementation note: a term with no wildcard characters still uses the original fast exact-match (content) / substring-match (filename/topic) path in both scripts — wildcards only trigger the slower full-vocabulary scan when actually present in the term. ISE (posts) doesn't have this yet because its matching runs through QueryParser.py → Tokeniser.py → ANDMatcher.py → IndexLookup.py, a different pipeline from ISEpdf/ISEmisc's direct dict-lookup approach — wildcard support there needs its own implementation against those modules, not a copy of the ISEpdf/ISEmisc approach.
/sort)Built in v9.3.0 — see changelog.md. v9.2 shipped the /help page text promising this (/sort newest|oldest|newmod|oldmod) before the code actually did it; v9.3.0 closes that gap for real. ISEmedia (built later, 2026-09-20 onward) picked up the same newest/oldest support via the same shared SortEngine.py — see the ISEmedia-specific note below, its behavior isn't a plain drop-in of the other four sources' pattern.
/sort newest — poster_time descending/sort oldest — poster_time ascending/sort newmod — modified_time descending, falling back to poster_time for any post that's never been edited (modified_time is 0/absent) — ISE (posts) only/sort oldmod — same fallback, ascending — ISE (posts) onlyTyped directly in the search box, same convention as /help and the older #ise-order-<mode> syntax (still works, kept for back-compat; /sort wins if a query somehow contains both). Parsing happens in Python, inside each source's own get_ranked_results() — via the new shared SortEngine.extract_order_mode() — not at the PHP bridge layer; the bridges still just pass the raw query string straight through unmodified, same as they always have.
Why newmod/oldmod is posts-only: ISEpdf/ISEmisc attachments carry poster_time only — attachments themselves aren't edited the way a forum post is, so there's no modified_time field anywhere in pdf_store.json/misc_store.json to sort by. newest/oldest work on all three individual sources.
ISEmega: newest/oldest work as a genuine flat chronological merge across the three sources it covers — posts, PDF, misc; ISEmega does not include ISEmedia at all, see §1 — via ise_render.merge_by_date(). Not a re-sort layered on top of the usual score-interlaced merge. The two are different merge strategies: interlacing exists specifically to stop one source dominating a relevance-score merge, which isn't a concern once you're sorting by date, so a date sort bypasses interlace_results() entirely rather than re-sorting its output. newmod/oldmod are not offered at the mega level — there's no coherent way to merge a posts-only "last edited" ordering against two sources that don't have the concept at all; use ISE (posts) search directly for those.
ISEmedia — a real behavioral difference, not just a missing feature: newest/oldest work, but only ever among items that already passed a literal keyword-match gate (v1.2.0) when the query has real search terms. For ISE/ISEpdf/ISEmisc/ISEmega, "sort by date, ignore relevance score" is always safe, because each source's own index lookup already narrowed the candidate list down to genuine matches before any sort runs. ISEmedia has no equivalent lookup — its CLIP similarity search treats every indexed image/video as a candidate regardless of query — so without the v1.2.0 gate, /sort newest brown would have silently shown every recent upload, "brown" or not. Fixed live, same night it was caught (2026-09-23) — see ISE_Python_Scripts_Manual.md's MediaSearch.py section for the full incident and the accepted trade-off (a genuinely relevant but uncaptioned image no longer surfaces for a worded query). A bare /sort newest with no other search term is unaffected — nothing to gate by, so every item's a candidate, same as it always was.
The "Why is this ranked #N?" explainer on every result card correctly reflects whichever sort is active — it says "sorted by newest, not relevance score" (etc.) instead of claiming a relevance ranking that isn't what actually produced the order.
position="before"/position="after" are inverted in this fork's package manager compared to stock SMF — before actually inserts content after the matched text, and vice versa. install.xml is already written to account for this; don't "fix" it back to stock semantics.<search>/<replace> (no position attribute) silently no-ops — always use position="before|after|replace" + <add> instead, even though the install preview shows "Test successful" either way.pdftotext or similar must pass encoding="utf-8", errors="replace" explicitly, or it will crash on real-world PDFs containing accented characters, curly quotes, em-dashes, etc. Doesn't apply to ISEmedia's three scripts — MediaIndexer.py/MediaProcessor.py/MediaSearch.py run under python3.8, a separate interpreter installed specifically for the torch/open_clip stack (see §2). Worth remembering before "fixing" an encoding issue in one of those three that's actually a different problem — this specific 3.6 default doesn't apply to them.<search position="before">/<add> entries one at a time against the same target line, over several install cycles, tends to leave that line's whitespace/alignment inconsistent — array entries and $txt[...] assignments end up crammed onto one line with no newlines between them. The next mod's search then silently fails to match ("Skipping Search"), because it's looking for clean, single-purpose text that no longer exists in that form. Prefer whole-block position="replace" over incremental adds whenever a settings array or a labeled string block (like the BIOHAZARD:CustomSearch block in Modifications.english.php) is likely to be touched by more than one install over its lifetime — replace the entire known block in one atomic operation instead of layering adds onto it.install.xml (a <modification>-type document) must only contain <file>/<operation> blocks. <uninstall>, <require-file>, <require-dir>, and <code> tags belong exclusively in package-info.xml — a stray <uninstall> block was found duplicated inside install.xml itself in an earlier version (including a nonsensical self-reference back to install.xml), which is invalid structure for that file type and was removed.require-dir on install merges, it never deletes. Any file already present in the destination stays untouched unless the package's own zip contains a file with that exact same name, in which case only that one file gets overwritten. This means the ISE_Data/ folder packed inside any zip must contain only .htaccess — never real index/store/log filenames (word_index*.json, *_store.json, ise_trace.log) — or a completely routine reinstall would silently overwrite live search data with the package's bundled (empty/stale) copies, with no checkbox or warning involved. pdfjs/, by contrast, should contain the full library every time — that's vendored code meant to refresh on every install.<!DOCTYPE package-info SYSTEM "http://..."> external DTD reference can fail to resolve depending on the server's libxml/PHP config (blocked outbound access, stricter defaults) — SMF doesn't need the DOCTYPE to parse the file, so it's safe to drop.zip -X -D (no extended attributes, no directory entries) — this keeps the archive at plain PKZIP 2.0 (extract version 20), no Zip64.qf_search_bridge.php used to write every ISE (posts) search to the same ISE_results.html path — meaning every search hit the exact same URL, which browsers readily cache. After an update that changes what that page contains (e.g. the theme/dropdown work), a browser can keep serving its old cached copy indefinitely with no visible error. Fixed by giving ISE's bridge the same unique-per-request filename pattern (..._<pid>_<microtime>.html) that qf_PDFsearch_bridge.php/ qf_Miscsearch_bridge.php already used — if a results page ever looks stale after a code update, check whether its bridge still writes to a fixed filename before assuming the deployed code itself is wrong./help and the footer link)All three search UIs share one built-in help page, The_ISE_Project/ISE_help.html — a single static file (web-accessible under The_ISE_Project/ per the .htaccess *.html rule) that explains each source's search methods and, accurately, how unquoted/'single'/ "double" quoting and */? wildcards behave differently across ISE, ISEpdf, and ISEmisc (including the real gap where single-quoting a word in ISE search returns nothing — documented plainly rather than glossed over). It carries its own theme switcher (same 13 themes as the results pages) and picks up whichever theme was last selected on any of the three sources' results pages.
Two ways to reach it:
/help (unquoted, case/whitespace-insensitive) as the query in any of the three search boxes. search.py, PDFsearch.py, and MiscSearch.py all detect this exact query and short-circuit before running any real search — they copy ISE_help.html's content straight into whatever --html path the PHP bridge requested, so the normal bridge readfile() flow serves it with no PHP changes needed./help keyword exists.| Symptom | Likely cause |
|---|---|
| Same search results regardless of query | Bridge PHP silently failed and served a stale cached output file |
| PDF search always returns ≤20 results | --limit cap in PDFsearch.py (should default to unlimited) |
| Search returns almost nothing on a populated forum | Index wasn't rebuilt after content changed, or IndexBuilder.py/PDFIndexer.py/MiscIndexer.py pointed at the wrong instance's Settings.php/attachments folder |
| One instance's search shows another instance's data | A script has a hardcoded absolute path instead of resolving relative to its own file location — check ATTACH_DIR, WORD_INDEX_PATH, *_STORE_PATH, and the bridge's $script variable |
| Dropdown missing ISE/ISEpdf/ISEmisc options after install | index.template.php operation didn't match — check tab count/whitespace against the live file with cat -A |
install_ise.php runs but nothing happens |
Check file_exists() on the target script paths — likely means The_ISE_Project wasn't extracted yet (code step must run after the extract steps in package-info.xml) |
| Checkbox appears on the settings page but with no label text (blank) | The language-string <search> anchor in install.xml/a mini-mod didn't match — usually a whitespace/alignment mismatch against the live file's actual (often hand-edited, inconsistently spaced) content. Always verify the anchor byte-for-byte against a fresh copy of the live file (cat -A or od -c), not a pasted/remembered version of it |
| Package Manager shows "Skipping Search" for a file operation | The <search> text genuinely wasn't found in the target file at install time — pull the file straight off the server and diff it against the anchor text exactly; don't assume the anchor that worked in a previous package version still matches, especially for blocks that get touched by more than one mod over time (see §7) |
| A patch appears to have installed successfully (no warning, shows in Package Manager's installed list) but the change isn't visible on the live page | Check the raw file on the server directly (not the rendered browser page) to isolate whether the source was actually patched. If the source was patched correctly but the page still shows the old version, suspect PHP OPcache or SMF's own settings/menu cache serving stale content — clear both (PHP-FPM/Apache restart or opcache_reset(), plus SMF's Admin > Maintenance cache-clear) before concluding the patch itself failed |
| "A broad ISE/ISEpdf/ISEmisc search never seems capped, but ISEmega/ISEmedia stop at 180" | Not a bug — result caps genuinely differ by source. ISE/ISEpdf/ISEmisc are uncapped by default via the UI (their bridges never pass --limit); ISEmega defaults to a 180-result soft cap (bypassable via the page's "Show all" link) and ISEmedia's bridge always passes --limit 180 explicitly. Everything is still searched/ranked in full either way — the cap only limits how many result cards get drawn. See ISE_Python_Scripts_Manual.md for the exact mechanism per source (confirmed against real source, 2026-09-23). |
Standalone/SSH PDFsearch.py "some query with no matches" prints "ISE ERROR — see ..." instead of "No matches." |
Confirmed real bug, fixed in v7.5.2 (2026-09-23): get_ranked_results() returned a bare [] on a genuine zero-match instead of the (results, order_mode) tuple every caller expects, so the unpack itself raised and got caught by the outer exception handler. Via the UI bridge this was invisible (the fallback path shows the same "No results found" text either way); via MegaSearch.py it was silently absorbed but logged misleadingly as "pdf source failed". Fixed by returning [], order_mode on that path; see ISE_Python_Scripts_Manual.md's PDFsearch.py section. |
orphan_check.py fails immediately, every mode (CLI/--html/cron), with ImportError: cannot import name 'BOARD_URL' from 'PDFsearch' |
Confirmed real bug, fixed in v7.4.2 (2026-09-24): PDFsearch.py's base-URL constant was renamed BOARD_URL → BASE_URL when its settings-loading moved onto ise_settings.py (v4.0, 2026-08-21) — two days after orphan_check.py's last edit — and orphan_check.py was never updated. BOARD_URL was never actually used anywhere in the file, so the fix drops it from the import entirely rather than re-importing it unused. See ISE_Python_Scripts_Manual.md's orphan_check.py section. |
ISEmedia search fails with ModuleNotFoundError: No module named 'numpy' (or torch) |
Wrong Python interpreter — MediaSearch.py/MediaIndexer.py/MediaProcessor.py need python3.8 specifically; a bare python3 under Apache's thinner shell_exec() PATH can resolve to the system default instead. qf_Mediasearch_bridge.php resolves this itself (resolve_media_python()); if you're testing manually over SSH, use the full path (/usr/local/bin/python3.8), not a bare python3/python3.8 alias — sudo's own secure_path may not include /usr/local/bin either. See §2/§3 and ISE_Python_Scripts_Manual.md. |
ISEmedia search fails with PermissionError: [Errno 13] ... /.cache |
HF cache not reachable by apache. open_clip tries to download/cache its CLIP checkpoint under the invoking user's home directory by default; apache's home (/var/www) isn't writable. install_ise.php v9.5 self-provisions a shared, apache-readable ISE_Data/hf_cache on install (see §3) — if this error shows up anyway, check that hf_cache/hub/models--timm--vit_base_patch32_clip_224.openai actually exists and is apache:apache-owned, and check ISE_Data/hf_cache_setup_status.log for whether both of install_ise.php's provisioning paths failed to even start (e.g. no root cache to copy from and no outbound network access to huggingface.co). |
| ISEmedia search returns results with no visible relation to the query word | Check the version. Pre-v1.2.0, MediaSearch.py ranked by pure CLIP semantic similarity with no required literal keyword match — a real, now-fixed gap (v1.1.0 added a score boost, v1.2.0 made it a hard gate for every sort mode). If a live instance is still showing this behavior, it's running a version older than v1.2.0 — update it, don't assume the ranking logic itself needs another look. |