A custom search mod for SMF, adding forum-wide post search ("ISE"), PDF-attachment search ("ISEpdf"), and text-attachment search ("ISEmisc") as options in the existing CustomSearch dropdown.
.md/.txt/.text/.json/ .html — both content and filename/topic subject; .html has its tags/scripts/styles stripped before indexing so only visible text is searched) plus a much wider set of code/config attachment types by filename and file type only (.py/.php/.js/.sh/.sql/.css/ .c/.cpp/.java/.rb/.go/.rs/.pl/.xml/.yml/.yaml/.ini/ .conf — no content read, no full-text search, just viewable via the "Open file" link and filename/topic searchable), with results linking straight to the owning forum post.All three bypass the database at query time — they read pre-built JSON indexes for speed, so the indexes must be rebuilt manually after posts or attachments change (see §4).
<SMF root>/
├── Sources/Search.php — dispatcher: case 'ISE' / 'ISEpdf' / 'ISEmisc'
├── Sources/ManageSettings.php — CustomSearch_use_ISE / _ISEpdf / _ISEmisc
│ checkboxes, plus ise_reset_pdfjs /
│ ise_reset_isedata reset checkboxes
├── Sources/Admin.php — registers CustomSearch in admin menu
├── Themes/default/index.template.php — dropdown options
├── Themes/default/languages/Modifications.english.php — label strings
│ (also patched at Themes/default/language/ — singular — for installs
│ that use that spelling instead)
├── ISE_Data/ — generated indexes + logs (777, apache-owned)
│ ├── word_index.json / post_store.json — forum post index
│ ├── word_index_pdf.json / pdf_store.json — PDF index
│ ├── word_index_misc.json / misc_store.json — Misc (.md/.txt/.text/.json/.html content, plus filename-only code/config types) index
│ ├── ise_trace.log — shared tracing log, all sources
│ └── *.log — other trace/debug logs
├── The_ISE_Project/ — all scripts
│ ├── qf_search_bridge.php — web entry point for ISE
│ ├── qf_PDFsearch_bridge.php — web entry point for ISEpdf
│ ├── qf_Miscsearch_bridge.php — web entry point for ISEmisc
│ ├── search.py — forum post search (called by bridge)
│ ├── PDFsearch.py — PDF search (called by bridge)
│ ├── MiscSearch.py — Misc search (called by bridge)
│ ├── ISE_help.html — shared help page, all three sources
│ │ (see §8) — served on "/help" query,
│ │ linked from every results page footer
│ ├── IndexBuilder.py — builds word_index.json / post_store.json
│ ├── PDFIndexer.py — builds word_index_pdf.json / pdf_store.json
│ ├── MiscIndexer.py — builds word_index_misc.json / misc_store.json
│ ├── IndexLookup.py — shared index-reading helper
│ ├── ise_trace.py — shared tracing module, all three sources
│ ├── reset_ise_data.php — install-time code hook, see §3
│ └── RankingEngine.py, QueryParser.py, etc. — scoring/parsing internals
└── pdfjs/ — bundled PDF.js viewerEach SMF instance (live, smf20, smf21, ...) has its own complete, independent copy of The_ISE_Project/ and ISE_Data/ — nothing is shared between instances by design.
readme.txt in the package): Board-color-and-icons, Automatic Package Version Emulation, CustomSearch.ISE_V<version>.zip via SMF's package manager as normal.install_ise.php runs automatically and creates The_ISE_Project/ and ISE_Data/ with 777 permissions, then kicks off IndexBuilder.py, PDFIndexer.py, and MiscIndexer.py in the background (non-blocking — the install request itself returns immediately). Progress/output for each lands in its own log file under ISE_Data/.Uninstalling reverses the install.xml file edits and deletes The_ISE_Project/ only. ISE_Data/ and pdfjs/ are deliberately left untouched on uninstall — both hold real generated data (indexes, trace logs, PDF.js viewer assets) that shouldn't vanish just because the mod entry is removed from Package Manager's list.
cd <SMF root>/The_ISE_Project
python3 IndexBuilder.py --monitor # forum posts → word_index.json / post_store.json
python3 PDFIndexer.py --monitor # PDF attachments → word_index_pdf.json / pdf_store.json
python3 MiscIndexer.py --monitor # .md/.txt/.text/.json → word_index_misc.json / misc_store.jsonIndexBuilder.py and MiscIndexer.py are both full-rebuild-only — no resume/incremental mode, safe to re-run anytime. PDFIndexer.py has its own resume-by-default behavior; see the Python Scripts Manual for details.
Suggested cron schedule (adjust to your forum's actual posting/upload volume):
# Every 2 hours (on the hour) — rebuild the Post index
0 */2 * * * python3 /var/www/html/The_ISE_Project/IndexBuilder.py >> /var/www/html/ISE_Data/indexbuilder.log 2>&1
# Every 2 hours (on the half-hour) — rebuild the Misc/text index
30 */2 * * * python3 /var/www/html/The_ISE_Project/MiscIndexer.py --monitor >> /var/www/html/ISE_Data/misc_indexer.log 2>&1
# Every 6 hours — fast rebuild of the PDF index
5 */6 * * * python3 /var/www/html/The_ISE_Project/PDFIndexer.py --fast --monitor >> /var/www/html/ISE_Data/pdfindexer_fast.log 2>&1
# Every 2 days — full union rebuild (both extractors) of the PDF index
10 0 */2 * * python3 /var/www/html/The_ISE_Project/PDFIndexer.py --monitor >> /var/www/html/ISE_Data/pdfindexer_union.log 2>&1Two checkboxes on the CustomSearch settings page control a one-shot, self-clearing reset mechanism for the next install/reinstall:
pdfjs/ on the next install. Since the library refresh itself already happens on every normal install (see §3), this checkbox only matters when you want a genuine clean slate — including clearing out anything not part of the vendored library, such as custom "ln" symlinks. Ticking this means you'll need to re-establish those links afterward — the checkbox label says so directly.ISE_Data/ on the next install, wiping all three indexes and the trace log. You'll need to rebuild all three indexes from scratch afterward (§4).Both default to unchecked. Each is a one-shot flag: tick it, save, run the install/reinstall, and it clears itself back to unchecked automatically once it fires — so it can't accidentally wipe data again on a later reinstall you didn't intend to reset.
Supported for ISEpdf and ISEmisc; not yet for ISE (forum posts) — see the in-app help page for the user-facing explanation of */? syntax and per-source behavior: <boardurl>/The_ISE_Project/ISE_help.html, or type /help in any of the three search boxes (§8).
Implementation note: a term with no wildcard characters still uses the original fast exact-match (content) / substring-match (filename/topic) path in both scripts — wildcards only trigger the slower full-vocabulary scan when actually present in the term. ISE (posts) doesn't have this yet because its matching runs through QueryParser.py → Tokeniser.py → ANDMatcher.py → IndexLookup.py, a different pipeline from ISEpdf/ISEmisc's direct dict-lookup approach — wildcard support there needs its own implementation against those modules, not a copy of the ISEpdf/ISEmisc approach.
position="before"/position="after" are inverted in this fork's package manager compared to stock SMF — before actually inserts content after the matched text, and vice versa. install.xml is already written to account for this; don't "fix" it back to stock semantics.<search>/<replace> (no position attribute) silently no-ops — always use position="before|after|replace" + <add> instead, even though the install preview shows "Test successful" either way.pdftotext or similar must pass encoding="utf-8", errors="replace" explicitly, or it will crash on real-world PDFs containing accented characters, curly quotes, em-dashes, etc.<search position="before">/<add> entries one at a time against the same target line, over several install cycles, tends to leave that line's whitespace/alignment inconsistent — array entries and $txt[...] assignments end up crammed onto one line with no newlines between them. The next mod's search then silently fails to match ("Skipping Search"), because it's looking for clean, single-purpose text that no longer exists in that form. Prefer whole-block position="replace" over incremental adds whenever a settings array or a labeled string block (like the BIOHAZARD:CustomSearch block in Modifications.english.php) is likely to be touched by more than one install over its lifetime — replace the entire known block in one atomic operation instead of layering adds onto it.install.xml (a <modification>-type document) must only contain <file>/<operation> blocks. <uninstall>, <require-file>, <require-dir>, and <code> tags belong exclusively in package-info.xml — a stray <uninstall> block was found duplicated inside install.xml itself in an earlier version (including a nonsensical self-reference back to install.xml), which is invalid structure for that file type and was removed.require-dir on install merges, it never deletes. Any file already present in the destination stays untouched unless the package's own zip contains a file with that exact same name, in which case only that one file gets overwritten. This means the ISE_Data/ folder packed inside any zip must contain only .htaccess — never real index/store/log filenames (word_index*.json, *_store.json, ise_trace.log) — or a completely routine reinstall would silently overwrite live search data with the package's bundled (empty/stale) copies, with no checkbox or warning involved. pdfjs/, by contrast, should contain the full library every time — that's vendored code meant to refresh on every install.<!DOCTYPE package-info SYSTEM "http://..."> external DTD reference can fail to resolve depending on the server's libxml/PHP config (blocked outbound access, stricter defaults) — SMF doesn't need the DOCTYPE to parse the file, so it's safe to drop.zip -X -D (no extended attributes, no directory entries) — this keeps the archive at plain PKZIP 2.0 (extract version 20), no Zip64./help and the footer link)All three search UIs share one built-in help page, The_ISE_Project/ISE_help.html — a single static file (web-accessible under The_ISE_Project/ per the .htaccess *.html rule) that explains each source's search methods and, accurately, how unquoted/'single'/ "double" quoting and */? wildcards behave differently across ISE, ISEpdf, and ISEmisc (including the real gap where single-quoting a word in ISE search returns nothing — documented plainly rather than glossed over). It carries its own theme switcher (same 13 themes as the results pages) and picks up whichever theme was last selected on any of the three sources' results pages.
Two ways to reach it:
/help (unquoted, case/whitespace-insensitive) as the query in any of the three search boxes. search.py, PDFsearch.py, and MiscSearch.py all detect this exact query and short-circuit before running any real search — they copy ISE_help.html's content straight into whatever --html path the PHP bridge requested, so the normal bridge readfile() flow serves it with no PHP changes needed./help keyword exists.| Symptom | Likely cause |
|---|---|
| Same search results regardless of query | Bridge PHP silently failed and served a stale cached output file |
| PDF search always returns ≤20 results | --limit cap in PDFsearch.py (should default to unlimited) |
| Search returns almost nothing on a populated forum | Index wasn't rebuilt after content changed, or IndexBuilder.py/PDFIndexer.py/MiscIndexer.py pointed at the wrong instance's Settings.php/attachments folder |
| One instance's search shows another instance's data | A script has a hardcoded absolute path instead of resolving relative to its own file location — check ATTACH_DIR, WORD_INDEX_PATH, *_STORE_PATH, and the bridge's $script variable |
| Dropdown missing ISE/ISEpdf/ISEmisc options after install | index.template.php operation didn't match — check tab count/whitespace against the live file with cat -A |
install_ise.php runs but nothing happens |
Check file_exists() on the target script paths — likely means The_ISE_Project wasn't extracted yet (code step must run after the extract steps in package-info.xml) |
| Checkbox appears on the settings page but with no label text (blank) | The language-string <search> anchor in install.xml/a mini-mod didn't match — usually a whitespace/alignment mismatch against the live file's actual (often hand-edited, inconsistently spaced) content. Always verify the anchor byte-for-byte against a fresh copy of the live file (cat -A or od -c), not a pasted/remembered version of it |
| Package Manager shows "Skipping Search" for a file operation | The <search> text genuinely wasn't found in the target file at install time — pull the file straight off the server and diff it against the anchor text exactly; don't assume the anchor that worked in a previous package version still matches, especially for blocks that get touched by more than one mod over time (see §7) |
| A patch appears to have installed successfully (no warning, shows in Package Manager's installed list) but the change isn't visible on the live page | Check the raw file on the server directly (not the rendered browser page) to isolate whether the source was actually patched. If the source was patched correctly but the page still shows the old version, suspect PHP OPcache or SMF's own settings/menu cache serving stale content — clear both (PHP-FPM/Apache restart or opcache_reset(), plus SMF's Admin > Maintenance cache-clear) before concluding the patch itself failed |