ISE - Intelligent Search Engine for Posts and PDFs! Current version:
ISE_v5.5 for
SMF 2.0.19 -- Download using the table below:
📎 Click on me to view the list of attachments Theme: Deep Indigo Pink & Purple Slate Tech Dark Cyan ISE Project — Checkpoint Paste this whole file into a fresh chat to resume. It reflects state as of the end of the PDF-indexing/search work session (PDFIndexer.py two-engine union + resume/locking, PDFsearch.py filename/topic search, orphan_check.py, cron cleanup). Older ISE post-search work (search.py/IndexBuilder.py/ RankingEngine.py) is included for completeness but was not touched this session — treat those as stable/working unless told otherwise.
Environment: live box is CentOS 6.10, Python 3.6.3, 2GB RAM. This OS's age is the root cause of most dependency fights below — expect the same pattern (no wheel, needs Rust, needs Python 3.8+) with any new tool.
File / component table File Location Status Use it? PDFIndexer.py/var/www/html/The_ISE_Project/Rebuilt this session: two-engine (PyMuPDF+pypdf) union extraction, --fast mode, checkpointing, resume-by-default, --start-at, single-instance file lock Yes — current, deployed, workingPDFsearch.py/var/www/html/The_ISE_Project/Rebuilt this session: adds filename/topic search merged with content search, live progress Yes — current, deployed, workingorphan_check.py/var/www/html/The_ISE_Project/New this session. Weekly cron job, avatar-excluded, diffs against orphan_state.json Yes — deployed, in crontabword_index_pdf.json / pdf_store.jsonISE_Data/Rebuilt from scratch with the new two-engine extractor; 497/676 PDFs indexed, 179 skipped (genuinely image-only, confirmed) Yes — current on diskorphan_state.jsonISE_Data/State file for orphan_check.py, auto-managed Don't touch by hand all_orphans.py/var/www/html/The_ISE_Project/Old one-off diagnostic (no avatar filter, no diffing) No — superseded by orphan_check.py, removed from crontab, left on disk unusedsearch.py, IndexBuilder.py, IndexLookup.py, Tokeniser.py, RankingEngine.py, ResultFormatter.py, ise_trace.py/var/www/html/The_ISE_Project/Older ISE post-search engine (ISE 5.1). Not touched this session. Yes — assumed stable; re-verify only if something seems brokenqf_search_bridge.php, qf_PDFsearch_bridge.phpSMF theme/Sources PHP bridges wiring the dropdown search-engine options to the Python scripts Yes — working as of ISE 5.1Search.php, index_template.php (SMF core, patched)SMF Dropdown values 'ISE' / 'ISEpdf' wired in Yes — workingpdfplumber tried, rejected Needs Rust to build cryptography from source on this box; no working path found No pypdfium2 tried, rejected No compatible wheel at any version for this platform No tesseract 3.04 (EPEL) / ocrmypdf 12.7.2 tried, parked ocrmypdf requires tesseract ≥4.0 (dead end); tesseract 3.04 direct-CLI use never actually tested Parked — user said "I don't need OCR", not pursuing
Verified facts — don't re-check Live box: CentOS 6.10, Python 3.6.3, 2GB RAM, swap present. dmesg confirmed an actual OOM kill of a PDFIndexer.py run (913MB resident) — this is a real, hit-once constraint, not theoretical. Package pins that are load-bearing , don't let anything upgrade them without re-testing:pymupdf==1.19.2 (no newer wheel for this Python/platform)pypdf==3.1.0 (5.x requires typing.Protocol, Python 3.8+, crashes on import here) poppler-utils on this box is 0.12.4 (2010) and silently mis-decodes PDFs with malformed ToUnicode CMaps into garbage tokens instead of erroring — this was the original root cause of the very first bug in this session (search for "listatt" finding nothing despite the word being visibly present).The two-engine union (PyMuPDF + pypdf) is genuinely more thorough than either alone — confirmed via per-engine unique-term tracking built into PDFIndexer.py --monitor output, not just assumed. --fast (PyMuPDF only) is real and works — confirmed correct index output, ~3-20x faster depending on file, meaningfully lighter on memory.The two schedules (--fast every 6h, full union every 2 days) write to the same word_index_pdf.json/pdf_store.json — the single-instance lock (/tmp/pdfindexer.lock, fcntl) was added and tested (confirmed: a second instance exits immediately with a clear message rather than racing) specifically to stop one overwriting the other's progress if they ever overlap. 179 of 676 PDFs are skipped as having no extractable text — confirmed via direct testing (both engines return empty) these are genuinely image-only/scanned, not an extraction bug. Orphaned-attachment root cause confirmed via direct DB query: e.g. attach-4868's id_msg (58213) doesn't resolve to any row in smf209_messages — the post was deleted. SMF's dlattach action requires both topic= and attach= to serve a file; confirmed (user tested directly) there is no working fallback without a valid topic. This is a genuine, currently-unfixable-in-code data gap, not a PDFsearch.py bug. ~360 attachments have id_msg=0 — confirmed this is SMF's normal avatar convention (profile pictures), not orphaning. Only ~6 of the original 360-ish flagged rows were real orphaned content PDFs. all_orphans.py (old diagnostic) has no avatar filter and no diff logic — confirmed strictly redundant with orphan_check.py. User agreed to remove it from cron after initially wanting to keep it.Current live crontab (as of this session, MAILTO=apache, all jobs now redirect output to per-job log files instead of flooding mail):0 */2 * * * python3 IndexBuilder.py (post index, every 2h)5 */6 * * * python3 PDFIndexer.py --fast --monitor (PDF index, fast, every 6h)10 0 */2 * * python3 PDFIndexer.py --monitor (PDF index, full union, every 2 days)15 */12 * * * attach_filetypes.sh /var/www/html/attachments (every 12h)0 6 * * 1 python3 orphan_check.py --html ... --quiet-if-empty (weekly Monday 06:00)all_orphans.py's 07:00 Monday line was removed.
Open items (decisions, not blockers) OCR for the 179 image-only PDFs — parked, not rejected. Tesseract 3.04 (EPEL) is installed and confirmed present; direct-CLI use (bypassing ocrmypdf, which needs tesseract ≥4.0) was never actually tested end-to-end. Revisit only if/when wanted.OS migration — user twice expressed mild wishful thinking about a newer OS given the repeated dependency-version fights, but has not decided to act on it. Not a live plan, don't push it.Tokeniser is still a plain .split() in PDFIndexer.py — not punctuation-aware ([listatt] and listatt index as separate tokens). Known limitation, never prioritized for a fix.pdf_store.json's filename field is always Noneresolve_filename() exists but isn't wired into PDFIndexer.py's main(). Doesn't block search (PDFsearch.py resolves real filenames itself via direct DB query) but is a known gap in the index file's own completeness.Markdown manual requested and delivered this session (ISE_Python_Scripts_Manual.md, covering PDFIndexer.py/PDFsearch.py/orphan_check.py CLI options) — exists as an output file; not saved into project memory/repo, worth confirming it's actually been kept somewhere durable if it's meant to be a lasting reference.Dev-box replication — user's stated plan is to follow this session's saved transcript (after text extraction) to redo this same PDFIndexer/PDFsearch work on the separate dev box (/var/www/html/smf20/The_ISE_Project). Not started.Next actual action None queued — user explicitly paused ISE work this session ("no more ISE!"). When resumed, the next actual action is: replicate the PDFIndexer.py/PDFsearch.py two-engine-union + locking/resume work on the dev box , using this session's saved transcript as the reference.
⚠️ No .md attachment at index 1 for message #49422 (currentMsgId() returned this value).
ISE - Intelligent Search Engine for Posts and PDFs
WHAT THIS IS Adds a custom search engine option called "ISE" (Intelligent Search
Engine) to your forum's search-engine drop-down box, alongside the
existing Default/Google/Yandex/Nigma options.
ISE is a private, self-hosted search tool: a pure Python search engine (QueryParser -> Tokeniser -> ANDMatcher -> SortEngine -> ResultFormatter) that reads directly from your forum's own posts and returns matching results as clickable links, sorted newest first.
This mod also adds search-query logging (via syslog) for both the
native SMF search box and the ISE dropdown option, so every real search your users type gets recorded for later review.
]REQUISITE MODS 1.
Board-color-and-icons (Version 1.1) 2.
Automatic Package Version Emulation (Version 1.6) 3.
CustomSearch (Version 1.0) -- Install via Emulating 2.0
Install the above, in order, before installing this package. FEATURES - WORD matching: any_search_term
- PHRASE matching: "an exact quoted phrase" (whitespace-exact, no
normalization -- a double space inside quotes is
matched literally)
- Mixed queries: word1 word2 "an exact phrase" -- all terms must
match (AND logic) for a post to qualify
- Results sorted newest-first
- Every search (native SMF box, ISE dropdown, and the site's Google
search widget) is logged via syslog: username/Guest, IP address,
timestamp, and the raw query string
WHAT GETS INSTALLED - Sources/QueryFrontend.php -- syslog logging function
- Sources/Search.php (patched) -- adds the "ISE" case to the search-engine dropdown handler
- Sources/ManageSettings.php (patched) -- adds an admin toggle, "Use ISE search"
- Themes/default/index.template.php (patched) -- loads
widget_write_log.js and adds the "ISE" option to the dropdown
- Themes/default/scripts/widget_write_log.js -- logs searches made
through the site's separate Google-search widget
- Themes/default/languages/Modifications.english.php (patched)
-- adds the "ISE" and "Use ISE search" language strings
(both languages/ and language/ directory spellings are patched,
since this varies by SMF install)
- $boarddir/The_ISE_Project/ -- the Python search engine itself
(QueryParser.py, Tokeniser.py, IndexBuilder.py, IndexLookup.py,
ANDMatcher.py, SortEngine.py, ResultFormatter.py, search.py,
qf_search_bridge.php, daily_queries.py, query_log_reader.py)
-- protected from direct web access via its own .htaccess
- $boarddir/ISE_Data/ -- holds the generated search index
(word_index.json, post_store.json) -- created with 0700
permissions and its own .htaccess, since this data is a full,
searchable copy of every post on your forum
- The full Mozilla PDF Javascript libraries at $boarddier/pdfjs
REQUIREMENTS -- READ BEFORE INSTALLING This mod's PHP/dropdown/logging pieces install automatically through
the Package Manager. The Python search engine itself does NOT run
automatically -- it needs a few things set up on the server first,
by hand, once:
- Python 3.6+ available on the server, with the PyMySQL package
installed for whichever interpreter you'll use to run these
scripts:
python3 -m pip install pymysql AFTER INSTALLING The Python search engine itself does NOT run by default so folow this carefully:
1. cd ..whatever-your-webroot-is../The_ISE_Project
python3 IndexBuilder.py
2. cd ..whatever-your-webroot-is../The_ISE_Project
python3 PDFIndexer.py --monitor
3. Go to
Admin > Configuration > Modificication Settings > CustomSearch > ISE and ISEpdf uncheck both and Save then recheck both and Save again .
4. Go to the top right of the page and now the dropdown will have ISE and ISEpdf so now you can use it but wait until the indexing has finished.
5. Test a search using the ISE option from the dropdown to confirm everything is wired up correctly before relying on it.
6. crontab -u apache -e
MAILTO=apache
0 */2 * * * python3 /var/www/html/The_ISE_Project/IndexBuilder.py
0 3 * * 0 python3 /var/www/html/The_ISE_Project/PDFIndexer.py --monitor
NOTES - Search queries are logged via syslog (facility LOCAL0, tag smf-queries) -- typically landing in /var/log/messages depending on your syslog configuration. Two small scripts are included to help review this log:
- The_ISE_Project/daily_queries.py -- lists today's searches
- The_ISE_Project/query_log_reader.py -- lists the full history
PRIVACY NOTE Once enabled, every real user's search terms are logged, including guests (identified by IP address rather than username). Given the sensitive nature of a harm-reduction forum's likely search content, review who has access to your server's syslog output and how long you want to retain this history before enabling this for your full userbase.
Post Merged: Yesterday at 11:52:17 PM ═══════════════════════════════════════════════ISE / SMF MOD WORK — RELOADABLE CHECKPOINT(paste this into a fresh chat to resume)═══════════════════════════════════════════════## FILE/COMPONENT STATUS TABLE| File | Location | Status | Use? ||---|---|---|---|| PDFsearch.py | The_ISE_Project/ | Fixed: no result cap, WORD_INDEX_PATH/PDF_STORE_PATH auto-resolve via ISE_ROOT (Path(__file__).resolve().parent.parent) | YES — confirmed working on smf20 + live || PDFIndexer.py | The_ISE_Project/ | Fixed: ATTACH_DIR auto-resolves via ISE_ROOT; UTF-8 decode fix for pdftotext; em-dash print fix | YES — confirmed working, 670 PDFs/live, 2 PDFs/smf20 || IndexBuilder.py | The_ISE_Project/ | Fixed: Settings.php path auto-resolves via os.path.dirname(__file__) | YES — confirmed working, 45,069 posts/live, 7 posts/smf20 || qf_PDFsearch_bridge.php | The_ISE_Project/ | Fixed: $script uses __DIR__ (was hardcoded to live); unique filename per request (no race condition); debug log → ISE_Data/qf_debug.txt (was /tmp/, permission denied) | YES || qf_search_bridge.php | The_ISE_Project/ | Fixed: debug log path same as above | YES || install_ise.php | package root | REVERTED to original simple version — just creates The_ISE_Project + ISE_Data dirs (0777). Automatic background indexer launch was built then explicitly abandoned (Andrew's call — too many failure modes: buffering, process detachment, stale root-owned log files blocking apache writes) | YES (simple version only) || uninstall_ise.php | package root | NEW — recursively deletes The_ISE_Project/ISE_Data/pdfjs even with leftover generated files inside (plain require-dir reverse fails silently on non-empty dirs) | YES, not yet tested live || install.xml | package root | Contains 5 operations: ManageSettings.php (ISE+ISEpdf checkboxes), Search.php (case 'ISE'/'ISEpdf' dispatcher), index.template.php (dropdown options), Modifications.english.php (lang strings), Admin.php (CustomSearch menu registration) | YES — confirmed installing correctly on smf20 || package-info.xml | package root | <code>install_ise.php</code> now runs AFTER extract steps (was before — scripts didn't exist yet); <code>uninstall_ise.php</code> added to uninstall block; <readme type="bbc">readme.txt</readme> added | YES || readme.txt | package root | Lists 3 prerequisite mods (Board-color-and-icons v1.1, Automatic Package Version Emulation v1.6, CustomSearch v1.0) + manual post-install indexer commands (Andrew added these himself) | YES — readme display on install screen never confirmed working (may be an unsupported tag on this fork) || Admin.php | Sources/ | No changes needed beyond install.xml's operation | n/a, not a standalone artifact |## VERIFIED FACTS — DON'T RE-CHECK- This SMF fork has **inverted position semantics**: `position="before"` in install.xml actually inserts content AFTER the matched text; `position="after"` inserts BEFORE. Confirmed via direct testing on index.template.php and ManageSettings.php. Order only matters for index.template.php (PHP ternary chain) — doesn't matter for ManageSettings.php/Modifications.english.php (independent array/assignment lines).- Bare `<search>`/`<replace>` (no position attribute) silently no-ops on this fork despite showing "Test successful" — must use `position="before|after"` + `<add>` instead.- Live SMF root = `/var/www/html/`, DB = `smf209`. Test install = `/var/www/html/smf20/`, DB = `smf20` (own separate DB, own separate Settings.php, own attachments folder).- Three parallel SMF installs exist: `/var/www/html/` (live), `/var/www/html/smf20/`, `/var/www/html/smf21/` — each fully independent, no shared files by design (Andrew's explicit requirement).- Python 3.6 on this server: `subprocess.run(..., universal_newlines=True)` defaults to strict ASCII decoding — must use `encoding="utf-8", errors="replace"` explicitly for any subprocess call touching real-world text (PDFs, etc.).- `shell_exec` and subprocess execution are NOT blocked by disable_functions or open_basedir on either PHP CLI or the real web-facing PHP (both confirmed empty via direct test).- readme.txt uses BBCode `[b]...[/b]` — added `type="bbc"` to the `<readme>` tag but never confirmed it actually renders on the install screen.- PDF search bug (results always capped at 20) — root cause was `--limit` argparse default=20 in PDFsearch.py; fixed to default=None.- "Same 20 results regardless of query" bug — root cause was qf_PDFsearch_bridge.php reading a stale shared HTML output file when shell_exec silently failed; fixed via unique-filename-per-request + readfile + unlink.- Main forum search "only 2 results for 'drugs'" bug — root cause was IndexBuilder.py connecting to smf20's test Settings.php via a hardcoded path instead of live's; unrelated to any of tonight's later bugs.## OPEN ITEMS (decisions, not blockers)- Whether to re-attempt automatic background indexing later, or keep it permanently manual (currently: manual, by Andrew's choice).- Whether readme.txt's BBCode actually renders on this fork's install screen — never visually confirmed.- `[lmv=inline]` BBCode tag — designed (unparsed_content tag feeding straight into MDParser.php) but not yet built.- `[lmv]` mod currently interferes with the PDF viewer (PDF disappears when [lmv] present) — reported, not yet diagnosed. Need PDFBBC.php + LMVBBC.php uploaded to find the collision (likely shared class/function/CSS name, since LMV mirrors PDF's architecture closely).- `[listatt]` "Open with" request — real fix is likely server-side `Content-Disposition: inline` header instead of `attachment`, not an OS-level right-click hook (browsers don't let a page control that menu). Not yet built.- Andrew has an already-written "redact tool" — mentioned in passing, zero details on what it does or where it lives.## NEXT ACTUAL ACTIONRebuild the package zip using package-info.xml + install.xml + readme.txt + install_ise.php (simple version) + uninstall_ise.php + the fixed PDFsearch.py/PDFIndexer.py/IndexBuilder.py/qf_PDFsearch_bridge.php/qf_search_bridge.php — all currently only verified on the *running* smf20 and live filesystems, NOT yet copied into Andrew's actual package source folder. Do that copy first, rebuild the zip, do one full fresh install (vanilla+CustomSearch) end to end, then manually run IndexBuilder.py and PDFIndexer.py per the readme instructions and confirm both complete cleanly.Post Merged: Today at 03:00:08 PM Checkpoint as at ISE v5.6 and sit on it as it is stable, packaged and system independent.
Home
Help
Search
Login
Register
Need help or a chat ?
