smf-search-query.md

📝 smf-search-query.mdTheme:

name: smf-search-query description: The ISE Project — Andrew's custom Python-based SMF search engine (posts/PDF/Misc); v8.1 closed out as a "bug exposing release" (BASE_URL/ise_settings fix), v8.2 starting next. See [[isemedia-clone]] for the separate ISEmedia/Clone-VM work. sources: [chat] aliases: [query, enhanced search, ISE]

Checkpoint 2026-08-21 (v8.1 closed out — "bug exposing release": BASE_URL bug chase → ise_settings.py v4.0 fix, not yet deployed)

[stated] v8.1 closed out and named a "bug exposing release" — testing ISE end-to-end from Dev (not just Live) surfaced several independent dormant issues at once: (1) Dev's mirror/sync silently drops dotfiles, so every .htaccess across the tree was missing on Dev only — fixed by copying Live's .htaccess over; mirror.sh itself still needs a dotfile-exclusion fix (2) a disk resize to 120GB on Dev never took effect at the LVM PV level (pvs still ~52GB) — unresolved (3) three separate real pdfjs directory copies existed on Dev — consolidated into one real dir + symlinks
[stated] Real code bug: ResultFormatter.py/MiscSearch.py/IndexBuilder.py each had their own older $boardurl-scraping guess for building SMF links, wrong on Dev (double /The_ISE_Project). PDFsearch.py had BASE_URL hardcoded to Live's domain, redirecting Dev clicks to Live
[stated] Fix built (not yet deployed): ise_settings.py rewritten to v4.0 — single hardcoded _SITE_ROOTS table keyed by {environment: Live|Dev, install: main|smf20|smf21} via each install's own Settings_ISE.json (sits next to Settings.php); unrecognised combo raises loud instead of guessing. ResultFormatter.py/MiscSearch.py/IndexBuilder.py/PDFsearch.py all import get_base_url()/get_project_url() from this one shared module — no per-file copies left. Settings_ISE.json is per-install/user-edited; _SITE_ROOTS table is shared code, redeployed everywhere on any environment change (accepted trade-off vs. typo-prone free-text URLs). All 4 Settings_ISE.json files made and excluded from mirror.sh alongside Settings.php
[stated] v8.2 rollout order: Live Test/smf20 first, then Dev, then Dev Test/smf20, then Live. Dev/smf20 leg: IndexBuilder.py/MiscIndexer.py ran clean; PDFIndexer.py crashed with ModuleNotFoundError: fitz (pymupdf) — unrelated dependency gap on Dev, not the URL fix
[stated] pymupdf on Dev (32-bit): confirmed conclusively no path via yum/pip (no mupdf package in any repo incl. EPEL, no PyPI wheel at any version, SCLo/devtoolset unavailable for i386). Compiled MuPDF 1.11 from source successfully, but pymupdf==1.11.2 against it hit "dereferencing pointer to incomplete type" — the downloaded MuPDF 1.11 source doesn't structurally match what that pymupdf version's SWIG wrapper expects; needs finding the exact matching MuPDF snapshot, real research. Deferred to a future session
[stated] Session wrapped: URL-resolution fix confirmed solid (posts/misc clean on both smf20 installs, no URL errors). ISEpdf/PDF indexing on Dev remains the clear next item
[stated] New feature idea floated, not scoped: user-selectable AND/OR query-operator panel for ISEpdf (possibly posts/misc too) — NAND/NOR floated then dropped as likely needing real matcher rework
[stated] AND/OR built and delivered for ISEpdf (PDFsearch.py v7.5.0): ISEpdf's search() was already soft-OR by default; added match_mode param ("or" default unchanged, "and" = hard filter requiring every term, quoted phrase = one unit) through search()/get_ranked_results()/main()'s --mode CLI flag. Verified against real synthetic word-index data: OR/AND/phrase-as-unit/wildcard-as-unit all confirmed correct. Design: no UI toggle — a bare unquoted "AND" in the query string itself switches mode (discoverable via ISE_help.html, not advertised in UI). Not yet built: stripping/detecting AND from the query string at the bridge layer (bridge file not yet uploaded). Longer-term idea, deferred indefinitely: +/- term weighting
[stated] Built [lmv=help][/lmv]/[lmv=?][/lmv] this session (LMVBBC.php v3.3.0, see [[lmv-bbcode-mod]]). Andrew wants a shared "common place" help mechanism across all BBCode mods eventually — settled plan: each module's own bundled readme.txt baked into a data file at package-install time; Master Post becomes just a "see full history" link. Not yet built, deferred
[stated] Duplicate PDFs skew search results (each is its own index entry, can rank differently). Andrew wants to KEEP duplicate files as-is but collapse them into one "master copy" result at search/ranking time (group by SHA-256 via [[pdf-duplicate-spotter]], master-selection rule undecided, optional "also posted in N other places" note). Touches RankingEngine.py and PDFsearch.py's dedupe_by_attachment(). Deferred
[stated] Andrew handling version numbering himself going forward; archiving sessions into the master ISE forum post; treating the uploaded zip (labeled v8.0) as the real v8.2 baseline, not v8.1
[stated] Andrew runs 6 SMF environments total (3 on the production VPS, mirrored 3 on Dev) — full topology in [[forum-infrastructure]]. A board-color mismatch this session turned out to be smf20 never having that board's color configured locally, not a code bug — worth checking which of the 6 environments any future report is actually from
[stated] Real bug fixed: PDFsearch.py's _parse_php_dir_array() only handled SMF's PHP-serialized attachmentUploadDir format — added a plain-string fallback; MiscIndexer.py shares the fix
[stated] ise_post_download.php was actually deployed as v1.0 (not the v1.2 an earlier checkpoint assumed) — rebuilt as v1.1 combining the Formatted/Raw split + generic bracket-tag BBCode-strip fix, confirmed deployed. Same "described in chat, never actually deployed" pattern hit ResultFormatter.py's Download button — patched live via safety-checked find-and-replace scripts over SSH (verify anchor matches exactly once, back up first) — reusable pattern for future live patches
[stated] MegaSearch.py's posts card now delegates to ResultFormatter.py's real _render_cluster_item() instead of a hand-copied parallel version (the parallel copy is exactly how Formatted/Raw buttons silently went missing before). Real coupling: assumes that function's first returned line is always the title div — flagged in code. PDF/Misc cards in MegaSearch.py NOT delegated this way (no equivalent shared function exists), same drift risk not yet hit
[stated] Detail-level toggle simplified 3 tiers -> 2 (Minimal/Full) since delegated posts cards return one unsplittable block
[stated] New-tab standard: anything navigating away from a results page to the live forum opens in a new tab; downloads never get a target attribute. Applied to PDFsearch.py's Board link and ResultFormatter.py's title/best-match/match-evidence links (confirmed); ResultFormatter.py's Board link patch delivered, run-confirmation pending; MegaSearch.py's own cards not yet audited
[stated] Browser caching caused 3 false "regression" reports this session — rule out with hard refresh/incognito before deep-diving any reported regression right after a patch
[stated] Plan: consolidate live server's actual current state (not any prior chat's output files — several live patches were never round-tripped back) into a v8.0 master, then start v8.1 from there; per-file version stamps currently inconsistent, worth a normalization pass

Earlier history (condensed)

[stated] Started as "query" — Python search processor built with ChatGPT, formalized as "The ISE Project" at /var/www/html/The_ISE_Project. Three sources: ISE (posts, RankingEngine.py additive scoring), ISEpdf (PDFsearch.py, PyMuPDF+pypdf union), ISEmisc (MiscSearch.py, content-indexed .md/.txt/.html/.json vs filename-only code/config). Shared infra: ise_theme.css/js (15 themes), ISE_help.html, ISE_text_viewer.html, wildcard search on all 3 sources (posts wildcard added 2026-08-20 via IndexLookup.py's fnmatch fix — quoted phrases never support wildcards on any source)
[stated] v7.0→v7.2→v7.3→v7.4.1#5→v8.0(ISEmega, in progress)→v8.1(closed 08-21)→v8.2(next). ISEmega = combined "search everything" page, results interlaced by raw score across Posts/PDF/Misc (not grouped), shared ise_render.py Open+Download module, per-card theme scoping (each card matches its own source's saved theme via data-ise-source/ise_mega_theme.js), separate shell-exclusive theme picker (aurora/vaporwave/goldrush/toxic/none) for ISEmega's own page shell. Each source paginates independently at different page sizes (4/2/~3) — bridge will need its own page-size decision, accepted as "a little uneven is OK"
[stated] v7.3 (completed): board display audit (MiscSearch.py/PDFIndexer.py needed board-color/board-awareness added, posts already had it), post download solved via ise_post_download.php generating a .txt on the fly (mirrors view_attach.php's permission pattern), attachment badges on post cards, fixed IndexBuilder.py's BBCODE_TAG_RE data-pollution bug (custom tags like [lmv] were becoming searchable indexed words purely from tag use), fixed a PDFsearch.py crash (undefined BASE_URL variable silently zeroing every ISEpdf search)
[stated] v7.4→v7.4.1#5 (ranking/search-quality overhaul): wired PDFsearch.py/MiscSearch.py into RankingEngine.py (shared scoring engine, "Why is this ranked #N?" explainer), unified case-sensitivity/quoting across all 3 sources, fixed PDFIndexer.py's tokenizer, normalized versioning to one package-wide number. Real ranking bugs found via live QA and fixed: result.get(key,"") not catching present-but-None fields; single-word searches under-scoring due to occurrences always =1; an Occurrence plateau bug; Recency swing overriding real matches (scaled to 1/4 weight for PDF); final fix rebuilt Occurrence to score by page-coverage ratio, no ceiling problem at any doc size. Standing preference: every fix verified with an actual runnable test against real/synthetic data, not just reasoned about — screenshots of live results are more useful than descriptions
[stated] Live box constraints: CentOS 6.10/Python 3.6/2GB RAM, EOL dependency ceilings (poppler 0.12.4, no Rust compiler). PDFIndexer.py has checkpointing/resume/single-instance lock after one OOM kill. Live crontab: IndexBuilder.py every 2h, PDFIndexer.py --fast every 6h, full union every 2 days, orphan_check.py weekly
[stated] Abandoned: search-history dropdown (ise_search_history.js) — never deployed correctly, reverted. Was awake 6 days straight building v7.0 originally (self-reported)

Security — permission-filtering gap (v8.1 polish item, still not built)

[stated] Confirmed real gap: none of the 3 search scripts filter results by board permission at the data layer. Severity confirmed narrow: actual content/file access stays properly gated regardless (SMF's own real permission checks independently re-validate on every request) — the only real leak is restricted-board result titles/excerpts appearing in listings even though clicking through is correctly denied (information-disclosure, not access-bypass)
[stated] Wants it enforced identically across all output methods including any future JSON mode — filtering must happen at the data layer before format-specific rendering
[stated] Fix design agreed, not built: indexers add id_board to every entry (largely done via v7.3's board-display work) → PHP bridge computes the user's allowed-board-ID list via SMF's real permission functions → Python filters every result against that list before writing output. Security-critical computation stays in PHP; Python only does list-membership filtering

New open thread (2026-09-08): nested boolean query syntax

[stated] Andrew wants nested AND/OR query syntax typed in the search box itself (e.g. keyword1 /and (keyword2 /or keyword3)), not a UI toggle — decided a syntax beats a form control since it also supports grouping a UI toggle couldn't
[stated] Also wants a case-insensitive phrase search; date sort (blocked on him confirming SMF edit-flag behavior for modified_time) and a fuzzy/nearest-neighbor search mode remain other open asks in the same session, still unscoped
[stated] Still undecided: whether the parser should support arbitrary nesting depth or just one level of parens, and whether this applies to all three sources (ISE/posts, ISEpdf, ISEmisc) or a subset

Other open items

[stated] Wants ISE downloads logged via WhoDownloadedAttachment (member downloads already captured automatically via real dlattach hooks; guest downloads are a known gap in that mod, moot since guest downloading is off). Standing policy: never patch a third-party/requisite mod's own source, report gaps to the mod's author instead — see [[requisite-mods]]
[stated] ISEmega (v8.0) design fully agreed but backend bridge/fusion wiring not started — needs qf_MegaSearch_bridge.php + ise_render.py
[stated] Frustrated with the current results-page design: thinks topic-cluster grouping (one "best match" + collapsed "Show N replies") is bad — wants plain-text discussion, no more tap-option cards. Hasn't landed on the alternative yet — asked to state it when ready

Checkpoint 2026-09-09 — PDFsearch.py phrase-mode case-sensitivity bug (verified + fixed, not yet deployed)

[stated] Root cause confirmed: word_index_pdf.json is lowercase-only at the key level; phrase-mode ("literal") lookup in search() did a raw case-sensitive exact-key check ([pw] if pw in word_index else []), never matching a capitalized quoted term like "Ian" against the lowercase-only index — word-mode already folded case correctly via folded_index, phrase-mode was never updated to match
[stated] Fix verified via isolated test harness (test_phrase_bug.py, using the real tokenise()/search() code against synthetic data shaped like the real confirmed word_index_pdf.json, including real data for "ian"/attach 6204): one-line fix matched_keys = folded_index.get(pw.lower(), []) resolves quoted "Ian"/"Ian " with no regression on already-lowercase phrases
[stated] Fix applied to a patched copy of PDFsearch.py this session; not yet deployed to Dev or Live
[stated] Open item: MiscSearch.py / posts-search not yet checked for the same case-sensitivity pattern against their own indexes, despite case-sensitivity having been recorded as "unified across all 3 sources" in the v7.4 checkpoint
[stated] Andrew's diagnostic process this session: confirmed attach 6204 present in pdf_store.json/word_index_pdf.json via grep, confirmed "ian" key present with correct match list via direct Python inspection, ruled out OCR/indexing/caching before finding the actual bug by running the real CLI query and hitting the case-sensitive quoting behavior directly