A fuller narrative companion to the condensed memory checkpoint — written so a future session (or a future Andrew) can reconstruct not just what happened but why, including the dead ends, without re-deriving any of it from scratch.
Picked up from checkpoint_closing_v8.1.md / changelog_closing_v8.1.md: v8.1 closed out as a "bug exposing release." The real code bug identified there — ResultFormatter.py, MiscSearch.py, IndexBuilder.py each carrying their own separate, older $boardurl-scraping "self-correcting" guess to build SMF links, wrong on Dev — had a fix designed (ise_settings.py v4.0, a hardcoded _SITE_ROOTS table keyed by (environment, install)) but explicitly marked "built and tested, not yet deployed."
Rollout order agreed at the start of tonight: Live Test/smf20 → Dev → Dev Test/smf20 → Live, testing each leg before moving to the next.
The ISE_Code_v8_2.zip Andrew uploaded turned out to be the pre-fix v8.1 baseline — labeled v8.2 at upload time, but ise_settings.py inside it was still v1.0 (the scraping version), and ResultFormatter.py/ MiscSearch.py/IndexBuilder.py still each had their own separate $boardurl-scraping regex. PDFsearch.py imported get_base_url() but from the old v1.0 module — centralized scraping, not the real fix. Both bridges (qf_search_bridge.php, qf_MegaSearch_bridge.php) still had the putenv('ISE_BASE_URL=...') calls that were supposed to be removed.
Confirmed by grep before touching anything — this wasn't a guess.
Andrew supplied the 4 real $boardurl values (Live/main, Live/smf20, Dev/main, Dev/smf20), and ise_settings.py v4.0 was rewritten properly: a flat _SITE_ROOTS table, two whitelisted words (environment, install) as the lookup key, raising loud with a list of valid options on any unrecognised combination — never scraping, never guessing.
All 4 downstream files updated to import get_base_url()/get_project_url() from this one shared module. Both bridges had their putenv() calls removed (kept ISE_ROOT, which is unrelated).
Verified, not just read-through: ran ise_settings.py against a simulated Live/smf20 install — confirmed https://forum.drugs-and-users.org/smf20 (base) and .../smf20/The_ISE_Project (project), no double segment. Confirmed an unrecognised (environment, install) pair raises immediately with the full valid-options list. ResultFormatter.py import-tested end-to-end against that same simulated layout.
IndexBuilder.py, MiscSearch.py, PDFsearch.py had their doc-block @version bumped to 7.4.2 but not their separate, actually-printed VERSION constant — caught because Dev's own IndexBuilder.py run output still showed v7.4.1 after "deploying" the fix. This is exactly the failure mode the project's doc-block convention exists to catch: the doc-block and the visible runtime stamp are two different things, and only one had been updated. Fixed and redelivered.
IndexBuilder.py and MiscIndexer.py both ran clean, no URL errors — 9 posts / 952 words indexed, 5 misc attachments / 1046 words indexed. This is the real confirmation that the fix holds on Dev, the environment the whole v8.1 bug chase started from.Dev Test/smf20 and Live (final legs) were not reached tonight — the PDF-indexing detour (below) consumed the rest of the session.
Running MegaIndexer.sh on Dev/smf20 (to confirm the URL fix held for PDF search too), PDFIndexer.py crashed: ModuleNotFoundError: No module named 'fitz'. Andrew was confident PDF indexing used to work on Dev.
pip3 install pymupdf — failed compiling from source (gctx undeclared in fitz_wrap.c, thread-local-storage-related). Confirmed: PyMuPDF has never published 32-bit manylinux wheels for any recent version — not a recent regression, a permanent architecture gap. Dev is 32-bit CentOS 6.10; pip was always going to fall back to a source compile here.1.11.2. --only-binary :all: returns nothing at any version, confirming no wheel exists anywhere in PyMuPDF's published history for this platform.yum search mupdf / mupdf-devel — nothing in any enabled repo, including EPEL (checked with --enablerepo=*).centos-sclo-rh repo already present in /etc/yum.repos.d/ (from an earlier session), but both the i386 and x86_64 SCLo trees 404 in the CentOS vault (confirmed via curl -sIL, following the redirect through to the real status code). CentOS 6 is EOL; SCLo simply isn't archived there for either architecture. Devtoolset is off the table via yum, full stop.libmupdf.a/ libmupdfthird.a built clean under Dev's stock gcc 4.4.7. The only build failure was the optional OpenGL desktop viewer (platform/gl, needs GLFW headers Dev doesn't have) — irrelevant to pymupdf, skipped via HAVE_GLFW=no.gctx/TLS wall entirely (real progress — this proved gcc 4.4.7 genuinely can compile MuPDF's C code, contrary to the working assumption up to that point). But hit a new, harder, different failure: dereferencing pointer to incomplete type on fz_point_s/ fz_rect_s/fz_stext_sheet_s. This means the downloaded MuPDF 1.11 source release doesn't structurally match what pymupdf 1.11.2's SWIG-generated fitz_wrap.c actually expects — PyMuPDF's version numbers in that era didn't reliably track MuPDF's own release tags. Not a compiler-age problem anymore; would need finding the exact MuPDF source snapshot pymupdf 1.11.2 was built against..gitmodules. Cloned the 1.11.2 tag specifically to check. Confirmed: this tag has no mupdf/ submodule directory at all — the pinned-submodule approach started later in PyMuPDF's history. Dead end, conclusively (not a wrong-directory mistake — verified with find / -iname "PyMuPDF*" turning up nothing but a PDF doc and an icon file).Net result: every real path — yum, every PyPI version, SCL/devtoolset, standalone MuPDF source compilation, and the git submodule route — was tried and ruled out with evidence, not abandoned on a guess. This is a genuine, narrow version-compatibility wall specific to 32-bit CentOS 6 + this era of PyMuPDF, not a skill or effort gap.
Mid-troubleshooting, a pip3 install pymupdf==1.11.2 was accidentally run on Live (production, not Dev) — terminal prompts had become hard to track across two boxes late at night. Pip's own failure handling auto-rolled back and restored the pre-existing working install. Confirmed safe afterward: import fitz and fitz.open() both work cleanly on Live, unchanged. No lasting damage, but a real reminder of how mistakes creep in under fatigue and cross-terminal confusion — this is the moment the session's pace deliberately slowed down.
PDF indexing on Dev remains unresolved and deferred, not abandoned. The clear next real avenue, floated late in the session and not yet attempted: build a small, modern 64-bit VM (Ubuntu/Debian, minimal) in VMware alongside the existing 32-bit Dev VM, run pymupdf there via a normal wheel install (no compiling anything), and expose it as a small HTTP microservice that PDFIndexer.py/PDFsearch.py call over the network instead of importing fitz in-process. Docker inside that new VM was discussed as a natural fit for reproducibility, not a replacement for the VM itself. This needs its own scoped session — a real (if modest) architecture and rewrite, not a same-night bolt-on.
While scoping a "user-selectable AND/OR" feature idea (prompted by Andrew wanting ISEpdf/Deep Search to actually live up to its name), reading the real code turned up something neither of us expected:
ANDMatcher.py): has always been strict AND — every term must match, via set intersection.PDFsearch.py's own search()): was already soft-OR the whole time — any page matching at least one term gets included, with match_count only feeding the ranking score, never filtering. There was never a hard "all terms must match" mode in ISEpdf at all.This wasn't documented anywhere before tonight. It changed the shape of the feature: rather than adding OR to both surfaces symmetrically, the real gap was adding an AND mode to ISEpdf specifically (posts already had it; PDF never did).
PDFsearch.py v7.5.0)search(search_terms, word_index, pdf_store, match_mode="or") — new parameter, "or" default preserves every existing caller's behavior exactly (MegaSearch.py, cron jobs, saved bookmarks — nothing changes for them).match_mode="and" — a new filter step applied after the existing hit-collection logic: keeps only (id_attach, page) entries where every term in the query matched. A quoted "phrase" term counts as one required unit, not one requirement per word inside it (it already internally requires its own words to co-occur to register a hit at all).get_ranked_results(query, limit=None, match_mode="or") and a new main() CLI flag, --mode {or,and}, defaulting to or.Extracted tokenise()/search() and their small dependencies straight from the real file and ran them against synthetic word-index data (not a live DB — this environment can't reach one):
"meth synthesis" — all 3 pages returned (any-term match), as expected."hot plate" reaction) — correctly treated the phrase as one required unit alongside the bare word, found only the one matching page.meth*) combined with a bare word in AND mode — initially wrote an assertion that turned out to encode a wrong assumption about the synthetic data (expected only 1 result, got 2); investigating showed the code was actually correct — a second document genuinely had both a meth*-matching word and the other term, so correctly appearing in AND results wasn't a bug. Confirmed wildcards behave as one required unit in AND mode, same as phrases.Andrew's explicit call, twice reinforced ("let's not advertise — let them discover," and confirmed again later in the conversation): no checkbox/radio panel on the search page. Instead, the plan (not yet built) is to recognize a bare, unquoted AND typed directly in the query string itself — the same pattern as Google's typed OR — strip it before normal tokenizing, and set match_mode="and". Documented in ISE_help.html for anyone curious enough to read it; invisible to everyone else. This needs the actual web bridge file (not yet uploaded this session) to wire in properly.
Longer-term idea floated for the same "the tool teaches its own logic as you use it" philosophy: +/- term weighting (Google-style), explicitly deferred — "some other day... refine it over the years."
ANDMatcher.py/QueryParser.py) OR mode — the mirror-image gap (posts has AND, lacks OR) — not scoped in detail yet, deliberately deferred alongside the above.Andrew's framing, worth preserving verbatim in spirit: tonight's PyMuPDF wall "makes for a fabulous case study — a real world example where we use the very tools we developed" on the project itself. The instinct to document the dead end as thoroughly as the win is deliberate — a transcript that only records successes is much less useful later than one that also shows exactly what was ruled out and why, so a future session (or collaborator) doesn't have to re-run the same experiments to rediscover the same wall.