ISE Project — v7.4 → v7.4.1#5 Checkpoint

Context: Session covered a large ranking/search-quality overhaul across ISE's three search surfaces (post search, ISEpdf, ISEmisc), following on from an earlier session that fixed a Dev/Live rsync throughput problem and an ISEpdf crash (BASE_URL bug).

What changed, in order

v7.4 (initial pass)

• Wired PDFsearch.py and MiscSearch.py into RankingEngine.py (the same scoring engine post search already used) instead of a simple match-count sort.
• Added the "Why is this ranked #N?" explainer (score, confidence, numeric breakdown, plain-language checklist) to all three search surfaces — extracted into a reusable render_ranking_explainer() in ResultFormatter.py so all three share one implementation.
• Unified case-sensitivity/quoting: normal words case-insensitive, "double-quoted phrases" case-sensitive, consistently across ISE/ ISEpdf/ISEmisc. ISEmisc got real phrase matching against stored file content; ISEpdf got phrase-as-AND-on-same-page (true adjacent-phrase matching isn't possible for PDF — its index has no stored page text).
• Fixed PDFIndexer.py's tokenizer (a long-standing TODO): was a bare .split() with no lowercasing and no punctuation stripping.

v7.4.1 (bugfix round after PDFIndexer.py's own fix regressed)

• PDFIndexer.py's new tokenizer copied a regex from MiscIndexer.py that allowed "." inside tokens — correct for misc file content, wrong for PDF text, where footnote/citation markers glue onto words with no whitespace ("meth.135"). Fixed by dropping "." from PDF's pattern.
• Extended "every module shows its version when it runs" to every script in the package that displays/traces/logs anything — several had no version tracking at all before (search.py, IndexBuilder.py, orphan_check.py, daily_queries.py, query_log_reader.py, and the small library files' debug entry points).
• Normalized every module's version marker to a single package-wide number (7.4.1) at Andrew's request, retiring the per-file sub-numbering (7.4#2, 7.3#5, etc.) some files had picked up.

v7.4.1#2 through #5 (real ranking bugs, found via Andrew's live QA)

1. Crash — RankingEngine.py's rules used result.get(key, ""), which only falls back on a missing key, not a present-but-None one. Orphaned PDF attachments (no resolvable topic) have subject/text/board_name explicitly None, so .lower() crashed every search that touched one. Fixed across 5 rules.
2. "Chemistry" search didn't surface the actual chemistry book — occurrences was len(matched_terms), which is always 1 for a single-word query regardless of how saturated the document is. Switched to page_count (pages actually matched).
3. That fix immediately hit a plateau bug — OccurrenceRule's point formula capped out by ~page 7, so a 12-page and 62-page match scored identically. Widened the cap (35→80 points) — helped, but incompletely.
4. RecencyRule was overriding real content matches — its swing (up to 29 points) was large enough that a newer, smaller document beat an older, larger match. Scaled Recency to 1/4 weight for PDF.
5. The real, permanent fix — any fixed additive cap on Occurrence was always going to plateau eventually; a sufficiently dense document (PIHKAL, per Andrew: "packed full of every Phenethylamine invented") still hit the raised ceiling. Rebuilt Occurrence to score by coverage — matched pages ÷ the document's own total page count (added total_pages to PDFsearch.py's result data from pdf_store.json) — which has no ceiling problem at any document size.

Current state

• Package is at v7.4.1#5 (RankingEngine.py) / v7.4.1#3 (PDFsearch.py), everything else at v7.4.1.
• Andrew has been installing by swapping the whole The_ISE_Project folder each round, no --fresh reindex needed for any of the RankingEngine/search-side fixes (only the earlier PDFIndexer.py tokenizer fix required reindexing).
• Coverage-based Occurrence scoring (#5) was tested against synthetic data before packaging but not yet confirmed against Andrew's real "chemistry" search results — that's the natural next check.

Style/working notes for this thread

• Andrew wants every fix verified with an actual runnable test (Python one-liner reproducing the exact scenario) before packaging, not just reasoned about — several rounds this session were fixes that looked right on paper but didn't survive real data.
• He's been pasting real screenshots of live results as evidence, which has been far more useful than descriptions — worth asking for a screenshot early if a ranking complaint comes up again.
• Mandatory doc-block/version convention applies to every touched file, including ones that never had a version before.