The ISE (Intelligent Search Engine) Project

[smfadmin (OP)]

Called the "QueryFrontend"

Yes. Let's call the first component QueryFrontend.

For now it is not a full SMF mod. It is a tiny instrumentation patch that adds one capability:

> Capture the raw search query as soon as SMF receives it, then let SMF continue normally.

Create:

QueryFrontend.php

with:

Code: [Select]

<?php

/*
 * QueryFrontend
 *
 * Phase 1:
 * Capture raw SMF search queries via syslog.
 *
 * Target:
 * SMF 2.0.19
 */

function qf_log_query($search_string)
{
    openlog(&#39;smf-queries&#39;, LOG_PID, LOG_LOCAL0);
    syslog(LOG_INFO, $search_string);
    closelog();
}

?>

Test first:

Code: [Select]

php -r "openlog('smf-queries', LOG_PID, LOG_LOCAL0); syslog(LOG_INFO,'test123'); closelog();"
grep test123 /var/log/messages

Take a backup:

Code: [Select]

cd /var/www/html/Sources $
cp Search.php Search.php-orig

After this:

Code: [Select]

function PlushSearch2()
{
        global $boardurl, $scripturl, $modSettings, $sourcedir, $txt, $db_connection;
        global $user_info, $context, $options, $messages_request, $boards_can;
        global $excludedWords, $participants, $smcFunc, $search_versions, $searchAPI;

Add this:

Code: [Select]

// ---- Log every search attempt, error or not
          require_once($sourcedir . '/QueryFrontend.php');
          qf_log_query($_REQUEST['search']);
// ---- Log end

That's the entire first version.
The important design decision:
No modification to search logic
No filtering
No tokenising
No database changes
No Python yet
It is purely a sensor.
Once installed on Dev, test:

1. Perform a search:

database access denied

2. Check:

grep smf-queries /var/log/messages

Expected:

Jul 22 14:xx:xx server smf-queries[xxxx]: database access denied

After this works, we can package QueryFrontend as an SMF 2.0.19 mod with:

install.xml
file patch
uninstall support

The nice part is this first component should remain useful forever — even when the AI retrieval layer exists. It becomes the query telemetry feed for the system.

The modules

QueryFrontend
QueryParser
Tokeniser
TokenNormaliser
IndexBuilder
IndexLookup
ANDMatcher
SortEngine
ResultFormatter

Notice that each module has exactly one responsibility:

Module------------------------------------Responsibility

QueryFrontend------------------------Accept the raw query from SMF.
QueryParser----------------------------Parse the user's query into a structured form.
Tokeniser---------------------------------Split the query into tokens.
TokenNormaliser---------------------Lowercase, remove punctuation, stemming (later).
IndexBuilder-----------------------------Build and maintain the search index from SMF posts.
IndexLookup----------------------------Retrieve the posting lists for each token.
ANDMatcher----------------------------Compute the intersection of the posting lists.
SortEngine-------------------------------Order the matching posts (newest first initially).
ResultFormatter-----------------------Return the results in a format SMF or another client can use.

The nice thing is that you can develop and test each one independently.

For example, ANDMatcher doesn't care where the token lists came from:

token1 = {5, 9, 14, 22}
token2 = {2, 5, 14}
token3 = {5, 14, 30}

result = token1 & token2 & token3

# {5, 14}

If that works, the module is done.

Later, you can swap implementations without changing the interfaces:

TokenNormaliser might later add stemming.

SortEngine might later use a relevance score instead of just date.

IndexLookup might move from MySQL to another storage mechanism.

Nothing else has to change.

One suggestion: I'd introduce one extra module between QueryFrontend and

QueryParser

QueryFrontend
    │
    ▼
QueryValidator
    │
    ▼
QueryParser

Its only job would be to reject or clean obviously invalid input before the parser sees it. Right now it might do almost nothing, but having that separation gives you a natural place to add things later (length limits, malformed queries, unsupported syntax, etc.) without complicating the parser.

Overall, this is starting to look less like "some Python scripts" and more like a well-structured information retrieval engine. That's a solid foundation for whatever features you decide to add later.
.
[Ok, that's great now that you know just how complex it needs to be.

The modules

1. QueryParser
2.Tokeniser
3. TokenNormaliser
4. IndexBuilder
5. IndexLookup
6. ANDMatcher
7. SortEngine
8. ResultFormatter

Notice that each module has exactly one responsibility:

Module   Responsibility

QueryFrontend   Accept the raw query from SMF.
QueryParser   Parse the user's query into a structured form.
Tokeniser   Split the query into tokens.
TokenNormaliser   Lowercase, remove punctuation, stemming (later).
IndexBuilder   Build and maintain the search index from SMF posts.
IndexLookup   Retrieve the posting lists for each token.
ANDMatcher   Compute the intersection of the posting lists.
SortEngine   Order the matching posts (newest first initially).
ResultFormatter   Return the results in a format SMF or another client can use.


The nice thing is that you can develop and test each one independently.

For example, ANDMatcher doesn't care where the token lists came from:

token1 = {5, 9, 14, 22}
token2 = {2, 5, 14}
token3 = {5, 14, 30}

result = token1 & token2 & token3

# {5, 14}

If that works, the module is done.

Later, you can swap implementations without changing the interfaces:

TokenNormaliser might later add stemming.

SortEngine might later use a relevance score instead of just date.

IndexLookup might move from MySQL to another storage mechanism.


Nothing else has to change.

One suggestion: I'd introduce one extra module between QueryFrontend and QueryParser:

QueryFrontend
    │
    ▼
QueryValidator
    │
    ▼
QueryParser

Its only job would be to reject or clean obviously invalid input before the parser sees it. Right now it might do almost nothing, but having that separation gives you a natural place to add things later (length limits, malformed queries, unsupported syntax, etc.) without complicating the parser.

Overall, this is starting to look less like "some Python scripts" and more like a well-structured information retrieval engine. That's a solid foundation for whatever features you decide to add later.

Post Merged: July 22, 2026, 05:25:42 PM
Me:

But please build me some psuedocode for the QueryParser

Sure. Let's define the QueryParser as a very simple first version. Its job is not to search, tokenise, or rank. It only understands the structure of the user's query and passes a clean representation to the next module.

QueryParser — responsibility

Input:

"database access denied"

Output:

{
    "terms": [
        "database",
        "access",
        "denied"
    ],
    "operator": "AND"
}


---

Pseudocode

MODULE QueryParser


FUNCTION parse_query(raw_query):

    IF raw_query is empty:
        RETURN error("No query supplied")


    query = trim(raw_query)


    query = remove_extra_spaces(query)


    IF query contains explicit operators:
       
        detect operators:
            AND
            OR
            NOT

        build query structure


    ELSE:

        assume default operator = AND


    extract search terms


    FOR each term:
        remove invalid characters


    RETURN ParsedQuery


---

Example flows

Simple query

Input:

database access denied

Parser produces:

ParsedQuery
{
    operator: AND

    terms:
       database
       access
       denied
}


---

Future query with operators

Input:

database AND access AND denied

Parser produces:

ParsedQuery
{
    operator: AND

    terms:
       database
       access
       denied
}


---

Future OR support

Input:

mysql OR mariadb

Output:

ParsedQuery
{
    operator: OR

    terms:
       mysql
       mariadb
}


---

Important design decision

The QueryParser should not do:

❌ lowercase conversion
❌ stemming
❌ database lookup
❌ matching posts
❌ sorting results

Those belong elsewhere.

The pipeline stays:

Raw text
   |
   v
QueryParser
   |
   v
Tokeniser
   |
   v
TokenNormaliser
   |
   v
IndexLookup
   |
   v
ANDMatcher
   |
   v
SortEngine
   |
   v
ResultFormatter

This keeps the modules replaceable. A simple QueryParser today can later evolve into a full query language parser without forcing changes throughout the search engine.

Me:

I need it to support searches like: "drugs of addiction "harm reduction" Australia"

Ah, that changes the design significantly. The parser cannot just split on spaces. It needs to understand phrases.

Your example:

drugs of addiction "harm reduction" Australia

should not become:

drugs
of
addiction
harm
reduction
Australia

because "harm reduction" is a single search unit.

The QueryParser should produce something like:

ParsedQuery

operator: AND

terms:
    drugs
    of
    addiction
    Australia

phrases:
    "harm reduction"

Then later modules decide how to search those.

A better pseudocode:

MODULE QueryParser


FUNCTION parse_query(raw_query):

    IF raw_query is empty:
        return error


    query = trim(raw_query)


    phrases = extract_quoted_phrases(query)

        Example:
        "harm reduction"

        becomes:

        phrases:
            harm reduction


    remove phrases from query


    remaining_words = split_on_spaces(query)


    terms = []

    FOR each word in remaining_words:

        IF word is not empty:
            terms.append(word)


    detect operators:

        IF query contains AND:
            operator = AND

        ELSE IF query contains OR:
            operator = OR

        ELSE:
            operator = AND


    RETURN:

        {
          operator: operator,
          terms: terms,
          phrases: phrases
        }

For your example:

Input

drugs of addiction "harm reduction" Australia

Output

{
 "operator": "AND",

 "terms": [
   "drugs",
   "of",
   "addiction",
   "Australia"
 ],

 "phrases": [
   "harm reduction"
 ]
}

Then the next stages can decide:

Tokeniser:

drugs → token

addiction → token

Australia → token


PhraseMatcher (a future module):

verify "harm reduction" appears together and in that order



This is actually the beginning of a proper search language.

A possible future query syntax:

drugs addiction "harm reduction" Australia

means:

> Find posts containing all words AND the exact phrase "harm reduction".



Later:

meth OR amphetamine "harm reduction"

means:

> Find posts containing either meth or amphetamine AND the phrase.



So I would slightly extend the architecture:

QueryFrontend
      |
QueryParser
      |
Tokeniser
      |
TokenNormaliser
      |
IndexBuilder
      |
IndexLookup
      |
ANDMatcher
      |
PhraseMatcher
      |
SortEngine
      |
ResultFormatter

The good news is your example is exactly the kind of real query that forces the design to become useful rather than just a toy token matcher.

Me:

Ok, that's great now that you know just how complex it needs to be.

Next I'll run it past "Programming" 😋 and see how well it translates to Python code. Your input is appreciated.

😄 Good plan. This is exactly the point where it is worth getting a programming-focused review.

The important thing is that the design is now language-independent. Python is just the implementation vehicle.

When you hand it over, the key points for "Programming" to preserve are:

QueryParser contract

Input

raw_query: string

Example:

drugs of addiction "harm reduction" Australia


---

Output

A structured object, for example:

Code: [Select]

{
    "operator": "AND",
    "terms": [
        "drugs",
        "of",
        "addiction",
        "Australia"
    ],
    "phrases": [
        "harm reduction"
    ]
}

---

The main Python tasks are actually quite manageable:

1. Extract quoted phrases

Find text between ".

Remove those phrases from the remaining string.



2. Parse remaining words

Split into tokens.

Preserve ordering for now.



3. Detect operators

Default:

AND

Later:

OR
NOT


4. Return a predictable structure


The beauty of doing it this way is that you can test it without touching SMF or the database:

Code: [Select]

Input:
mysql "access denied"

Expected:
terms:
    mysql

phrases:
    access denied

If the parser passes those tests, the next module can be developed independently.

One thing I would tell the Python implementer: don't over-engineer the first version. Avoid jumping straight into NLP libraries. For your use case, a small, transparent parser you can understand and modify is probably more valuable initially.

The really interesting parts come later:


building the inverted index,

deciding what to store,

ranking results,

adding fuzzy matching,

using NLP intelligently.


But the parser is the doorway. Get that clean and the rest of the engine has a solid foundation. 👍

Me:

I like his pseudocode, despite being imprecise,  I think it would be AI friendly.

I agree with your instinct. Pseudocode does not have to be executable to be valuable — especially when the next reader is another AI or a programmer.

In fact, for an AI-assisted project, good pseudocode is often better than prematurely detailed code because it captures the intent without locking the implementation into one language.

The QueryParser pseudocode we have is useful because it defines:

Purpose — what the module is responsible for.

Boundaries — what it must not do.

Input — raw user query.

Output — structured query object.

Examples — what behaviour is expected.


Those examples are particularly important. An AI can infer a lot from:

Input:
drugs of addiction "harm reduction" Australia

Output:
terms:
    drugs
    of
    addiction
    Australia

phrases:
    harm reduction

operator:
    AND

That gives it a behavioural contract.

Where I would refine it for an AI handoff is by making the ambiguity explicit:

NOTE:
Quoted text is treated as an exact phrase.
Unquoted words are individual search terms.
Default operator between terms is AND.
Parser does not perform matching.
Parser does not access the database.

That prevents the next model from "helpfully" adding search logic into the parser.

This is actually similar to how good software teams work: a module specification is often more important than the first implementation. Once the contract is clear, multiple people (or models) can implement it and still produce compatible components.

Your checkpoint-file idea fits perfectly here too. A future session could load:

Code: [Select]

Project: SmartSearch

Current module:
QueryParser

Contract:
[input/output]

Examples:
[tests]

Do not:
[database access, ranking, NLP]
[/quote]
and immediately understand the task without replaying the entire history.

For a multi-model workflow, these small design documents may end up being one of the most valuable assets you create.

[Chip]

That answers decision #1 — yes, quoted phrases need to be treated as one exact unit, mixed in with loose words. Here's the parsing logic for that:

```

Code: [Select]

MODULE QueryParser

INPUT: raw_query — e.g. drugs of addiction "harm reduction" Australia

FUNCTION parse(raw_query):
    IF raw_query is empty/whitespace-only:
        RETURN error "empty query"

    tokens = []
    scan raw_query left to right:
        IF a "quoted phrase" is found:
            extract contents between the quotes as ONE token, marked type=PHRASE
        ELSE:
            each whitespace-separated word becomes its own token, marked type=WORD

    RETURN ParsedQuery(
        raw = raw_query,
        tokens = [
            {type: WORD,   value: "drugs"},
            {type: WORD,   value: "of"},
            {type: WORD,   value: "addiction"},
            {type: PHRASE, value: "harm reduction"},
            {type: WORD,   value: "Australia"},
        ]
    )
```

Real Python for that part (regex does the quote-aware splitting in one pass):

```python

Code: [Select]

import re

def parse(raw_query: str):
    raw_query = raw_query.strip()
    if not raw_query:
        raise ValueError("empty query")

    tokens = []
    for match in re.finditer(r'"([^"]+)"|(\S+)', raw_query):
        phrase, word = match.group(1), match.group(2)
        if phrase is not None:
            tokens.append({"type": "PHRASE", "value": phrase})
        else:
            tokens.append({"type": "WORD", "value": word})

    return {"raw": raw_query, "tokens": tokens}m```

Downstream effect on the rest of the pipeline: `Tokeniser.py` normalizes each `WORD` individually (lowercase, strip punctuation), but a `PHRASE` token needs to be indexed/matched as a **consecutive run**, not just AND'd — meaning `IndexLookup`/`ANDMatcher` need to check that "harm" and "reduction" appear adjacent in a post, not just that both appear somewhere. That's a meaningfully bigger lift than plain AND-of-words (needs word-position data in the index, not just token→post_id).

Given that's a real jump in complexity — do you want phrase matching to require true adjacency (harder, needs positional index), or is "both words present, order doesn't matter" close enough for v1 ?

order doesn't matter

Good — that simplifies things a lot. If order/adjacency doesn't matter, a "phrase" is really just a shorthand for "these words must all appear," which means it collapses into the exact same AND logic as everything else. You don't even need a separate `PHRASE` token type downstream — `QueryParser` can just split the quoted content into individual words and merge them into the same flat word-list as everything else:

```
python

Code: [Select]

import re

def parse(raw_query: str):
    raw_query = raw_query.strip()
    if not raw_query:
        raise ValueError("empty query")

    words = []
    for match in re.finditer(r'"([^"]+)"|(\S+)', raw_query):
        phrase, word = match.group(1), match.group(2)
        if phrase is not None:
            words.extend(phrase.split())   # "harm reduction" -> harm, reduction
        else:
            words.append(word)

    return {"raw": raw_query, "words": words}

```

From our example:

```
drugs of addiction "harm reduction" Australia
```

→ `["drugs", "of", "addiction", "harm", "reduction", "Australia"]` — five plain words, all AND'd together, quotes only ever mattered as a way to group multi-word concepts visually while typing.

That means `QueryParser`'s whole job really is just "strip the quote syntax, flatten everything into one word list" — `Tokeniser.py` then normalizes each word the same way regardless of whether it came from inside quotes or not, and `ANDMatcher.py` never needs to know phrases existed at all.

Much simpler module than I first sketched. Ready to move to `Tokeniser.py` next, or do you want to sit with `QueryParser` a bit longer?

Me:

No hurry but I like the thought of my own Tokeniser pleasing.

No rush at all — enjoying the design work is a good reason on its own to sit with it. `Tokeniser.py` is a genuinely satisfying one to get right, since it's the piece that decides what actually counts as "the same word" across your whole archive.

Basic version, since it feeds off the flat word-list `QueryParser` now hands it:

```
python

Code: [Select]

import re

def tokenise(words: list) -> list:
    tokens = []
    for w in words:
        w = w.lower()
        w = re.sub(r'[^\w]', '', w)   # strip punctuation, keep letters/digits/underscore
        if w:
            tokens.append(w)
    return tokens

```

For your example, `["drugs", "of", "addiction", "harm", "reduction", "Australia"]` → `["drugs", "of", "addiction", "harm", "reduction", "australia"]` — just lowercased and punctuation-stripped.

A few real decisions that shape it, worth pinning down while it's cheap:

1. **Stopwords** — "of," "the," "a," "and" etc. Drop them, or keep everything? Dropping shrinks your index and speeds lookups, but for *your own* forum with domain-specific phrasing, sometimes a stopword carries meaning you'd search for.
2. **Numbers** — keep `"2026"` or dosage-like `"5mg"` as valid tokens, or strip anything non-alphabetic? Given your content, dosage/number tokens are probably worth keeping.
3. **Stemming** — `"reducing"` vs `"reduction"` vs `"reduce"` — do you want those to match each other, or keep it literal for v1? Stemming is a bigger lift (needs a stemming library or hand-rolled suffix rules) — my instinct is skip it for v1, add later if you notice yourself missing results because of it.

Given "basic and pleasing" — my gut says keep numbers, drop nothing else, skip stemming for now. Want to lock those three in, or think out loud on any of them?

[Chip]

Code: [Select]

import re

def parse(raw_query: str) -> dict:
    """
    Parses a raw search string into a mix of loose words and
    intact quoted phrases. Quoted phrases are kept as single
    units (with their embedded blank preserved) so downstream
    matching can require true adjacency, not just presence.
    """
    raw_query = raw_query.strip()
    if not raw_query:
        raise ValueError("empty query")

    terms = []
    for match in re.finditer(r'"([^"]+)"|(\S+)', raw_query):
        phrase, word = match.group(1), match.group(2)
        if phrase is not None:
            terms.append({"type": "PHRASE", "value": phrase})
        else:
            terms.append({"type": "WORD", "value": word})

    return {"raw": raw_query, "terms": terms}


if __name__ == "__main__":
    import sys
    query = " ".join(sys.argv[1:])
    result = parse(query)
    print(result)


Code: [Select]

python3 QueryParser.py '11 22 "33 44"'


Code: [Select]

{'raw': '11 22 "33 44"',
 'terms': [
    {'type': 'WORD', 'value': '11'},
    {'type': 'WORD', 'value': '22'},
    {'type': 'PHRASE', 'value': '33 44'},
 ]}


The ISE Project (Intelligent Search Engine) — Checkpoint (v1)

Forum: drugs-and-users.org (self-hosted SMF). Official project directory: /var/www/html/The_ISE_Project on the dev box (CentOS 6.10, i686/32-bit). Live forum runs SMF 2.0.19.

Purpose: a private, personal search tool for Andrew — not a replacement for SMF's native search, not for the general userbase. Pure Python search engine (stdlib + PyMySQL only, no web framework/API/daemon), reading SMF's own MySQL tables as its data source, completely decoupled from PHP at query time. A separate PHP-side syslog logger captures every real search string submitted through the live forum, feeding a query corpus for later tuning. A future, deliberately separate phase adds an AI retrieval layer using two chosen models — Claude Sonnet 5 and Gemini — via an orchestrator.py (not yet started, not a near-term blocker).

Architecture (settled)

SMF (PHP + MySQL)                     You      |                                |      | (source data, one-way,         v      |  read only)                search.py      v                                |IndexBuilder.py                        v      |                          QueryParser.py      v                                |  Search Index  <-------------------- (feeds into ANDMatcher.py)                                        |                                        v                                    Results

• PHP/SMF = forum application (unchanged, except the tiny logger patch below)
• MySQL = shared data store (SMF's own tables + the search index tables)
• Python = the entire search engine, stdlib + PyMySQL, no other dependencies
• No PHP↔Python IPC, no daemon, no API, no web service — this is a CLI tool

  run manually, e.g. python3 search.py "database access denied"

• Index freshness is manual/on-demand (IndexBuilder.py re-run whenever

  wanted), not live-triggered by new posts — accepted tradeoff given   single-user scope

Files/components — status and whether to use

File	Status	Use it?
QueryFrontend.php	Working, confirmed on dev. Contains qf_log_query($search_string) — opens syslog LOCAL0, logs the raw string, closes. Simple, unchanged since first written.	Yes
Sources/Search.php — PlushSearch2()	Working, confirmed on dev. Real search entry point (NOT PlushSearch1(), which only renders the search form). Logger call sits right after the global declarations: require_once($sourcedir . '/QueryFrontend.php'); qf_log_query($_REQUEST['search']); — guard removed (the !empty($_REQUEST['search']) check was blocking real logging on this fork and was deliberately dropped)	Yes — needs porting to live (dev-confirmed only so far, see open items)
QueryParser.py	Written, NOT yet tested. Parses raw query string into a list of terms, each `{'type': 'WORD'	'PHRASE', 'value': ...}. Quoted phrases preserved as literal, whitespace-exact units (see verified facts).rawfield retained unmodified for logging. Has a workingif name == "main":` block for CLI testing.	Yes — pending first real test run
Tokeniser.py	Not yet built. Design needs revising post-phrase-matching reversal — must handle WORD and PHRASE terms differently, not treat everything as one flat word list (see open items)	Not started
IndexBuilder.py	Not yet built. Needs to read smf_messages/smf_topics (and optionally smf_boards) from MySQL, strip BBCode/HTML entities from body before indexing	Not started
IndexLookup.py / ANDMatcher.py	Not yet built. Must handle mixed term types: WORD → presence check; PHRASE → literal, whitespace-exact substring match (NOT a word-position/adjacency index — see verified facts)	Not started
SortEngine.py	Not yet built. Simple: sort qualifying posts by post date, newest first	Not started
ResultFormatter.py	Not yet built. Builds SMF permalinks: https://drugs-and-users.org/index.php?topic={id_topic}.msg{id_msg}#msg{id_msg}	Not started
daily_queries.py	Working, confirmed correct. Lists today's searches by grep-parsing smf-queries lines out of /var/log/messages, filtered by today's date	Yes
query_log_reader.py	Written, broken — produces no output at all. Same general approach as daily_queries.py (parse smf-queries lines from /var/log/messages) but something in its logic/regex isn't matching. Root cause not yet diagnosed.	No — needs debugging before use
orchestrator.py	Not started. Future AI retrieval layer, models: Claude Sonnet 5 + Gemini only (not ChatGPT/chatty, not Copilot — those are used for drafting/dreaming, not chosen as orchestrator models)	Future phase, not a near-term blocker

Verified facts — don't re-check

• Real SMF search-submission entry point is PlushSearch2() in Sources/Search.php, mapped from index.php's action array: 'search2' => array('Search.php', 'PlushSearch2'). PlushSearch1() is a decoy for this purpose — it only renders the search form, never fires on an actual submitted query.
• The !empty($_REQUEST['search']) guard around qf_log_query() was removed in live PlushSearch2() because logging silently failed to fire with the guard in place. Tradeoff accepted: PHP will throw an undefined-index notice on Search.php loads with no search term — not yet cleaned up, not urgent.
• Syslog logging confirmed fully working end-to-end on the 2.0.19 dev box: openlog('smf-queries', LOG_PID, LOG_LOCAL0); syslog(LOG_INFO, $search_string); closelog(); lands correctly in /var/log/messages (LOCAL0 falls through to the default catch-all on this box, no rsyslog config changes needed).
• daily_queries.py (parses smf-queries lines from /var/log/messages, filters by today's date) is confirmed correct and working.
• query_log_reader.py is confirmed broken — no output at all — despite daily_queries.py's near-identical parsing approach working fine. This means the bug is specific to query_log_reader.py's own logic, not the underlying log format/data.
• Phrase-matching design — final, reversed from an earlier session decision: quoted phrases (e.g. "harm reduction") must be preserved as one literal unit, whitespace included exactly as typed. An earlier decision to flatten phrases into separately-AND'd words (order/adjacency doesn't matter) was explicitly reversed — the point of quoting is precisely to require the embedded blank.
• Whitespace inside phrases must NOT be normalized/collapsed. Vanilla SMF's own native search was tested and found to collapse multiple consecutive blanks down to one before matching — Andrew explicitly decided AGAINST replicating that behavior. A double blank in a quoted search phrase must remain a literal double-blank requirement, matched exactly as typed.
• Practical implication of the above, not yet built: because whitespace must be honored exactly, phrase matching cannot use a simple word-position/adjacency index (which would treat "harm" and "reduction" as adjacent regardless of exact spacing) — it needs a literal substring check against stored post text instead.
• QueryParser.py's current design (confirmed correct via manual trace, not yet run):

  - Regex "([^"]+)"|(\S+) scans left to right   - Every match becomes a term: {'type': 'PHRASE', 'value': <exact text between quotes, unmodified>} or {'type': 'WORD', 'value': <bare word>}   - raw field retains the original query string exactly as typed (with quote characters intact) — this is for logging/display only, NOT used for matching   - Example: '11 22 "33 44"' → terms: [WORD:11, WORD:22, PHRASE:"33 44"] (confirmed correct by manual trace)

• Mobile SSH (JuiceSSH) paste reliability is poor for multi-line code — both direct nano paste and heredoc (cat >> file << EOF) attempts produced mangled/misformatted output. No fix found yet; recommend deferring code-writing to a proper keyboard session rather than fighting mobile paste.
• ChatGPT (chatty) is good for architecture/phased-planning conversations but is explicitly NOT a coder — every actual code fix in this project came from Andrew's own hands-on debugging, not from chatty's suggestions directly.
• Dev box (CentOS 6.10, i686) has broken/dead SCL yum repos (centos-sclo-rh, centos-sclo-sclo both pointing at dead mirror.centos.org URLs) and a gcc alias to colourify/grc that was suspected but ruled out as the cause of a bizarre, unresolved issue where compiled binaries vanish immediately after linking (confirmed via direct /usr/bin/gcc calls bypassing the alias — same failure). This does NOT happen on the live box. Root cause never found; investigation abandoned in favor of building Pythontooling on live instead, since dev's environment is uniquely broken in ways unrelated to the actual 32-bit/CentOS 6 constraints.
• Getting Python 3.6 onto CentOS 6.10 i686 via SCL doesn't work — rh-python36 isn't built for i386/i686 at all (SCL's newer collections are effectively x86_64-only on CentOS 6). Building from source via make altinstall (never make install) is the safe path that doesn't touch system Python 2.6, needed because CentOS 6's yum itself depends on Python 2.6.
• The ChatGPT full-conversation-history export (Settings → Data controls → Export data, ~24hr turnaround) is the only reliable way to get a complete transcript of a ChatGPT conversation — Chrome's "Save as MHTML" only captures whatever's currently mounted in the DOM, and ChatGPT's web UI virtualizes/unmounts long conversations outside the viewport, so MHTML saves of long conversations are silently incomplete (confirmed: an MHTML save of the "AND-based token matching" conversation only captured 5 of 50 real messages).

Open items — decisions, not blockers

• QueryParser.py has never actually been run/tested. Written and manually traced through by hand, believed correct, but no real execution yet (blocked by mobile SSH paste reliability — deferred to a proper keyboard session).
• query_log_reader.py is broken — produces no output, root cause not yet diagnosed. daily_queries.py's near-identical logic works fine, so the bug is localized to this specific script, not the log format.
• Live forum still needs the QueryFrontend.php + PlushSearch2() logger patch ported over from dev. Steps already given (backup Search.php, copy QueryFrontend.php, add the two lines, php -l validate, test) — not yet confirmed done on live.
• Live query logging privacy consideration, flagged but not yet decided: once live, the syslog logger captures every real user's search terms, not just Andrew's own — worth being deliberate about retention/access given the forum's harm-reduction context, before it's silently running against real visitor traffic (if it isn't already).
• Tokeniser.py design needs rework post phrase-matching reversal — must treat WORD and PHRASE terms differently rather than one flat lowercase/punctuation-stripped word list. Not yet started.
• IndexBuilder.py/IndexLookup.py/ANDMatcher.py all need a design decision locked in, following from the whitespace-literal phrase requirement: phrase terms need literal substring matching against stored post text, NOT a word-position/adjacency index. Schema/approach not yet finalized.
• .htaccess (or equivalent) not yet confirmed added to /var/www/html/The_ISE_Project to block direct web access, since the project directory sits inside the live web-accessible document root. Exact Deny/Require all denied lines were given; not yet confirmed applied.

Next actual action

Get QueryParser.py actually running for the first time, from a proper keyboard (not mobile SSH) — run:

python3 QueryParser.py '11 22 "33 44"'

Confirm it prints:

{'raw': '11 22 "33 44"', 'terms': [    {'type': 'WORD', 'value': '11'},    {'type': 'WORD', 'value': '22'},    {'type': 'PHRASE', 'value': '33 44'}, ]}

Once confirmed working, move to Tokeniser.py (redesigned for mixed WORD/PHRASE terms), then lock in the literal-substring approach for phrase matching in IndexBuilder.py/ANDMatcher.py.

Post Merged: July 26, 2026, 04:39:26 PMee Tokeniser.py

Code: [Select]

import re

def tokenise(parsed_query: dict) -> dict:
    """
    Takes QueryParser's output and normalizes each term:
    - WORD terms: lowercased, punctuation stripped
    - PHRASE terms: lowercased (respecting case_sensitive flag),
      but internal whitespace is left completely untouched —
      exact spacing is part of what must match later.
    """
    case_sensitive = parsed_query.get("flags", {}).get("case_sensitive", False)
    tokens = []

    for term in parsed_query["terms"]:
        value = term["value"]

        if term["type"] == "WORD":
            cleaned = re.sub(r'[^\w]', '', value)
            if not case_sensitive:
                cleaned = cleaned.lower()
            if cleaned:
                tokens.append({"type": "WORD", "value": cleaned})

        elif term["type"] == "PHRASE":
            # No whitespace touched, no punctuation stripped —
            # this needs to match literally later
            cleaned = value if case_sensitive else value.lower()
            tokens.append({"type": "PHRASE", "value": cleaned})

    return {
        "version": parsed_query.get("version", 1),
        "raw": parsed_query["raw"],
        "tokens": tokens,
    }


if __name__ == "__main__":
    import sys
    from QueryParser import parse

    query = " ".join(sys.argv[1:])
    parsed = parse(query)
    result = tokenise(parsed)
    print(result)Post Merged: Yesterday at 05:45:54 AMNow you get to the fun part. 😄

The core search engine is the **engine room**. The "really cool" stuff is what you build *on top of it*.

I would avoid randomly adding AI everywhere. The best upgrades are the ones that make your retrieval engine feel intelligent while keeping the results trustworthy.

Here are some ideas, roughly in order of impact:

---

## 1. Relevance scoring (big upgrade)

Right now:

```text
term exists → post matches
```

Make it:

```text
term exists → how important is this match?
```

Example:

Query:

```text
"harm reduction" Australia drugs
```

Post A:

* contains exact phrase "harm reduction"
* contains all words
* recent

Post B:

* contains only "drugs"

Both match, but A should rank higher.

A simple scoring engine:

```text
score =
    +10 exact phrase match
    +5  all query words present
    +3  title match
    +2  recent post
    +1  author/thread relevance
```

Now it feels like a modern search engine.

---

## 2. Snippet generation

Instead of returning:

```text
Post #45861
```

return:

```text
Post #45861

"...we have always promoted harm reduction principles because
people need accurate information about drugs..."

[Read full post]
```

This makes the archive immediately usable.

---

## 3. Search history intelligence

You already have QueryFrontend logging.

Now you can analyse:

* most searched topics
* failed searches
* common terminology
* changes over time

Example:

```text
Most common searches this month:

harm reduction       84
methamphetamine      61
withdrawal           45
```

Your own archive starts telling you what people are interested in.

---

## 4. "Related posts"

This one would be very cool.

User views:

> Post about meth harm reduction

System suggests:

```text
Related discussions:

- Harm reduction strategies (2019)
- Experiences with different routes of administration (2021)
- Medical advice discussion (2023)
```

No AI required initially. Shared tokens can do this.

---

## 5. AI-assisted answers from your archive

This is where your Claude/Gemini/Groq orchestra becomes valuable.

Flow:

```text
User question
      |
      v
SmartSearch retrieves 20 relevant posts
      |
      v
AI models analyse ONLY those posts
      |
      v
Answer with references
```

The AI is no longer guessing from general knowledge. It is working from your 12-year archive.

---

## 6. Search explanations

A very "smart engine" feature:

Instead of only:

```text
Found 42 results
```

show:

```text
Why these results:

Matched:
✓ harm reduction (phrase)
✓ Australia
✓ drugs

Sorted by:
✓ phrase relevance
✓ newest first
```

Users trust systems more when they can see why.

---

## 7. The really ambitious one: "Ask the Archive"

A special interface:

```text
Ask:
"What have we learned about harm reduction over the last decade?"
```

System:

1. Retrieves relevant posts.
2. Groups them by topic/date.
3. Sends them to multiple AI models.
4. Produces a synthesis.

That would genuinely make a 12-year forum archive feel alive.

---

If I were choosing the next three upgrades, I'd do:

1. **SortEngine with relevance scoring**
2. **ResultFormatter with snippets/highlighting**
3. **AI layer using retrieved posts as context**

Because those three would transform it from:

> "a better search box"

into:

> "a knowledge retrieval system for the entire forum."

The foundation you've built is exactly what makes those features possible.

[Chip]

All modules working and [index-Rebuild][/index-Rebuild] trigger coded.

Project complete - use the drop down box and select ISE.

Allows multiple keywords and phrases delimited by double-quotes.

Entities are ANDes and are case-sensitive.

Search index rebuilds every 10 minutes.

Enjoy.

The ISE Project (Intelligent Search Engine) — Checkpoint (v2, supersedes v1)

Forum: drugs-and-users.org (self-hosted SMF, live host: forum.drugs-and-users.org). Live box: 64-bit, SMF 2.0.19 (table prefix confirmed non-default — check actual prefix before assuming smf_). Separate dev box: CentOS 6.10, i686/32-bit. Project directory: /var/www/html/The_ISE_Project (pure code, mod-packaged). Generated index data lives separately at /var/www/html/ISE_data (NOT packaged — deliberately kept out of the mod).

Open ambiguity, flag before acting further: Andrew stated late in this session "the ISE was a one-off for Dev," which is not fully reconciled with the rest of this checkpoint — most of the build, testing, and the live cron/dropdown integration described below happened against the live box (root@forum prompts throughout). Clarify with Andrew whether the packaged-mod effort, the live dropdown integration, or something else specifically is considered the "one-off," before assuming any of this should be rolled back.

Purpose: a private, personal search tool for Andrew — not a replacement for SMF's native search, not primarily for the general userbase (though it is now wired into the live search dropdown as an option). Pure Python search engine (stdlib + PyMySQL only), reading SMF's own MySQL tables, decoupled from PHP except via shell_exec() at query time and at scheduled-rebuild time.

Architecture (built, working, confirmed end-to-end)

SMF (PHP + MySQL)                     You / SMF dropdown      |                                |      | (source data, one-way,         v      |  read only)              qf_search_bridge.php      v                                |IndexBuilder.py  <--- cron */10min     v      |                          search.py      v                                |  word_index.json              QueryParser.py -> Tokeniser.py  post_store.json                      |  (in /ISE_data, NOT packaged)         v                                  ANDMatcher.py (uses IndexLookup.py)                                        |                                        v                                  SortEngine.py                                        |                                        v                                ResultFormatter.py -> clickable links

Full pipeline built as separate files per module (kept separate on Andrew's explicit "single approach, but separate files" calls — see verified facts). No daemon/API/web service for the search itself; search.py is a single CLI-style entry point shelled out to by PHP.

Files/components — status and whether to use

File	Status	Use it?
QueryFrontend.php (in Sources/)	Working. qf_log_query($search_string, $username, $ip) — logs to syslog LOCAL0 as username | ip | query	Yes
Sources/Search.php — PlushSearch2()	Working. Real search entry point (not PlushSearch1(), which only renders the form). Guard-less logging call, includes guest/IP handling	Yes
Sources/Search.php — case 'ISE': in the $_POST['custom'] switch	Working, live. Decodes HTML entities (html_entity_decode(..., ENT_QUOTES)) before logging/redirecting — this was essential; without it, " leaked into shell commands and broke phrase queries	Yes
qf_search_bridge.php (in The_ISE_Project/)	Working, live. Shells out to search.py --json, decodes entities as a second safety net, renders results as clickable links. Debug lines (error_log/file_put_contents to /tmp/qf_debug.txt) were added during troubleshooting and still need removing from the live file (not present in the packaged zip)	Yes, but strip debug logging first
widget_write_log.js (theme scripts/ dir, loaded via index.template.php)	Working. Logs both the free-text widget search box and the 12 quick-topic tag clicks. Had to be moved out of the Simple Portal inline <script> block because SMF/Simple Portal's sanitizer was stripping inline <script> tags	Yes
QueryParser.py	Working, tested. Outputs {version, raw, terms: [{type: WORD|PHRASE, value}], flags: {case_sensitive}}. Phrases preserved as one literal unit — whitespace and punctuation NOT normalized inside phrases (deliberate, reversed an earlier decision)	Yes
Tokeniser.py	Working, tested. WORD terms: lowercased + punctuation-stripped. PHRASE terms: lowercased only (respects case_sensitive flag) — whitespace/punctuation untouched	Yes
IndexBuilder.py	Working, tested against live DB. Streaming cursor (SSDictCursor) required — default buffered cursor made 46,005-row fetch look hung. Cleanup folded in from a separate NLP-adjacent script (ai-preproc.py's Stage 1 only — mojibake fix, precise BBCode-tag stripping, whitespace collapse; Stage 2's spaCy-based canonicalization was deliberately NOT used, wrong problem for short search queries). Now includes timing/size stats + syslog summary (ise-rebuild tag, LOCAL0). Paths hardcoded to /var/www/html/ISE_data/... (Andrew's deliberate choice over relative-pathing)	Yes
IndexLookup.py	Working, tested. Separate file from ANDMatcher.py (Andrew's explicit choice — rejected combining them). lookup_word(), lookup_phrase() — phrase lookup is a literal substring check against post_store.json's stored text, O(n) over all posts (not indexed positionally — see verified facts on why)	Yes
ANDMatcher.py	Working, tested. Imports from IndexLookup.py, applies AND/intersection logic across mixed WORD+PHRASE token types, short-circuits on first empty match	Yes
SortEngine.py	Working, tested. Sorts matched id_msg list by poster_time, newest first	Yes
ResultFormatter.py	Working, tested. Builds https://forum.drugs-and-users.org/index.php?msg={id_msg} links (SMF auto-redirects bare ?msg= to the right topic)	Yes
search.py	Working, tested, live. Single entry point chaining the whole pipeline. Supports --json flag for machine-readable output (required for the PHP bridge)	Yes
daily_queries.py	Working. Lists today's searches from /var/log/messages, filtered by date	Yes
query_log_reader.py	Working (fixed). Originally silently broken (no output at all) — root cause never fully diagnosed but resolved during a later rewrite; now correctly parses the full multi-field log format (timestamp, username, ip, query)	Yes
[ixRebuild][/ixRebuild] trigger in Subs-Post.php (create_post())	Working, live, deliberately NOT packaged. Admin-only, plain strpos() string check (no BBC tag registered, no parser involvement) — on save, if the tag is present, shells out to IndexBuilder.py in the background and replaces the tag with a timestamp in the saved post. Andrew explicitly wants this as a permanent hand-edit to Subs-Post.php, not part of the mod package	Yes, but stays a manual/unpackaged edit
Cron job — full rebuild every 10 minutes	Working, live. flock -n /tmp/ise_rebuild.lock python3 /var/www/html/The_ISE_Project/IndexBuilder.py > /var/www/html/ISE_data/rebuild.log 2>&1 — rebuild.log is overwritten (not appended) each run; a durable one-line summary also goes to syslog (ise-rebuild tag) every run	Yes
The_ISE_Project/.htaccess	Working. Blocks direct web access to the whole directory, with an explicit <Files> exception carved out for qf_search_bridge.php (which must stay web-reachable since the dropdown redirects the browser to it)	Yes
ISE_data/.htaccess	Working. Deny from all / Require all denied — protects the raw extracted post content and index files from direct web access. ISE_data itself created with 0700 permissions (tightened from an initial 0777 in an earlier package draft)	Yes
Mod package (ISE_V1_0.zip: package-info.xml, install.xml, install_ise.php)	Built by Meta.ai with Andrew, reviewed and confirmed clean. Fixed issues: removed a redundant duplicate <require-file> for qf_search_bridge.php (already covered by <require-dir>), added the missing CustomSearch_ISE language string (patches both languages/ and language/ directory-naming variants), tightened ISE_data permissions, ships ISE_data/.htaccess as a real packaged file rather than PHP-generated	Yes — status re: the "one-off for Dev" comment is the open ambiguity flagged above
readme.txt	Built, reviewed. Still references bare python3 throughout the manual setup steps — Andrew has since settled on "no hardcoded executable paths, rely on default PATH" as final policy, so this is actually already consistent, no change needed	Yes
orchestrator.py (future AI retrieval layer)	Not started. Models: Claude Sonnet 5 + Gemini only (not chatty/ChatGPT, not Copilot). No near-term work planned	Future phase, not a blocker
NLTK / spaCy (live box, my36project venv)	Installed, confirmed working, but NOT integrated into the ISE pipeline — considered and explicitly rejected for this project (wrong tool for short keyword/phrase queries; the useful parts of an NLP-adjacent script were cherry-picked into IndexBuilder.py's cleanup instead)	Available, not used by ISE directly

Verified facts — don't re-check

• Real SMF search-submission entry point is PlushSearch2(), mapped in index.php's action array ('search2' => array('Search.php', 'PlushSearch2')) — PlushSearch1() only renders the search form, never fires on a real submitted query.
• SMF's own dropdown-based custom-search handler uses $_POST['custom'] with a switch statement inside Search.php (pre-existing code, alongside Google/Yandex/Nigma) — the case 'ISE': block was added there, not anywhere else.
• Critical, easy-to-miss gotcha: $_POST['search'] arrives already run through htmlspecialchars() by SMF itself — a literal " becomes ". This broke phrase-query matching (queries with quotes silently returned nothing) until html_entity_decode(..., ENT_QUOTES) was applied before the query reaches qf_log_query() and before it's URL-encoded and handed to qf_search_bridge.php. qf_search_bridge.php also independently decodes $_GET['q'] as a second safety net.
• PyMySQL's default DictCursor silently "hangs" (actually just slowly buffers) on large result sets — with 46,005 rows in smf_messages, this looked indistinguishable from a genuine hang. Fixed by switching to pymysql.cursors.SSDictCursor (server-side streaming cursor).
• Phrase-matching final design, reversed from an earlier in-session decision: quoted phrases ("harm reduction") are one literal unit — whitespace-exact, punctuation-exact, no normalization at all, including deliberately NOT collapsing multiple consecutive spaces (vanilla SMF's own native search does collapse multiple spaces; Andrew explicitly chose not to replicate that).
• Because whitespace must be exact, phrase matching cannot use a word-position/adjacency index — it's implemented as a literal substring check against each post's full stored text in post_store.json, which is O(n) over every post per phrase term. Accepted as fine at this project's scale/usage pattern.
• Confirmed via real production testing: unquoted 11 22 "33 44" (bash strips the quotes) → 14 results (loose word match on all four numbers). Quoted '11 22 "33 44"' (bash preserves them) → 3 results (proper phrase match) — a correct subset, proving the whole pipeline handles mixed WORD+PHRASE AND logic correctly with real data.
• Shell-quoting discipline confirmed repeatedly as an ongoing gotcha: any query containing embedded double quotes must be wrapped in single quotes at the bash prompt, or bash strips the inner quotes before Python ever sees them.
• Andrew's explicit, repeated architectural calls, all final: (1) single unified typed terms/tokens list, not split parallel arrays — preserves order, avoids duplicate-loop logic; (2) IndexLookup.py and ANDMatcher.py kept as separate files despite no functional need — Andrew's explicit preference, not to be "simplified" back into one file; (3) rejected chatty's proposed full lexer/parser/normaliser/planner/executor staged pipeline as over-engineered for this project's actual scope; (4) rejected chatty's rolling 1/3/5/7/14/21/28/35-day incremental-reindex scheduler as unnecessary complexity — full rebuild via frequent cron is simpler and the VPS has spare capacity for it.
• No hardcoded executable paths policy, final and explicit: every script/cron entry uses bare python3 (relying on default $PATH resolution), not full interpreter paths — this reverses an earlier-session workaround where /python/my36project/bin/python3 was hardcoded to route around a broken system Python wrapper. That underlying breakage has since been fixed (system Python 3.6.3 was accidentally deleted during a cleanup session, then rebuilt from source and restored), so bare python3 is now safe to rely on again.
• IndexBuilder.py takes zero command-line arguments — always run as python3 IndexBuilder.py, no flags.
• CPU load: a full IndexBuilder.py rebuild spikes the live VPS to ~95% CPU. Andrew has explicitly decided this is acceptable ("paying for the vCPUs, gonna use 'em") and wants the aggressive 10-minute cron cadence kept — but has also explicitly said this would be "first thing to go" if it ever noticeably degrades real visitor page-load performance. Not yet observed to cause user-facing problems.
• [ixRebuild][/ixRebuild] is NOT a registered BBCode tag — it's a plain strpos() string check in create_post(), admin-gated, fires IndexBuilder.py in the background (trailing &), then replaces the tag with a save-time timestamp before the post is stored. Deliberately chosen over a real BBC tag specifically because BBC tags re-fire on every render/view, not just once on save.
• The mod package (ISE_V1_0.zip) was reviewed file-by-file (package-info.xml, install.xml, install_ise.php) and confirmed clean — no outstanding issues found in the packaging itself.

Open items — decisions, not blockers

• Resolve the "ISE was a one-off for Dev" ambiguity raised late in this session — unclear whether this means the packaging effort, the live dropdown integration, or something else should be treated differently going forward. Nothing has been rolled back yet; everything described above is still live and working as of this checkpoint.
• Remove the leftover debug logging (/tmp/qf_debug.txt writes) from the live qf_search_bridge.php file — confirmed NOT present in the packaged zip version, so this is purely a live-server cleanup task, not a packaging issue.
• Board-permission / private-board filtering is NOT implemented at all. IndexBuilder.py's query has no board-permission awareness — it indexes every approved post regardless of board privacy. This is safe today only because Andrew is effectively the sole real user of ISE; it becomes a real information-leak risk the moment the "Use ISE search" admin toggle is enabled for the general userbase. Flagged as important, not yet addressed.
• [ixRebuild] trigger — new-post-only, not edits. Currently hooked into create_post(); SMF's edit path (modifyPost()) is a separate function and does NOT currently trigger a rebuild. Open question, not yet decided, whether this should also fire on edits.
• Cron cadence (currently 10 minutes) may need revisiting if real user-facing performance ever degrades — Andrew has pre-committed to backing this off as the first response if that happens, but nothing currently monitors for it automatically.
• orchestrator.py / AI retrieval layer (Claude Sonnet 5 + Gemini) remains a deliberately deferred future phase, no active work.

Next actual action

Remove the debug logging lines from the live qf_search_bridge.php (error_log(...) / file_put_contents(...) writing to /tmp/qf_debug.txt), and delete the existing /tmp/qf_debug.txt file. Then clarify with Andrew exactly what "one-off for Dev" was meant to scope, before making any further changes to the live dropdown integration, the mod package, or the cron/[ixRebuild] setup — all of which are currently live, working, and untouched pending that clarification.