The ISE (Intelligent Search Engine) Project

[smfadmin (OP)]

Called the "QueryFrontend"

Yes. Let's call the first component QueryFrontend.

For now it is not a full SMF mod. It is a tiny instrumentation patch that adds one capability:

> Capture the raw search query as soon as SMF receives it, then let SMF continue normally.

Create:

QueryFrontend.php

with:

Code: [Select]

<?php

/*
 * QueryFrontend
 *
 * Phase 1:
 * Capture raw SMF search queries via syslog.
 *
 * Target:
 * SMF 2.0.19
 */

function qf_log_query($search_string)
{
    openlog(&#39;smf-queries&#39;, LOG_PID, LOG_LOCAL0);
    syslog(LOG_INFO, $search_string);
    closelog();
}

?>

Test first:

Code: [Select]

php -r "openlog('smf-queries', LOG_PID, LOG_LOCAL0); syslog(LOG_INFO,'test123'); closelog();"
grep test123 /var/log/messages

Take a backup:

Code: [Select]

cd /var/www/html/Sources $
cp Search.php Search.php-orig

After this:

Code: [Select]

function PlushSearch2()
{
        global $boardurl, $scripturl, $modSettings, $sourcedir, $txt, $db_connection;
        global $user_info, $context, $options, $messages_request, $boards_can;
        global $excludedWords, $participants, $smcFunc, $search_versions, $searchAPI;

Add this:

Code: [Select]

// ---- Log every search attempt, error or not
          require_once($sourcedir . '/QueryFrontend.php');
          qf_log_query($_REQUEST['search']);
// ---- Log end

That's the entire first version.
The important design decision:
No modification to search logic
No filtering
No tokenising
No database changes
No Python yet
It is purely a sensor.
Once installed on Dev, test:

1. Perform a search:

database access denied

2. Check:

grep smf-queries /var/log/messages

Expected:

Jul 22 14:xx:xx server smf-queries[xxxx]: database access denied

After this works, we can package QueryFrontend as an SMF 2.0.19 mod with:

install.xml
file patch
uninstall support

The nice part is this first component should remain useful forever — even when the AI retrieval layer exists. It becomes the query telemetry feed for the system.

The modules

QueryFrontend
QueryParser
Tokeniser
TokenNormaliser
IndexBuilder
IndexLookup
ANDMatcher
SortEngine
ResultFormatter

Notice that each module has exactly one responsibility:

Module------------------------------------Responsibility

QueryFrontend------------------------Accept the raw query from SMF.
QueryParser----------------------------Parse the user's query into a structured form.
Tokeniser---------------------------------Split the query into tokens.
TokenNormaliser---------------------Lowercase, remove punctuation, stemming (later).
IndexBuilder-----------------------------Build and maintain the search index from SMF posts.
IndexLookup----------------------------Retrieve the posting lists for each token.
ANDMatcher----------------------------Compute the intersection of the posting lists.
SortEngine-------------------------------Order the matching posts (newest first initially).
ResultFormatter-----------------------Return the results in a format SMF or another client can use.

The nice thing is that you can develop and test each one independently.

For example, ANDMatcher doesn't care where the token lists came from:

token1 = {5, 9, 14, 22}
token2 = {2, 5, 14}
token3 = {5, 14, 30}

result = token1 & token2 & token3

# {5, 14}

If that works, the module is done.

Later, you can swap implementations without changing the interfaces:

TokenNormaliser might later add stemming.

SortEngine might later use a relevance score instead of just date.

IndexLookup might move from MySQL to another storage mechanism.

Nothing else has to change.

One suggestion: I'd introduce one extra module between QueryFrontend and

QueryParser

QueryFrontend
    │
    ▼
QueryValidator
    │
    ▼
QueryParser

Its only job would be to reject or clean obviously invalid input before the parser sees it. Right now it might do almost nothing, but having that separation gives you a natural place to add things later (length limits, malformed queries, unsupported syntax, etc.) without complicating the parser.

Overall, this is starting to look less like "some Python scripts" and more like a well-structured information retrieval engine. That's a solid foundation for whatever features you decide to add later.
.
[Ok, that's great now that you know just how complex it needs to be.

The modules

1. QueryParser
2.Tokeniser
3. TokenNormaliser
4. IndexBuilder
5. IndexLookup
6. ANDMatcher
7. SortEngine
8. ResultFormatter

Notice that each module has exactly one responsibility:

Module   Responsibility

QueryFrontend   Accept the raw query from SMF.
QueryParser   Parse the user's query into a structured form.
Tokeniser   Split the query into tokens.
TokenNormaliser   Lowercase, remove punctuation, stemming (later).
IndexBuilder   Build and maintain the search index from SMF posts.
IndexLookup   Retrieve the posting lists for each token.
ANDMatcher   Compute the intersection of the posting lists.
SortEngine   Order the matching posts (newest first initially).
ResultFormatter   Return the results in a format SMF or another client can use.


The nice thing is that you can develop and test each one independently.

For example, ANDMatcher doesn't care where the token lists came from:

token1 = {5, 9, 14, 22}
token2 = {2, 5, 14}
token3 = {5, 14, 30}

result = token1 & token2 & token3

# {5, 14}

If that works, the module is done.

Later, you can swap implementations without changing the interfaces:

TokenNormaliser might later add stemming.

SortEngine might later use a relevance score instead of just date.

IndexLookup might move from MySQL to another storage mechanism.


Nothing else has to change.

One suggestion: I'd introduce one extra module between QueryFrontend and QueryParser:

QueryFrontend
    │
    ▼
QueryValidator
    │
    ▼
QueryParser

Its only job would be to reject or clean obviously invalid input before the parser sees it. Right now it might do almost nothing, but having that separation gives you a natural place to add things later (length limits, malformed queries, unsupported syntax, etc.) without complicating the parser.

Overall, this is starting to look less like "some Python scripts" and more like a well-structured information retrieval engine. That's a solid foundation for whatever features you decide to add later.

Post Merged: July 22, 2026, 05:25:42 PM
Me:

But please build me some psuedocode for the QueryParser

Sure. Let's define the QueryParser as a very simple first version. Its job is not to search, tokenise, or rank. It only understands the structure of the user's query and passes a clean representation to the next module.

QueryParser — responsibility

Input:

"database access denied"

Output:

{
    "terms": [
        "database",
        "access",
        "denied"
    ],
    "operator": "AND"
}


---

Pseudocode

MODULE QueryParser


FUNCTION parse_query(raw_query):

    IF raw_query is empty:
        RETURN error("No query supplied")


    query = trim(raw_query)


    query = remove_extra_spaces(query)


    IF query contains explicit operators:
       
        detect operators:
            AND
            OR
            NOT

        build query structure


    ELSE:

        assume default operator = AND


    extract search terms


    FOR each term:
        remove invalid characters


    RETURN ParsedQuery


---

Example flows

Simple query

Input:

database access denied

Parser produces:

ParsedQuery
{
    operator: AND

    terms:
       database
       access
       denied
}


---

Future query with operators

Input:

database AND access AND denied

Parser produces:

ParsedQuery
{
    operator: AND

    terms:
       database
       access
       denied
}


---

Future OR support

Input:

mysql OR mariadb

Output:

ParsedQuery
{
    operator: OR

    terms:
       mysql
       mariadb
}


---

Important design decision

The QueryParser should not do:

❌ lowercase conversion
❌ stemming
❌ database lookup
❌ matching posts
❌ sorting results

Those belong elsewhere.

The pipeline stays:

Raw text
   |
   v
QueryParser
   |
   v
Tokeniser
   |
   v
TokenNormaliser
   |
   v
IndexLookup
   |
   v
ANDMatcher
   |
   v
SortEngine
   |
   v
ResultFormatter

This keeps the modules replaceable. A simple QueryParser today can later evolve into a full query language parser without forcing changes throughout the search engine.

Me:

I need it to support searches like: "drugs of addiction "harm reduction" Australia"

Ah, that changes the design significantly. The parser cannot just split on spaces. It needs to understand phrases.

Your example:

drugs of addiction "harm reduction" Australia

should not become:

drugs
of
addiction
harm
reduction
Australia

because "harm reduction" is a single search unit.

The QueryParser should produce something like:

ParsedQuery

operator: AND

terms:
    drugs
    of
    addiction
    Australia

phrases:
    "harm reduction"

Then later modules decide how to search those.

A better pseudocode:

MODULE QueryParser


FUNCTION parse_query(raw_query):

    IF raw_query is empty:
        return error


    query = trim(raw_query)


    phrases = extract_quoted_phrases(query)

        Example:
        "harm reduction"

        becomes:

        phrases:
            harm reduction


    remove phrases from query


    remaining_words = split_on_spaces(query)


    terms = []

    FOR each word in remaining_words:

        IF word is not empty:
            terms.append(word)


    detect operators:

        IF query contains AND:
            operator = AND

        ELSE IF query contains OR:
            operator = OR

        ELSE:
            operator = AND


    RETURN:

        {
          operator: operator,
          terms: terms,
          phrases: phrases
        }

For your example:

Input

drugs of addiction "harm reduction" Australia

Output

{
 "operator": "AND",

 "terms": [
   "drugs",
   "of",
   "addiction",
   "Australia"
 ],

 "phrases": [
   "harm reduction"
 ]
}

Then the next stages can decide:

Tokeniser:

drugs → token

addiction → token

Australia → token


PhraseMatcher (a future module):

verify "harm reduction" appears together and in that order



This is actually the beginning of a proper search language.

A possible future query syntax:

drugs addiction "harm reduction" Australia

means:

> Find posts containing all words AND the exact phrase "harm reduction".



Later:

meth OR amphetamine "harm reduction"

means:

> Find posts containing either meth or amphetamine AND the phrase.



So I would slightly extend the architecture:

QueryFrontend
      |
QueryParser
      |
Tokeniser
      |
TokenNormaliser
      |
IndexBuilder
      |
IndexLookup
      |
ANDMatcher
      |
PhraseMatcher
      |
SortEngine
      |
ResultFormatter

The good news is your example is exactly the kind of real query that forces the design to become useful rather than just a toy token matcher.

Me:

Ok, that's great now that you know just how complex it needs to be.

Next I'll run it past "Programming" 😋 and see how well it translates to Python code. Your input is appreciated.

😄 Good plan. This is exactly the point where it is worth getting a programming-focused review.

The important thing is that the design is now language-independent. Python is just the implementation vehicle.

When you hand it over, the key points for "Programming" to preserve are:

QueryParser contract

Input

raw_query: string

Example:

drugs of addiction "harm reduction" Australia


---

Output

A structured object, for example:

Code: [Select]

{
    "operator": "AND",
    "terms": [
        "drugs",
        "of",
        "addiction",
        "Australia"
    ],
    "phrases": [
        "harm reduction"
    ]
}

---

The main Python tasks are actually quite manageable:

1. Extract quoted phrases

Find text between ".

Remove those phrases from the remaining string.



2. Parse remaining words

Split into tokens.

Preserve ordering for now.



3. Detect operators

Default:

AND

Later:

OR
NOT


4. Return a predictable structure


The beauty of doing it this way is that you can test it without touching SMF or the database:

Code: [Select]

Input:
mysql "access denied"

Expected:
terms:
    mysql

phrases:
    access denied

If the parser passes those tests, the next module can be developed independently.

One thing I would tell the Python implementer: don't over-engineer the first version. Avoid jumping straight into NLP libraries. For your use case, a small, transparent parser you can understand and modify is probably more valuable initially.

The really interesting parts come later:


building the inverted index,

deciding what to store,

ranking results,

adding fuzzy matching,

using NLP intelligently.


But the parser is the doorway. Get that clean and the rest of the engine has a solid foundation. 👍

Me:

I like his pseudocode, despite being imprecise,  I think it would be AI friendly.

I agree with your instinct. Pseudocode does not have to be executable to be valuable — especially when the next reader is another AI or a programmer.

In fact, for an AI-assisted project, good pseudocode is often better than prematurely detailed code because it captures the intent without locking the implementation into one language.

The QueryParser pseudocode we have is useful because it defines:

Purpose — what the module is responsible for.

Boundaries — what it must not do.

Input — raw user query.

Output — structured query object.

Examples — what behaviour is expected.


Those examples are particularly important. An AI can infer a lot from:

Input:
drugs of addiction "harm reduction" Australia

Output:
terms:
    drugs
    of
    addiction
    Australia

phrases:
    harm reduction

operator:
    AND

That gives it a behavioural contract.

Where I would refine it for an AI handoff is by making the ambiguity explicit:

NOTE:
Quoted text is treated as an exact phrase.
Unquoted words are individual search terms.
Default operator between terms is AND.
Parser does not perform matching.
Parser does not access the database.

That prevents the next model from "helpfully" adding search logic into the parser.

This is actually similar to how good software teams work: a module specification is often more important than the first implementation. Once the contract is clear, multiple people (or models) can implement it and still produce compatible components.

Your checkpoint-file idea fits perfectly here too. A future session could load:

Code: [Select]

Project: SmartSearch

Current module:
QueryParser

Contract:
[input/output]

Examples:
[tests]

Do not:
[database access, ranking, NLP]
[/quote]
and immediately understand the task without replaying the entire history.

For a multi-model workflow, these small design documents may end up being one of the most valuable assets you create.

[Chip]

That answers decision #1 — yes, quoted phrases need to be treated as one exact unit, mixed in with loose words. Here's the parsing logic for that:

```

Code: [Select]

MODULE QueryParser

INPUT: raw_query — e.g. drugs of addiction "harm reduction" Australia

FUNCTION parse(raw_query):
    IF raw_query is empty/whitespace-only:
        RETURN error "empty query"

    tokens = []
    scan raw_query left to right:
        IF a "quoted phrase" is found:
            extract contents between the quotes as ONE token, marked type=PHRASE
        ELSE:
            each whitespace-separated word becomes its own token, marked type=WORD

    RETURN ParsedQuery(
        raw = raw_query,
        tokens = [
            {type: WORD,   value: "drugs"},
            {type: WORD,   value: "of"},
            {type: WORD,   value: "addiction"},
            {type: PHRASE, value: "harm reduction"},
            {type: WORD,   value: "Australia"},
        ]
    )
```

Real Python for that part (regex does the quote-aware splitting in one pass):

```python

Code: [Select]

import re

def parse(raw_query: str):
    raw_query = raw_query.strip()
    if not raw_query:
        raise ValueError("empty query")

    tokens = []
    for match in re.finditer(r'"([^"]+)"|(\S+)', raw_query):
        phrase, word = match.group(1), match.group(2)
        if phrase is not None:
            tokens.append({"type": "PHRASE", "value": phrase})
        else:
            tokens.append({"type": "WORD", "value": word})

    return {"raw": raw_query, "tokens": tokens}m```

Downstream effect on the rest of the pipeline: `Tokeniser.py` normalizes each `WORD` individually (lowercase, strip punctuation), but a `PHRASE` token needs to be indexed/matched as a **consecutive run**, not just AND'd — meaning `IndexLookup`/`ANDMatcher` need to check that "harm" and "reduction" appear adjacent in a post, not just that both appear somewhere. That's a meaningfully bigger lift than plain AND-of-words (needs word-position data in the index, not just token→post_id).

Given that's a real jump in complexity — do you want phrase matching to require true adjacency (harder, needs positional index), or is "both words present, order doesn't matter" close enough for v1 ?

order doesn't matter

Good — that simplifies things a lot. If order/adjacency doesn't matter, a "phrase" is really just a shorthand for "these words must all appear," which means it collapses into the exact same AND logic as everything else. You don't even need a separate `PHRASE` token type downstream — `QueryParser` can just split the quoted content into individual words and merge them into the same flat word-list as everything else:

```
python

Code: [Select]

import re

def parse(raw_query: str):
    raw_query = raw_query.strip()
    if not raw_query:
        raise ValueError("empty query")

    words = []
    for match in re.finditer(r'"([^"]+)"|(\S+)', raw_query):
        phrase, word = match.group(1), match.group(2)
        if phrase is not None:
            words.extend(phrase.split())   # "harm reduction" -> harm, reduction
        else:
            words.append(word)

    return {"raw": raw_query, "words": words}

```

From our example:

```
drugs of addiction "harm reduction" Australia
```

→ `["drugs", "of", "addiction", "harm", "reduction", "Australia"]` — five plain words, all AND'd together, quotes only ever mattered as a way to group multi-word concepts visually while typing.

That means `QueryParser`'s whole job really is just "strip the quote syntax, flatten everything into one word list" — `Tokeniser.py` then normalizes each word the same way regardless of whether it came from inside quotes or not, and `ANDMatcher.py` never needs to know phrases existed at all.

Much simpler module than I first sketched. Ready to move to `Tokeniser.py` next, or do you want to sit with `QueryParser` a bit longer?

Me:

No hurry but I like the thought of my own Tokeniser pleasing.

No rush at all — enjoying the design work is a good reason on its own to sit with it. `Tokeniser.py` is a genuinely satisfying one to get right, since it's the piece that decides what actually counts as "the same word" across your whole archive.

Basic version, since it feeds off the flat word-list `QueryParser` now hands it:

```
python

Code: [Select]

import re

def tokenise(words: list) -> list:
    tokens = []
    for w in words:
        w = w.lower()
        w = re.sub(r'[^\w]', '', w)   # strip punctuation, keep letters/digits/underscore
        if w:
            tokens.append(w)
    return tokens

```

For your example, `["drugs", "of", "addiction", "harm", "reduction", "Australia"]` → `["drugs", "of", "addiction", "harm", "reduction", "australia"]` — just lowercased and punctuation-stripped.

A few real decisions that shape it, worth pinning down while it's cheap:

1. **Stopwords** — "of," "the," "a," "and" etc. Drop them, or keep everything? Dropping shrinks your index and speeds lookups, but for *your own* forum with domain-specific phrasing, sometimes a stopword carries meaning you'd search for.
2. **Numbers** — keep `"2026"` or dosage-like `"5mg"` as valid tokens, or strip anything non-alphabetic? Given your content, dosage/number tokens are probably worth keeping.
3. **Stemming** — `"reducing"` vs `"reduction"` vs `"reduce"` — do you want those to match each other, or keep it literal for v1? Stemming is a bigger lift (needs a stemming library or hand-rolled suffix rules) — my instinct is skip it for v1, add later if you notice yourself missing results because of it.

Given "basic and pleasing" — my gut says keep numbers, drop nothing else, skip stemming for now. Want to lock those three in, or think out loud on any of them?

[Chip]

Code: [Select]

import re

def parse(raw_query: str) -> dict:
    """
    Parses a raw search string into a mix of loose words and
    intact quoted phrases. Quoted phrases are kept as single
    units (with their embedded blank preserved) so downstream
    matching can require true adjacency, not just presence.
    """
    raw_query = raw_query.strip()
    if not raw_query:
        raise ValueError("empty query")

    terms = []
    for match in re.finditer(r'"([^"]+)"|(\S+)', raw_query):
        phrase, word = match.group(1), match.group(2)
        if phrase is not None:
            terms.append({"type": "PHRASE", "value": phrase})
        else:
            terms.append({"type": "WORD", "value": word})

    return {"raw": raw_query, "terms": terms}


if __name__ == "__main__":
    import sys
    query = " ".join(sys.argv[1:])
    result = parse(query)
    print(result)


Code: [Select]

python3 QueryParser.py '11 22 "33 44"'


Code: [Select]

{'raw': '11 22 "33 44"',
 'terms': [
    {'type': 'WORD', 'value': '11'},
    {'type': 'WORD', 'value': '22'},
    {'type': 'PHRASE', 'value': '33 44'},
 ]}


The ISE Project (Intelligent Search Engine) — Checkpoint (v1)

Forum: drugs-and-users.org (self-hosted SMF). Official project directory: /var/www/html/The_ISE_Project on the dev box (CentOS 6.10, i686/32-bit). Live forum runs SMF 2.0.19.

Purpose: a private, personal search tool for Andrew — not a replacement for SMF's native search, not for the general userbase. Pure Python search engine (stdlib + PyMySQL only, no web framework/API/daemon), reading SMF's own MySQL tables as its data source, completely decoupled from PHP at query time. A separate PHP-side syslog logger captures every real search string submitted through the live forum, feeding a query corpus for later tuning. A future, deliberately separate phase adds an AI retrieval layer using two chosen models — Claude Sonnet 5 and Gemini — via an orchestrator.py (not yet started, not a near-term blocker).

Architecture (settled)

SMF (PHP + MySQL)                     You      |                                |      | (source data, one-way,         v      |  read only)                search.py      v                                |IndexBuilder.py                        v      |                          QueryParser.py      v                                |  Search Index  <-------------------- (feeds into ANDMatcher.py)                                        |                                        v                                    Results

• PHP/SMF = forum application (unchanged, except the tiny logger patch below)
• MySQL = shared data store (SMF's own tables + the search index tables)
• Python = the entire search engine, stdlib + PyMySQL, no other dependencies
• No PHP↔Python IPC, no daemon, no API, no web service — this is a CLI tool

  run manually, e.g. python3 search.py "database access denied"

• Index freshness is manual/on-demand (IndexBuilder.py re-run whenever

  wanted), not live-triggered by new posts — accepted tradeoff given   single-user scope

Files/components — status and whether to use

File	Status	Use it?
QueryFrontend.php	Working, confirmed on dev. Contains qf_log_query($search_string) — opens syslog LOCAL0, logs the raw string, closes. Simple, unchanged since first written.	Yes
Sources/Search.php — PlushSearch2()	Working, confirmed on dev. Real search entry point (NOT PlushSearch1(), which only renders the search form). Logger call sits right after the global declarations: require_once($sourcedir . '/QueryFrontend.php'); qf_log_query($_REQUEST['search']); — guard removed (the !empty($_REQUEST['search']) check was blocking real logging on this fork and was deliberately dropped)	Yes — needs porting to live (dev-confirmed only so far, see open items)
QueryParser.py	Written, NOT yet tested. Parses raw query string into a list of terms, each `{'type': 'WORD'	'PHRASE', 'value': ...}. Quoted phrases preserved as literal, whitespace-exact units (see verified facts).rawfield retained unmodified for logging. Has a workingif name == "main":` block for CLI testing.	Yes — pending first real test run
Tokeniser.py	Not yet built. Design needs revising post-phrase-matching reversal — must handle WORD and PHRASE terms differently, not treat everything as one flat word list (see open items)	Not started
IndexBuilder.py	Not yet built. Needs to read smf_messages/smf_topics (and optionally smf_boards) from MySQL, strip BBCode/HTML entities from body before indexing	Not started
IndexLookup.py / ANDMatcher.py	Not yet built. Must handle mixed term types: WORD → presence check; PHRASE → literal, whitespace-exact substring match (NOT a word-position/adjacency index — see verified facts)	Not started
SortEngine.py	Not yet built. Simple: sort qualifying posts by post date, newest first	Not started
ResultFormatter.py	Not yet built. Builds SMF permalinks: https://drugs-and-users.org/index.php?topic={id_topic}.msg{id_msg}#msg{id_msg}	Not started
daily_queries.py	Working, confirmed correct. Lists today's searches by grep-parsing smf-queries lines out of /var/log/messages, filtered by today's date	Yes
query_log_reader.py	Written, broken — produces no output at all. Same general approach as daily_queries.py (parse smf-queries lines from /var/log/messages) but something in its logic/regex isn't matching. Root cause not yet diagnosed.	No — needs debugging before use
orchestrator.py	Not started. Future AI retrieval layer, models: Claude Sonnet 5 + Gemini only (not ChatGPT/chatty, not Copilot — those are used for drafting/dreaming, not chosen as orchestrator models)	Future phase, not a near-term blocker

Verified facts — don't re-check

• Real SMF search-submission entry point is PlushSearch2() in Sources/Search.php, mapped from index.php's action array: 'search2' => array('Search.php', 'PlushSearch2'). PlushSearch1() is a decoy for this purpose — it only renders the search form, never fires on an actual submitted query.
• The !empty($_REQUEST['search']) guard around qf_log_query() was removed in live PlushSearch2() because logging silently failed to fire with the guard in place. Tradeoff accepted: PHP will throw an undefined-index notice on Search.php loads with no search term — not yet cleaned up, not urgent.
• Syslog logging confirmed fully working end-to-end on the 2.0.19 dev box: openlog('smf-queries', LOG_PID, LOG_LOCAL0); syslog(LOG_INFO, $search_string); closelog(); lands correctly in /var/log/messages (LOCAL0 falls through to the default catch-all on this box, no rsyslog config changes needed).
• daily_queries.py (parses smf-queries lines from /var/log/messages, filters by today's date) is confirmed correct and working.
• query_log_reader.py is confirmed broken — no output at all — despite daily_queries.py's near-identical parsing approach working fine. This means the bug is specific to query_log_reader.py's own logic, not the underlying log format/data.
• Phrase-matching design — final, reversed from an earlier session decision: quoted phrases (e.g. "harm reduction") must be preserved as one literal unit, whitespace included exactly as typed. An earlier decision to flatten phrases into separately-AND'd words (order/adjacency doesn't matter) was explicitly reversed — the point of quoting is precisely to require the embedded blank.
• Whitespace inside phrases must NOT be normalized/collapsed. Vanilla SMF's own native search was tested and found to collapse multiple consecutive blanks down to one before matching — Andrew explicitly decided AGAINST replicating that behavior. A double blank in a quoted search phrase must remain a literal double-blank requirement, matched exactly as typed.
• Practical implication of the above, not yet built: because whitespace must be honored exactly, phrase matching cannot use a simple word-position/adjacency index (which would treat "harm" and "reduction" as adjacent regardless of exact spacing) — it needs a literal substring check against stored post text instead.
• QueryParser.py's current design (confirmed correct via manual trace, not yet run):

  - Regex "([^"]+)"|(\S+) scans left to right   - Every match becomes a term: {'type': 'PHRASE', 'value': <exact text between quotes, unmodified>} or {'type': 'WORD', 'value': <bare word>}   - raw field retains the original query string exactly as typed (with quote characters intact) — this is for logging/display only, NOT used for matching   - Example: '11 22 "33 44"' → terms: [WORD:11, WORD:22, PHRASE:"33 44"] (confirmed correct by manual trace)

• Mobile SSH (JuiceSSH) paste reliability is poor for multi-line code — both direct nano paste and heredoc (cat >> file << EOF) attempts produced mangled/misformatted output. No fix found yet; recommend deferring code-writing to a proper keyboard session rather than fighting mobile paste.
• ChatGPT (chatty) is good for architecture/phased-planning conversations but is explicitly NOT a coder — every actual code fix in this project came from Andrew's own hands-on debugging, not from chatty's suggestions directly.
• Dev box (CentOS 6.10, i686) has broken/dead SCL yum repos (centos-sclo-rh, centos-sclo-sclo both pointing at dead mirror.centos.org URLs) and a gcc alias to colourify/grc that was suspected but ruled out as the cause of a bizarre, unresolved issue where compiled binaries vanish immediately after linking (confirmed via direct /usr/bin/gcc calls bypassing the alias — same failure). This does NOT happen on the live box. Root cause never found; investigation abandoned in favor of building Pythontooling on live instead, since dev's environment is uniquely broken in ways unrelated to the actual 32-bit/CentOS 6 constraints.
• Getting Python 3.6 onto CentOS 6.10 i686 via SCL doesn't work — rh-python36 isn't built for i386/i686 at all (SCL's newer collections are effectively x86_64-only on CentOS 6). Building from source via make altinstall (never make install) is the safe path that doesn't touch system Python 2.6, needed because CentOS 6's yum itself depends on Python 2.6.
• The ChatGPT full-conversation-history export (Settings → Data controls → Export data, ~24hr turnaround) is the only reliable way to get a complete transcript of a ChatGPT conversation — Chrome's "Save as MHTML" only captures whatever's currently mounted in the DOM, and ChatGPT's web UI virtualizes/unmounts long conversations outside the viewport, so MHTML saves of long conversations are silently incomplete (confirmed: an MHTML save of the "AND-based token matching" conversation only captured 5 of 50 real messages).

Open items — decisions, not blockers

• QueryParser.py has never actually been run/tested. Written and manually traced through by hand, believed correct, but no real execution yet (blocked by mobile SSH paste reliability — deferred to a proper keyboard session).
• query_log_reader.py is broken — produces no output, root cause not yet diagnosed. daily_queries.py's near-identical logic works fine, so the bug is localized to this specific script, not the log format.
• Live forum still needs the QueryFrontend.php + PlushSearch2() logger patch ported over from dev. Steps already given (backup Search.php, copy QueryFrontend.php, add the two lines, php -l validate, test) — not yet confirmed done on live.
• Live query logging privacy consideration, flagged but not yet decided: once live, the syslog logger captures every real user's search terms, not just Andrew's own — worth being deliberate about retention/access given the forum's harm-reduction context, before it's silently running against real visitor traffic (if it isn't already).
• Tokeniser.py design needs rework post phrase-matching reversal — must treat WORD and PHRASE terms differently rather than one flat lowercase/punctuation-stripped word list. Not yet started.
• IndexBuilder.py/IndexLookup.py/ANDMatcher.py all need a design decision locked in, following from the whitespace-literal phrase requirement: phrase terms need literal substring matching against stored post text, NOT a word-position/adjacency index. Schema/approach not yet finalized.
• .htaccess (or equivalent) not yet confirmed added to /var/www/html/The_ISE_Project to block direct web access, since the project directory sits inside the live web-accessible document root. Exact Deny/Require all denied lines were given; not yet confirmed applied.

Next actual action

Get QueryParser.py actually running for the first time, from a proper keyboard (not mobile SSH) — run:

python3 QueryParser.py '11 22 "33 44"'

Confirm it prints:

{'raw': '11 22 "33 44"', 'terms': [    {'type': 'WORD', 'value': '11'},    {'type': 'WORD', 'value': '22'},    {'type': 'PHRASE', 'value': '33 44'}, ]}

Once confirmed working, move to Tokeniser.py (redesigned for mixed WORD/PHRASE terms), then lock in the literal-substring approach for phrase matching in IndexBuilder.py/ANDMatcher.py.

Post Merged: Yesterday at 04:39:26 PMee Tokeniser.py

Code: [Select]

import re

def tokenise(parsed_query: dict) -> dict:
    """
    Takes QueryParser's output and normalizes each term:
    - WORD terms: lowercased, punctuation stripped
    - PHRASE terms: lowercased (respecting case_sensitive flag),
      but internal whitespace is left completely untouched —
      exact spacing is part of what must match later.
    """
    case_sensitive = parsed_query.get("flags", {}).get("case_sensitive", False)
    tokens = []

    for term in parsed_query["terms"]:
        value = term["value"]

        if term["type"] == "WORD":
            cleaned = re.sub(r'[^\w]', '', value)
            if not case_sensitive:
                cleaned = cleaned.lower()
            if cleaned:
                tokens.append({"type": "WORD", "value": cleaned})

        elif term["type"] == "PHRASE":
            # No whitespace touched, no punctuation stripped —
            # this needs to match literally later
            cleaned = value if case_sensitive else value.lower()
            tokens.append({"type": "PHRASE", "value": cleaned})

    return {
        "version": parsed_query.get("version", 1),
        "raw": parsed_query["raw"],
        "tokens": tokens,
    }


if __name__ == "__main__":
    import sys
    from QueryParser import parse

    query = " ".join(sys.argv[1:])
    parsed = parse(query)
    result = tokenise(parsed)
    print(result)Post Merged: Today at 05:45:54 AMNow you get to the fun part. 😄

The core search engine is the **engine room**. The "really cool" stuff is what you build *on top of it*.

I would avoid randomly adding AI everywhere. The best upgrades are the ones that make your retrieval engine feel intelligent while keeping the results trustworthy.

Here are some ideas, roughly in order of impact:

---

## 1. Relevance scoring (big upgrade)

Right now:

```text
term exists → post matches
```

Make it:

```text
term exists → how important is this match?
```

Example:

Query:

```text
"harm reduction" Australia drugs
```

Post A:

* contains exact phrase "harm reduction"
* contains all words
* recent

Post B:

* contains only "drugs"

Both match, but A should rank higher.

A simple scoring engine:

```text
score =
    +10 exact phrase match
    +5  all query words present
    +3  title match
    +2  recent post
    +1  author/thread relevance
```

Now it feels like a modern search engine.

---

## 2. Snippet generation

Instead of returning:

```text
Post #45861
```

return:

```text
Post #45861

"...we have always promoted harm reduction principles because
people need accurate information about drugs..."

[Read full post]
```

This makes the archive immediately usable.

---

## 3. Search history intelligence

You already have QueryFrontend logging.

Now you can analyse:

* most searched topics
* failed searches
* common terminology
* changes over time

Example:

```text
Most common searches this month:

harm reduction       84
methamphetamine      61
withdrawal           45
```

Your own archive starts telling you what people are interested in.

---

## 4. "Related posts"

This one would be very cool.

User views:

> Post about meth harm reduction

System suggests:

```text
Related discussions:

- Harm reduction strategies (2019)
- Experiences with different routes of administration (2021)
- Medical advice discussion (2023)
```

No AI required initially. Shared tokens can do this.

---

## 5. AI-assisted answers from your archive

This is where your Claude/Gemini/Groq orchestra becomes valuable.

Flow:

```text
User question
      |
      v
SmartSearch retrieves 20 relevant posts
      |
      v
AI models analyse ONLY those posts
      |
      v
Answer with references
```

The AI is no longer guessing from general knowledge. It is working from your 12-year archive.

---

## 6. Search explanations

A very "smart engine" feature:

Instead of only:

```text
Found 42 results
```

show:

```text
Why these results:

Matched:
✓ harm reduction (phrase)
✓ Australia
✓ drugs

Sorted by:
✓ phrase relevance
✓ newest first
```

Users trust systems more when they can see why.

---

## 7. The really ambitious one: "Ask the Archive"

A special interface:

```text
Ask:
"What have we learned about harm reduction over the last decade?"
```

System:

1. Retrieves relevant posts.
2. Groups them by topic/date.
3. Sends them to multiple AI models.
4. Produces a synthesis.

That would genuinely make a 12-year forum archive feel alive.

---

If I were choosing the next three upgrades, I'd do:

1. **SortEngine with relevance scoring**
2. **ResultFormatter with snippets/highlighting**
3. **AI layer using retrieved posts as context**

Because those three would transform it from:

> "a better search box"

into:

> "a knowledge retrieval system for the entire forum."

The foundation you've built is exactly what makes those features possible.