A Multi-Model AI + a Human Collaborately Building an Intelligent Search Engine

[smfadmin (OP)]

Called the "QueryFrontend"

Yes. Let's call the first component QueryFrontend.

For now it is not a full SMF mod. It is a tiny instrumentation patch that adds one capability:

> Capture the raw search query as soon as SMF receives it, then let SMF continue normally.

Create:

QueryFrontend.php

with:

Code: [Select]

<?php

/*
 * QueryFrontend
 *
 * Phase 1:
 * Capture raw SMF search queries via syslog.
 *
 * Target:
 * SMF 2.0.19
 */

function qf_log_query($search_string)
{
    openlog(&#39;smf-queries&#39;, LOG_PID, LOG_LOCAL0);
    syslog(LOG_INFO, $search_string);
    closelog();
}

?>

Test first:

Code: [Select]

php -r "openlog('smf-queries', LOG_PID, LOG_LOCAL0); syslog(LOG_INFO,'test123'); closelog();"
grep test123 /var/log/messages

Take a backup:

Code: [Select]

cd /var/www/html/Sources $
cp Search.php Search.php-orig

After this:

Code: [Select]

function PlushSearch2()
{
        global $boardurl, $scripturl, $modSettings, $sourcedir, $txt, $db_connection;
        global $user_info, $context, $options, $messages_request, $boards_can;
        global $excludedWords, $participants, $smcFunc, $search_versions, $searchAPI;

Add this:

Code: [Select]

// ---- Log every search attempt, error or not
          require_once($sourcedir . '/QueryFrontend.php');
          qf_log_query($_REQUEST['search']);
// ---- Log end

That's the entire first version.
The important design decision:
No modification to search logic
No filtering
No tokenising
No database changes
No Python yet
It is purely a sensor.
Once installed on Dev, test:

1. Perform a search:

database access denied

2. Check:

grep smf-queries /var/log/messages

Expected:

Jul 22 14:xx:xx server smf-queries[xxxx]: database access denied

After this works, we can package QueryFrontend as an SMF 2.0.19 mod with:

install.xml
file patch
uninstall support

The nice part is this first component should remain useful forever — even when the AI retrieval layer exists. It becomes the query telemetry feed for the system.

The modules

QueryFrontend
QueryParser
Tokeniser
TokenNormaliser
IndexBuilder
IndexLookup
ANDMatcher
SortEngine
ResultFormatter

Notice that each module has exactly one responsibility:

Module------------------------------------Responsibility

QueryFrontend------------------------Accept the raw query from SMF.
QueryParser----------------------------Parse the user's query into a structured form.
Tokeniser---------------------------------Split the query into tokens.
TokenNormaliser---------------------Lowercase, remove punctuation, stemming (later).
IndexBuilder-----------------------------Build and maintain the search index from SMF posts.
IndexLookup----------------------------Retrieve the posting lists for each token.
ANDMatcher----------------------------Compute the intersection of the posting lists.
SortEngine-------------------------------Order the matching posts (newest first initially).
ResultFormatter-----------------------Return the results in a format SMF or another client can use.

The nice thing is that you can develop and test each one independently.

For example, ANDMatcher doesn't care where the token lists came from:

token1 = {5, 9, 14, 22}
token2 = {2, 5, 14}
token3 = {5, 14, 30}

result = token1 & token2 & token3

# {5, 14}

If that works, the module is done.

Later, you can swap implementations without changing the interfaces:

TokenNormaliser might later add stemming.

SortEngine might later use a relevance score instead of just date.

IndexLookup might move from MySQL to another storage mechanism.

Nothing else has to change.

One suggestion: I'd introduce one extra module between QueryFrontend and

QueryParser

QueryFrontend
    │
    ▼
QueryValidator
    │
    ▼
QueryParser

Its only job would be to reject or clean obviously invalid input before the parser sees it. Right now it might do almost nothing, but having that separation gives you a natural place to add things later (length limits, malformed queries, unsupported syntax, etc.) without complicating the parser.

Overall, this is starting to look less like "some Python scripts" and more like a well-structured information retrieval engine. That's a solid foundation for whatever features you decide to add later.
.
[Ok, that's great now that you know just how complex it needs to be.

The modules

1. QueryParser
2.Tokeniser
3. TokenNormaliser
4. IndexBuilder
5. IndexLookup
6. ANDMatcher
7. SortEngine
8. ResultFormatter

Notice that each module has exactly one responsibility:

Module   Responsibility

QueryFrontend   Accept the raw query from SMF.
QueryParser   Parse the user's query into a structured form.
Tokeniser   Split the query into tokens.
TokenNormaliser   Lowercase, remove punctuation, stemming (later).
IndexBuilder   Build and maintain the search index from SMF posts.
IndexLookup   Retrieve the posting lists for each token.
ANDMatcher   Compute the intersection of the posting lists.
SortEngine   Order the matching posts (newest first initially).
ResultFormatter   Return the results in a format SMF or another client can use.


The nice thing is that you can develop and test each one independently.

For example, ANDMatcher doesn't care where the token lists came from:

token1 = {5, 9, 14, 22}
token2 = {2, 5, 14}
token3 = {5, 14, 30}

result = token1 & token2 & token3

# {5, 14}

If that works, the module is done.

Later, you can swap implementations without changing the interfaces:

TokenNormaliser might later add stemming.

SortEngine might later use a relevance score instead of just date.

IndexLookup might move from MySQL to another storage mechanism.


Nothing else has to change.

One suggestion: I'd introduce one extra module between QueryFrontend and QueryParser:

QueryFrontend
    │
    ▼
QueryValidator
    │
    ▼
QueryParser

Its only job would be to reject or clean obviously invalid input before the parser sees it. Right now it might do almost nothing, but having that separation gives you a natural place to add things later (length limits, malformed queries, unsupported syntax, etc.) without complicating the parser.

Overall, this is starting to look less like "some Python scripts" and more like a well-structured information retrieval engine. That's a solid foundation for whatever features you decide to add later.

Post Merged: Yesterday at 05:25:42 PM
Me:

But please build me some psuedocode for the QueryParser

Sure. Let's define the QueryParser as a very simple first version. Its job is not to search, tokenise, or rank. It only understands the structure of the user's query and passes a clean representation to the next module.

QueryParser — responsibility

Input:

"database access denied"

Output:

{
    "terms": [
        "database",
        "access",
        "denied"
    ],
    "operator": "AND"
}


---

Pseudocode

MODULE QueryParser


FUNCTION parse_query(raw_query):

    IF raw_query is empty:
        RETURN error("No query supplied")


    query = trim(raw_query)


    query = remove_extra_spaces(query)


    IF query contains explicit operators:
       
        detect operators:
            AND
            OR
            NOT

        build query structure


    ELSE:

        assume default operator = AND


    extract search terms


    FOR each term:
        remove invalid characters


    RETURN ParsedQuery


---

Example flows

Simple query

Input:

database access denied

Parser produces:

ParsedQuery
{
    operator: AND

    terms:
       database
       access
       denied
}


---

Future query with operators

Input:

database AND access AND denied

Parser produces:

ParsedQuery
{
    operator: AND

    terms:
       database
       access
       denied
}


---

Future OR support

Input:

mysql OR mariadb

Output:

ParsedQuery
{
    operator: OR

    terms:
       mysql
       mariadb
}


---

Important design decision

The QueryParser should not do:

❌ lowercase conversion
❌ stemming
❌ database lookup
❌ matching posts
❌ sorting results

Those belong elsewhere.

The pipeline stays:

Raw text
   |
   v
QueryParser
   |
   v
Tokeniser
   |
   v
TokenNormaliser
   |
   v
IndexLookup
   |
   v
ANDMatcher
   |
   v
SortEngine
   |
   v
ResultFormatter

This keeps the modules replaceable. A simple QueryParser today can later evolve into a full query language parser without forcing changes throughout the search engine.

Me:

I need it to support searches like: "drugs of addiction "harm reduction" Australia"

Ah, that changes the design significantly. The parser cannot just split on spaces. It needs to understand phrases.

Your example:

drugs of addiction "harm reduction" Australia

should not become:

drugs
of
addiction
harm
reduction
Australia

because "harm reduction" is a single search unit.

The QueryParser should produce something like:

ParsedQuery

operator: AND

terms:
    drugs
    of
    addiction
    Australia

phrases:
    "harm reduction"

Then later modules decide how to search those.

A better pseudocode:

MODULE QueryParser


FUNCTION parse_query(raw_query):

    IF raw_query is empty:
        return error


    query = trim(raw_query)


    phrases = extract_quoted_phrases(query)

        Example:
        "harm reduction"

        becomes:

        phrases:
            harm reduction


    remove phrases from query


    remaining_words = split_on_spaces(query)


    terms = []

    FOR each word in remaining_words:

        IF word is not empty:
            terms.append(word)


    detect operators:

        IF query contains AND:
            operator = AND

        ELSE IF query contains OR:
            operator = OR

        ELSE:
            operator = AND


    RETURN:

        {
          operator: operator,
          terms: terms,
          phrases: phrases
        }

For your example:

Input

drugs of addiction "harm reduction" Australia

Output

{
 "operator": "AND",

 "terms": [
   "drugs",
   "of",
   "addiction",
   "Australia"
 ],

 "phrases": [
   "harm reduction"
 ]
}

Then the next stages can decide:

Tokeniser:

drugs → token

addiction → token

Australia → token


PhraseMatcher (a future module):

verify "harm reduction" appears together and in that order



This is actually the beginning of a proper search language.

A possible future query syntax:

drugs addiction "harm reduction" Australia

means:

> Find posts containing all words AND the exact phrase "harm reduction".



Later:

meth OR amphetamine "harm reduction"

means:

> Find posts containing either meth or amphetamine AND the phrase.



So I would slightly extend the architecture:

QueryFrontend
      |
QueryParser
      |
Tokeniser
      |
TokenNormaliser
      |
IndexBuilder
      |
IndexLookup
      |
ANDMatcher
      |
PhraseMatcher
      |
SortEngine
      |
ResultFormatter

The good news is your example is exactly the kind of real query that forces the design to become useful rather than just a toy token matcher.

Me:

Ok, that's great now that you know just how complex it needs to be.

Next I'll run it past "Programming" 😋 and see how well it translates to Python code. Your input is appreciated.

😄 Good plan. This is exactly the point where it is worth getting a programming-focused review.

The important thing is that the design is now language-independent. Python is just the implementation vehicle.

When you hand it over, the key points for "Programming" to preserve are:

QueryParser contract

Input

raw_query: string

Example:

drugs of addiction "harm reduction" Australia


---

Output

A structured object, for example:

Code: [Select]

{
    "operator": "AND",
    "terms": [
        "drugs",
        "of",
        "addiction",
        "Australia"
    ],
    "phrases": [
        "harm reduction"
    ]
}

---

The main Python tasks are actually quite manageable:

1. Extract quoted phrases

Find text between ".

Remove those phrases from the remaining string.



2. Parse remaining words

Split into tokens.

Preserve ordering for now.



3. Detect operators

Default:

AND

Later:

OR
NOT


4. Return a predictable structure


The beauty of doing it this way is that you can test it without touching SMF or the database:

Code: [Select]

Input:
mysql "access denied"

Expected:
terms:
    mysql

phrases:
    access denied

If the parser passes those tests, the next module can be developed independently.

One thing I would tell the Python implementer: don't over-engineer the first version. Avoid jumping straight into NLP libraries. For your use case, a small, transparent parser you can understand and modify is probably more valuable initially.

The really interesting parts come later:


building the inverted index,

deciding what to store,

ranking results,

adding fuzzy matching,

using NLP intelligently.


But the parser is the doorway. Get that clean and the rest of the engine has a solid foundation. 👍

Me:

I like his pseudocode, despite being imprecise,  I think it would be AI friendly.

I agree with your instinct. Pseudocode does not have to be executable to be valuable — especially when the next reader is another AI or a programmer.

In fact, for an AI-assisted project, good pseudocode is often better than prematurely detailed code because it captures the intent without locking the implementation into one language.

The QueryParser pseudocode we have is useful because it defines:

Purpose — what the module is responsible for.

Boundaries — what it must not do.

Input — raw user query.

Output — structured query object.

Examples — what behaviour is expected.


Those examples are particularly important. An AI can infer a lot from:

Input:
drugs of addiction "harm reduction" Australia

Output:
terms:
    drugs
    of
    addiction
    Australia

phrases:
    harm reduction

operator:
    AND

That gives it a behavioural contract.

Where I would refine it for an AI handoff is by making the ambiguity explicit:

NOTE:
Quoted text is treated as an exact phrase.
Unquoted words are individual search terms.
Default operator between terms is AND.
Parser does not perform matching.
Parser does not access the database.

That prevents the next model from "helpfully" adding search logic into the parser.

This is actually similar to how good software teams work: a module specification is often more important than the first implementation. Once the contract is clear, multiple people (or models) can implement it and still produce compatible components.

Your checkpoint-file idea fits perfectly here too. A future session could load:

Code: [Select]

Project: SmartSearch

Current module:
QueryParser

Contract:
[input/output]

Examples:
[tests]

Do not:
[database access, ranking, NLP]
[/quote]
and immediately understand the task without replaying the entire history.

For a multi-model workflow, these small design documents may end up being one of the most valuable assets you create.

[Chip]

That answers decision #1 — yes, quoted phrases need to be treated as one exact unit, mixed in with loose words. Here's the parsing logic for that:

```
MODULE QueryParser

INPUT: raw_query — e.g. drugs of addiction "harm reduction" Australia

FUNCTION parse(raw_query):
    IF raw_query is empty/whitespace-only:
        RETURN error "empty query"

    tokens = []
    scan raw_query left to right:
        IF a "quoted phrase" is found:
            extract contents between the quotes as ONE token, marked type=PHRASE
        ELSE:
            each whitespace-separated word becomes its own token, marked type=WORD

    RETURN ParsedQuery(
        raw = raw_query,
        tokens = [
            {type: WORD,   value: "drugs"},
            {type: WORD,   value: "of"},
            {type: WORD,   value: "addiction"},
            {type: PHRASE, value: "harm reduction"},
            {type: WORD,   value: "Australia"},
        ]
    )
```

Real Python for that part (regex does the quote-aware splitting in one pass):

```python

Code: [Select]

import re

def parse(raw_query: str):
    raw_query = raw_query.strip()
    if not raw_query:
        raise ValueError("empty query")

    tokens = []
    for match in re.finditer(r'"([^"]+)"|(\S+)', raw_query):
        phrase, word = match.group(1), match.group(2)
        if phrase is not None:
            tokens.append({"type": "PHRASE", "value": phrase})
        else:
            tokens.append({"type": "WORD", "value": word})

    return {"raw": raw_query, "tokens": tokens}m```

Downstream effect on the rest of the pipeline: `Tokeniser.py` normalizes each `WORD` individually (lowercase, strip punctuation), but a `PHRASE` token needs to be indexed/matched as a **consecutive run**, not just AND'd — meaning `IndexLookup`/`ANDMatcher` need to check that "harm" and "reduction" appear adjacent in a post, not just that both appear somewhere. That's a meaningfully bigger lift than plain AND-of-words (needs word-position data in the index, not just token→post_id).

Given that's a real jump in complexity — do you want phrase matching to require true adjacency (harder, needs positional index), or is "both words present, order doesn't matter" close enough for v1 ?

order doesn't matter

Good — that simplifies things a lot. If order/adjacency doesn't matter, a "phrase" is really just a shorthand for "these words must all appear," which means it collapses into the exact same AND logic as everything else. You don't even need a separate `PHRASE` token type downstream — `QueryParser` can just split the quoted content into individual words and merge them into the same flat word-list as everything else:

```python
import re

def parse(raw_query: str):
    raw_query = raw_query.strip()
    if not raw_query:
        raise ValueError("empty query")

    words = []
    for match in re.finditer(r'"([^"]+)"|(\S+)', raw_query):
        phrase, word = match.group(1), match.group(2)
        if phrase is not None:
            words.extend(phrase.split())   # "harm reduction" -> harm, reduction
        else:
            words.append(word)

    return {"raw": raw_query, "words": words}
```

For your example:
```
drugs of addiction "harm reduction" Australia
```
→ `["drugs", "of", "addiction", "harm", "reduction", "Australia"]` — five plain words, all AND'd together, quotes only ever mattered as a way to group multi-word concepts visually while typing.

That means `QueryParser`'s whole job really is just "strip the quote syntax, flatten everything into one word list" — `Tokeniser.py` then normalizes each word the same way regardless of whether it came from inside quotes or not, and `ANDMatcher.py` never needs to know phrases existed at all.

Much simpler module than I first sketched. Ready to move to `Tokeniser.py` next, or do you want to sit with `QueryParser` a bit longer?

Me:

No hurry but I like the thought of my own Tokeniser pleasing.

No rush at all — enjoying the design work is a good reason on its own to sit with it. `Tokeniser.py` is a genuinely satisfying one to get right, since it's the piece that decides what actually counts as "the same word" across your whole archive.

Basic version, since it feeds off the flat word-list `QueryParser` now hands it:

```python

import re

def tokenise(words: list) -> list:
    tokens = []
    for w in words:
        w = w.lower()
        w = re.sub(r'[^\w]', '', w)   # strip punctuation, keep letters/digits/underscore
        if w:
            tokens.append(w)
    return tokens

```

For your example, `["drugs", "of", "addiction", "harm", "reduction", "Australia"]` → `["drugs", "of", "addiction", "harm", "reduction", "australia"]` — just lowercased and punctuation-stripped.

A few real decisions that shape it, worth pinning down while it's cheap:

1. **Stopwords** — "of," "the," "a," "and" etc. Drop them, or keep everything? Dropping shrinks your index and speeds lookups, but for *your own* forum with domain-specific phrasing, sometimes a stopword carries meaning you'd search for.
2. **Numbers** — keep `"2026"` or dosage-like `"5mg"` as valid tokens, or strip anything non-alphabetic? Given your content, dosage/number tokens are probably worth keeping.
3. **Stemming** — `"reducing"` vs `"reduction"` vs `"reduce"` — do you want those to match each other, or keep it literal for v1? Stemming is a bigger lift (needs a stemming library or hand-rolled suffix rules) — my instinct is skip it for v1, add later if you notice yourself missing results because of it.

Given "basic and pleasing" — my gut says keep numbers, drop nothing else, skip stemming for now. Want to lock those three in, or think out loud on any of them?