RAG for Terry: what it is, how it copes with changing documents, and how we plan to use it for the mentok store

Part 1: What RAG means and why it exists

RAG stands for Retrieval-Augmented Generation. Before an AI answers a question, the system first looks up the relevant passages in your own documents, then the AI writes its answer from what was found. It's an open-book exam instead of answering from memory.

The purpose is to let an AI answer from your material, such as a company's policies or manuals, which it never saw in training. It also stops the AI from inventing plausible-sounding answers, and it costs far less than retraining the AI whenever your documents change.

To make the lookup work, each document is cut into small pieces called chunks. Each chunk is turned into a list of numbers capturing its meaning, called an embedding (re-doing this is "reembedding"). That lets the system find chunks about the same topic as a question, even when the wording differs.

The steps are:

1. Documents are cut into chunks (a paragraph or two each).
2. Each chunk is converted into an embedding.
3. When someone asks a question, the system finds the handful of chunks closest in meaning and hands them to the AI along with the question.
4. The AI writes its answer from those chunks, so it's grounded in your actual material.

Part 2: The problem of continuously changing documents

Documents get edited all the time. If the system still holds the old version of a chunk, the bot confidently quotes last week's policy.

The lazy fix is to reembed everything after every edit. With 500 documents (about 10,000 chunks), that means redoing all 10,000 chunks to fix one sentence. It works, but it's slow and wasteful.

The better answer has three layers.

Layer 1: Detect what changed

• A webhook is an automatic instant ping the app sends the moment a file is edited, so the system doesn't have to keep checking.
• A hash is a short fingerprint calculated from a chunk's text. Change one character and the fingerprint changes completely.
• Compare each chunk's new fingerprint with the old one. If they match, skip the chunk. If they differ, that chunk changed.

Layer 2: Update only that chunk

• Reembed just the changed chunk and store it under the same ID (its unique label).
• The original text says "absurd", which is almost certainly a mistranscription of upsert, a database term meaning "overwrite if it exists, insert if it doesn't".
• If a document is deleted, delete all its chunks using the document's ID.
• The result is one chunk reembedded instead of 10,000.

Layer 3: Safety nets

• Tag every chunk with a version number, and search only the latest versions, so a stale chunk can't slip into an answer.
• Webhooks can occasionally be missed, so also run a scheduled full resync, a periodic complete check that catches anything the pings missed.
• The payoff: if the policy was changed to 14 days, the bot now says 14 days, not the old figure.

The takeaway: don't rebuild everything. Detect what changed, update only that, and double-check on a schedule.

Part 3: Addendum — how the same ideas apply to our mentok store

Our store doesn't hold documents. It holds small entries made of single words, word pairs (binomials) and mini-phrases, which we call mentoks. The database manager's job is to keep that store current as new conversations arrive, so the AI stops repeating mistakes it has been corrected on. Below is how the three layers map onto that job. This is the proposed plan, and the details are still open.

The main difference

In the text above, the unit being managed is a chunk of a document. Here, every entry is already tiny, so there's no chunking. And nothing is "edited" the way a file is. Instead, every new question and answer passing through the system changes the entries: counts go up, new words appear, and corrections replace older wording. The change trigger is a new query and reply arriving, not a webhook from an app.

Layer 1: Detect what changed

• A small program sits between the question and the AI, like a middleman, and watches each query and reply go past. The word and phrase extractor lifts out the notable words and mini-phrases.
• Each entry gets a fingerprint (hash) from its normalised text. Normalised means lower-case with spacing tidied, so "Same Thing" and "same thing" are one entry.
• This lets the fingerprint double as the entry's ID. The same phrase always lands on the same ID, so "is this new, or one we already hold?" answers itself.

Layer 2: Update only what changed

• An existing entry is overwritten, for example its count goes up. A new one is inserted. This is the upsert from Part 2.
• Each entry records which conversation or transcript it came from. If a source is later corrected or withdrawn, everything it contributed can be found and removed by that source ID, without touching anything else.
• Reembedding only applies to entries that are new or changed. Because the store is tiny, the cost is trivial.

Layer 3: Safety nets

• Versions: when a correction supersedes an older entry, the new one is tagged as the latest and searches see only the latest. The older entry is kept, dated, rather than erased. If the two ever conflict, they're shown in dated sequence so the operator decides.
• Rebuild check: the archive of conversation transcripts is the source of truth, so the store can always be rebuilt from it. A scheduled full resync compares the live store against a fresh rebuild and flags any difference. That catches anything the live watching missed.

Where it sits

• It would be one separate stage among the experiment's stages (context definition, lookup, second AI, main AI), kept apart so each can be traced and debugged on its own. The middleman program picks it up at start-up from a stages folder, as the other stages do.
• The store itself would live in the existing memory-store home on the Clone machine.

Two honest caveats

1. The store is small, so a full rebuild is cheap. The change-detection layers matter most as the store grows, and until then they act more as insurance than necessity. That doesn't make them wasted, since building the habit now means growth won't force a redesign.
2. "Search only the latest version" works against keeping history. The plan above resolves it by keeping old versions but hiding them from normal searches. If history isn't wanted, the design gets simpler.

Install on the live system stays the operator's call, and nothing here is built yet.