hotwords.py — manual

Version 2.1.0 — 2026-10-27 — Andrew.human & Claude.ai

Written for humans and for other AIs/machines. Everything a reader needs to run, check or extend the tool is here.

1. Purpose

hotwords.py does two jobs for the Influence experiments:

1. Reduce each query or reply of LLM1 to one small ranked array of its hottest words and phrases, and store the array. No text is kept.
2. Build the pseudo query for LLM2 (pseudo): the new query's own hot array, followed by the terms RAG chose from the stored arrays that match it.

The pseudo query is a theoretical query. Its value is in what it ranks as important. It is not meant to read as a sentence and nothing checks that it is coherent; making the final words coherent is LLM1's job. For that reason the tool has no coherence step: order and weights are the content.

2. Requirements and install

• Python 3.6 or later, standard library only. Written to avoid anything newer than 3.6 and anything newer than SQLite 3.6.20 (no UPSERT, no JSON1, no window functions).
• Copy hotwords.py to the folder you run your scripts from (e.g. /root/memstore). Nothing else to install.
• The script prints its version once when it starts (to stderr), and every error message carries the version.
• Output is always UTF-8, whatever the shell's locale (Python 3.6 on a POSIX/C locale would otherwise crash on non-ASCII text).

3. Where things live

Item	Default	Override
Folder	/root/memstore	env MEMDIR
Database	$MEMDIR/hotwords.db	env HOTWORDS_DB
Stopword list	$MEMDIR/hotwords_stopwords.txt	env HOTWORDS_STOPWORDS

The database path is printed at the start of every command. The database file is created with mode 600 and a missing folder with mode 700. The stopword file is created on first use from a built-in English list; edit it freely (one word per line, # starts a comment). Its hash is recorded with every array.

An older database is refused (version 1, or the 2.0.0 database), not converted, and is left untouched. Point HOTWORDS_DB at a new file.

4. Commands

add

Reduce a query or reply to its hot array and store it.

hotwords.py add --kind query --text "Why is the forum slow after the upgrade?"
hotwords.py add --kind reply --file reply1.txt reply2.txt --label run42
hotwords.py add --jsonl pairs.jsonl
cat reply.txt | hotwords.py add --kind reply --file -

• --kind query|reply is required with --text and --file. With --jsonl each line may carry its own kind; --kind is the default.
• --file takes one or more paths, one entry per file. - means standard input.
• --jsonl lines look like {"kind": "reply", "text": "...", "label": "run42"}.
• --label is free text stored with the array (a run id, a source). It is used for nothing else.
• The stored array is printed.

Counted once. Two things stop repeats from skewing the counts:

• The same text (whitespace collapsed) added again changes nothing except times_seen. Experiment 000 repeats one query about 20 times; it stays one entry.
• A different text that reduces to the same array (same terms in the same order, same kind) is stored as that array, with times_seen increased and nothing else changed.

Exit code 0 on success, 1 if any text was skipped (no countable words), 2 on an error.

pseudo

Build the pseudo query for LLM2. Stores nothing.

hotwords.py pseudo --text "Is serotonin syndrome likely at a normal MDMA dose?"
hotwords.py pseudo --file query.txt --format json
echo "..." | hotwords.py pseudo --file - --format table

Options: --kind (what the input is; default query), --source query|reply|both (which stored arrays RAG may draw on; default both), --arrays N (default 5), --extra N (default 12), --min-match X (default 0.05), --format line|table|json (default line), --weights (with line: add each term's weight).

line (what goes to LLM2) is two lines. Phrases are quoted so their boundaries are clear:

query: likely | normal | dose | serotonin | syndrome | mdma
related: "serotonin syndrome" | dangerous | risk | worries | combined | maois

• query: the new query's own hot array, most important first.
• related: RAG's choices, most important first, or (none).

With --weights each term carries its weight, so faint terms stay visibly faint:

query: seretonin:1.00 | sindrome:0.95 | 2-cb:0.56
related: "serotonin syndrome":0.81 | risk:0.40 | combined:0.34 | "2c-b dose":0.22 | dose:0.21

A query term's weight is its relative score (1.0 for the top term). A related term's weight is array match × its relative score in that array, summed over the arrays used. Both are on a 0..1 scale, so they can be compared. Influence is rarely overt: a weak match is kept in proportion to its weight rather than cut, which is why the floor is low.

table shows the same with counts, scores, every fuzzy match used, which stored arrays were used and where each related term came from. json carries everything (every score and weight, the fuzzy matches, the arrays chosen with their match, the settings) and is the record to save with a run.

The query's own array is built exactly as add would build it, so pseudo followed by add of the same text gives the same array.

show

One stored array, with its settings: hotwords.py show 17.

top

Hottest terms across the stored arrays, ranked by the sum of their relative scores: hotwords.py top [--kind query|reply|both] [--n 20] [--words | --phrases].

stats

What the database holds, the array rules and RAG defaults in force, and how many arrays were made under a different stopword list or different rules.

5. How an array is made

5.1 Reading the text

1. Fenced code blocks and web addresses are removed.
2. The text is lower-cased and split into segments at punctuation (. , ; : ! ? ( ) [ ] { } " / \ |, dashes, line breaks). Phrases never cross a segment.
3. Words are runs of letters and digits, with an internal hyphen or apostrophe kept, so 2c-b, 5-ht2a and don't stay whole. A trailing 's is dropped.
4. Spelling key. Every term also gets a key: the term with accents, hyphens and apostrophes removed. 2C-B, 2-CB and 2cb all have the key 2cb. Terms whose keys are equal are the same term. Inside one text they merge (counts add up, the earliest position counts, the form typed first is shown). The key is also how terms are stored, counted for rarity and matched.
5. Position is the word index of a term's first occurrence, counting every word read (stopwords included): the first word is 0.

5.2 Which terms are candidates

• A content word has at least 2 characters, is not all digits, and is not a stopword. Every content word is a candidate hot word.
• A phrase is 2 or 3 consecutive words that start and end on a content word (a stopword may sit in the middle). To be a candidate it must occur at least twice inside its own text.
• A two-word phrase that only ever appears inside an equally frequent three-word phrase is dropped in favour of the three-word phrase.

5.3 Score

score = (1 + ln count) × rarity × position weight

• count: occurrences in the text.
• rarity(word) = ln((N + 1) / (df + 1)) + 1. N is the number of texts counted, df the number of them containing the word. The text being processed is counted in as if already stored (N+1, df+1), so pseudo and add agree. A phrase's rarity is the average rarity of its content words. With nothing counted yet, rarity is 1.0 for everything.
• position weight = 1 / (1 + first_index / 20): 1.0 for the first word, 0.5 at word 20, about 0.25 at word 60. Early words weigh more.

5.4 How many terms an array keeps — relevance, not space

Terms are ranked by score. The array keeps every term scoring at least 40% of the top term, but never fewer than 3 (if the text has them) and never more than 12. rel is a term's score divided by the top score, so it is 1.0 for the top term.

5.5 Stored per array

Kind, a hash of the term list, label, words read, stopword hash, tool version, the settings used, times_seen, UTC timestamps, and per term: rank, term, word or phrase, count, first index, score, rel. Scores and rel are a snapshot from the moment the array was made.

5.6 The word counter

One extra table, vocab, counts for every content word how many texts contained it. It exists only to compute rarity. It holds words, not texts.

6. How RAG chooses (pseudo)

1. Take the query's own array; each term's weight is its rel.
2. Matching terms. A query term matches a stored term when
   • their spelling keys are equal (factor 1.0), or
   • the query term is a plain word (letters only) of 5 or more letters and a word the database has counted differs by a few letters (a fuzzy match). Allowed: 1 edit for words of 5 to 8 letters, 2 edits for words of 9 or more. An edit is a letter added, dropped or changed, or two neighbouring letters swapped. The factor is 1.0 minus 0.1 per edit (0.9, 0.8), and at most the 3 closest near-spellings of each word are used. Phrases match when each of their words matches, with the factors multiplied.
   • Anything containing a digit is never fuzzy-matched, and neither are words shorter than 5 letters. One character separates different drugs (2C-B/2C-I, 25I-NBOMe/25B-NBOMe), so those match only on the spelling key.
3. Find stored arrays (of the allowed kinds) holding matching terms. An array identical to the query's own (same kind, same terms) is skipped.
4. Match = the share of the query's total weight that the stored array covers, each matched term counted as (query rel) × (stored rel) × factor, using each query term's best match in that array. The score is 0..1. Because rel already includes rarity and position, a shared rare, early term counts for more than a shared common, late one.
5. The best --arrays stored arrays with match at least --min-match are used.
6. Their terms that do not match a query term are weighted match × stored rel, added up across arrays, and the best --extra become the related: terms. A stored term that fuzzy-matched a query word is not offered as related, since it is the same word.

All lookups use the small array_terms table (indexed by spelling key) and the list of counted words; no text and no embeddings are read. Every fuzzy match used is listed in table and json.

7. Privacy

The database holds hot terms, not text, but a secret that is a single rare word can appear in an array. The file is mode 600. There is no remove command in 2.0.0; delete the database and re-add what you want to keep.

8. Known limits in 2.0.0

• Rarity comes from the counted texts themselves. Once the archive is heavily about one subject, that subject's own words become common and are marked down against generic words. Early on the effect is small; it grows with the archive. The earlier background-seeding command was removed, so there is no outside reference.
• A phrase in a short query almost never repeats, so queries yield mostly words. Phrases enter through replies and through longer texts.
• Fuzzy matching is about spelling only. It does not know that two different words mean the same thing; that needs embeddings, not built. Words with a digit and words under 5 letters never fuzzy-match, so a misspelt MDMA or LSD will not find its stored arrays. Token ids and vectors per term are not stored yet.
• The fuzzy search reads the list of counted words on each pseudo run and compares it letter by letter against each long query word. That is quick for thousands of words; it should be revisited if the list grows to hundreds of thousands.
• Scores in stored arrays are snapshots; they do not change as the archive grows.
• English stopwords only. No remove command. No --json for add, show, top.

9. Version history

Version	Date	Change
2.1.0	2026-10-27	Fuzzy matching. Spelling key (accents, hyphens, apostrophes removed) so 2C-B, 2-CB and 2cb are one term: merged inside a text, matched across texts, rarity counted per key. Near-spellings of plain words of 5+ letters match at a reduced weight (1 edit up to 8 letters, 2 from 9, a swap counts as one); words with a digit and short words never fuzzy-match. Faint matches count: RAG defaults changed to 5 arrays, 12 extra terms, match floor 0.05 (were 3, 8, 0.15). New --weights flag for the line format. Fuzzy matches listed in table and json. Database schema 3 (adds the spelling key); older databases are refused, not converted.
2.0.0	2026-10-27	Redesigned around small arrays instead of documents: one ranked hot-term array per query or reply, no text kept. Array length set by relevance (40% of the top score, 3 to 12 terms). Position weight added (early words score higher). Phrases must repeat within their own text. Dedupe by text hash and by array hash. New pseudo command builds the pseudo query for LLM2 (own array plus RAG's chosen terms), stores nothing. Removed seed-background and the document tables; rarity comes from a word counter. Settings recorded with every array. Version 1 databases are refused, not converted.
1.0.1	2026-10-27	Fix: non-ASCII text no longer crashes under a POSIX/C locale (Python 3.6): UTF-8 handled explicitly for arguments, standard input and output. Timestamps use timezone-aware UTC.
1.0.0	2026-10-27	First release: add, top, show, stats, seed-background; separate query and reply counts; counted-once rule; stopword file with recorded hash.