Sessions: Wed 14/10/2026 20:56 to 21:11, Fri 16/10/2026 11:15 to 11:51 (Sydney time). Andrew's ideas are in bold with a rating line beneath. Ratings are Claude's judgement, not experimental results. Nothing here has been tested, and the closest prior art (SuTRA, arXiv 2608.18087) could not be read.
| # | Idea (Andrew's words, condensed) | Novelty | Merit | Testability |
|---|---|---|---|---|
| 1 | Split words on etymological boundaries | 5 | 6 | 7 |
| 2 | Combine it with the cooperative-attention (LLM1/LLM2) design | 3 | 3 | 3 |
| 3 | Should cut retraining time for derivatives | 5 | 6 | 8 |
| 4 | "Has to be a benefit, it's fundamental to language" | n/a | 4 | n/a |
| 5 | Nomenclature-based pre-classified tables/indexes | 6 | 8 | 8 |
| 6 | Coprolalia as the illustrative case | n/a | 7 | n/a |
| 7 | A basic multi-LLM setup could deploy it for richer context | 4 | 5 | 6 |
Overall: the core instinct is sound, and idea 5 is the strongest because it fixes the main practical weakness of idea 1. Idea 4 is the weakest as an argument, though its conclusion is plausible. Ideas 2 and 7 add machinery that a plain lookup table mostly replaces.
I just had a potentially brilliant idea for the next generation of LLMs, unless of course, it's old hat ...
Claude: [October 14, 2026, 08:56:19 PM]
Go ahead and lay it out. I'll tell you plainly whether it's old hat, who has done something similar, and where it would break if not. A few sentences is enough to start, and the mechanism matters more than the polish.
split words up on their etymological boundaries
Idea 1 · Novelty 5 · Merit 6 · Testability 7. Morphology-aware tokenization is established (Turkish, Finnish, Spanish, Danish work). The etymological layer under it, Greek and Latin roots rather than current-language affixes, is the less-explored part. Merit is real for technical vocabulary and weak for ordinary English, where BPE already learns common affixes. Testable with a small model and a rare-term benchmark.
Claude: [October 14, 2026, 08:57:20 PM]
Partly old hat, partly not.
The old part is morphology-aware tokenization: splitting on prefixes, stems and suffixes instead of BPE's frequency-driven merges. Morfessor-style segmenters, MorphPiece and similar work have all tried it. The results are modest. They help most in morphologically rich languages (Turkish, Finnish) and barely at all in English, because BPE already learns common affixes like un-, -ing and -tion by frequency.
The part I haven't seen done properly is etymological rather than morphological boundaries. Morphology splits at the word-formation level of the current language. Etymology reaches under that to the Greek and Latin roots, which is where technical vocabulary lives: pharmaco-logy, neuro-trans-mitter, phen-ethyl-amine. BPE shreds these into arbitrary fragments, so a model has to learn each word's meaning almost from scratch. With root-aligned tokens, -logy and neuro- would carry meaning across thousands of words, which should help rare technical terms, new coinages and cross-word generalization. Your pharmacology and neurochemistry interests are a good test case, since chemical nomenclature is already systematically compositional.
Where it would break:
Likely fix for the length and misleading-split problems: don't make the split hard. Keep the whole-word token and add the root embeddings to it, similar to how fastText sums subword n-gram vectors. That way common words stay cheap, and rare or technical words inherit meaning from their roots. A fair test would be a small model trained on a technical corpus (pharmacology papers, say), compared against plain BPE on held-out rare-term tasks.
If you tell me whether you meant hard splits or something softer, I can look for the closest published prior art.
ok then, get Attention to consider it then
Idea 2 · Novelty 3 · Merit 3 · Testability 3. Folding the idea into the existing LLM1/LLM2 cooperative-attention design. It maps neatly on paper, but logit-bias injection cannot carry root meaning into how a frozen model reads its input, so the second model adds complexity without a clear gain. Andrew dropped LLM2 two turns later, which was the right call.
Claude: [October 14, 2026, 08:59:08 PM]
I've taken this to mean putting the etymology idea inside your LLM1/LLM2 pairing design. It fits that design better than it fits a tokenizer.
How it maps. LLM1's salient token (say the BPE fragments of pharmacokinetics) gets paired by LLM2 with root-level memtoks (pharmako-, kinesis, -ics). Your static datastore is a Wiktionary-style etymology table, and your "generate a map once, then look up" refinement is exactly how you'd build the root index. The pair's weight is inherited from LLM1's half and acts on inference.
Why this beats the tokenizer version. Last turn's main objection was that hard splits mislead when roots have drifted (understand, butterfly). Your own insight that relevance must be judged against the current context handles that. LLM2 would only surface the roots when they're relevant to what LLM1 is doing right now. For technical vocabulary it fires, and for understand it stays quiet. It also needs no retraining and no new tokenizer, which matches your no-weight-changes constraint.
The weak point. Logit bias works on output tokens, so it can't carry "this word means drug + movement" into the model's reading of the input. Real influence on understanding means injecting at embedding or KV level, and a frozen model has never learned to use roots that arrive that way. It would most likely treat them as noise. Etymology helps most when it's present during training, which is the very thing this design avoids.
Cheap test first. In this setup LLM2 can be a plain dictionary lookup, with no second model and no hardware strain. Take one small frozen model and a set of rare technical terms, then compare three conditions on definition or usage questions:
If the text gloss doesn't beat the plain prompt, injection almost certainly won't either. If it does, you've shown the signal is worth the harder engineering. Text glosses are the "just more text" approach you want to move beyond, but they make the right baseline.
forget about LLM2
Claude: [October 14, 2026, 09:01:10 PM]
Dropping LLM2 leaves a single model with etymology built in. There are two places it could go.
1. In the attention itself. Add a bias to the attention scores whenever two tokens in the context share a root, the same way relative-position bias works. The mechanism is well trodden: relation-aware attention (Shaw et al.), syntax-aware attention and graph-biased transformers like Graphormer all do this. Applying it to etymological relations is new as far as I know, but I expect a weak payoff. It only helps when related words appear in the same context window. The real gain from etymology is across the whole vocabulary, and attention doesn't touch that.
2. In the embeddings. Each token's input vector becomes its own embedding plus the sum of its root embeddings, so pharmacology, pharmacist and pharmacokinetics all share a learned pharmako- component. This is fastText's subword idea with etymological roots instead of character n-grams. It is where I'd put the effort. It acts on every occurrence of every word, not just co-occurring ones, and rare technical terms inherit meaning from their frequent relatives.
The fix for my earlier objection. I said a frozen model would treat injected roots as noise. Without LLM2 the model doesn't need to be frozen. Initialise the root embeddings to zero so the model starts out identical to the original, then continue training on a technical corpus (pharmacology papers, say). Zero-init means the roots can only help or stay silent, and the model learns when to use them. That also covers the understand/butterfly problem, since it can learn to ignore roots that carry no meaning.
Experiment. Take a small open model (a few hundred million parameters) and run continued pretraining twice on the same corpus, with and without root embeddings. Then compare on held-out rare technical terms, using perplexity and definition-completion. This does need real GPU time, so a rented GPU is more realistic than the Clone box.
The root lookup is the one piece of heavy preparation. Wiktionary's etymology data would need parsing into a word-to-roots table, and that table's quality will largely decide whether the experiment works.
it should boost retraining time saved for derivatives
Idea 3 · Novelty 5 · Merit 6 · Testability 8. A specific, falsifiable claim, which is its strength. Plausible for small models, domain corpora and newly coined words, since a derived word's vector is composed from known roots at once. Likely to shrink at frontier scale, where models have seen most derivations many times. Measurable by counting fine-tuning steps needed to reach a target score on held-out derived words.
Claude: [October 14, 2026, 09:02:45 PM]
I read "derivatives" as derived and newly coined words, such as a new drug name built from familiar roots. On that reading the claim is plausible, and it's the strongest argument for the idea.
Why it should save training. A new word like -amine compounded onto a novel prefix arrives in a plain model as arbitrary BPE fragments with no learned relationship to its relatives, so the model needs many examples before it knows what the word is. With root embeddings, the new word's vector is composed from known roots the moment it's spelled out. It starts out roughly right, and fine-tuning only has to correct it.
Prior art. Initialising a new token's embedding from the average of its subword embeddings is already a standard trick when extending a vocabulary, and fastText handles unseen words the same way. Your version makes the pieces semantically meaningful (roots) instead of frequency accidents (BPE fragments). So the mechanism is established, and the etymological twist on it is the new part.
Where I'd expect it to disappoint:
How to measure it. Hold out a set of derived words the model has never seen, then count how many fine-tuning steps or examples each version needs to reach a target score on them. If the root-embedding model gets there in a fraction of the steps, the saving is real. Fewer steps to match is a sharper test than final accuracy, because both models may end up at the same place.
it's got to be a benefit somehow as it's fundament yo language and these are LLMs !
Idea 4 · Merit 4. The conclusion (some benefit) is plausible; the argument is not. LLMs already absorb etymological structure statistically from raw text, so "fundamental to language" cuts both ways: it is the reason to expect a benefit and the reason the benefit may be small. Rated as an argument, not as the underlying idea.
Claude: [October 14, 2026, 09:04:25 PM]
Etymology being fundamental to language is the reason to expect a benefit, but it's also the reason the benefit may be smaller than you'd hope.
LLMs learn language from raw text, and etymology is already written all over that text. After seeing pharmacology, pharmacist and pharmacokinetics thousands of times, a model absorbs that these share something, even when BPE chops them up inconsistently. Models can already tell you what -logy means. So the structure isn't missing from the model. It's being learned the slow, statistical way, and your idea hands it over directly.
That puts the benefit in the margin between learned and given, and the size of that margin depends on how much data the model has had:
So the instinct is sound, but it's an expectation that the experiment has to confirm, not something language guarantees. The experiment is also what turns "might be old hat" into a yes or no, so it's worth running on a small model before building anything larger around it.
nomenclature ?
Idea 5 (first mention) · see next turn for the full form. A one-word prompt that redirected the whole project, and the most useful single contribution in the conversation. Systematic nomenclature (IUPAC, WHO INN drug stems, Latin binomials, anatomical terms) is built by rule, so decomposition is authoritative and mostly deterministic. That removes the data-quality problem that threatened the Wiktionary version.
Claude: [October 14, 2026, 09:05:16 PM]
Nomenclature is the best place to start, and it fixes the biggest weakness I raised.
My main worry was the root table. General etymology data from Wiktionary is patchy and inconsistent. Systematic nomenclature is built by rule, so the decomposition is authoritative and mostly deterministic:
Two cautions. INN stems are conventions that encode pharmacological class, not true etymology, which doesn't matter for the idea but is worth naming accurately. And chemistry-specific tokenization isn't untouched ground: SMILES tokenizers, ChemBERTa-style models and Galactica's special chemical tokens exist. What I haven't seen is the soft-sum design (whole-word token plus shared component embeddings) applied across drug names, IUPAC names and general text together.
As a first experiment this beats general etymology. The data is clean, the gain is measurable (can the model classify a held-out drug from its name alone?), and it's your home territory in pharmacology. If it works there, general etymology becomes the next, harder step.
nomenclature-based pre-classified tables/indexes
Idea 5 · Novelty 6 · Merit 8 · Testability 8. The strongest idea in the conversation. A versioned, auditable table sourced from the nomenclature authorities, updatable without retraining (a new WHO stem is a new row), and usable three ways: training-time embeddings, inference-time lookup, or auxiliary supervision. Novelty is held at 6 because chemistry-specific tokenizers exist; the combination with shared component embeddings across drug names, IUPAC names and general text is the part not seen. Main risks: ambiguous short stems (-ine, -ide), trivial and brand names with no structure, and stems that encode naming convention rather than mechanism.
Claude: [October 14, 2026, 09:06:39 PM]
That makes the root table a first-class component, built from the nomenclature authorities and not scraped from dictionaries. It can feed the model three ways, and they're worth keeping separate because each tests something different:
What makes it attractive. The table is versioned and auditable, and it updates without retraining. When WHO adds a stem, you add a row. That suits your provenance-minded way of working, since every decomposition can be traced to a source entry.
Schema. Each row needs the pattern, its position (prefix, infix or suffix), the class, a parent class for hierarchy (benzodiazepine → GABA-A modulator), the source and the version. The hierarchy matters because it lets the embeddings share structure at several levels, not just one.
Where the tables will bite:
A first version could be small: a few hundred INN stems plus the IUPAC functional-group fragments. That's enough to run the held-out-drug test without building the full thing.
dinner time thought based on copralalia
Idea 6 · Merit 7 (as an illustration). Not a mechanism but a well-chosen test word. Coprolalia splits cleanly into copro- (dung) and -lalia (speech), each with relatives (echolalia, palilalia, coprophagia, coprolite), and a BPE tokenizer may produce a cop fragment that points at police. That makes the misleading-fragment problem concrete in a way that understand or butterfly did not. Caveat: the exact BPE split depends on the tokenizer, and the example is hypothetical until checked against a real vocabulary.
Claude: [October 14, 2026, 09:08:08 PM]
Coprolalia is a good test word for the idea, so I'll take it as the example. It splits cleanly into copro- (Greek kopros, dung) and -lalia (talk, speech). Each half has relatives:
A model with shared components gets the whole family from two small pieces, and it can read a word it has never met, like coprographia, correctly on first sight.
A typical BPE tokenizer will chop it into something like cop + ro + lalia, depending on the vocabulary. The cop fragment points the model toward police, which has nothing to do with the word. That's the misleading-fragment problem in a nutshell, and it's a better illustration than understand or butterfly because the damage is plain to see.
It also sits well with the nomenclature tables, since medical terminology like this is built on exactly this Greek compounding.
If you meant a separate thought for dinner and not this example, give me that one and I'll treat it on its own.
no, that's it - I'm sure that perhaps with a basic multiLLM configuration could deploy this for richer context
Idea 7 · Novelty 4 · Merit 5 · Testability 6. Pragmatic: it runs on frozen models with no retraining, which suits the hardware available. The split Claude proposed (table lookup first, a small LLM only for context-dependent judgement, compact annotations to the main model) keeps the second model narrow. The honest weakness is that a plain table lookup may deliver most of the gain, in which case the second model is not earning its place. It also improves answers about rare terms without changing how the main model represents words internally.
Claude: [October 14, 2026, 09:10:45 PM]
Yes, and this version has a practical advantage: it runs on frozen models you can already get, with no retraining. It's the inference-time route from earlier, with a second model in a narrower job than the LLM2 you shelved. Here it's an annotator in front of the main model.
Division of labour.
What it can and can't do. It improves answers about rare terms, because the main model gets the structure explicitly. It can't change how the main model represents words internally. That still needs the embedding route, and the benefit stays at the context level. It also costs extra latency and tokens per query, so it's worth running only on words flagged as rare or technical, not on every token.
Test. It slots straight into the experiment from before, as a fourth condition next to plain prompt, text gloss and root embeddings. The question to ask is whether the small LLM beats the plain table lookup. If the table alone gives the same gain, the second model isn't earning its place, and you'd have a simpler system that's still useful.
I can't test anything ! I ain't got the hardware
(Claude ran two web searches for prior art and tried to read the SuTRA paper; the PDF returned no extractable text.)
Claude: [October 14, 2026, 09:12:08 PM]
No hardware is fine, because the part that decides "is it old hat?" needs none. I ran a quick prior-art search, and it was only partly successful.
What I found. A 2026 paper titled SuTRA: Structurally-Unified Tokenization with Root Awareness looks like the closest match. I couldn't read it, though: the PDF came back with no extractable text, so I don't know whether its roots are etymological or just morphological, or what it does with them. The other hits were morphology-aware work in specific languages, such as Morpheus for Turkish, a Spanish case study and Danish morpheme-level training. That fits what I said earlier: morphology-aware approaches exist, mostly for morphologically rich languages. I can't confirm anything about the nomenclature-table angle either way.
What you can do without hardware:
The paper abstract is the quickest next step, since it decides whether the idea is new before anything else gets built.
(Attached an Instagram video, 22031a00-instagram-1792108940024.mp4.) some info for ya
(Claude extracted frames and read them as a contact sheet. It then tried to install a speech-to-text package, and Andrew declined that tool use.)
shit, I could process it quicker with Qwen
Claude: [October 16, 2026, 11:17:14 AM]
Fair enough, I'll stop the install and not push transcription. I don't need the audio anyway, because the frames carry the content. It's a 2.4-minute explainer from "KGP Talkie" called LLM Concepts 001: How is a tokenizer trained?, and it walks through BPE:
This hits your idea directly. Steps 3 to 5 show why the coprolalia problem exists. Merges are chosen purely by pair frequency, so the vocabulary has no idea what a root is. Th, in and ing get tokens because they're common, not because they mean anything. A fragment like cop survives because it's frequent in other words.
It also points at where your idea could attach. Step 4 is the only place where the choice of merges is made, so the root table could constrain it. Don't merge across a root boundary, and give each known root its own protected token. That's a hard-split variant, and it needs no new embedding machinery, only a different training step for the tokenizer. It's the version closest to the morphology-aware tokenizers I mentioned, so the SuTRA paper is still the thing to check.
If you want, run the video through Qwen and send me the transcript, in case the narration says something the visuals don't.
(Pasted the video's narration, reproduced as given.)
Today in LLM concepts, let's see how a tokenizer is trained.
Before an LLM reads a single word, it needs a tokenizer. And here's the surprising part: the tokenizer isn't written by hand. It's trained on text before the model's own training even starts.
Step One: COLLECT THE TEXT
—A big sample of the same kind of text the model will read—web pages, books, code, many languages.
Step Two: START TINY
The first vocabulary is just single characters, or in GPT's case, the 256 possible bytes. So at the start, every word is cut into its smallest pieces.
Step Three: COUNT
Across the whole text, inside every word, the trainer counts every pair of neighboring pieces and how often each pair appears. Say "I" followed by "N" shows up millions of times.
Step Four: MERGE
The winning pair becomes one new token, "in," and the rule is saved: "I plus N becomes in." Then count again, merge again. "In plus G becomes aing." Round after round, common chunks like "aing" and "tion" and whole words like "the" each become a single token.
Step Five: STOP AT THE TARGET SIZE
The vocabulary size is fixed before training, like 32,000 or 50,000 tokens. Once the vocabulary reaches that number, training is done.
Only two things come out: the vocabulary, every token with its ID, and the merge rules in order.
That's the whole tokenizer, and then it's frozen. The model learns one embedding row for every token ID.
So if you change the tokenizer later, the model reads nonsense.
One more effect: a tokenizer only learns what it saw. If the training text had little Hindi, Hindi words never got merged into big chunks, so the same sentence in Hindi turns into many more tokens. Count and merge.
That recipe is called BPE.
But BERT's tokenizer picks its merges differently, and sentence piece doesn't even split on spaces.
In today's LLM interview reel, BPE versus word piece versus sentence piece.
Which one does your model use ?
Comment and tell me.
Which model's tokenizer should we open up next ?
Claude: [October 16, 2026, 11:49:52 AM]
The transcript matches the frames, and the narration adds nothing that changes my read. Two details are worth having.
It confirms the weak point for your idea. The narrator says it plainly: the tokenizer "only learns what it saw", and step four chooses merges purely by pair count. Nothing in the process has any notion of meaning. That's the gap your etymology and nomenclature tables would fill.
It also shows where the cheapest intervention sits. The narration says only two things come out of training, the vocabulary and the ordered merge rules. So a root-aware tokenizer is a change to that one output, and the model downstream stays standard:
That's the hard-split design again, now stated as a patch to this exact procedure, so a person could prototype it in a short script. The merge loop is a few dozen lines of Python, and running it on a small text sample needs no GPU, only a CPU. That's the first piece of this idea you could build and inspect on the hardware you have.
Two cautions. The video states that changing a tokenizer after training makes the model read nonsense, so a root-aware tokenizer needs its own model trained from scratch, or at least heavy continued training. The soft-sum embedding route avoids that, which is why I'd still keep it as the main option. And the video stops at BPE, WordPiece and SentencePiece, none of which use meaning-based merges, so it says nothing either way about whether your variant exists. That's still the SuTRA question.
Transcribe this conversation and emphasize my ideas and rate them, in raw md