(continued from page 63)
…the first token a convenient place. Your report shows it as a huge value at position 0 (1,636 against about 15 for the rest at layer 6). I'm giving that as the usual explanation, and I haven't proved it on this model.
what is the sink ?
Claude: [October 18, 2026, 10:17:32 AM]
The sink is the very first token of the text, which soaks up attention it doesn't need to spend. In your prompt it was The, but it's whatever token comes first.
Why it exists. Each head must split 100% of its attention across the earlier tokens. It can't say "nothing here is relevant" and look at nothing, so it has to put the attention somewhere. Training settled on the first token as the place to dump it. That token's contribution is made small, so attention that lands there changes almost nothing. It works as a "do nothing" option.
Why the first token and not another.
What your report shows. In the middle layers the sink took most of the attention: 82% at layer 11 and 88% at layer 16. In those layers most heads were mostly idle at that moment. That's why the inspector excludes it when counting memory: attention on the sink is an idle head, not interest in anything.
Why it matters in practice. If you removed the first token, or threw it out of the stored context, the model would degrade badly. Long-chat systems are known to keep the first few tokens when they trim old text for that reason. I believe llama.cpp's "keep the first N tokens" setting is there for the same purpose. I haven't checked that in your build.
now, do you know how the llm//attncame to be ? it wasn't someones eureka moment, it evolved ... i know but do you ?
Claude: [October 18, 2026, 10:19:38 AM]
Yes, I know the outline, and you're right that it was a chain of fixes to specific problems. This is from memory. I'm confident of the sequence and of the main years and names, and less so of the smaller details.
1. The problem (2014): the bottleneck. The leading translation systems read a sentence into one fixed-size list of numbers and then wrote the translation from that. Long sentences didn't fit, and quality dropped sharply with length.
2. The first attention (2014). Bahdanau, Cho and Bengio let the writing side look back over all the reading side's word-by-word states and take a weighted mix, with the weights learned. That's the attention you've been looking at, in its first form, bolted onto a recurrent network. Related ideas appeared around the same time, in Graves's Neural Turing Machine (which read memory by similarity) and in memory networks. The memory-lookup framing matters for your LLM2 idea, because attention began as a way to look things up.
3. Refinements (2015–2016). Luong and colleagues simplified the scoring. Others used attention for image captioning and reading comprehension. Several groups let a sequence attend to itself ("self-attention") as an add-on to recurrent networks.
4. The break (2017): "Attention Is All You Need." The Google paper by Vaswani and colleagues removed the recurrent part completely. The reason was practical: recurrent networks must process words one after another, which wastes the parallel hardware. Multi-head attention, positional encodings and the layer layout came together there. That's the transformer.
5. Left to right and scale (2018 onward). GPT (2018) used only the left-to-right half for text generation, and BERT (2018) used the other half. GPT-2, GPT-3 and the scaling-law papers showed that more parameters and data kept helping. RLHF-style tuning (2022) turned raw text predictors into assistants. Open weights such as LLaMA and Qwen followed.
Supporting inventions along the way that the transformer leans on: word embeddings (2013), residual connections (2015) and layer normalisation (2016).
A caution on credit. The history is argued over. Some researchers point to earlier work from the early 1990s on "fast weights" as an ancestor of the same maths, and priority claims are disputed. I'd treat the chain above as the mainstream account, not the last word.
no, go back further
Claude: [October 18, 2026, 10:20:15 AM]
Going back further, the history splits into three roots that merged. Same caveat as before: this is from memory, and I'm confident of the main years and names.
Root 1: predicting the next word. This is the task your Qwen is still doing.
Root 2: learning by adjusting weights.
Root 3: sequences and memory.
A link to your memory idea. In 2020, Ramsauer and colleagues showed that a modern version of the Hopfield network does the same computation as transformer attention. So the lineage runs from associative memory to attention. Attention is, in a precise sense, a memory lookup.
Then two things made it work: GPUs (around 2012, with AlexNet) and large text collections. The rest followed from there.
what application didit support ?
Claude: [October 18, 2026, 10:20:34 AM]
Attention itself was built for machine translation. Bahdanau's 2014 paper was about English-to-French. The 2017 transformer paper was also tested on translation (English to German and French) and won on quality and training speed. Text generation like Qwen's came afterwards, once people realised that predicting the next word at scale was enough for chat.
Each earlier root had its own first job, and these are from memory:
| work | application |
|---|---|
| Markov's letter chains (1913) | analysing the structure of a Russian poem |
| Shannon's predictions (1948–51) | communications engineering, meaning how efficiently text could be coded and sent |
| IBM n-gram models (1970s–90s) | speech recognition, choosing the likeliest word from ambiguous sounds |
| Perceptron (1958) | recognising simple patterns and images, with funding from the US Navy |
| Backpropagation with convolutional nets (1989) | reading handwritten ZIP-code digits for the US Postal Service |
| Hopfield networks (1982) | recalling stored patterns from partial or noisy input |
| LSTM (1997) | sequence tasks such as speech and handwriting, years before translation |
The pattern is that each step was a fix for a concrete failing in some practical job: sound to words, digits to numbers, sentences to translations. The general-purpose assistant came last.
YES !
Claude: [October 18, 2026, 10:20:51 AM]
Good. That pattern bears on your own work. Your two-LLM idea is also a fix for a concrete failing: a stateless model with a limited context window. The history says the ideas that lasted began as answers to a specific problem, then proved themselves on a task.
Your pending items are unchanged: the top-20 test (/root/top20.json), then qwen_chat.py v2.0.0, and optionally the Coder 1.5B or Qwen3 1.7B download.
now this Long Term Short Memory model has the whackiest name going - i might as well call myself a very intoxicated sober person
Claude: [October 18, 2026, 10:23:01 AM]
It's "Long Short-Term Memory", and the name does make sense once the hyphen is in the right place. It means "a short-term memory that lasts long". It doesn't describe two kinds of memory.
An ordinary recurrent network carried a short-term memory, its hidden state, from word to word. That state faded within a few steps because the training signal shrank as it was passed back through each step. This was the problem Hochreiter analysed. LSTM added a separate channel, the cell state, plus gates that decide what to write, what to keep and what to forget. Short-term memory could then survive for hundreds of steps.
So the name is a pun on the problem, and it was the same word order everyone then hung "memory" on. You're not far off, though. "Very intoxicated sober person" is a better description of the original paper's title than most abstracts manage.
it is a pure illogical contradiction
Claude: [October 18, 2026, 10:23:44 AM]
As wording, yes, it's a contradiction. As meaning, it isn't one, and the confusion comes from the phrase "short-term memory" having two readings.
In psychology, short-term memory is a category, the scratchpad that holds things for seconds while you work on them. It is contrasted with long-term memory, which is storage that persists. The "long" in the name describes that scratchpad: a scratchpad that holds on for longer than a plain recurrent network's did. The memory is still working memory. It's wiped when the sequence ends and keeps nothing permanent.
So nothing in the design is contradictory. The name is just awkward because a category term got an adjective that sounds like its opposite.