Attention/LLM Original Application and Timelines

[smfadmin (OP)]

📝 Inline Markdownv4.6.2 · 2026-10-28Theme: Deep IndigoNo Theme (Plain)Pink & PurpleSlate TechEmerald FreshWarm AmberCrimson RedSolarized LightCoffee BeanAntique SepiaSunset GlowLavender MistMint CreamPeach PuffSky BlueRose QuartzLemon ChiffonSpring BudAqua MarineCoral ReefLilac DreamPearl White

now, do you know how the llm//attn came to be ? it wasn't someones eureka moment, it evolved ... i know but do you ?

1. The problem (2014): the bottleneck. The leading translation systems read a sentence into one fixed-size list of numbers and then wrote the translation from that. Long sentences didn't fit, and quality dropped sharply with length.

2. The first attention (2014). Bahdanau, Cho and Bengio let the writing side look back over all the reading side's word-by-word states and take a weighted mix, with the weights learned. That's the attention you've been looking at, in its first form, bolted onto a recurrent network. Related ideas appeared around the same time, in Graves's Neural Turing Machine (which read memory by similarity) and in memory networks. The memory-lookup framing matters for your LLM2 idea, because attention began as a way to look things up.

3. Refinements (2015–2016). Luong and colleagues simplified the scoring. Others used attention for image captioning and reading comprehension. Several groups let a sequence attend to itself ("self-attention") as an add-on to recurrent networks.

4. The break (2017): "Attention Is All You Need." The Google paper by Vaswani and colleagues removed the recurrent part completely. The reason was practical: recurrent networks must process words one after another, which wastes the parallel hardware. Multi-head attention, positional encodings and the layer layout came together there. That's the transformer.

5. Left to right and scale (2018 onward). GPT (2018) used only the left-to-right half for text generation, and BERT (2018) used the other half. GPT-2, GPT-3 and the scaling-law papers showed that more parameters and data kept helping. RLHF-style tuning (2022) turned raw text predictors into assistants. Open weights such as LLaMA and Qwen followed.

Supporting inventions along the way that the transformer leans on: word embeddings (2013), residual connections (2015) and layer normalisation (2016).

A caution on credit. The history is argued over. Some researchers point to earlier work from the early 1990s on "fast weights" as an ancestor of the same maths, and priority claims are disputed. I'd treat the chain above as the mainstream account, not the last word.

no, go back further

Claude: [October 18, 2026, 10:20:15 AM]

Going back further, the history splits into three roots that merged. Same caveat as before: this is from memory, and I'm confident of the main years and names.

Root 1: predicting the next word. This is the task your Qwen is still doing.

• 1913: Markov counted letter-to-letter transitions in Pushkin's Eugene Onegin. It's the first statistical model of text I know of.

• 1948 and 1951: Shannon's information theory modelled English as a chain of probabilities. His "guessing game" experiments had people predict the next letter of a text.

• 1970s–90s: IBM's speech-recognition group, with Jelinek as the best-known name, built n-gram models. These use the last few words to estimate the next one. That was the standard for decades.

• 2003: Bengio and colleagues built a neural-network version that gave each word a learned list of numbers (word embeddings).

Root 2: learning by adjusting weights.

• 1943: McCulloch and Pitts proposed the artificial neuron.

• 1949: Hebb proposed that connections strengthen with use.

• 1958: Rosenblatt's perceptron learned from examples.

• 1969: Minsky and Papert showed single-layer networks' limits, and funding collapsed for years.

• 1970s–86: Backpropagation, the method that trains every layer, was developed by Linnainmaa (1970) and

Werbos (1974), and popularised by Rumelhart, Hinton and Williams in 1986.

Root 3: sequences and memory.

• 1982: Hopfield networks stored patterns and recalled them from partial cues, an associative memory.

• 1990: Elman's recurrent networks carried a hidden state across a sentence.

• 1991–97: Hochreiter analysed why recurrent nets forget, and LSTM (1997) was the fix. Schmidhuber's fast-weight work in the early 1990s also belongs here.

• 2014: Seq2seq translation with the fixed-vector bottleneck, which led to attention.

A link to your memory idea. In 2020, Ramsauer and colleagues showed that a modern version of the Hopfield network does the same computation as transformer attention. So the lineage runs from associative memory to attention. Attention is, in a precise sense, a memory lookup.

Then two things made it work: GPUs (around 2012, with AlexNet) and large text collections. The rest followed from there.

what application didit support ?

Claude: [October 18, 2026, 10:20:34 AM]

Attention itself was built for machine translation. Bahdanau's 2014 paper was about English-to-French. The 2017 transformer paper was also tested on translation (English to German and French) and won on quality and training speed. Text generation like Qwen's came afterwards, once people realised that predicting the next word at scale was enough for chat.

Each earlier root had its own first job, and these are from memory:

work	application
Markov's letter chains (1913)	analysing the structure of a Russian poem

| Shannon's predictions (1948–51) | communications engineering, meaning how effi ciently text could be coded and sent |

| IBM n-gram models (1970s–90s) | speech recognition, choosing the likeliest word from ambiguous sounds |

| Perceptron (1958) | recognising simple patterns and images, with funding from the US Navy |

| Backpropagation with convolutional nets (1989) | reading handwritten ZIP-code digits for the US Postal Service | | Hopfield networks (1982) | recalling stored patterns from partial or noisy input |

| LSTM (1997) | sequence tasks such as speech and handwriting, years before translation |

The pattern is that each step was a fix for a concrete failing in some practical job: sound to words, digits to numbers, sentences to translations. The general-purpose assistant came last.