New Search Options: ISE (posts), +pdf, +misc (text), +mega (all 3), +media (audio & visual) non-Default dropdown options (enter '/help' within ISE) 🧾āØ Link to our Forum Charter: Read, Respect, Reflect Solicitation and Dealing of Drugs is Strictly Prohibited ! Please email smfadmin if you wish to advertise here Non drug topics are also very welcome ! All Terms and Conditions are at the Bottom of the Page
« Last post by smfadmin on October 28, 2026, 04:05:38 PM »
š Inline Markdownv4.6.3 · 2026-11-01
now, do you know how the llm//attn came to be ? it wasn't someones eureka moment, it evolved ... i know but do you ?
1. The problem (2014): the bottleneck. The leading translation systems read a sentence into one fixed-size list of numbers and then wrote the translation from that. Long sentences didn't fit, and quality dropped sharply with length.
2. The first attention (2014). Bahdanau, Cho and Bengio let the writing side look back over all the reading side's word-by-word states and take a weighted mix, with the weights learned. That's the attention you've been looking at, in its first form, bolted onto a recurrent network. Related ideas appeared around the same time, in Graves's Neural Turing Machine (which read memory by similarity) and in memory networks. The memory-lookup framing matters for your LLM2 idea, because attention began as a way to look things up.
3. Refinements (2015ā2016). Luong and colleagues simplified the scoring. Others used attention for image captioning and reading comprehension. Several groups let a sequence attend to itself ("self-attention") as an add-on to recurrent networks.
4. The break (2017): "Attention Is All You Need." The Google paper by Vaswani and colleagues removed the recurrent part completely. The reason was practical: recurrent networks must process words one after another, which wastes the parallel hardware. Multi-head attention, positional encodings and the layer layout came together there. That's the transformer.
5. Left to right and scale (2018 onward). GPT (2018) used only the left-to-right half for text generation, and BERT (2018) used the other half. GPT-2, GPT-3 and the scaling-law papers showed that more parameters and data kept helping. RLHF-style tuning (2022) turned raw text predictors into assistants. Open weights such as LLaMA and Qwen followed.
Supporting inventions along the way that the transformer leans on: word embeddings (2013), residual connections (2015) and layer normalisation (2016).
A caution on credit. The history is argued over. Some researchers point to earlier work from the early 1990s on "fast weights" as an ancestor of the same maths, and priority claims are disputed. I'd treat the chain above as the mainstream account, not the last word.
no, go back further
Claude: [October 18, 2026, 10:20:15 AM]
Going back further, the history splits into three roots that merged. Same caveat as before: this is from memory, and I'm confident of the main years and names.
Root 1: predicting the next word. This is the task your Qwen is still doing.
1913: Markov counted letter-to-letter transitions in Pushkin's Eugene Onegin. It's the first statistical model of text I know of.
1948 and 1951: Shannon's information theory modelled English as a chain of probabilities. His "guessing game" experiments had people predict the next letter of a text.
1970sā90s: IBM's speech-recognition group, with Jelinek as the best-known name, built n-gram models. These use the last few words to estimate the next one. That was the standard for decades.
2003: Bengio and colleagues built a neural-network version that gave each word a learned list of numbers (word embeddings).
Root 2: learning by adjusting weights.
1943: McCulloch and Pitts proposed the artificial neuron.
1949: Hebb proposed that connections strengthen with use.
1958: Rosenblatt's perceptron learned from examples.
1969: Minsky and Papert showed single-layer networks' limits, and funding collapsed for years.
1970sā86: Backpropagation, the method that trains every layer, was developed by Linnainmaa (1970) and
Werbos (1974), and popularised by Rumelhart, Hinton and Williams in 1986.
Root 3: sequences and memory.
1982: Hopfield networks stored patterns and recalled them from partial cues, an associative memory.
1990: Elman's recurrent networks carried a hidden state across a sentence.
1991ā97: Hochreiter analysed why recurrent nets forget, and LSTM (1997) was the fix. Schmidhuber's fast-weight work in the early 1990s also belongs here.
2014: Seq2seq translation with the fixed-vector bottleneck, which led to attention.
A link to your memory idea. In 2020, Ramsauer and colleagues showed that a modern version of the Hopfield network does the same computation as transformer attention. So the lineage runs from associative memory to attention. Attention is, in a precise sense, a memory lookup.
Then two things made it work: GPUs (around 2012, with AlexNet) and large text collections. The rest followed from there.
what application didit support ?
Claude: [October 18, 2026, 10:20:34 AM]
Attention itself was built for machine translation. Bahdanau's 2014 paper was about English-to-French. The 2017 transformer paper was also tested on translation (English to German and French) and won on quality and training speed. Text generation like Qwen's came afterwards, once people realised that predicting the next word at scale was enough for chat.
Each earlier root had its own first job, and these are from memory:
work
application
Markov's letter chains (1913)
analysing the structure of a Russian poem
| Shannon's predictions (1948ā51) | communications engineering, meaning how effi ciently text could be coded and sent |
| IBM n-gram models (1970sā90s) | speech recognition, choosing the likeliest word from ambiguous sounds |
| Perceptron (1958) | recognising simple patterns and images, with funding from the US Navy |
| Backpropagation with convolutional nets (1989) | reading handwritten ZIP-code digits for the US Postal Service | | Hopfield networks (1982) | recalling stored patterns from partial or noisy input |
| LSTM (1997) | sequence tasks such as speech and handwriting, years before translation |
The pattern is that each step was a fix for a concrete failing in some practical job: sound to words, digits to numbers, sentences to translations. The general-purpose assistant came last.
« Last post by AidanField on October 28, 2026, 01:51:56 PM »
If you have ever felt your heart race while watching a tiny square navigate through an endless series of obstacles, you have probably encountered the addictive world of geometry-based rhythm games. These deceptively simple yet incredibly challenging games have captivated millions of players worldwide, combining music, timing, and pure determination into one hypnotic experience. Whether you are a complete beginner or someone looking to improve your skills, this guide will help you understand what makes these games so engaging and how to get better at them. Understanding the Core Experience At its heart, a geometry jump game like Geometry Dash is about synchronizing your actions with a musical beat while navigating through increasingly complex obstacles. The beauty of these games lies in their straightforward premise: you control a geometric shape, usually a cube or ball, that moves automatically through a level. Your job is to jump at precisely the right moments to avoid spikes, navigate gaps, and survive until the end of the song. What makes this experience so captivating is the harmony between sound and movement. Every jump you make should connect with the music, creating a satisfying rhythm that keeps you engaged from start to finish. The visual design reinforces this connection, with obstacles appearing and disappearing in time with the beat. This synchronization is what separates a geometry jump game from a typical platformer, creating a unique gameplay loop that feels almost dance-like in its execution. Breaking Down the Gameplay Mechanics When you start playing geometry jump games, you will encounter several core mechanics that form the foundation of the experience. The basic jump is your primary tool, and mastering its timing is essential. Unlike traditional platformers where you have time to think, these games demand split-second reactions. You will need to develop an instinctive feel for when to press the jump button, often before you even consciously register the obstacle approaching. As you progress through levels, new mechanics gradually introduce themselves. You might encounter portals that transform your shape into different forms, each with unique jumping mechanics. A cube might handle differently from a ball or a ship-like object that requires tap-and-hold controls to navigate through space. These transformations prevent the game from becoming repetitive and force you to adapt your playstyle continuously. The level design itself tells a story through progression. Early levels teach you the basics with forgiving layouts and clear timing cues. Mid-tier levels introduce faster speeds, tighter jumps, and sections where you must sustain your concentration for extended periods. Later levels demand near-perfect execution, with no room for mistakes and increasingly creative obstacle patterns. This gradual difficulty curve makes the game accessible to newcomers while providing endless challenge for dedicated players. Essential Tips for Improvement Your journey toward mastery starts with understanding that these games reward patience and practice over raw talent. Here are proven strategies to help you progress. First, focus on consistent practice rather than marathon sessions. Spending thirty minutes daily will develop better muscle memory than sporadic intensive play. Your fingers and brain need time to absorb the patterns and timings. When you feel frustrated, take a break. Walking away for an hour often helps you return with fresh perspective and better performance. Second, pay close attention to the audio cues. While visual obstacles are important, the music itself communicates when jumps should happen. Many players make the mistake of focusing exclusively on what they see, missing the rhythmic patterns embedded in the soundtrack. Try to feel the beat, anticipate changes, and let your body respond instinctively rather than consciously calculating every jump. Third, learn from each failure. Every time you crash into an obstacle, you have gained information about timing or mechanics. Instead of feeling discouraged, study exactly what went wrong. Did you jump too early? Did you misjudge the spacing? This analytical approach transforms frustration into learning. Fourth, customize your settings to match your preferences. Whether you use keyboard controls, mouse, or touch input, find what feels most natural. Some players prefer sensitivity adjustments that make controls more responsive, while others enjoy having precise control. Experiment until you find your sweet spot. Finally, watch experienced players tackle levels you find challenging. Observing how they time their jumps and navigate tricky sections can reveal techniques you had not considered. Many online communities share gameplay videos that break down difficult sections into manageable parts. The Mental Game Beyond mechanics, success in Geometry Dash requires mental resilience. These games will humiliate you. You will fail within seconds of a level's end. You will make the same mistake repeatedly. This is normal, and every player experiences it. The difference between those who improve and those who quit comes down to persistence. Cultivate patience with yourself. Acknowledge that these games are genuinely difficult. Celebrate small wins. When you finally clear a section that previously destroyed you, take a moment to recognize that achievement. Build confidence gradually, and do not pressure yourself to progress faster than feels natural. Conclusion Playing a geometry jump game offers more than entertainment. It provides a space where dedication directly translates to improvement, where seemingly impossible challenges become conquered goals, and where music and gameplay merge into something genuinely special. Whether you are seeking stress relief, a creative challenge, or simply fun, these games deliver. Start with manageable levels, focus on enjoying the experience rather than perfecting every jump immediately, and remember that every expert player began exactly where you are now. The addictive appeal of these games stems from their perfect balance between accessibility and challenge. Pick up a geometry jump game today, let the music guide your actions, and discover why millions of players worldwide cannot put them down. Your journey toward mastery awaits, one perfectly timed jump at a time.
RAG for Terry: what it is, how it copes with changing documents, and how we plan to use it for the mem-tok store
Part 1: What RAG means and why it exists
RAG stands for Retrieval-Augmented Generation. Before an AI answers a question, the system first looks up the relevant passages in your own documents, then the AI writes its answer from what was found. It's an open-book exam instead of answering from memory.
The purpose is to let an AI answer from your material, such as a company's policies or manuals, which it never saw in training. It also stops the AI from inventing plausible-sounding answers, and it costs far less than retraining the AI whenever your documents change.
To make the lookup work, each document is cut into small pieces called chunks. Each chunk is turned into a list of numbers capturing its meaning, called an embedding (re-doing this is "reembedding"). That lets the system find chunks about the same topic as a question, even when the wording differs.
The steps are:
Documents are cut into chunks (a paragraph or two each).
Each chunk is converted into an embedding.
When someone asks a question, the system finds the handful of chunks closest in meaning and hands them to the AI along with the question.
The AI writes its answer from those chunks, so it's grounded in your actual material.
Part 2: The problem of continuously changing documents
Documents get edited all the time. If the system still holds the old version of a chunk, the bot confidently quotes last week's policy.
The lazy fix is to reembed everything after every edit. With 500 documents (about 10,000 chunks), that means redoing all 10,000 chunks to fix one sentence. It works, but it's slow and wasteful.
The better answer has three layers.
Layer 1: Detect what changed
A webhook is an automatic instant ping the app sends the moment a file is edited, so the system doesn't have to keep checking.
A hash is a short fingerprint calculated from a chunk's text. Change one character and the fingerprint changes completely.
Compare each chunk's new fingerprint with the old one. If they match, skip the chunk. If they differ, that chunk changed.
Layer 2: Update only that chunk
Reembed just the changed chunk and store it under the same ID (its unique label).
The original text says "absurd", which is almost certainly a mistranscription of upsert, a database term meaning "overwrite if it exists, insert if it doesn't".
If a document is deleted, delete all its chunks using the document's ID.
The result is one chunk reembedded instead of 10,000.
Layer 3: Safety nets
Tag every chunk with a version number, and search only the latest versions, so a stale chunk can't slip into an answer.
Webhooks can occasionally be missed, so also run a scheduled full resync, a periodic complete check that catches anything the pings missed.
The payoff: if the policy was changed to 14 days, the bot now says 14 days, not the old figure.
The takeaway: don't rebuild everything. Detect what changed, update only that, and double-check on a schedule.
Part 3: Addendum ā how the same ideas apply to our mem-tok store
Our store doesn't hold documents. It holds small entries made of single words, word pairs (binomials) and mini-phrases, which we call mem-toks (or Memory Tokens). The database manager's job is to keep that store current as new conversations arrive, so the AI stops repeating mistakes it has been corrected on. Below is how the three layers map onto that job. This is the proposed plan, and the details are still open.
The main difference
In the text above, the unit being managed is a chunk of a document. Here, every entry is already tiny, so there's no chunking. And nothing is "edited" the way a file is. Instead, every new question and answer passing through the system changes the entries: counts go up, new words appear, and corrections replace older wording. The change trigger is a new query and reply arriving, not a webhook from an app.
Layer 1: Detect what changed
A small program sits between the question and the AI, like a middleman, and watches each query and reply go past. The word and phrase extractor lifts out the notable words and mini-phrases.
Each entry gets a fingerprint (hash) from its normalised text. Normalised means lower-case with spacing tidied, so "Same Thing" and "same thing" are one entry.
This lets the fingerprint double as the entry's ID. The same phrase always lands on the same ID, so "is this new, or one we already hold?" answers itself.
Layer 2: Update only what changed
An existing entry is overwritten, for example its count goes up. A new one is inserted. This is the upsert from Part 2.
Each entry records which conversation or transcript it came from. If a source is later corrected or withdrawn, everything it contributed can be found and removed by that source ID, without touching anything else.
Reembedding only applies to entries that are new or changed. Because the store is tiny, the cost is trivial.
Layer 3: Safety nets
Versions: when a correction supersedes an older entry, the new one is tagged as the latest and searches see only the latest. The older entry is kept, dated, rather than erased. If the two ever conflict, they're shown in dated sequence so the operator decides.
Rebuild check: the archive of conversation transcripts is the source of truth, so the store can always be rebuilt from it. A scheduled full resync compares the live store against a fresh rebuild and flags any difference. That catches anything the live watching missed.
Where it sits
It would be one separate stage among the experiment's stages (context definition, lookup, second AI, main AI), kept apart so each can be traced and debugged on its own. The middleman program picks it up at start-up from a stages folder, as the other stages do.
The store itself would live in the existing memory-store home on the Clone machine.
Two honest caveats
The store is small, so a full rebuild is cheap. The change-detection layers matter most as the store grows, and until then they act more as insurance than necessity. That doesn't make them wasted, since building the habit now means growth won't force a redesign.
"Search only the latest version" works against keeping history. The plan above resolves it by keeping old versions but hiding them from normal searches. If history isn't wanted, the design gets simpler.
Install on the live system stays the operator's call, and nothing here is built yet.
š Inline Markdownv4.6.3 · 2026-11-01
So how do you handle continuously changing documents in RAG ?"
Here's the full answer. Get it wrong, and your bot quotes last week's policy.
The naive fix: reembed everything after every edit.
Say 500 docs, about 10,000 chunks --- all reembedded to fix one sentence. The answer they want has three layers.
Layer one: detect what changed. The app sends a webhook, an instant ping, when a file changes. Then compare each chunk's hash, a fingerprint of its text. Same hash, skip it; new hash that chunk changed.
Layer two: update only that chunk. Reembed it and absurd it under the same ID. Absurd means override if it exists, insert if not. Doc deleted, delete all its chunks by its doc ID. One chunk reembedded, not 10,000.
Layer three: safety nets. Tag every chunk with a version, search only the latest. Webhooks can be missed, so run a scheduled full resync. Now your bot says 14 days.
Don't rebuild everything, touch only what changed.
« Last post by Chip on October 28, 2026, 11:45:52 AM »
š Inline Markdownv4.6.3 · 2026-11-01
Have you ever wondered how tools like Chat GPT or Gemini can start answering your question so quickly ?
There are clever tricks that speed up generation and one of them is speculative decoding.
Normally, a language model generates text one token at a time, with each step depending on the previous one.
Speculative decoding adds a smaller, faster draft model.
It proposes several tokens ahead.
Then the larger model checks those proposed tokens together in parallel.
When it guesses match, multiple tokens can be accepted in one verification step.
If a guess diverges, the verification process corrects it.
That can reduce sequential decoding steps and make generation faster depending on the models and hardware and when implemented correctly, it preserves the target model's output distribution.
Small model drafts, large model verifies. That's speculative decoding.
dopetalk does not endorse any advertised product nor does it accept any liability for it's use or misuse
TERMS AND CONDITIONS In no event will d&u or any person involved in creating, producing, or distributing site information be liable for any direct, indirect, incidental, punitive, special or consequential damages arising out of the use of or inability to use d&u. You agree to indemnify and hold harmless d&u, its domain founders, sponsors, maintainers, server administrators, volunteers and contributors from and against all liability, claims, damages, costs and expenses, including legal fees, that arise directly or indirectly from the use of any part of the d&u site.
TO USE THIS WEBSITE YOU MUST AGREE TO THE TERMS AND CONDITIONS ABOVE
instagram-1793168847247.heic.jpg (261.22 kB . 1440x1919 - viewed 24 times)
Need help or a chat ?
Home
Help
Search
Login
Register
Unanswered Topics
