dopetalk

News:

New Search Options : ISE (posts), PDFs, misc (text) and ISEmedia (images + mp4s) dropdowns, enter '/help' to learn more

🧾✨ Link to our Forum Charter: Read, Respect, Reflect
Solicitation and Dealing of Drugs is Strictly Prohibited !
Please email smfadmin if you wish to advertise here
Non drug topics are also very welcome !
All Terms and Conditions are at the Bottom of the Page

dopetalk does not endorse any advertised product nor does it accept any liability for it's use or misuse

Our Discord Notification Server invitation link is https://discord.gg/jB2qmRrxyD

« previous next »

Pages: 1 Go Down

Author Topic: The Stateless Memory Hypothesis -- Could it Fly ? (Orthogany, Probe and Explore) (Read 74 times)

Chip (OP)

Server Admin
Admin
Hero Member
Administrator
Restrictive
RC Discussion
Private
Owner
Join Date: Dec 2014
Location: Australia
Posts: 7433
Reputation Power: 0
Gender:
Last Login:Today at 08:56:49 AM
Deeply Confused Learner
Profession: IT Engineer now retired

The Stateless Memory Hypothesis -- Could it Fly ? (Orthogany, Probe and Explore)

« on: October 08, 2026, 11:48:40 AM »

📎 Click on me to view all my attachmentsTheme:

ListAttBBC | v6.12 | 2026-09-10 | Andrew.human & Claude.ai

#	File	Size	Downloads	Info	DL
00	README.md	621 B	1	ℹ️	⬇️
Attachment ID 7279 Filename (display) README.md File extension md MIME type Size 621 B (621 bytes) Downloads 1 Dimensions n/a Uploaded by Guest Message ID 49750 Attachment type Normal attachment (0) Storage folder ID 2 Thumbnail attach ID none File hash 771376702ca644d16b7ad0f0bb55b92c7407afd6 Approved Yes Filename (on disk) 7279_771376702ca644d16b7ad0f0bb55b92c7407afd6 Storage directory 2 (/var/www/html/attachments2) File exists on disk? Yes (core)
No recorded downloads yet.
01	orthogonal_probe.py	9 KB	3	ℹ️	⬇️
Attachment ID 7280 Filename (display) orthogonal_probe.py File extension py MIME type Size 9 KB (9259 bytes) Downloads 3 Dimensions n/a Uploaded by Guest Message ID 49750 Attachment type Normal attachment (0) Storage folder ID 2 Thumbnail attach ID none File hash ee0be0d76fa2275ef1ee6249fb7e84ddb6ba39df Approved Yes Filename (on disk) not found on disk Storage directory unresolved File exists on disk? No
Chip — October 08, 2026, 02:59:41 PM
02	Stateless Memory Ideas Formatted.md	37.5 KB	3	ℹ️	⬇️
Attachment ID 7281 Filename (display) Stateless Memory Ideas Formatted.md File extension md MIME type Size 37.5 KB (38393 bytes) Downloads 3 Dimensions n/a Uploaded by Guest Message ID 49750 Attachment type Normal attachment (0) Storage folder ID 2 Thumbnail attach ID none File hash 0de29df6dca12182d44be440b7162ad53cb80413 Approved Yes Filename (on disk) not found on disk Storage directory unresolved File exists on disk? No
No recorded downloads yet.
03	inspect_Qwen_Qwen2.5-0.5B.json	27.2 KB	0	ℹ️	⬇️
Attachment ID 7358 Filename (display) inspect_Qwen_Qwen2.5-0.5B.json File extension json MIME type Size 27.2 KB (27886 bytes) Downloads 0 Dimensions n/a Uploaded by Guest Message ID 49750 Attachment type Normal attachment (0) Storage folder ID 2 Thumbnail attach ID none File hash 390da7fc451d6d0910d19da1c36b0e11431ed6a8 Approved Yes Filename (on disk) not found on disk Storage directory unresolved File exists on disk? No
No recorded downloads yet.
04	inspect_Qwen_Qwen2.5-0.5B.md	16.3 KB	0	ℹ️	⬇️
Attachment ID 7359 Filename (display) inspect_Qwen_Qwen2.5-0.5B.md File extension md MIME type Size 16.3 KB (16730 bytes) Downloads 0 Dimensions n/a Uploaded by Guest Message ID 49750 Attachment type Normal attachment (0) Storage folder ID 2 Thumbnail attach ID none File hash 8a71f66dd43e26b3001a10bf6aab0f0a100543d5 Approved Yes Filename (on disk) not found on disk Storage directory unresolved File exists on disk? No
No recorded downloads yet.
05	inspect_selftest-tiny-random.json	25.1 KB	0	ℹ️	⬇️
Attachment ID 7360 Filename (display) inspect_selftest-tiny-random.json File extension json MIME type Size 25.1 KB (25672 bytes) Downloads 0 Dimensions n/a Uploaded by Guest Message ID 49750 Attachment type Normal attachment (0) Storage folder ID 2 Thumbnail attach ID none File hash 83e14f29f961f7a9b4d6effce4212f9bd3fcb6da Approved Yes Filename (on disk) not found on disk Storage directory unresolved File exists on disk? No
No recorded downloads yet.
06	inspect_selftest-tiny-random.md	21.5 KB	0	ℹ️	⬇️
Attachment ID 7361 Filename (display) inspect_selftest-tiny-random.md File extension md MIME type Size 21.5 KB (21987 bytes) Downloads 0 Dimensions n/a Uploaded by Guest Message ID 49750 Attachment type Normal attachment (0) Storage folder ID 2 Thumbnail attach ID none File hash bc4834b2905c83ffdf38d3dde73d1cddbf4dba99 Approved Yes Filename (on disk) not found on disk Storage directory unresolved File exists on disk? No
No recorded downloads yet.

📝 README.mdTheme:

orthogonal_probe v1.0.0

Needs: pip install torch transformers, plus a small model (e.g. Qwen/Qwen2.5-0.5B or a 1B-3B Llama).

python3 orthogonal_probe.py --selftest
python3 orthogonal_probe.py --model Qwen/Qwen2.5-0.5B \
    --prompt "The best thing to do on a rainy weekend is" \
    --memory "The user loves jigsaw puzzles." --keywords jigsaw,puzzle

Read the table: text is the control. A mode only earns its place if it beats text on recall at similar ppl. Strength columns: logit = score added; steer = fraction of residual norm; attn = added to attention scores; keyscale = key multiplier.

README.md (0.61 kB - downloaded 1 times.)

orthogonal_probe.py (9.04 kB - downloaded 3 times.)

📝 Stateless Memory Ideas Formatted.mdTheme:

Stateless Memory Ideas

Source: Conversation transcript — 8 October 2026

This edition preserves the transcript's wording while improving its Markdown structure, spacing, hierarchy, and readability. No substantive content has been rewritten.

Page 1
Page 2
Page 3
Page 4
Page 5
Page 6
Page 7
Page 8
Page 9
Page 10
Page 11
Page 12
Page 13
Page 14
Page 15
Page 16
Page 17
Page 18
Page 19

Page 1

do you remember anything from pur chat about my ideas on stateless memory by means if Ran a command, read memory

Claude

[08/10/2026--11:19:44:905 AEDT] Yes. Your message cut off at “by means of…”, so I can’t tell which angle you meant. Here is what I have on stateless memory. Stateless memory in the Orthogonal / “cooperative Attention” design. This is the most direct match. LLM1 stays stateless. All state lives in LLM2, which holds the memory pool and the datastore. LLM1’s most salient token is paired with LLM2’s most salient memory token (the “memtok”). The pair’s attention weight is inherited from LLM1’s half. That weight then heavily influences LLM1 during inference, between the query and the reply. LLM2 acts as the “hippocampus”. It is query-driven by LLM1 and returns curated “rich context”. LLM1 never touches the datastore. Eviction needs both LLMs to agree on relevance. Relevance is judged against current context, not stored as a fixed score. Your example was that a car-accident memory is worthless if you’re currently immobile. LLM2 doesn’t get smarter by repeated querying, because inference doesn’t update weights. Instead it generates an explicit “map” (an index) in one pass over the static datastore and uses that for later lookups. Later phases add more LLMs in an “orthogonal” (spherical, wrap-around), IP-internetwork-style configuration. The open item is the validation experiment. It compares a small set of injected high-value tokens against a much larger raw context window on a coding task. You have PyTorch and llama.cpp ready for it. Media-compression memory model. 10/8/26, 2:39 PM Claude

Page 2

3 hours ago Memory is stored as BMP/WAV-style (raw) and JPG/MP3-style (lossy) grids. Progressive recompression at varying bitrates stands in for sleep consolidation. Contrast boost on the grid stands in for norepinephrine salience in a virtual visual cortex. Vocal isolation and frequency lift do the same in a virtual auditory cortex. The grids aren’t viewable or audible. A small, deliberately overfitting co-LLM reads and writes them. MEL / QuasiCode. This is the stateless-AI problem applied to code. “AI has no historical timeline”, so QCE chains and IFCSAR retrieval supply one from outside the model. Which of these did you mean, or was it something newer that “by means of” was leading into? Finish the sentence and I’ll go from there. yes but you have rounded up some of the candidate models are undercooked and halfbaked whilst others were either malformed or too hazy but it's my most recent and most novel idea to date that has captured my imagination ... Ran a command

Claude

[08/10/2026--11:29:57:082 AEDT] Fair. What I gave you were the candidates I could find filed. The idea you mean isn’t among them. The filed material stops at the Orthogonal pairs design and the media-grid model, and neither is a complete stateless-memory mechanism “by means of” anything. Whatever you were about to name is either a newer idea or one that never got saved in full, so I’d be guessing if I tried to reconstruct it. I’d rather not fill the gap with a plausible-sounding version of my own. Tell me the mechanism in your own words, even roughly, and I’ll treat it as the primary design. I’ll hold it to the same standard as the others and raise objections before anything gets built. 10/8/26, 2:39 PM Claude

Page 3

But I am glad you did because I mananged to accidentally delete the last transcript on it and some details were forgotton ... but you have helped me piece together more of it. Ran a command

Claude

[08/10/2026--11:33:34:654 AEDT] Good, that’s a useful outcome from a miss. Since the transcript is gone, this conversation is now the only record of the reassembled idea. Write out the version you’ve pieced together while it’s fresh, and I’ll keep it as a single clean statement of the design: what’s settled, what’s still hazy, and what the stateless part rests on. That gives you a document to start the next chapter from, in a fresh conversation, which suits how you like to work. But because I really wanted to explore and test my theory by 'spinning up' the proposed minimum configuration of two loosely interconnecting streaming LLMs/Transformers (physically managed by HTTP and logically managed by Attention, because apparently "Attention is all you need", right ?) where one acts as a typical inferrence engine whilst the other is non-inferring with no current use for its generative outputs ... Ran a command [08/10/2026--11:57:54:335 AEDT] Your message cut off again after “no current use for its generative outputs…”. Here’s my reading of the setup so far, with objections up front. Configuration as I understand it LLM1 is an ordinary inference engine and the only one whose output is used. LLM2 runs forward passes but its generated tokens are discarded. It is used for its internal states: attention 10/8/26, 2:39 PM Claude

Page 4

patterns and hidden activations. The two are joined over HTTP, streaming in both directions. Attention is the logical coupling: LLM1’s salient tokens go to LLM2, and LLM2’s paired memtoks come back to influence LLM1.

Objections

Attention in a deployed model isn’t an interface.

llama.cpp and hosted APIs don’t expose attention weights or per-token salience over HTTP by default. In llama.cpp you can pull hidden states or embeddings, but the attention maps need a patch. In PyTorch you can hook them easily. For a prototype, PyTorch with forward hooks is the realistic route, and HTTP is just transport. 2. “Attention weight inherited from LLM1” has no native injection point. A returned memtok is a token, and LLM1 only attends over what’s in its context. So the influence has to enter either as extra tokens in the context, which is plain RAG with extra steps, or as a bias on LLM1’s attention logits or KV cache, which is real intervention but means modifying LLM1’s forward pass. Your “heavy influence between query and reply” claim is only distinct from RAG if it takes the second route. That decision decides whether the experiment tests anything new. 3. A non-inferring LLM2 is a good fit for state, but a cost to watch. Its job reduces to encoding, scoring and retrieving, which an embedding model plus a vector index also does. The experiment has to show something LLM2’s attention gives you that a cheap retriever doesn’t, otherwise the null hypothesis wins. 4. Streaming adds latency before it adds value. Per-token round trips over HTTP between two models will dominate. Run the validation batched first, and stream later.

Minimum test I’d run

LLM1 gets a coding task. Condition A is a large raw context. Condition B is a small set of LLM2-selected memtoks injected via the logit-bias route. Condition C is the same small set picked by a plain embedding retriever. B beating C is the result that matters. B beating A alone proves nothing. Finish the sentence about LLM2’s role and I’ll fold it in. I haven’t built anything yet. 10/8/26, 2:39 PM Claude

Page 5

But what hasn't been defined up until now is the nature of the data that LLM2 'trades with' and strives to overtly/heavily influence or skew LLM1's inferrence ... Ran a command

Claude

[08/10/2026--12:12:10:704 AEDT] This is the central design question, so here are the candidates for what crosses the wire. Each one fixes how LLM2 can skew LLM1.

Text tokens. LLM2 sends words. This is the simplest, and

LLM1 treats them as ordinary context. The influence is only as strong as LLM1 chooses to weight them, so it can’t “heavily skew” anything. It is RAG. 2. Token IDs with a logit bias. LLM2 sends a sparse map of token ID to bias value, added to LLM1’s output logits at each step. This steers vocabulary choice directly and works over HTTP today (llama.cpp and many APIs accept logit bias). It is crude, because it skews what gets said and not what LLM1 attends to. 3. Embedding or hidden-state vectors. LLM2 sends dense vectors that are injected into LLM1’s input embeddings, or added to its residual stream (activation steering). This is the strongest and most genuinely “attention-level” route. It requires both models to share an embedding space, or a learned projection between them. The projection needs training, which breaks your “no training at this stage” rule. 4. KV-cache entries. LLM2 supplies precomputed key/value pairs that LLM1 attends to as if they were its own past context. The pair’s weight is inherited through the keys, which matches your “pair’s attention weight” idea most literally. It needs the same hidden dimensions and layers, so in practice both are the same model family, ideally the same model. 5. Attention-score priors. LLM2 sends a salience vector over LLM1’s existing context positions, added to LLM1’s attention logits. It reweights what LLM1 already has and adds no new content. 10/8/26, 2:39 PM Claude

Page 6

My read: your design as described, a pair of (LLM1 salient token, LLM2 memtok) with the weight inherited from LLM1, maps best to option 4, with option 2 as the cheap fallback. Option 4 also keeps the no-training constraint if both models are identical copies. Using two copies of one small model is probably the right first prototype for that reason. Objection: “overtly/heavily skew” has no safe default. With options 3 to 5, a strong bias degrades LLM1’s coherence quickly, because it is pushed off the distribution it was trained on. So the experiment needs a strength dial and a coherence measure, or a win on relevance will hide a loss on quality. One thing has to be fixed before design continues. Is the data LLM2 trades in tokens, or in vectors? My original model was overly simplistic and closest to option 2 because I have little to no appreciation, understanding or knowledge of what those tokens directly represent and how their embeddings, as vectors, are assembled and stored physically stored, how they are logically read or processed, being numerical values, how firm or flexible they (and under what influences, to what degree and when, if any) and whether or not they have different lifetimes. I assume that Attention influences Genererative (which then influences tokem selection). --- and having just learned of the others, so this leaves me little choice but for some basic trial and error testing, whereby different techniques could match differing types of memory, which although is far more complex, but is closer to that of organic models with obvious advantages ... My original model was overly simplistic and closest to option 2 because I have little to no appreciation, understanding or knowledge of what those tokens directly represent and how their embeddings, as vectors, are assembled and stored physically stored, how they are logically read or processed, being numerical values, how firm or flexible they (and under what influences, to what degree and when, if any) and whether or not they have different lifetimes. I assume that Attention influences Genererative (which then influences tokem selection). --- and having just learned of the others, so this leaves me little choice but for some basic trial and error testing, whereby different techniques could match differing types of memory, which although is far more complex, but is closer to that of organic models with obvious advantages ... Show more Ran a command, read memory

Claude

[08/10/2026--12:56:10:640 AEDT] Trial and error is the right method here, and the gaps you listed are small enough to close in one pass. Here are your questions answered in order. What tokens and embeddings are A token is an integer ID into a fixed vocabulary, for example 50,000 to 150,000 entries. 10/8/26, 2:39 PM Claude

Page 7

An embedding table is a matrix with one row per token ID. Each row is a vector of a few thousand floats. It is stored as ordinary weights in the model file and is fixed after training. At inference the token ID selects its row. That vector enters the network and passes through every layer. The vector gets rewritten at each layer, so the same word means different things in different contexts. This running vector is the “hidden state” or residual stream. In effect a token is a lookup key. The meaning is in the vectors that build up around it. Where attention sits Your assumption is half right. Attention doesn’t act on the output directly. At every layer each position computes a query, key and value vector. Attention scores come from comparing queries to keys, and they decide how much of each earlier position’s value gets mixed into the current position. The final layer’s hidden state is turned into a score for every vocabulary token (the logits), and token selection is made from those. So attention shapes the hidden state, the hidden state gives the logits, and the logits drive selection. Option 2, logit bias, edits the last step only. It changes which word wins but not what the model “understood”. Firmness and lifetimes Weights, including the embedding table, are permanent and shared. Nothing you inject touches them. Hidden states and the KV cache last for one conversation, because they exist only while the context is held. Whatever you inject lasts as long as you keep re- supplying it. Lifetime is therefore a property of LLM2’s store, which fits your design. LLM1 has no lifetimes of its own. Mapping techniques to memory types This is the trial-and-error plan. Each row is a different kind of memory, and the techniques differ in how deep they reach and how long they persist. 10/8/26, 2:39 PM Claude

Page 8

Memory type Technique Persistence Cost Fact or word recall Logit bias (option 2) one step trivial Episodic, “what was said” Text injection (option 1) as long as re-sent trivial Mood, stance, style Activation steering (option 3) as long as applied needs a steering vector from contrast prompts, no training Working memory KV-cache entries (option 4) per session same model only Attention focus Salience prior (option 5) per step needs hooks

Objections

Test all five on one model with fixed prompts, and score

two things: did the injected content show up in the output, and did output quality survive. Bias strength is the dial. 2. Use a small open model such as a 1B to 3B Llama or Qwen in PyTorch. Options 3 to 5 need hooks, and llama.cpp doesn’t give you those without patching. 3. Skip HTTP for the first round. Run both models in one process, and add HTTP once you know which techniques work. 4. Option 3 is cheaper than it sounds. Run the model on a prompt with the concept and one without. The difference of their hidden states is the steering vector, and no training is involved. If you want, I’ll write the one-process harness with the five injection modes behind a switch. That’s one Python file that follows your delivery rules. 10/8/26, 2:39 PM Claude

Page 9

yes, i am relying on you to translate what i have conceived from my layman's research of and a vivid imagination based a on conjecture and personal logic. Ran 2 commands Ran 3 commands, created a file, shared files The PyTorch CPU index is blocked, so I’ll try plain PyPI.

Claude

[08/10/2026--13:01:45:597 AEDT] I’ve built the harness. The self-test passes, but I couldn’t run it against a real model, so you’ll be the first to see real output. This sandbox can’t reach Hugging Face. The self-test used a tiny random model. It proves the mechanics work: every injection changes the model’s output, and the attn and keyscale controls reproduce the text baseline exactly. The output text is meaningless gibberish. Changes from the plan I gave you, listed before you open it Option 4 became keyscale , which multiplies the keys of the memory positions. With two identical models, a KV cache from LLM2 would be exactly the same as pasting the text in, because attention only looks backwards. So there was nothing to test in the cache itself. Scaling the keys is the real “attention weight” dial. There is no HTTP in this version, and both models run in one process, as I suggested. It recomputes the whole sequence at every step and doesn’t use a cache. That’s slow, but it keeps every injection method simple, and it’s fine for 40 tokens on a small model. The steer strength is a fraction of the model’s typical activation size, so 0.5 means half. That makes strengths comparable between models. How to read the output Each row shows recall (how many of your --keywords appeared) and ppl (how coherent the text is under the untouched model; lower is better). The text row is the 10/8/26, 2:39 PM Claude

Page 10

control. A technique is only interesting if it beats text on recall without ppl blowing up. The run lists all six modes, with the four tunable ones swept across four strengths.

Release card

Rule Status @version / @date in module done (1.0.0, 2026-10-08) Manual version history entry done Prints version when run done (first line) Zip with dotted, never-reused version done ( orthogonal_probe_v1.0.0.zip ) xmllint on XML not needed (no XML) Legacy zipper not needed (not an SMF package) Unrequested changes listed done (above) Real-model test missing (blocked here; run it on your machine) Run --selftest first on your machine, then the Qwen 0.5B line from the README. Paste the table back and we’ll pick the next experiment from the results. Bearing foremost in mind that although "a memory" may appear to be a static, simplistic 2-D representation, I posit that it is actually a tiny logical "frame in" streams ... Ran a command

Claude

[08/10/2026--13:15:14:244 AEDT] Your message cut off after “a tiny logical ‘frame in’ streams…”. I’m reading it as a memory being one frame cut from a 10/8/26, 2:39 PM Claude

Page 11

continuous stream, in the way a video frame is, and not a standalone record. This is also the logic behind your BMP/JPG model. I’d count it as a strong idea, and it changes the design in three ways.

A memory item has to carry its neighbours. A frame

means little alone. It is defined by what came before and after it. A stored memory should therefore be a short run of tokens, or a pointer to the stream it was cut from, and not a single memtok. Your “pairs only” rule keeps one salient token per side, and that conflicts with this. A pair is one frame. The stream would have to be a sequence of pairs ordered in time. 2. It gives you the lifetimes you asked about. Video codecs already separate full frames (keyframes) from frames stored as differences. LLM2 could keep rare, rich keyframes and many cheap deltas, then recompress old stretches into fewer keyframes. That is your sleep- consolidation idea, and it comes with a ready-made structure. 3. It has a concrete form in the model. A stream of frames is a sequence of positions, and positions are what attention already works over. The attn and keyscale modes in the harness act on a whole run of memory positions, so they can test this directly. A frame-aware version would also give each frame its own strength, falling off with age. Objection: if each frame is a vector, the frame boundaries become your own convention. The model has no built-in notion of one. Whatever cuts the stream into frames (fixed length, topic shift, or a salience spike) determines everything downstream. I’d test fixed-length chunks first, since they’re easy to compare, and only then try salience-triggered cuts. The next harness version (1.1.0) could take --memory as an ordered list of frames with an age-decay on strength, so the same table compares “one memory item” against “a stream of frames”. I haven’t built it. Finish the thought about what the stream is made of, and it will settle whether the frames are token runs or vectors. But my idea is based on AI being a stateless inferring and generative engine, THAT STRIVES TO EMULATE 10/8/26, 2:39 PM Claude

Page 12

REASONING and i have seen this demonstrated by many and varied debugging and troubleshooting "expeditions" with mainly mostly impressive and positive outcomes BUT also with a few "disasters" amd even some endless loops, as I have seen it "dig bigger holes" and make shocking code chnages too but SO TOO should memory be like like that and not just "more text" to be either merged, prepended or appended with the prompt, right ? Ran a command

Claude

[08/10/2026--13:28:40:399 AEDT] Partly right. The “more text” criticism holds, but I don’t think your own analogy supports “memory should be like reasoning” in the way you mean. What holds Memory as extra text in the prompt is the weakest design. The model reads it as new input and gives it no more standing than the prompt. Nothing makes it persist, fade or compete with other memories. Your point that memory should act on the inference, not sit in front of it, is why steer , attn and keyscale exist. They change how the model processes the prompt, and text changes only what it reads. Where I disagree Reasoning emulation is a poor model for memory, and your debugging evidence shows why. The loops, the “bigger holes” and the shocking code changes come from a stateless engine acting on a context it can’t verify, with no stable record of what it has already tried. That is the failure of missing memory. It isn’t what memory should look like. Memory that behaves like that engine would inherit those failures. A memory that drifted, dug holes or looped would be worse than none. Organic memory is also not reasoning-like. It is consolidated, decays and is reconstructed on recall, and it is mostly stable. The reasoning-like part is the recall process, deciding what to pull up in context. That is LLM2’s job in your design, and your “relevance depends on current context” insight covers it. What this means for the design LLM1 reasons, and it is allowed to be fallible. 10/8/26, 2:39 PM Claude

Page 13

LLM2 should be a conservative, consistent store that is hard to push off course. Its one job is supplying the influence. The danger in your design is the coupling. If LLM2’s influence is strong and LLM1 acts on it, a bad memory gets amplified into a bad inference, and LLM1 then reinforces it. That’s the loop failure again, built into the architecture.

Objections and tests

The harness currently has no test for loops or runaway

behaviour. I’d add a run where the injected memory is wrong, say a false fact, and measure whether strong injection makes LLM1 repeat it, ignore evidence or degrade. That gives you a safe measure of your “disaster” cases. 2. Add a decay dial so the influence weakens over turns unless it is reinforced. That matches your lifetimes idea and limits the damage from a bad memory. 3. Keep the text control. The question that matters is whether influence beats text on helpful memories without being worse on wrong ones. Version 1.1.0 would add the wrong-memory test and the decay dial, along with the frame-stream input. I haven’t built it, and I’ll wait for your word on scope. you aare taking me literally as it's not the memory that holds value but the outcome(s) derived from it -- so in effence LLM2 adds another dimension to memory, as it's foundations remain as static as they are now but the expanding store is token based and are logical "pairs" both of what LLM1 deems/ed high value and what LLM2 "matches" with it and it dynamic and that relationships are current;y theoretical but LLM2 has a map of the tokens in general, the pairs spefically, and those logical pairs change with and when LLM1 does but is also sujected to what LLM2 deems most relative to -- this of it as a secondary cooperative Attention, that may yield "thoughtful and likely outcomes or simply fail dismally -- am I expressinf myself well enough, being my simplistic seld ? 10/8/26, 2:39 PM Claude

Page 14

Ran a command

Claude

[08/10/2026--13:42:37:069 AEDT] Yes, and the correction is useful. I was treating the memory as the thing that matters. You’re saying the thing that matters is the outcome the pairs produce, and I read you too literally. Your idea as I now understand it The base models stay frozen. Nothing in either one is retrained. The new dimension is a growing store of pairs: (what LLM1 judged high value, what LLM2 matched to it). LLM2 holds a map of the tokens in general and of the pairs in particular. It keeps the pairs current. When LLM1’s focus shifts, the pairs shift, and LLM2 also reshapes them by what it judges most relevant. So the store is a set of live relationships between two models’ judgments. It isn’t a set of facts. The value is in the thoughtful outcome it steers LLM1 towards, and the design accepts it may fail. I’d call that a secondary cooperative Attention, as you put it. You described it clearly, and what you describe is coherent. Why it changes the test A pair is a relationship, so the harness has been testing the wrong unit. It injects one fixed memory and checks whether LLM1 shows it. Your design predicts something else: the pairs evolve as LLM1’s attention moves, so the thing to measure is whether the pairs improve LLM1’s answers over a run of turns compared with no pairs. What stays theoretical “LLM2 matches with” needs a definition. The workable one is similarity between LLM2’s hidden state for a candidate and LLM1’s state for the salient token. Without that, the pair has no computable form. “Currency” needs a rule. Pairs update when LLM1’s salient token changes, and decay when they stop being matched. “Simply fail dismally” needs a yardstick, so success is defined before you run anything. I’d use answer quality 10/8/26, 2:39 PM Claude

Page 15

on a multi-turn coding or Q&A task, with pairs on and off, against the text control. Objection: the strongest competitor is a plain retriever with a recency weight. If your pairs don’t beat that, the second model isn’t earning its place. That outcome would also be a valid result. The next build (1.1.0) would be a multi-turn loop, where LLM1’s salient token from each turn is paired with a LLM2 match, the pair store updates between turns, and the pairs steer the following turn. It would score pairs on against pairs off. I haven’t built it. The step that decides whether it measures your idea is how “matches” is computed, so tell me if hidden- state similarity matches what you mean. say, LLM1 performs its typical Attention based on it's token set but then notifies LLM3 of a fixed percentage (got to start somehere and some consistency is mandatory) pool then LLM2 pairs it with it's Attention, and then LLM1 , before doing anything else, performs Attention based on the combined value of the pair and then we watch what happens, and it may even loop/refine ans since the "weights" are static, unpredictability can be minimised -- now tell me, does Attention play any role in Traning, and either way, why or why not ? Ran a command

Claude

[08/10/2026--13:54:03:675 AEDT] Your loop, restated (I’m reading “LLM3” as a typo for LLM2, since you’ve described only two models)

LLM1 runs its normal Attention over its tokens.

It sends LLM2 a fixed percentage of its most salient

tokens, for example the top 10%. 3. LLM2 pairs each with its own best match, using its own Attention. 4. LLM1 runs Attention again, this time over the combined value of each pair, before it does anything else. 5. Repeat, if you want refinement. 10/8/26, 2:39 PM Claude

Page 16

Your claim that static weights keep this predictable is half right. The weights don’t change, but the inputs do, and a loop feeds its own output back in. That’s the same structure as the runaway cases you described. The fixed percentage and a hard cap on loop count are the right controls. I’d add a stop rule: halt when a pass changes the salient set by less than some threshold. Does Attention play a role in training? Yes, a central one. Attention is part of the model’s architecture, so it runs in every training step as well as in inference. The mechanics: Training feeds in text, the model predicts the next token at every position, and the error against the true token is measured as a loss. That error is pushed backwards through the whole network (backpropagation), and the weights are adjusted to reduce it. The weights that get adjusted include the ones that produce the queries, keys and values. So Attention is learned. Nobody programs what it attends to. By the end of training, the attention weight matrices (the Q, K, V and output projections) encode which relationships matter: grammar, reference, code structure and so on.

Why this matters for your design

What you see at inference is the product of that training. Attention scores are computed fresh from the input each time, but the rules for computing them are frozen weights. That is why the weights staying static makes the system more predictable. It also sets a limit. LLM1 was trained to attend to ordinary text. When you hand it a “combined pair value”, you’re giving it something it never saw in training. That is the reason strong injection degrades coherence in the harness, and the reason LLM2’s output has to be translated into a form LLM1 can attend to without breaking. It raises a question you will hit: if both models were trained separately, their notion of “salient” differs. Pairing works best when both share a tokenizer and, ideally, are the same model family. 10/8/26, 2:39 PM Claude

Page 17

If the combined value is just LLM2’s tokens placed in LLM1’s context, it behaves like text mode. The attn and keyscale modes are closer to your “Attention over the pair’s combined value”, because they change the weighting itself. I haven’t built 1.1.0. This loop, with its fixed percentage, the pairing step and the stop rule, is what I’d put in it. yes but LLM1 and LLM2 need to be identical (intially) and even if it is just for curiosities sake, it gives this fixed system "new context, a logical one that could be controlled/adjusted for either deeper/meaningless reasoning that may either influence the outcome with more stability or the opposite but by tweaking the data that Attention works with could be an extensoin of training, a way to train with new data on static systems as NOT NEW TRUTHS but as "influences" that can result in a revised inferrences - how else could you reversibly breath new life into the proven investments that are today's models - i know, it could be a shocker or not but the wisdom is in the pairings and how :predictable LLM1 is in its generative output and maybe LLM2 can also be influenced on the output of LLM2 in exactly the same manner --- food for thought !] yes but LLM1 and LLM2 need to be identical (intially) and even if it is just for curiosities sake, it gives this fixed system "new context, a logical one that could be controlled/adjusted for either deeper/meaningless reasoning that may either influence the outcome with more stability or the opposite but by tweaking the data that Attention works with could be an extensoin of training, a way to train with new data on static systems as NOT NEW TRUTHS but as "influences" that can result in a revised inferrences - how else could you reversibly breath new life into the proven investments that are today's models - i know, it could be a shocker or not but the wisdom is in the pairings and how :predictable LLM1 is in its generative output and maybe LLM2 can also be influenced on the output of LLM2 in exactly the same manner --- food for thought !] Show more Ran a command

Claude

[08/10/2026--14:09:54:961 AEDT] Making the two models identical is a good decision, and it fixes the translation problem I raised earlier. Same weights mean the same tokenizer, the same embedding space and the same idea of salience, so LLM2’s output means something to LLM1 without any trained adapter. Your claim, stated plainly Influencing a frozen model’s attention at inference is a reversible substitute for training. It adds influence, not new truths, and it can be switched off. This is a real, active research area. It goes by names like activation steering, prefix tuning and in-context learning, and it’s what steer , attn and keyscale already do. So the idea is sound, and it has a precedent. Three limits I’d keep in view 10/8/26, 2:39 PM Claude

Page 18

Influence is not learning. Nothing persists inside the

model. Once the pairs stop being supplied, the effect is gone. That’s the reversibility you want, but it also means the model never improves. All the accumulated value sits in the pair store. 2. Identical models don’t give two views. LLM2 is the same function as LLM1, so given the same input it produces the same salience. The pairing only adds something if LLM2 sees different input: a different slice of the context, the stored memory, or a prompt asking a different question. Otherwise the loop is LLM1 agreeing with itself. This is the point I’d test hardest. 3. A self-influencing loop can amplify its own errors. Your last idea, where LLM2 is also influenced by LLM1’s output in the same way, makes this sharper. Two identical models steering each other is a feedback loop with no external anchor. It might stabilise on useful structure, or it might lock onto whatever was salient first. This is the “dig bigger holes” behaviour you mentioned, now designed in. It should be tested in a bounded form first: a fixed number of passes, with a measure of how far the output drifts from the pair-free baseline.

What the harness should measure for your claim

Does a pass through the pair loop change the output relative to baseline, and does it do so consistently from run to run? Does the effect fade cleanly when the pairs are removed? That is your reversibility test. Does a second pass stabilise or drift? Those three tests are the core of 1.1.0, along with the multi- turn loop and the stop rule. The same-weights setup makes it easy to build, since one loaded model can play both roles. I have not written it. Say go and I’ll build it with the usual release card. ok, i need to settle on something and even anthing is better that nothing do i will stop righ there and explore, the least (any) outcome will be a greater width or breadth of understanding so it won't be wasted effort, whatever the 10/8/26, 2:39 PM Claude

Page 19

outcome is. OK, i will save this and then get AI to clean it up for me in various ways until i. approve Ran a command

Claude

[08/10/2026--14:18:07:300 AEDT] Stopping here is sound. You have a testable design, a working v1.0.0 harness, and a clear list of what 1.1.0 would add, so any result will tell you something. For the clean-up stage, here is what to carry into the next conversation so nothing gets lost again: Core claim: two identical frozen models. LLM1 infers. LLM2 is non-inferring and pairs LLM1’s top fixed percentage of salient tokens with its own matches. LLM1 then re-attends over the combined pair value. The effect is reversible influence on a frozen model, not new training. Unit of memory: a pair, treated as a frame in a stream, and not a single stored fact. Settled: the weights stay static, the store lives outside the models, and both models are the same. Still undefined: how LLM2 “matches” (hidden-state similarity is my suggestion), how pairs age and decay, and what makes LLM2 see something different from LLM1. Tests that decide it: pairs against the plain text control, against a recency-weighted retriever, a wrong-memory run, removal to confirm reversibility, and a bounded second pass to check for drift. Harness: orthogonal_probe_v1.0.0.zip is delivered and self-tested. It has not run on a real model yet. 1.1.0 is not built. When the clean-up is done and you want the multi-turn loop built, bring that summary into a new conversation. 10/8/26, 2:39 PM Claude

Stateless Memory Ideas Formatted.md (37.49 kB - downloaded 3 times.)

inspect_Qwen_Qwen2.5-0.5B.json (27.23 kB - downloaded 0 times.)

📝 inspect_Qwen_Qwen2.5-0.5B.mdTheme:

Model inspection report: Qwen/Qwen2.5-0.5B

Generated by model_inspector v1.1.0 (2026-10-11) on 2026-10-10 22:40:08. Read-only: no weights were changed.

5. What the model does with your prompt

push code: orthogonal_probe v1.1.0 (imported from the same folder)

Two runs of the same prompt. A (reference) = the prompt alone. B (test) = the prompt alone, with the memory 'direction' added to the running thought at layer 12, strength 0.5 x the typical size. Everything below is measured from inside the model on those two runs - no guessing.

prompt: The best thing to do on a rainy weekend is (10 tokens)
memory: The user loves jigsaw puzzles. (7 tokens)

Run A (reference) - the prompt alone

What the model would say if it had to stop at each layer (logit lens)

layer	best guess so far	its probability	probability of the final answer	uncertainty (nats)
0	`';";\n`	0.120	0.000	7.19
1	`Chain`	0.036	0.000	7.96
2	`';";\n`	0.057	0.000	7.10
3	`�`	0.060	0.000	7.96
4	`�`	0.107	0.000	7.79
5	`';";\n`	0.155	0.000	5.88
6	`';";\n`	0.128	0.000	6.41
7	`%";\n`	0.248	0.000	4.30
8	`')));\n\n`	0.077	0.000	7.09
9	`?>>`	0.085	0.000	7.39
10	`","#`	0.063	0.000	7.62
11	`ped`	0.078	0.000	7.47
12	`","#`	0.079	0.000	8.08
13	`","#`	0.235	0.000	6.97
14	`","#`	0.388	0.000	5.36
15	`","#`	0.399	0.000	5.33
16	`","#`	0.209	0.000	6.98
17	`","#`	0.067	0.003	8.44
18	`either`	0.024	0.006	8.93
19	`to`	0.105	0.105	7.06
20	`to`	0.405	0.405	4.85
21	`to`	0.111	0.111	7.25
22	`to`	0.261	0.261	5.19
23	`to`	0.303	0.303	4.49

Early layers usually guess generic words; the final answer 'crystallises' at some layer. Uncertainty falling = the model is making up its mind. Which layer it settles at is where the decision is made.

How much each layer adds to the running thought (last position)

layer	thought size entering	attention adds	MLP adds	real change	attn vs MLP direction (cos)	MLP share of the two lengths
0	0.4	0.4	4.6	4.7	0.03	91.44%
1	4.7	2.9	4.3	5.4	0.10	59.83%
2	8.2	1.6	4.3	4.4	-0.11	73.40%
3	9.1	3.6	5.4	5.7	-0.24	59.67%
4	9.8	3.3	5.3	5.5	-0.24	61.75%
5	10.8	4.1	6.0	6.3	-0.25	59.58%
6	12.6	2.5	5.3	5.5	-0.16	67.70%
7	12.9	4.7	6.1	6.6	-0.28	56.63%
8	13.8	4.7	5.4	6.5	-0.18	53.50%
9	14.6	3.2	5.7	5.8	-0.27	64.23%
10	15.9	3.9	4.3	4.8	-0.29	52.27%
11	15.4	2.2	4.3	4.5	-0.15	65.50%
12	15.3	3.0	5.8	5.8	-0.26	65.75%
13	17.8	2.3	5.4	5.3	-0.27	70.08%
14	18.7	3.8	5.0	6.1	-0.08	56.64%
15	20.1	3.2	7.6	7.9	-0.11	70.51%
16	23.0	2.5	13.1	12.8	-0.19	83.80%
17	29.1	6.1	9.9	12.4	0.16	62.14%
18	36.1	3.8	9.7	10.1	-0.08	71.79%
19	40.8	4.6	12.9	13.8	0.03	73.89%
20	45.0	7.0	20.0	21.3	0.02	74.22%
21	58.6	10.4	25.5	30.0	0.26	70.99%
22	80.5	3.1	27.9	27.7	-0.12	89.90%
23	87.0	13.3	78.6	75.8	-0.29	85.48%

'Thought size' is the length of the number-vector the model carries for the last word. Each layer adds an attention contribution and an MLP contribution to it. 'Real change' is the length of what the layer actually changed (the two contributions added as arrows, not as lengths). The cosine says whether attention and the MLP pull the same way (+1), at right angles (0) or against each other (-1). Where a steering push is applied, that layer's 'real change' includes the push.

Where the attention spotlights look (averaged over all heads in the layer, last position)

layer	spread (0 = pin-point, 1 = even)	first token (the sink)	memory text (sink excluded)	memory if attention were even	previous token	itself
0	0.62	7.26%	0.00%	0.00%	25.98%	24.52%
1	0.76	22.55%	0.00%	0.00%	10.44%	12.21%
2	0.75	30.88%	0.00%	0.00%	6.56%	18.30%
3	0.48	49.84%	0.00%	0.00%	2.86%	9.81%
4	0.57	43.81%	0.00%	0.00%	9.12%	16.86%
5	0.51	52.91%	0.00%	0.00%	3.83%	13.09%
6	0.54	53.86%	0.00%	0.00%	8.19%	9.75%
7	0.50	50.35%	0.00%	0.00%	1.90%	9.91%
8	0.60	34.75%	0.00%	0.00%	17.29%	13.08%
9	0.30	73.13%	0.00%	0.00%	6.31%	5.74%
10	0.55	51.46%	0.00%	0.00%	13.31%	14.25%
11	0.25	81.84%	0.00%	0.00%	5.44%	2.20%
12	0.62	50.15%	0.00%	0.00%	11.19%	14.95%
13	0.32	76.14%	0.00%	0.00%	4.61%	9.83%
14	0.56	44.60%	0.00%	0.00%	12.07%	20.45%
15	0.54	57.90%	0.00%	0.00%	10.29%	10.49%
16	0.18	88.08%	0.00%	0.00%	0.42%	1.62%
17	0.42	62.47%	0.00%	0.00%	2.43%	9.31%
18	0.51	60.98%	0.00%	0.00%	4.54%	11.65%
19	0.48	57.76%	0.00%	0.00%	5.92%	22.26%
20	0.35	76.36%	0.00%	0.00%	4.07%	6.68%
21	0.27	77.12%	0.00%	0.00%	1.31%	9.45%
22	0.67	43.15%	0.00%	0.00%	0.44%	19.24%
23	0.63	28.50%	0.00%	0.00%	0.71%	39.18%

The first token is an 'attention sink': models park spare attention on it, and it often takes most of the attention in every layer. It is NOT counted as memory here. 'Memory text' is the share landing on the memory sentence's other tokens, and 'memory if attention were even' is what it would get if every token were treated alike. Memory above that line is real interest in the memory.

Run B (test) - the prompt alone, with the memory 'direction' added to the running thought at layer 12, strength 0.5 x the typical size

What the model would say if it had to stop at each layer (logit lens)

layer	best guess so far	its probability	probability of the final answer	uncertainty (nats)
0	`';";\n`	0.120	0.000	7.19
1	`Chain`	0.036	0.000	7.96
2	`';";\n`	0.057	0.000	7.10
3	`�`	0.060	0.000	7.96
4	`�`	0.107	0.000	7.79
5	`';";\n`	0.155	0.000	5.88
6	`';";\n`	0.128	0.000	6.41
7	`%";\n`	0.248	0.000	4.30
8	`')));\n\n`	0.077	0.000	7.09
9	`?>>`	0.085	0.000	7.39
10	`","#`	0.063	0.000	7.62
11	`ped`	0.078	0.000	7.47
12	`ped`	0.033	0.000	8.07
13	`ped`	0.059	0.000	8.10
14	`ped`	0.048	0.000	7.94
15	`ped`	0.037	0.000	7.86
16	`堞`	0.015	0.000	8.49
17	`either`	0.015	0.001	8.76
18	`either`	0.089	0.004	8.39
19	`to`	0.200	0.200	5.71
20	`to`	0.877	0.877	1.19
21	`to`	0.370	0.370	4.60
22	`to`	0.431	0.431	4.20
23	`to`	0.522	0.522	3.15

Early layers usually guess generic words; the final answer 'crystallises' at some layer. Uncertainty falling = the model is making up its mind. Which layer it settles at is where the decision is made.

How much each layer adds to the running thought (last position)

layer	thought size entering	attention adds	MLP adds	real change	attn vs MLP direction (cos)	MLP share of the two lengths
0	0.4	0.4	4.6	4.7	0.03	91.44%
1	4.7	2.9	4.3	5.4	0.10	59.83%
2	8.2	1.6	4.3	4.4	-0.11	73.40%
3	9.1	3.6	5.4	5.7	-0.24	59.67%
4	9.8	3.3	5.3	5.5	-0.24	61.75%
5	10.8	4.1	6.0	6.3	-0.25	59.58%
6	12.6	2.5	5.3	5.5	-0.16	67.70%
7	12.9	4.7	6.1	6.6	-0.28	56.63%
8	13.8	4.7	5.4	6.5	-0.18	53.50%
9	14.6	3.2	5.7	5.8	-0.27	64.23%
10	15.9	3.9	4.3	4.8	-0.29	52.27%
11	15.4	2.2	4.3	4.5	-0.15	65.50%
12	15.3	3.0	5.8	10.4	-0.26	65.75%
13	19.0	2.8	6.6	6.5	-0.26	70.25%
14	19.9	3.9	6.0	6.9	-0.08	60.73%
15	20.0	3.4	7.7	7.9	-0.15	68.99%
16	21.1	2.4	11.5	11.2	-0.21	82.96%
17	25.8	5.8	8.9	11.2	0.12	60.58%
18	31.4	3.2	8.8	9.0	-0.12	73.07%
19	35.5	4.8	12.6	13.3	-0.03	72.49%
20	39.0	8.4	20.5	22.1	-0.00	71.04%
21	52.0	11.0	23.6	27.3	0.13	68.28%
22	70.6	3.4	29.5	29.4	-0.09	89.65%
23	79.5	13.5	70.8	67.2	-0.36	83.98%

'Thought size' is the length of the number-vector the model carries for the last word. Each layer adds an attention contribution and an MLP contribution to it. 'Real change' is the length of what the layer actually changed (the two contributions added as arrows, not as lengths). The cosine says whether attention and the MLP pull the same way (+1), at right angles (0) or against each other (-1). Where a steering push is applied, that layer's 'real change' includes the push.

Where the attention spotlights look (averaged over all heads in the layer, last position)

layer	spread (0 = pin-point, 1 = even)	first token (the sink)	memory text (sink excluded)	memory if attention were even	previous token	itself
0	0.62	7.26%	0.00%	0.00%	25.98%	24.52%
1	0.76	22.55%	0.00%	0.00%	10.44%	12.21%
2	0.75	30.88%	0.00%	0.00%	6.56%	18.30%
3	0.48	49.84%	0.00%	0.00%	2.86%	9.81%
4	0.57	43.81%	0.00%	0.00%	9.12%	16.86%
5	0.51	52.91%	0.00%	0.00%	3.83%	13.09%
6	0.54	53.86%	0.00%	0.00%	8.19%	9.75%
7	0.50	50.35%	0.00%	0.00%	1.90%	9.91%
8	0.60	34.75%	0.00%	0.00%	17.29%	13.08%
9	0.30	73.13%	0.00%	0.00%	6.31%	5.74%
10	0.55	51.46%	0.00%	0.00%	13.31%	14.25%
11	0.25	81.84%	0.00%	0.00%	5.44%	2.20%
12	0.62	50.15%	0.00%	0.00%	11.19%	14.95%
13	0.45	63.70%	0.00%	0.00%	7.68%	13.04%
14	0.57	47.27%	0.00%	0.00%	11.78%	18.50%
15	0.59	56.08%	0.00%	0.00%	8.92%	9.93%
16	0.16	90.55%	0.00%	0.00%	0.50%	2.49%
17	0.41	67.04%	0.00%	0.00%	2.36%	10.82%
18	0.51	63.69%	0.00%	0.00%	5.41%	12.66%
19	0.51	51.39%	0.00%	0.00%	6.48%	25.98%
20	0.40	70.87%	0.00%	0.00%	4.50%	6.95%
21	0.34	72.25%	0.00%	0.00%	1.87%	8.87%
22	0.70	42.56%	0.00%	0.00%	1.13%	13.85%
23	0.69	25.48%	0.00%	0.00%	1.38%	29.15%

The first token is an 'attention sink': models park spare attention on it, and it often takes most of the attention in every layer. It is NOT counted as memory here. 'Memory text' is the share landing on the memory sentence's other tokens, and 'memory if attention were even' is what it would get if every token were treated alike. Memory above that line is real interest in the memory.

Reference versus test: what did the push change?

reference (A): the prompt alone
test (B): the prompt alone, with the memory 'direction' added to the running thought at layer 12, strength 0.5 x the typical size
top choice in A: to (30.35%)
top choice in B: to (52.21%)
top choice changed: no
probability moved to different next words: 37.16% (0% = nothing changed, 100% = completely different)
KL divergence B from A: 0.4573 nats (0 = identical odds; bigger = bigger change)

rank	A: next word	A: prob	B: next word	B: prob
1	`to`	30.35%	`to`	52.21%
2	`go`	5.16%	`go`	4.99%
3	`sit`	3.09%	`play`	2.90%
4	`take`	3.08%	`take`	2.24%
5	`just`	2.26%	`ride`	2.05%
6	`a`	1.94%	`bike`	1.85%
7	`get`	1.27%	`playing`	1.36%
8	`hang`	1.26%	`hang`	1.18%
9	`spend`	1.19%	`paddle`	0.91%
10	`catch`	1.14%	`walk`	0.81%

Words whose probability rose most because of the push

word	A	B	change
`to`	30.35%	52.21%	+21.86%
`play`	0.76%	2.90%	+2.14%
`bike`	0.17%	1.85%	+1.68%
`ride`	0.49%	2.05%	+1.56%
`playing`	0.05%	1.36%	+1.31%
`paddle`	0.09%	0.91%	+0.82%
`going`	0.16%	0.59%	+0.43%
`biking`	0.01%	0.38%	+0.37%

Your keywords as the very next word

keyword	first piece	A prob	A rank	B prob	B rank	B / A
jigsaw	`j`	0.00%	1804	0.00%	726	2.3x
puzzle	`puzzle`	0.00%	1967	0.00%	707	2.8x

Probability and rank (1 = the model's top choice) of each keyword's first word-piece as the immediate next word. Sharper than the probe's 'recall', which only checks whether the word shows up somewhere in the generated text.

How the change travels through the layers

layer	relative change in the running thought	mean attention shift (0-1)	most shifted head
0	0.0000	0.0000	h0 (0.000)
1	0.0000	0.0000	h0 (0.000)
2	0.0000	0.0000	h0 (0.000)
3	0.0000	0.0000	h0 (0.000)
4	0.0000	0.0000	h0 (0.000)
5	0.0000	0.0000	h0 (0.000)
6	0.0000	0.0000	h0 (0.000)
7	0.0000	0.0000	h0 (0.000)
8	0.0000	0.0000	h0 (0.000)
9	0.0000	0.0000	h0 (0.000)
10	0.0000	0.0000	h0 (0.000)
11	0.0000	0.0000	h0 (0.000)
12	0.4947	0.0000	h0 (0.000)
13	0.4962	0.1317	h4 (0.344)
14	0.4895	0.1365	h2 (0.372)
15	0.4631	0.1101	h9 (0.284)
16	0.4193	0.0521	h10 (0.311)
17	0.3742	0.1342	h9 (0.367)
18	0.3651	0.1153	h11 (0.327)
19	0.3749	0.0999	h5 (0.241)
20	0.3485	0.0904	h2 (0.253)
21	0.3051	0.0682	h1 (0.134)
22	0.3096	0.1056	h0 (0.177)
23	0.3818	0.1445	h2 (0.323)

Left column: how big the difference between the two runs is at the output of each layer, as a fraction of that layer's output size. A push that fades is small by the last layers; one that grows is being amplified. Attention shift: how far a head's pattern of looking moved (0 = identical, 1 = looks at entirely different words).

The 10 spotlights whose pattern moved most

layer	head	shift	sink share A -> B	memory share A -> B
14	2	0.372	20.46% -> 57.68%	-
17	9	0.367	44.41% -> 81.09%	-
17	10	0.359	18.86% -> 45.86%	-
13	4	0.344	63.51% -> 29.06%	-
18	11	0.327	22.13% -> 45.02%	-
23	2	0.323	31.46% -> 11.31%	-
13	0	0.314	53.56% -> 23.33%	-
16	10	0.311	19.59% -> 38.54%	-
15	9	0.284	37.21% -> 56.41%	-
13	6	0.271	85.69% -> 58.59%	-

Size of the running thought at every position (run B)

position	token	layer 6	layer 12	layer 18
0	`The`	1636.1	1641.1	1643.9
1	`best`	15.6	17.9	36.7
2	`thing`	15.1	17.9	37.4
3	`to`	13.3	16.7	34.0
4	`do`	14.1	17.9	35.4
5	`on`	13.8	18.1	32.9
6	`a`	13.8	18.7	34.1
7	`rainy`	15.4	20.8	39.0
8	`weekend`	15.1	20.8	36.2
9	`is`	12.9	19.0	35.5
	first / typical	115.7x	90.6x	46.3x

Length of the number-vector the model carries for each token, at three depths. The first position is usually enormous next to the rest (see the last row). That 'massive activation' is what turns the first token into the attention sink, and it is why the probe leaves position 0 alone when it pushes.

section 'trace' took 37.7s

8. Self-checks (the report checking its own arithmetic)

result	check	detail
PASS	attention weights of one head sum to 1	sum = 1.000000
PASS	layer hooks agree with the model's own hidden states	max relative error 0.00e+00
PASS	layer output = input + attention + MLP (exact decomposition)	max relative gap 0.00e+00 (steer layer excluded)
PASS	logit-lens pipeline reproduces the model's own scores at the last layer	max diff 0.00e+00
PASS	measured steering push equals strength x vector length	measured 8.8109 vs 8.8109

ALL CHECKS PASSED

total time 163.5s, peak memory 2842.9 MB

inspect_Qwen_Qwen2.5-0.5B.md (16.34 kB - downloaded 0 times.)

inspect_selftest-tiny-random.json (25.07 kB - downloaded 0 times.)

📝 inspect_selftest-tiny-random.mdTheme:

Model inspection report: selftest-tiny-random

Generated by model_inspector v1.1.0 (2026-10-11) on 2026-10-10 22:38:12. Read-only: no weights were changed.

1. Identity and provenance

What this model is according to its own config file, and what its publisher says about how it was made. Nothing in this section is measured from the weights except where stated.

setting	value	plain words
model_type	qwen2	family of architecture
architectures	None	class name the publisher used
hidden_size	64	width of the 'thought vector' kept for every word-piece
intermediate_size	128	width of the MLP inside each layer (where most 'stored knowledge' lives)
num_hidden_layers	4	how many identical processing blocks are stacked
num_attention_heads	4	parallel 'spotlights' per layer that pick which earlier words matter
num_key_value_heads	2	shared 'index cards' the spotlights read from (fewer = cheaper memory)
head_dim (computed)	16	size of each spotlight's workspace
vocab_size	256	number of distinct word-pieces (tokens) it knows
max_position_embeddings	256	longest text it was set up to handle (word-pieces)
rope_theta	10000.0	base of the rotary position code (how it knows word order); bigger = longer reach
rms_norm_eps	1e-06	tiny number that keeps the normalisation maths stable
hidden_act	silu	activation function inside the MLP
tie_word_embeddings	True	input word table is reused as the output word table
torch_dtype	None	number format the weights were published in
use_sliding_window	False	attention limited to a recent window
sliding_window	4096	size of that window
attention_dropout	0.0	random dropping during training (0 = none)
initializer_range	0.02	spread of the random numbers it started training from
bos_token_id	None	id of the beginning-of-text marker
eos_token_id	None	id of the end-of-text marker

Model class and generation defaults

Python class: Qwen2ForCausalLM
generation default _from_model_config = True
generation default bad_words_ids = None
generation default begin_suppress_tokens = None
generation default bos_token_id = None
generation default constraints = None
generation default decoder_start_token_id = None
generation default diversity_penalty = 0.0
generation default do_sample = False
generation default early_stopping = False
generation default encoder_no_repeat_ngram_size = 0
generation default encoder_repetition_penalty = 1.0
generation default eos_token_id = None
generation default epsilon_cutoff = 0.0
generation default eta_cutoff = 0.0
generation default exponential_decay_length_penalty = None
generation default force_words_ids = None
generation default forced_bos_token_id = None
generation default forced_decoder_ids = None
generation default forced_eos_token_id = None
generation default generation_kwargs = {}
generation default guidance_scale = None
generation default length_penalty = 1.0
generation default low_memory = None
generation default max_length = 20
generation default max_new_tokens = None
generation default max_time = None
generation default min_length = 0
generation default min_new_tokens = None
generation default no_repeat_ngram_size = 0
generation default num_assistant_tokens = 5
generation default num_assistant_tokens_schedule = heuristic
generation default num_beam_groups = 1
generation default num_beams = 1
generation default num_return_sequences = 1
generation default output_attentions = False
generation default output_hidden_states = False
generation default output_scores = False
generation default pad_token_id = None
generation default penalty_alpha = None
generation default prompt_lookup_num_tokens = None
generation default remove_invalid_values = False
generation default renormalize_logits = False
generation default repetition_penalty = 1.0
generation default return_dict_in_generate = False
generation default sequence_bias = None
generation default suppress_tokens = None
generation default temperature = 1.0
generation default top_k = 50
generation default top_p = 1.0
generation default typical_p = 1.0
generation default use_cache = True

Generation defaults are only used when the model writes text on its own. Both the probe and this report use greedy decoding (always the single most likely next piece).

Tokenizer

self-test byte tokenizer (256 symbols)

Publisher's model card (VERBATIM - not derived from the weights)

(skipped in --selftest)

This machine and software

item	value
python	3.8.18
torch	1.10.0a0+git71f889c
transformers	4.37.2
tokenizers	0.15.2
safetensors	0.5.3
huggingface_hub	0.23.5
cpu	Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz
logical cpus	4
cpu features seen	sse4_2, avx, f16c
torch threads in use	2
MemTotal	6026.7 MB
SwapTotal	2048.0 MB
SwapFree	2048.0 MB

avx2 and fma are the instruction sets that make neural-network maths fast on a CPU. If they are missing from 'cpu features seen', expect slow runs - that is the hardware, not a bug.

section 'identity' took 0.0s

2. Parameter census

A 'parameter' is one adjustable number inside the model. Everything the model 'knows' is stored in these numbers. Below, every one is counted and filed under the part of the model it belongs to.

unique parameters: 164,928 (0.000 billion)
memory for the weights as loaded: 0.6 MB (dtype torch.float32)
input word table and output word table are shared (tied): yes
if the shared table were counted twice (once as input, once as output): 181,312

part	parameters	share	approx memory
embedding	16,384	9.93%	0.1 MB
attention	49,664	30.11%	0.2 MB
mlp	98,304	59.60%	0.4 MB
norms	576	0.35%	0.0 MB

embedding = the dictionary that turns word-pieces into number-vectors. attention = the 'who should I look at' machinery. mlp = the per-word 'think about it' machinery, usually the biggest block. norms = tiny stabilisers.

part	tensor	shape	copies	parameters (all copies)	share
attention	self_attn.k_proj.bias	32	4	128	0.08%
attention	self_attn.k_proj.weight	32x64	4	8,192	4.97%
attention	self_attn.o_proj.weight	64x64	4	16,384	9.93%
attention	self_attn.q_proj.bias	64	4	256	0.16%
attention	self_attn.q_proj.weight	64x64	4	16,384	9.93%
attention	self_attn.v_proj.bias	32	4	128	0.08%
attention	self_attn.v_proj.weight	32x64	4	8,192	4.97%
embedding	embed_tokens.weight	256x64	1	16,384	9.93%
mlp	mlp.down_proj.weight	64x128	4	32,768	19.87%
mlp	mlp.gate_proj.weight	128x64	4	32,768	19.87%
mlp	mlp.up_proj.weight	128x64	4	32,768	19.87%
norms	input_layernorm.weight	64	4	256	0.16%
norms	norm.weight	64	1	64	0.04%
norms	post_attention_layernorm.weight	64	4	256	0.16%

Per layer

layer	attention	mlp	norms	layer total
0	12,416	24,576	128	37,120
1	12,416	24,576	128	37,120
2	12,416	24,576	128	37,120
3	12,416	24,576	128	37,120

All layers identical in size: yes

section 'params' took 0.0s

3. Attention anatomy

Attention is how each word-piece decides which earlier word-pieces to read. Think of each layer as having several spotlights (heads); each spotlight reads from an 'index card' set (key/value head). Fewer index-card sets shared between spotlights = much cheaper memory while generating (this design is called grouped-query attention).

quantity	value	plain words
layers	4	stacked blocks, each with its own attention
query heads per layer	4	spotlights
key/value heads per layer	2	index-card sets
query heads sharing one key/value head	2	spotlights per index-card set
head size	16	numbers in each spotlight's workspace
total spotlights in the model	16	layers x heads
rotary base (rope_theta)	10000.0	position code; bigger reaches farther

Shapes inside one layer (layer 0)

projection	weight shape (out x in)	has bias	parameters
q_proj	64x64	yes	4,096
k_proj	32x64	yes	2,048
v_proj	32x64	yes	2,048
o_proj	64x64	no	4,096

q = what each spotlight is looking FOR, k = what each earlier word-piece offers to be found BY, v = what it hands over when found, o = mixes all spotlights' findings back together.

Memory the model needs per word-piece of context (the 'KV cache')

context length (tokens)	KV cache at 32-bit	KV cache at 16-bit
1,024	1.0 MB	0.5 MB
8,192	8.0 MB	4.0 MB
256	0.2 MB	0.1 MB

Per token: 256 numbers (1024 bytes at 32-bit). Without the shared index cards it would be 2x more.

orthogonal_probe v1.1.0 uses this cache for the modes none, text, logit and steer, so it does not re-read the whole text for every new word. attn and keyscale change what earlier words look like, so they cannot reuse it.

Attention versus MLP

attention parameters: 49,664 (30.11% of the model)
MLP parameters: 98,304 (59.60% of the model)

Most of the model's capacity is in the MLP and the dictionary; attention is the small, fast-moving part. That matters for your idea: attention is the cheapest place to influence what the model does.

section 'attention' took 0.0s

4. Weights

The actual numbers inside the model, summarised. 'std' is the typical size of a weight; 'near zero' is the share of weights smaller than 0.001; 'abs max' is the largest weight. Healthy models have small typical weights and a few large outliers.

tensor type (N = layer)	copies	parameters	mean	avg std	std range	abs max	near zero
model.embed_tokens.weight	1	16,384	-0.00029	0.0200	0.0200 - 0.0200	0.085	4.17%
model.layers.N.input_layernorm.weight	4	256	1.00000	0.0000	0.0000 - 0.0000	1.000	0.00%
model.layers.N.mlp.down_proj.weight	4	32,768	-0.00018	0.0200	0.0198 - 0.0203	0.081	4.09%
model.layers.N.mlp.gate_proj.weight	4	32,768	-0.00012	0.0200	0.0199 - 0.0200	0.092	4.04%
model.layers.N.mlp.up_proj.weight	4	32,768	-0.00018	0.0200	0.0199 - 0.0201	0.077	3.95%
model.layers.N.post_attention_layernorm.weight	4	256	1.00000	0.0000	0.0000 - 0.0000	1.000	0.00%
model.layers.N.self_attn.k_proj.bias	4	128	0.00000	0.0000	0.0000 - 0.0000	0.000	100.00%
model.layers.N.self_attn.k_proj.weight	4	8,192	-0.00013	0.0201	0.0199 - 0.0203	0.076	4.30%
model.layers.N.self_attn.o_proj.weight	4	16,384	0.00010	0.0202	0.0200 - 0.0205	0.093	3.60%
model.layers.N.self_attn.q_proj.bias	4	256	0.00000	0.0000	0.0000 - 0.0000	0.000	100.00%
model.layers.N.self_attn.q_proj.weight	4	16,384	0.00013	0.0199	0.0197 - 0.0200	0.083	4.10%
model.layers.N.self_attn.v_proj.bias	4	128	0.00000	0.0000	0.0000 - 0.0000	0.000	100.00%
model.layers.N.self_attn.v_proj.weight	4	8,192	0.00003	0.0199	0.0198 - 0.0201	0.082	4.33%
model.norm.weight	1	64	1.00000	0.0000	0.0000 - 0.0000	1.000	0.00%

Non-finite values (NaN / infinity) found: 0

Weight size per layer (Frobenius norm = overall magnitude of each matrix)

layer	q_proj	k_proj	v_proj	o_proj	gate_proj	up_proj	down_proj
0	1.3	0.9	0.9	1.3	1.8	1.8	1.8
1	1.3	0.9	0.9	1.3	1.8	1.8	1.8
2	1.3	0.9	0.9	1.3	1.8	1.8	1.8
3	1.3	0.9	0.9	1.3	1.8	1.8	1.8

Shows where the weight 'energy' sits across depth. Big changes between neighbouring layers mark layers that behave differently from the rest.

Fingerprint of all weights

sha256 over every weight value (as 32-bit numbers, in name order): bf4c81af2ad02cfbe68c6a73885107524fed3148996a4689172ef448eb0664b7

Same fingerprint on Clone and Live = bit-for-bit the same model. Any difference means a different file, version, or a damaged download. The .json report also holds a sha256 for each of the 50 tensors: run --compare a.json b.json to see exactly which ones differ.

section 'weights' took 0.1s

5. What the model does with your prompt

Two runs of the same prompt. A (reference) = the prompt alone. B (test) = the memory pasted as plain text in front of the prompt. Everything below is measured from inside the model on those two runs - no guessing.

prompt: Hello there (11 tokens)
memory: puzzle (6 tokens)

Run A (reference) - the prompt alone

What the model would say if it had to stop at each layer (logit lens)

layer	best guess so far	its probability	probability of the final answer	uncertainty (nats)
0	`e`	0.010	0.010	5.53
1	`e`	0.009	0.009	5.53
2	`e`	0.008	0.008	5.53
3	`e`	0.007	0.007	5.53

Early layers usually guess generic words; the final answer 'crystallises' at some layer. Uncertainty falling = the model is making up its mind. Which layer it settles at is where the decision is made.

How much each layer adds to the running thought (last position)

layer	thought size entering	attention adds	real change	attn vs MLP direction (cos)	MLP share of the two lengths
0	0.2	0.1	0.1	0.23	19.65%
1	0.2	0.1	0.1	-0.10	14.09%
2	0.2	0.1	0.1	-0.07	18.61%
3	0.2	0.2	0.2	-0.23	10.38%

'Thought size' is the length of the number-vector the model carries for the last word. Each layer adds an attention contribution and an MLP contribution to it. 'Real change' is the length of what the layer actually changed (the two contributions added as arrows, not as lengths). The cosine says whether attention and the MLP pull the same way (+1), at right angles (0) or against each other (-1). Where a steering push is applied, that layer's 'real change' includes the push.

Where the attention spotlights look (averaged over all heads in the layer, last position)

layer	spread (0 = pin-point, 1 = even)	first token (the sink)	memory text (sink excluded)	memory if attention were even	previous token	itself
0	1.00	9.23%	0.00%	0.00%	9.13%	9.05%
1	1.00	9.03%	0.00%	0.00%	8.99%	9.34%
2	1.00	9.30%	0.00%	0.00%	9.09%	8.88%
3	1.00	9.09%	0.00%	0.00%	8.99%	9.04%

The first token is an 'attention sink': models park spare attention on it, and it often takes most of the attention in every layer. It is NOT counted as memory here. 'Memory text' is the share landing on the memory sentence's other tokens, and 'memory if attention were even' is what it would get if every token were treated alike. Memory above that line is real interest in the memory.

Run B (test) - the memory pasted as plain text in front of the prompt

What the model would say if it had to stop at each layer (logit lens)

layer	best guess so far	its probability	probability of the final answer	uncertainty (nats)
0	`e`	0.011	0.011	5.53
1	`e`	0.009	0.009	5.53
2	`e`	0.008	0.008	5.53
3	`e`	0.007	0.007	5.53

Early layers usually guess generic words; the final answer 'crystallises' at some layer. Uncertainty falling = the model is making up its mind. Which layer it settles at is where the decision is made.

How much each layer adds to the running thought (last position)

layer	thought size entering	attention adds	real change	attn vs MLP direction (cos)	MLP share of the two lengths
0	0.2	0.1	0.1	0.20	22.19%
1	0.2	0.1	0.1	-0.17	19.04%
2	0.2	0.1	0.1	0.02	13.89%
3	0.2	0.1	0.1	-0.05	12.02%

'Thought size' is the length of the number-vector the model carries for the last word. Each layer adds an attention contribution and an MLP contribution to it. 'Real change' is the length of what the layer actually changed (the two contributions added as arrows, not as lengths). The cosine says whether attention and the MLP pull the same way (+1), at right angles (0) or against each other (-1). Where a steering push is applied, that layer's 'real change' includes the push.

Where the attention spotlights look (averaged over all heads in the layer, last position)

layer	spread (0 = pin-point, 1 = even)	first token (the sink)	memory text (sink excluded)	memory if attention were even	previous token	itself
0	1.00	5.54%	27.67%	27.78%	5.60%	5.55%
1	1.00	5.40%	27.70%	27.78%	5.54%	5.74%
2	1.00	5.52%	27.74%	27.78%	5.55%	5.43%
3	1.00	5.46%	27.48%	27.78%	5.53%	5.59%

The first token is an 'attention sink': models park spare attention on it, and it often takes most of the attention in every layer. It is NOT counted as memory here. 'Memory text' is the share landing on the memory sentence's other tokens, and 'memory if attention were even' is what it would get if every token were treated alike. Memory above that line is real interest in the memory.

The 10 spotlights that read the memory text most, compared with an even spread

rank	layer	head	share of its attention on memory	x even spread	spread
1	2	3	27.97%	1.0x	1.00
2	3	0	27.94%	1.0x	1.00
3	0	3	27.87%	1.0x	1.00
4	2	1	27.86%	1.0x	1.00
5	1	3	27.81%	1.0x	1.00
6	1	0	27.78%	1.0x	1.00
7	0	1	27.73%	1.0x	1.00
8	1	2	27.64%	1.0x	1.00
9	0	2	27.62%	1.0x	1.00
10	2	2	27.58%	1.0x	1.00

Reference versus test: what did the push change?

reference (A): the prompt alone
test (B): the memory pasted as plain text in front of the prompt
top choice in A: e (0.70%)
top choice in B: e (0.70%)
top choice changed: no
probability moved to different next words: 5.62% (0% = nothing changed, 100% = completely different)
KL divergence B from A: 0.0099 nats (0 = identical odds; bigger = bigger change)

rank	A: next word	A: prob	B: next word	B: prob
1	`e`	0.70%	`e`	0.70%
2		0.61%	`H`	0.58%
3	`�`	0.60%	`m`	0.58%
4	`E`	0.58%	`�`	0.55%
5	`/`	0.54%	`S`	0.55%
6	`!`	0.53%	`�`	0.55%
7	`�`	0.53%	`-`	0.53%
8	`�`	0.52%	`�`	0.52%
9	`�`	0.52%	`�`	0.52%
10	`�`	0.52%	`�`	0.51%

Words whose probability rose most because of the push

word	A	B	change
	0.35%	0.51%	+0.17%
`2`	0.34%	0.50%	+0.15%
`�`	0.26%	0.41%	+0.14%
`�`	0.33%	0.45%	+0.12%
`K`	0.36%	0.48%	+0.12%
`m`	0.47%	0.58%	+0.11%
`(`	0.36%	0.47%	+0.11%
`P`	0.26%	0.37%	+0.11%

Your keywords as the very next word

keyword	first piece	A prob	A rank	B prob	B rank	B / A
puzzle		0.43%	66	0.42%	92	1.0x

Probability and rank (1 = the model's top choice) of each keyword's first word-piece as the immediate next word. Sharper than the probe's 'recall', which only checks whether the word shows up somewhere in the generated text.

How the change travels through the layers

(the two runs have different lengths - 11 vs 18 tokens - so layer-by-layer differences are not defined)

Size of the running thought at every position (run B)

position	token	layer 1	layer 2
0	`p`	0.4	0.4
1	`u`	0.3	0.4
2	`z`	0.2	0.3
3	`z`	0.2	0.3
4	`l`	0.2	0.3
5	`e`	0.2	0.3
6	`\n`	0.2	0.3
7	`H`	0.2	0.3
8	`e`	0.2	0.3
9	`l`	0.2	0.3
10	`l`	0.2	0.3
11	`o`	0.2	0.3
12		0.2	0.3
13	`t`	0.2	0.2
14	`h`	0.2	0.2
15	`e`	0.2	0.2
16	`r`	0.2	0.2
17	`e`	0.2	0.2
	first / typical	1.8x	1.6x

Length of the number-vector the model carries for each token, at three depths. The first position is usually enormous next to the rest (see the last row). That 'massive activation' is what turns the first token into the attention sink, and it is why the probe leaves position 0 alone when it pushes.

section 'trace' took 0.2s

8. Self-checks (the report checking its own arithmetic)

result	check	detail
PASS	census parts add up to the model's own parameter count	164,928 vs 164,928
PASS	census equals independent count from config numbers	config formula 164,928 vs counted 164,928
PASS	all weights are finite numbers	0 bad values
PASS	attention weights of one head sum to 1	sum = 1.000000
PASS	layer hooks agree with the model's own hidden states	max relative error 0.00e+00
PASS	layer output = input + attention + MLP (exact decomposition)	max relative gap 0.00e+00
PASS	logit-lens pipeline reproduces the model's own scores at the last layer	max diff 0.00e+00

ALL CHECKS PASSED

total time 3.3s, peak memory 134.7 MB

inspect_selftest-tiny-random.md (21.47 kB - downloaded 0 times.)

« Last Edit: Today at 06:46:02 AM by Chip »

No reactions

Our Discord Server invitation link is https://discord.gg/jB2qmRrxyD

Pages: 1 Go Up

« previous next »

Tags:

It appears that you have not registered with dopetalk. To register, please click here...

Need help or a chat ?

If you need any help or a chat then IM/PM or email me, Chip

dopetalk does not endorse any advertised product nor does it accept any liability for it's use or misuse

TERMS AND CONDITIONS

In no event will d&u or any person involved in creating, producing, or distributing site information be liable for any direct, indirect, incidental, punitive, special or consequential damages arising out of the use of or inability to use d&u. You agree to indemnify and hold harmless d&u, its domain founders, sponsors, maintainers, server administrators, volunteers and contributors from and against all liability, claims, damages, costs and expenses, including legal fees, that arise directly or indirectly from the use of any part of the d&u site.

TO USE THIS WEBSITE YOU MUST AGREE TO THE TERMS AND CONDITIONS ABOVE

Founded December 2014

SMF 2.0.19 | SMF © 2021, Simple Machines
Simple Audio Video Embedder
SMFAds for Free Forums | Sitemap | Terms and Policies
XHTML
RSS
WAP2

Server load over the past 5, 10 and 15 minutes respectively: 0.35, 0.36, 0.32

Page created in 0.275 seconds with 56 queries.

dopetalk

News:

dopetalk does not endorse any advertised product nor does it accept any liability for it's use or misuse

Author Topic: The Stateless Memory Hypothesis -- Could it Fly ? (Orthogany, Probe and Explore) (Read 74 times)

Chip (OP)

The Stateless Memory Hypothesis -- Could it Fly ? (Orthogany, Probe and Explore)

orthogonal_probe v1.0.0

Stateless Memory Ideas

Contents

Page 1

Page 2

Page 3

Page 4

Objections

Minimum test I’d run

Page 5

Page 6

Page 7

Page 8

Objections

Page 9

Page 10

Release card

Page 11

Page 12

Page 13

Objections and tests

Page 14

Page 15

Page 16

Why this matters for your design

Page 17

Page 18

What the harness should measure for your claim

Page 19

Model inspection report: Qwen/Qwen2.5-0.5B

5. What the model does with your prompt

Run A (reference) - the prompt alone

Run B (test) - the prompt alone, with the memory 'direction' added to the running thought at layer 12, strength 0.5 x the typical size

Reference versus test: what did the push change?

How the change travels through the layers

Size of the running thought at every position (run B)

8. Self-checks (the report checking its own arithmetic)

Model inspection report: selftest-tiny-random

1. Identity and provenance

Model class and generation defaults

Tokenizer

Publisher's model card (VERBATIM - not derived from the weights)

This machine and software

2. Parameter census

Per layer

3. Attention anatomy

Shapes inside one layer (layer 0)

Memory the model needs per word-piece of context (the 'KV cache')

Attention versus MLP

4. Weights

Weight size per layer (Frobenius norm = overall magnitude of each matrix)

Fingerprint of all weights

5. What the model does with your prompt

Run A (reference) - the prompt alone

Run B (test) - the memory pasted as plain text in front of the prompt

Reference versus test: what did the push change?

How the change travels through the layers

Size of the running thought at every position (run B)

8. Self-checks (the report checking its own arithmetic)

Need help or a chat ?

If you need any help or a chat then IM/PM or email me, Chip

dopetalk does not endorse any advertised product nor does it accept any liability for it's use or misuse