Checkpoint — Cross-Model Token/Attention Exchange, Stateless Memory Fragment Design

Session date: 24/09/2026

1. Problem & Resolution

Problem: Andrew's "Orthogonal"/"Super-Attention" design (LLM1 selects high-value tokens, hands them to LLM2) assumed raw attention-internal tokens/vectors could be exported from one transformer and used immediately by another, with no training step.

Resolution reached: Raw hidden-state vectors (Q/K/V, residual-stream activations) are NOT portable between independently-trained models — different models land on unrelated internal coordinate systems (permutation invariance + independent training), so a vector from Model A is arithmetically meaningless to Model B's attention matrices. What IS portable, with zero training: (a) token/span text, (b) scalar attention-weight scores. Design converged on:

• Drop vector exchange entirely.
• Unit of exchange = "Context Token": raw captured text + structured tags (geo, time, source, etc.), concatenated as token sequences (not embeddings).
• Salience is operator-defined, not model-derived — attention/LLM1 salience is at most an advisory first-pass filter, never the source of truth.
• Storage: stateless, fixed, untransformed text fragments — no persistent model state, no vector store feeding the LLM directly.
• Store upgraded from flat JSON/concatenation to a relational schema (fragments / fragment_tags / fragment_relations) so relationships between fragments are explicit rather than re-inferred at read time. Concatenation becomes a query (SELECT ... ORDER BY captured_at), not a fixed write-time format.
• Ordering: strict chronological, content never rewritten/summarized (preserves causal reasoning, keeps fragments as ground truth regardless of which co-LLM reads them).
• Influence on the co-LLM at inference: system/developer-role channel + imperative phrasing > end-of-prompt user-turn placement > mid-context tagged text. (Forced-token/logit biasing available as a blunt override, not part of the core design.)

2. Risk Analysis

• Tag reliability is inference-based, not architectural. [GEO:AU-NSW]-style tags are only respected because instruction-tuned models pattern-match on consistent formatting — not guaranteed, varies per co-LLM, can misfire under load. No training-free way to make this 100% reliable.
• Per-co-LLM tuning required. System-role weighting and tag-sensitivity differ by model/checkpoint — whatever wrapper scheme is chosen has to be empirically validated against the actual llama.cpp checkpoint(s) in use, not assumed.
• Chronological order vs. salience-boosted positioning are in tension. Can't have both strict time-order AND "most important fragment gets the high-attention end-of-prompt slot" when the most important fact isn't also the most recent. Resolved by keeping order chronological and marking importance via tag/role instead of position — but this is a weaker lever than positioning, and unproven at scale.
• Attention weight ≠ operator value. Confirmed and settled, not a residual risk — but worth flagging as the assumption that would break the design if reintroduced.
• Retrieval scaling not yet designed. Relational schema settled; the query pattern that selects which fragments make it into a given concatenation (and the relation_type vocabulary) is explicitly deferred, not yet built.

3. Design/Code Change

No code changed this session — architecture/theory discussion only. Deliverables produced: verbatim chat transcript (.md + .pdf, print-export style). No diff to log.

Next open item (deferred, not started): relation_type vocabulary for fragment_relations, and the read-time retrieval query pattern for assembling a concatenation set per inference call.