This was our testing command, data and the optimised result, ready for input to AI:
echo "Um, err, can you shed some light on what spaCy can really do when one- kind- of gets the hang of it ?" | python3 ai-preproc.py -
Shed light on what spaCy can really do when one kind of gets the hang of it
Fails when parsed a lot of data:
python3 ai-preproc.py transcript.txt
transcript.txt
AG is basically mandatory, not optional  anywhere requiring private, proprietary, or fast- changing data the model was never trained on: internal company wikis, live inventory/pricing, legal case law, customer support histories. Standalone LLMs have no access to any of that no matter how big their window is  that data doesn't exist in their weights, full stop.
Where it's skippable, and often is  general reasoning, coding help, writing assistance, ideation, or anything where the model's frozen training knowledge is already sufficient  adding retrieval here mostly adds latency and complexity without meaningfully improving the answer.
So the honest answer: RAG is dominant in enterprise/knowledge- work AI products specifically, genuinely optional or actively being replaced in some 2026 pipelines as context windows grow, and simply absent in the plain chatbot use case you and I have been running most of this conversation in.[/color]
[color=purple]Me: but if you use RAG to access another of my conversations, would this be considered RAG ?[/color]
[color=navy]AI: That's actually the exact case I described earlier under "using another LLM"  worth distinguishing it clearly from RAG's more common use, since the term gets applied to two genuinely different jobs.
RAG for external knowledge (the common case)  retrieval brings in facts the model was never trained on: your company's internal docs, live data, things outside the weights entirely. The point is adding knowledge that doesn't exist in the model.
RAG for context management (what you're asking about)  here the information isn't new  it's stuff that was already said earlier in the same conversation or session, but has scrolled outside the current context window. Retrieval's job isn't "give the model new facts," it's "give the model back something it technically already knows but can no longer see." Same mechanism (embed, store, retrieve by similarity, reinsert), completely different purpose.
This is exactly the "long- term store" and "hierarchical memory" pattern from earlier: old conversation turns get embedded and stored once they age out of the live window, and when something later in the conversation seems related, the retrieval step pulls the relevant fragment back in  not from an external knowledge base, but from the conversation's own history.
The genuinely useful distinction: when RAG is used for external knowledge, a "miss" means the model doesn't know a fact it was never told. When RAG is used for context management, a "miss" is worse in one sense  the information was said, the user reasonably expects the system to remember it, and a failed retrieval looks like forgetting rather than not- knowing. That's part of why "lost in the middle" and retrieval quality matter even more here than in the standard document- QA RAG case  the user's trust that the system is actually listening is directly on the line.
This is also functionally close to what my own memory system is doing with you across sessions, just at a much coarser grain  periodic summarization instead of fine- grained per- turn retrieval. Same underlying idea: don't force everything into the live window, selectively bring back what's relevant when it's needed.[/color]
[color=purple]Me: what is lost in the middle[/color]
[color=navy]AI: You actually asked about this a bit earlier in this conversation, so briefly, in case it didn't fully land the first time, then I'll go deeper if useful.
The core phenomenon: LLMs don't attend evenly across a long context. When relevant information sits at the very start or very end of a long prompt, models retrieve and use it reliably. When that same information sits buried in the middle, accuracy drops - sometimes sharply - even though the information is technically right there, fully within the context window, and the model has more than enough capacity to hold it.
Why it happens, mechanically: it traces back to training data patterns and how attention gets distributed. Models tend to develop a bias toward the start of a sequence (recency- from- the- beginning) and the end (recency- from- now), because those positions are disproportionately informative during training - the beginning often sets up the task, the end is what's freshest. The middle doesn't get that same "special treatment" baked in, so attention weights end up systematically weaker there, independent of the content's actual relevance.
Here is the final build of the ai-preproc.py that merges Claude's original preprocessor with Copilot's NLP preprocessor that Claude debugged
#!/usr/bin/env python3
"""
ai-preproc.py
Merged two-stage pipeline: token_preprocessor.py + canonical_rewriter.py
combined into one script, one command.
STAGE 1 (cleanup, from token_preprocessor.py):
- Fix double-encoded UTF-8 (mojibake, e.g. "a€"" -> "-")
- Strip BBCode tags ([b], [color=...], etc.)
- Fix broken-hyphenation PDF-extraction artifacts
- Collapse page-break markers
- Shorten verbose phrases ("utilize" -> "use", etc.)
- Collapse excess whitespace
STAGE 2 (canonicalization, from canonical_rewriter.py):
- Strip conversational/voice filler ("um", "please", "can you", etc.)
- Split into sentences
- Extract each sentence's core verb + object (walking ROOT, ccomp,
prep, pcomp, and advmod-mediated attachments - see inline notes,
each confirmed against real spaCy parses via debug_parse.py)
- Rebuild each sentence as a short canonical command phrase
Usage:
python3 ai-preproc.py input.txt output.txt
python3 ai-preproc.py input.txt # prints to stdout
cat input.txt | python3 ai-preproc.py - # read from stdin
python3 ai-preproc.py # runs built-in samples
"""
import re
import sys
import spacy
nlp = spacy.load("en_core_web_sm")
# =======================================================================
# STAGE 1 - Cleanup (from token_preprocessor.py)
# =======================================================================
def fix_mojibake(text: str) -> str:
"""Repair double-encoded UTF-8 (e.g. em-dashes turned into a€" )."""
try:
return text.encode('cp1252').decode('utf-8')
except (UnicodeEncodeError, UnicodeDecodeError):
return text
BBCODE_TAG_RE = re.compile(r'\[/?(?:color|b|i|u|size|url|quote|img)[^\]]*\]', re.IGNORECASE)
def strip_bbcode(text: str) -> str:
return BBCODE_TAG_RE.sub('', text)
BROKEN_HYPHEN_RE = re.compile(r'(\w)(?:\s+-\s*|\s*-\s+)(\w)')
def fix_broken_hyphenation(text: str) -> str:
prev = None
while prev != text:
prev = text
text = BROKEN_HYPHEN_RE.sub(r'\1 \2', text)
return text
PAGE_MARKER_RE = re.compile(r'=+\s*Page\s*\d+\s*=+', re.IGNORECASE)
def strip_page_markers(text: str) -> str:
return PAGE_MARKER_RE.sub('', text)
PHRASE_MAP = [
("in order to", "to"),
("due to the fact that", "because"),
("in the event that", "if"),
("with regard to", "about"),
("with respect to", "about"),
("in spite of the fact that", "although"),
("for the purpose of", "to"),
("at this point in time", "now"),
("in the process of", "while"),
("a large number of", "many"),
("the majority of", "most"),
("utilize", "use"),
("utilization", "use"),
("commence", "start"),
("terminate", "end"),
("facilitate", "help"),
("subsequently", "then"),
("approximately", "about"),
("in conjunction with", "with"),
("prior to", "before"),
("subsequent to", "after"),
]
def shorten_phrases(text: str) -> str:
for verbose, plain in PHRASE_MAP:
pattern = re.compile(r'\b' + re.escape(verbose) + r'\b', re.IGNORECASE)
text = pattern.sub(plain, text)
return text
def collapse_whitespace(text: str) -> str:
text = re.sub(r'[ \t]+', ' ', text)
text = re.sub(r'\n{3,}', '\n\n', text)
text = re.sub(r' +\n', '\n', text)
return text.strip()
def stage1_cleanup(text: str) -> str:
text = fix_mojibake(text)
text = strip_bbcode(text)
text = strip_page_markers(text)
text = fix_broken_hyphenation(text)
text = shorten_phrases(text)
text = collapse_whitespace(text)
return text
# =======================================================================
# STAGE 2 - Canonicalization (from canonical_rewriter.py)
# =======================================================================
VERB_MAP = {
"clarify": "explain",
"describe": "explain",
"tell": "explain",
"show": "demonstrate",
"give": "provide",
"list": "list",
"identify": "identify",
"explain": "explain",
"define": "define",
"summarize": "summarize",
"compare": "compare",
"analyze": "analyze",
}
FILLER_PHRASES = [
"before we proceed",
"at this point",
"if possible",
"could you",
"can you",
"would you",
"please",
"kindly",
"i was wondering",
"i want to know",
"i need to know",
"um,", "um",
"uh,", "uh",
"err,", "err",
"erm,", "erm",
"you know,", "you know",
"i mean,", "i mean",
# NOTE: "sort of" and "kind of" deliberately removed from this list.
# They're too ambiguous for blind string matching - "what KIND OF
# things spaCy can do" is genuine content (a noun phrase), not a
# hedge, but "it's kind of hard" is filler. A context-free string
# match can't tell these apart, and deleting real content is worse
# than missing an occasional hedge. Confirmed via live testing:
# this exact collision silently ate "kind of" out of a real sentence.
]
def remove_filler(text):
cleaned = text
for phrase in FILLER_PHRASES:
pattern = re.compile(re.escape(phrase), re.IGNORECASE)
cleaned = pattern.sub("", cleaned)
cleaned = re.sub(r"\s{2,}", " ", cleaned)
cleaned = re.sub(r"^[,\s]+", "", cleaned)
return cleaned.strip()
def extract_main_verb(doc):
for token in doc:
if token.pos_ == "VERB" and token.dep_ == "ROOT":
return token
for token in doc:
if token.pos_ == "VERB" and token.dep_ in ("ccomp", "advcl"):
return token
return None
def _expand_with_trailing_prep_phrase(token):
for child in token.children:
if child.dep_ == "prep":
for grandchild in child.children:
if grandchild.dep_ == "pobj":
start = token.left_edge.i
end = grandchild.right_edge.i
return token.doc[start:end + 1].text
return token.text
def _verb_level_prep_phrase(verb):
def _phrase_from_prep(prep_token):
for grandchild in prep_token.children:
if grandchild.dep_ in ("pobj", "pcomp"):
start = prep_token.left_edge.i
end = grandchild.right_edge.i
return prep_token.doc[start:end + 1].text
return None
for child in verb.children:
if child.dep_ == "prep":
phrase = _phrase_from_prep(child)
if phrase:
return phrase
for child in verb.children:
if child.dep_ == "advmod":
for grandchild in child.children:
if grandchild.dep_ == "prep":
phrase = _phrase_from_prep(grandchild)
if phrase:
return phrase
return None
def extract_object(verb):
direct = [child for child in verb.children if child.dep_ in ("dobj", "attr", "oprd")]
obj_text = _expand_with_trailing_prep_phrase(direct[0]) if direct else None
verb_prep_text = _verb_level_prep_phrase(verb)
if obj_text and verb_prep_text:
if verb_prep_text not in obj_text:
return f"{obj_text} {verb_prep_text}"
return obj_text
if obj_text:
return obj_text
if verb_prep_text:
return verb_prep_text
for child in verb.children:
if child.dep_ == "ccomp":
nested_obj = extract_object(child)
if nested_obj:
return nested_obj
nested_prep = _verb_level_prep_phrase(child)
if nested_prep:
return nested_prep
return None
def extract_entities(doc):
return [ent.text for ent in doc.ents]
def canonical_rewrite_sentence(sent):
verb = extract_main_verb(sent)
if not verb:
return None
canonical_verb = VERB_MAP.get(verb.lemma_, verb.lemma_)
obj = extract_object(verb)
entities = extract_entities(sent)
parts = [canonical_verb.capitalize()]
if obj:
parts.append(obj)
if entities:
new_entities = [e for e in entities if not (obj and e in obj)]
if new_entities:
parts.append("about " + ", ".join(new_entities))
return " ".join(parts)
def stage2_canonicalize(text: str) -> str:
cleaned = remove_filler(text)
doc = nlp(cleaned)
results = []
for sent in doc.sents:
rewritten = canonical_rewrite_sentence(sent)
if rewritten:
results.append(rewritten)
if not results:
return cleaned
return " | ".join(results)
# =======================================================================
# Combined pipeline
# =======================================================================
def run_pipeline(text: str) -> str:
stage1 = stage1_cleanup(text)
stage2 = stage2_canonicalize(stage1)
return stage2
if __name__ == "__main__":
if len(sys.argv) < 2:
print(__doc__)
print("With no arguments, runs the built-in sample sentences instead.\n")
samples = [
"Before we proceed, can you clarify what you meant earlier about memory?",
"Can you describe the process for token generation?",
"Please give me a summary of the KV cache splicing problem.",
"Um, err, can you shed some light on what spaCy can really do when one gets the hang of it?",
]
for sample in samples:
print("Original: ", sample)
print("Canonical:", run_pipeline(sample))
print()
sys.exit(0)
src = sys.argv[1]
if src == "-":
raw = sys.stdin.read()
else:
with open(src, "r", encoding="utf-8") as f:
raw = f.read()
output_text = run_pipeline(raw)
if len(sys.argv) >= 3:
with open(sys.argv[2], "w", encoding="utf-8") as f:
f.write(output_text)
print(f"Written to {sys.argv[2]}")
else:
print(output_text)
Here is the final build of canonical_rewriter.py whicj is Copilot's preprocessor that Claude debugged and fixed -
#!/usr/bin/env python3
"""
canonical_rewriter.py
General-purpose canonical query rewriter.
No per-query editing. No hard-coded samples.
Fixes applied to Copilot's version, and why:
1. remove_filler() used to lowercase the ENTIRE text before parsing.
spaCy's model relies on capitalization (proper nouns, acronyms,
sentence boundaries) to parse correctly - feeding it all-lowercase
text makes acronym/entity detection worse, not better.
Fix: strip filler phrases case-insensitively, but preserve the
original casing in the text that actually gets parsed.
2. Verb selection took the FIRST verb matching ROOT/ccomp/advcl in
linear order, so a subordinate clause verb appearing earlier in
the sentence could be picked over the real ROOT verb.
Fix: search for ROOT explicitly first, fall back to ccomp/advcl
only if no ROOT verb exists.
3. Object extraction only checked the verb's DIRECT children for
dobj/pobj/attr/oprd, missing objects nested inside prepositional
phrases (verb -> prep -> pobj, e.g. "clarify ... about memory"),
and missing further nested phrases attached to the object itself
(object -> prep -> pobj, e.g. "summary OF the KV cache problem").
Fix: walk both levels and return the full phrase span, not just
the bare head noun.
4. Entities that are already contained within the extracted object
phrase used to get appended a second time (e.g. "KV" appearing
both inside the object phrase and again as "about KV" from NER
tagging the acronym separately).
Fix: only append entities not already covered by the object text.
"""
import re
import spacy
nlp = spacy.load("en_core_web_sm")
# Universal polite -> command verb mapping
VERB_MAP = {
"clarify": "explain",
"describe": "explain",
"tell": "explain",
"show": "demonstrate",
"give": "provide",
"list": "list",
"identify": "identify",
"explain": "explain",
"define": "define",
"summarize": "summarize",
"compare": "compare",
"analyze": "analyze",
}
# Universal filler removal
FILLER_PHRASES = [
"before we proceed",
"at this point",
"if possible",
"could you",
"can you",
"would you",
"please",
"kindly",
"i was wondering",
"i want to know",
"i need to know",
# Voice-to-text verbal fillers (added for voice input support)
"um,",
"um",
"uh,",
"uh",
"err,",
"err",
"erm,",
"erm",
"you know,",
"you know",
"i mean,",
"i mean",
# NOTE: "sort of" and "kind of" deliberately removed - blind string
# match can't distinguish genuine content ("what KIND OF things...")
# from hedging filler ("it's kind of hard"). Confirmed via live
# testing that this collision silently deletes real content.
]
def remove_filler(text):
# Case-insensitive matching, but preserve original casing in the
# text that gets returned/parsed - spaCy needs real capitalization
# to parse acronyms and proper nouns correctly.
cleaned = text
for phrase in FILLER_PHRASES:
pattern = re.compile(re.escape(phrase), re.IGNORECASE)
cleaned = pattern.sub("", cleaned)
cleaned = re.sub(r"\s{2,}", " ", cleaned)
cleaned = re.sub(r"^[,\s]+", "", cleaned)
return cleaned.strip()
def extract_main_verb(doc):
for token in doc:
if token.pos_ == "VERB" and token.dep_ == "ROOT":
return token
for token in doc:
if token.pos_ == "VERB" and token.dep_ in ("ccomp", "advcl"):
return token
return None
def _expand_with_trailing_prep_phrase(token):
for child in token.children:
if child.dep_ == "prep":
for grandchild in child.children:
if grandchild.dep_ == "pobj":
start = token.left_edge.i
end = grandchild.right_edge.i
return token.doc[start:end + 1].text
return token.text
def _verb_level_prep_phrase(verb):
"""
Find a prepositional phrase associated with the verb. Checks two
attachment points, and two kinds of preposition complement:
Attachment points:
1. Direct child of the verb - e.g. "describe X FOR token generation".
2. Child of an adverb (advmod) modifying the verb - e.g.
"meant EARLIER ABOUT memory" (confirmed via debug_parse.py).
Complement types:
- pobj: a simple noun object ("about MEMORY").
- pcomp: a whole embedded clause ("on WHAT SPACY CAN REALLY DO
WHEN ONE GETS THE HANG OF IT") - confirmed via debug_parse.py
on the "shed light on..." sentence. When this happens we grab
the full span through the end of that clause's subtree, since
the "object" here isn't a single word.
"""
def _phrase_from_prep(prep_token):
for grandchild in prep_token.children:
if grandchild.dep_ == "pobj":
start = prep_token.left_edge.i
end = grandchild.right_edge.i
return prep_token.doc[start:end + 1].text
if grandchild.dep_ == "pcomp":
start = prep_token.left_edge.i
end = grandchild.right_edge.i
return prep_token.doc[start:end + 1].text
return None
# Case 1: prep directly on the verb.
for child in verb.children:
if child.dep_ == "prep":
phrase = _phrase_from_prep(child)
if phrase:
return phrase
# Case 2: prep nested under an adverb modifying the verb.
for child in verb.children:
if child.dep_ == "advmod":
for grandchild in child.children:
if grandchild.dep_ == "prep":
phrase = _phrase_from_prep(grandchild)
if phrase:
return phrase
return None
def extract_object(verb):
direct = [child for child in verb.children if child.dep_ in ("dobj", "attr", "oprd")]
obj_text = _expand_with_trailing_prep_phrase(direct[0]) if direct else None
verb_prep_text = _verb_level_prep_phrase(verb)
# A direct object and a verb-level prep phrase can coexist
# ("process" + "for token generation") - combine rather than
# treating them as either/or.
if obj_text and verb_prep_text:
if verb_prep_text not in obj_text:
return f"{obj_text} {verb_prep_text}"
return obj_text
if obj_text:
return obj_text
if verb_prep_text:
return verb_prep_text
# Nothing found on the verb itself - check if the verb has a ccomp
# (a nested complement clause, e.g. "clarify [what you meant about
# memory]") and recurse into THAT clause's own verb to find its
# object. This is what "clarify what you meant... about memory"
# needs: the real content is one level deeper than "clarify" itself.
for child in verb.children:
if child.dep_ == "ccomp":
nested_obj = extract_object(child)
if nested_obj:
return nested_obj
nested_prep = _verb_level_prep_phrase(child)
if nested_prep:
return nested_prep
return None
def extract_entities(doc):
return [ent.text for ent in doc.ents]
def canonical_rewrite_sentence(sent):
"""
Runs the verb/object/entity extraction on a single sentence span
(not a whole paragraph). Returns the canonical phrase for just
that sentence, or None if no usable verb was found.
"""
verb = extract_main_verb(sent)
if not verb:
return None
canonical_verb = VERB_MAP.get(verb.lemma_, verb.lemma_)
obj = extract_object(verb)
entities = extract_entities(sent) # entities within this sentence only
parts = [canonical_verb.capitalize()]
if obj:
parts.append(obj)
if entities:
new_entities = [e for e in entities if not (obj and e in obj)]
if new_entities:
parts.append("about " + ", ".join(new_entities))
return " ".join(parts)
def canonical_rewrite(text: str) -> str:
"""
Processes a whole paragraph: splits into sentences (spaCy's own
sentence boundary detection via doc.sents), canonicalizes each
sentence separately, then joins the results. This is what makes
it paragraph-safe rather than only picking up the first verb in
the entire blob and silently dropping every sentence after it.
"""
cleaned = remove_filler(text)
doc = nlp(cleaned)
results = []
for sent in doc.sents:
rewritten = canonical_rewrite_sentence(sent)
if rewritten:
results.append(rewritten)
if not results:
return cleaned # fallback: nothing extracted at all
return " | ".join(results)
if __name__ == "__main__":
import sys
if len(sys.argv) < 2:
print(__doc__)
print("Usage: python3 canonical_rewriter.py input.txt [output.txt]")
print(" python3 canonical_rewriter.py - (read stdin)")
print()
print("With no arguments, runs the built-in sample sentences instead.")
samples = [
"Before we proceed, can you clarify what you meant earlier about memory?",
"Can you describe the process for token generation?",
"Please give me a summary of the KV cache splicing problem.",
]
for sample in samples:
print("Original: ", sample)
print("Canonical:", canonical_rewrite(sample))
print()
sys.exit(0)
src = sys.argv[1]
if src == "-":
raw = sys.stdin.read()
else:
with open(src, "r", encoding="utf-8") as f:
raw = f.read()
# Treat the whole file as ONE paragraph - canonical_rewrite()
# handles splitting it into sentences internally now, so we don't
# split by line here (a forum paragraph isn't one query per line).
output_text = canonical_rewrite(raw)
if len(sys.argv) >= 3:
with open(sys.argv[2], "w", encoding="utf-8") as f:
f.write(output_text)
print(f"Written to {sys.argv[2]}")
else:
print(output_text)
This is debug_parse.py - Claude's debug tool
#!/usr/bin/env python3
"""
debug_parse.py
Prints the real dependency tree for a sentence: token, POS, dep label,
and head - so we can see exactly what's attached to what instead of
guessing.
"""
import spacy
nlp = spacy.load("en_core_web_sm")
sample = "ahh ... shed some light on what kind-of-things spaCy can really do when one gets the hang of it ?"
doc = nlp(sample)
print("SENTENCES spaCy detected:")
for i, sent in enumerate(doc.sents):
print(f" [{i}] {sent.text!r}")
print()
print(f"{'TOKEN':<12}{'POS':<8}{'DEP':<10}{'HEAD'}")
for token in doc:
print(f"{token.text:<12}{token.pos_:<8}{token.dep_:<10}{token.head.text}")
Home
Help
Search
Login
Register
Need help or a chat ?
