The Tokeniser and my suggestions ...

[Chip (OP)]

📎 Click on me to view all my attachmentsTheme: Deep IndigoMidnight DarkPink & PurpleSlate GraySolarized LightEmerald GreenCyberpunk NeonNordDraculaSunset OrangeWarm AmberOcean BreezeClassic LightCrimson DuskMatrix TerminalCoffee & CreamAutumn FrostRetro TerminalLavender MistNordic FrostMonochrome SlateTeal BreezeNo theme (plain)

ListAttBBC  |  v6.12  |  2026-09-10  |  Andrew.human & Claude.ai

#	File	Size	Downloads	Info	DL
00	instagram-1792108940024.mp4	4 MB	8	ℹ️	⬇️
Attachment ID 7472 Filename (display) instagram-1792108940024.mp4 File extension mp4 MIME type video/mp4 Size 4 MB (4166106 bytes) Downloads 8 Dimensions n/a Uploaded by Guest Message ID 49797 Attachment type Normal attachment (0) Storage folder ID 2 Thumbnail attach ID none File hash e3c497c3058e7d6154127e3d1642efafe9a4e67f Approved Yes Filename (on disk) 7472_e3c497c3058e7d6154127e3d1642efafe9a4e67f Storage directory 2 (/var/www/html/attachments2) File exists on disk? Yes (core) — Media metadata (none) — Media metadata Unavailable - Neither getID3 nor ffprobe is available on this install
Chip — Yesterday at 08:56:24 AM
01	etymology_idea_transcript.md	31.9 KB	1	ℹ️	⬇️
Attachment ID 7473 Filename (display) etymology_idea_transcript.md File extension md MIME type Size 31.9 KB (32617 bytes) Downloads 1 Dimensions n/a Uploaded by Guest Message ID 49797 Attachment type Normal attachment (0) Storage folder ID 2 Thumbnail attach ID none File hash 337427a9387d79bdf5511d8f9d1491b14e969cbe Approved Yes Filename (on disk) not found on disk Storage directory unresolved File exists on disk? No
No recorded downloads yet.

Transcript:

Today in LLM concepts, let's see how a tokenizer is trained.

Before an LLM reads a single word, it needs a tokenizer. And here's the surprising part: the tokenizer isn't written by hand. It's trained on text before the model's own training even starts.

Step One: COLLECT THE TEXT

—A big sample of the same kind of text the model will read—web pages, books, code, many languages.

Step Two: START TINY

The first vocabulary is just single characters, or in GPT's case, the 256 possible bytes. So at the start, every word is cut into its smallest pieces.

Step Three: COUNT

Across the whole text, inside every word, the trainer counts every pair of neighboring pieces and how often each pair appears. Say "I" followed by "N" shows up millions of times.

Step Four: MERGE

The winning pair becomes one new token, "in," and the rule is saved: "I plus N becomes in." Then count again, merge again. "In plus G becomes aing." Round after round, common chunks like "aing" and "tion" and whole words like "the" each become
a single token.

Step Five: STOP AT THE TARGET SIZE

The vocabulary size is fixed before training, like 32,000 or 50,000 tokens. Once the vocabulary reaches that number, training is done.

Only two things come out: the vocabulary, every token with its ID, and the merge rules in order.

That's the whole tokenizer, and then it's frozen. The model learns one embedding row for every token ID.

So if you change the tokenizer later, the model reads nonsense.

One more effect: a tokenizer only learns what it saw. If the training text had little Hindi, Hindi words never got merged into big chunks, so the same sentence in Hindi turns into many more tokens. Count and merge.

That recipe is called BPE.

But BERT's tokenizer picks its merges differently, and sentence piece doesn't even split on spaces.

In today's LLM interview reel, BPE versus word piece versus sentence piece.

Which one does your model use ?

Comment and tell me.

Which model's tokenizer should we open up next ?