You and the Model Never Read the Same Text

You type Tokenization is cool. The model never sees that string. It sees four integers — 3404, 2065, 374, 7155 — each pointing at an entry in a lookup table. Somewhere between your sentence and the transformer’s math, your words get shredded into sub-word chunks, mapped to numbers, and those numbers get turned into vectors. That shredding step is tokenization, and it quietly decides everything about an LLM: how much your API call costs, how long your context window actually is, how well the model handles emoji, typos, and Turkish, and whether your prompt-cache ever hits.

Most people treat it as plumbing. It’s not. It’s the part of the model you can actually inspect and control, and it’s where a shocking number of “the model is broken” bugs actually live.

This post walks through what tokenization is, the algorithms behind it (BPE, WordPiece, Unigram, SentencePiece), and then a pile of real, runnable examples — every number below came out of an actual tokenizer, not a textbook.

The One Picture

Here is the entire journey, for the string Tokenization is cool, using the tokenizer behind GPT-4-era models (cl100k_base):

 "Tokenization is cool"
        │  tokenizer.encode(text)
        ▼
┌──────────┬──────────┬──────────┬──────────┐
│   3404   │   2065   │   374    │   7155   │   ← token ids
├──────────┼──────────┼──────────┼──────────┤
│ "Token"  │"ization" │  " is"   │  " cool" │   ← the actual pieces
└──────────┴──────────┴──────────┴──────────┘
       │         │         │         │
       └────┬────┴────┬────┴────┬────┘
            ▼         ▼         ▼
     embedding lookup: every id pulls one
     learned vector out of a table
            │
            ▼
        transformer

Three things are going on here, and each one is a trap waiting for you later:

1. Your text is chopped into variable-size chunks, not words and not letters. Tokenization = Token + ization. Why? Because Token, ization, ` is and cool` all show up constantly in the training corpus, so the vocabulary keeps them as single entries.
2. Every chunk is an id, a row number into a fixed vocabulary. The ids are the only thing the neural network ever sees.
3. The spaces are glued to the front of chunks, not the back. Look closely: it’s ` is and cool`, with a leading space. That convention is load-bearing (more on that later).

The chunk between your raw text and the ids is called a token. And here’s the part that feels like a scam at first: the model does not see “words”, it sees these tokens — which means is and ` is` are different tokens even though they’re the same two letters.

Characters Are Too Raw, Words Are Too Proud

Why not just tokenize on characters or words? Both are tempting and both fail.

word level :  [unbreakable]                       1 token — but "unbreakabl" and "breakable"
                                                     aren't in the vocab → unknown token
char level :  u n b r e a k a b l e               11 tokens — long sequences, slow to train,
                                                     no shared structure learned
subword    :  [un] [##break] [##able]             3 tokens — reuses pieces seen everywhere

• Word-level blows up the vocabulary (English alone has ~500k words, and then you meet German compounds, names, and slang) and any word you didn’t see in training becomes a dead <unk>.
• Character-level needs a tiny vocabulary, but now “unbreakable” is 11 positions the transformer has to model, and long-range structure (like “un-“ being a negation prefix) never gets shared between words.
• Subword-level splits the difference: frequent words stay whole (low), rare words fall apart into frequent pieces (wid + est). The vocabulary stays small (~50k–200k entries), nothing is ever truly “unknown”, and morphology gets shared for free.

Every modern LLM is subword. The remaining differences are just which algorithm learned the chunks and how it marks word boundaries.

The Tokenizer Family Tree

There are basically four algorithms, and you’ve been using all of them without knowing:

A couple of vocab-size numbers worth remembering, because they come up in config files constantly:

• GPT-2: 50,257 tokens
• Llama 1 / Llama 2 (SentencePiece): 32,000
• GPT-4-era cl100k_base: 100,261 (≈100k — that’s the name)
• GPT-4o-era o200k_base: 200,000
• Llama 3: ~128k

The trend is telling: newer models buy a bigger vocabulary because it means fewer tokens per sentence, and tokens are literally money.

BPE by Hand

Let’s learn BPE for real, on a toy corpus. The algorithm is almost embarrassingly simple:

1. Start with every character as its own token.
2. Count every adjacent pair of tokens across the corpus.
3. Merge the most frequent pair into one token.
4. Repeat until you run out of vocabulary budget.

Here’s a complete, runnable implementation:

# mini_bpe.py — learn BPE merges on a toy corpus, then encode unseen words
from collections import Counter

corpus = ["low"] * 20 + ["lowest", "newer", "newest", "widest"]

def learn_bpe(words, num_merges):
    word_counts = Counter(words)                    # keep frequencies!
    vocab = {w: list(w) for w in word_counts}       # every word starts as its letters
    merges = []

    for _ in range(num_merges):
        stats = Counter()
        for word, count in word_counts.items():
            pieces = vocab[word]
            for i in range(len(pieces) - 1):        # count adjacent pairs...
                stats[(pieces[i], pieces[i + 1])] += count   # ...weighted by frequency
        if not stats:
            break
        best = stats.most_common(1)[0][0]           # most frequent pair wins
        merges.append(best)

        for word in word_counts:                    # apply the merge everywhere
            pieces = vocab[word]
            merged, i = [], 0
            while i < len(pieces):
                if i + 1 < len(pieces) and (pieces[i], pieces[i + 1]) == best:
                    merged.append(pieces[i] + pieces[i + 1])
                    i += 2
                else:
                    merged.append(pieces[i])
                    i += 1
            vocab[word] = merged
    return merges, vocab

def bpe_encode(text, merges):
    pieces = list(text)
    for a, b in merges:                             # replay merges in learned order
        out, i = [], 0
        while i < len(pieces):
            if i + 1 < len(pieces) and pieces[i] == a and pieces[i + 1] == b:
                out.append(a + b)
                i += 2
            else:
                out.append(pieces[i])
                i += 1
        pieces = out
    return pieces

merges, vocab = learn_bpe(corpus, 12)

Run learn_bpe(corpus, 12) and you get this merge history:

merge  1: 'l'  + 'o'    ->  'lo'       # low shows up 20+ times, 'lo' is everywhere
merge  2: 'lo' + 'w'    ->  'low'      # now 'low' is itself a frequent pair
merge  3: 'e'  + 's'    ->  'es'       # newest / widest / lowest share the suffix
merge  4: 'es' + 't'    ->  'est'
merge  5: 'n'  + 'e'    ->  'ne'
merge  6: 'ne' + 'w'    ->  'new'
merge  7: 'low' + 'est' ->  'lowest'   # whole frequent words become ONE token
merge  8: 'new' + 'e'   ->  'newe'
merge  9: 'newe' + 'r'  ->  'newer'
merge 10: 'new' + 'est' ->  'newest'
merge 11: 'w'  + 'i'    ->  'wi'
merge 12: 'wi' + 'd'    ->  'wid'

Notice what happened: low, lowest, newer, newest each collapsed into a single token because they were common enough, while widest only partially merged (wid + est). That’s the whole trick of BPE — frequency buys vocabulary slots. The fun part is encoding words that were never in the training corpus:

bpe_encode("slow", merges)          # ['s', 'low']     — reuses the 'low' token
bpe_encode("wider", merges)         # ['wid', 'e', 'r'] — falls back to smaller pieces
bpe_encode("lowestprice", merges)   # ['lowest', 'p', 'r', 'i', 'c', 'e']

No <unk> anywhere. An unseen word just decomposes into the longest known pieces. This single property is why subword tokenization lets models handle words they’ve never seen, in languages they were barely trained on. A real BPE tokenizer learns these merges on gigabytes of text, not six words — but it is exactly the same greedy loop.

Byte-Level BPE and the Sneaky Space

GPT-2 added two twists that almost every LLM after it copied.

Twist 1: operate on bytes, not characters. Before tokenizing, text is encoded to UTF-8 bytes. Why? Because no matter what language or emoji a user throws at the model, the bytes are always in a manageable set (256 of them), so the tokenizer never sees a truly novel character. Something exotic that never got a merged token simply stays split into single bytes. That is why tokenizers survive Unicode without exploding.

 🚀  =  UTF-8 bytes:  F0 9F 9A 80

 cl100k_base chops it into byte pieces:
      [0xF0,0x9F]  [0x9A]  [0x80]        →  3 tokens   ← every rocket costs ~3 tokens
 o200k_base (newer) has seen enough 🚀s to do better:
      [0xF0,0x9F,0x9A]  [0x80]           →  2 tokens

Twist 2: the leading-space convention. Real tokenizers put a space at the front of a token when it starts a new word. So a trained tokenizer never has to merge across word boundaries by accident — otherwise “New York” and “NewYork” could merge into the same weird blob. You saw it in the opening diagram: ` is and cool`, space included.

You can verify both twists in five lines:

# bytes_under_the_hood.py — what GPT-family tokenizers actually store
import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
text = "Tokenization is cool"

ids = enc.encode(text)
print(ids)                          # [3404, 2065, 374, 7155]
print(enc.decode_tokens_bytes(ids)) # [b'Token', b'ization', b' is', b' cool']
print(enc.decode(ids))              # "Tokenization is cool"  — lossless round trip

Note the bytes: b' is' and b' cool' — spaces glued to the front of the following piece. And the round trip is lossless: encode then decode and you get your exact string back, spaces and all.

Real Splits on Real Models

Here is where the rubber meets the road. These are genuine outputs — I ran every one of them.

# tokenizer_tour.py — GPT-4-era encoding
import tiktoken

cl100k = tiktoken.get_encoding("cl100k_base")   # GPT-3.5 / GPT-4 era
o200k  = tiktoken.get_encoding("o200k_base")    # GPT-4o era

cl100k.encode("hello world")    # [15339, 1917]
cl100k.encode("Hello, world!")  # [9906, 11, 1917, 0]
cl100k.encode("café")           # [936, 59958]   → b'ca' + b'f\xc3\xa9'
cl100k.encode("你好")            # [57668, 53901]

o200k.encode("hello world")     # [24912, 2375]
o200k.encode("café")            # [66, 103112]
o200k.encode("你好")             # [177519]       ← ONE token now

Read those three facts out loud:

1. Hello ≠ hello. Different first letter, different ids. Tokenizers are case-sensitive, and almost every character/space/case variation is a different token. Hello, is [9906, 11] while Hello. would be [9906, 13] — different trailing punctuation, different ids.
2. Accents get byte-split. café is not one token; the é becomes raw bytes \xc3\xa9 glued onto a piece. The newer o200k vocabulary still splits it, but into a nicer c + f\xc3\xa9.
3. Multilingual coverage is a moving target. 你好 (“hello”) costs 2 tokens on cl100k_base but a single token on o200k_base. This is the single biggest reason OpenAI shipped a 200k vocabulary — to make non-English text cheaper. If your users write in Turkish, Korean, or Arabic, tokenization is quietly taxing them 2–4x more than English speakers.

Llama-family tokenizers (SentencePiece) behave the same but mark spaces with ▁ instead of gluing them:

# llama_tokenizer_tour.py — SentencePiece mechanics (small test tokenizer, same as Llama 1/2)
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("hf-internal-testing/llama-tokenizer")
tok.tokenize("Hello, world!")   # ['▁Hello', ',', '▁world', '!']   ← ▁ = space
tok.encode("Hello, world!")     # [1, 15043, 29892, 3186, 29991]    ← 1 is the BOS token
tok.vocab_size                  # 32000

That stray leading 1 is a special token — the beginning-of-sentence marker added automatically. Which brings us to the invisible grammar.

Special Tokens: The Invisible Grammar

Tokenizers reserve a handful of ids that never map to readable text. They’re punctuation for the model — they tell it where the prompt starts, where the text ends, where your system prompt ends and the user’s question begins. Every chat model you’ve used is secretly a grammar of reserved tokens:

what you send:
    system: "You are a snarky assistant."
    user:   "What is a token?"

what the model actually parses:
    <s>  system: You are a snarky assistant.  <|sep|>
    user: What is a token?  <|sep|>
    assistant: <|gen|>

each <|...|> or <s> is ONE reserved token id — not readable text

Some real ones so you recognize them in logs:

• GPT-2: <|endoftext|> = id 50256
• cl100k_base: <|endoftext|> = 100257
• o200k_base: <|endoftext|> = 199999
• Llama 1/2: <s> = 1, </s> = 2

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
enc.encode("<|endoftext|>", allowed_special={"<|endoftext|>"})  # [100257]

Your framework adds these for you (that’s what the “chat template” does), but you pay for them: every message gets wrapped in reserved scaffolding that consumes context window and billable tokens on every single call. This is exactly why the automatic-prefix-caching post matters — the scaffold is identical across all of a conversation’s turns, so it’s the most cache-friendly text in your entire request. Break the scaffold and you break the cache.

Token Counting Is Your Budget

Tokens are the unit of everything that costs you: context windows (8k, 32k, 128k, 200k are all token limits, not word limits), API billing, KV-cache size, and time-to-first-token. So build a feel for them.

Rough rules of thumb that have served me well:

• English: ~4 characters per token, ~0.75 words per token. 100 tokens ≈ 75 words ≈ a short paragraph.
• Code: ~3.5 chars/token (punctuation-heavy, and punctuation rarely merges).
• Emoji and non-Latin scripts: 1–3+ tokens per symbol. A 4000-character Chinese news article and a 4000-character English one do not cost the same — the Chinese one can be 2–4x more tokens.

That’s not a curiosity, that’s a budget. A 100k-token window holds roughly 75,000 English words. The same content in Chinese costs ~1 token per character (vs ~0.25 per character for English), so a 100k-token budget buys roughly a quarter of the characters of text.

# budget.py — count before you send (use the SAME tokenizer the model uses)
import tiktoken

enc = tiktoken.get_encoding("o200k_base")   # match your model! cl100k ≠ o200k

def estimate_cost(text, price_per_million: float) -> float:
    n_tokens = len(enc.encode(text))
    return n_tokens / 1_000_000 * price_per_million

The cardinal sin is estimating with a random tokenizer. Every tokenizer gives different counts for the same sentence, and if you estimate with the wrong one you’ll discover it the moment the provider’s bill arrives.

Tokenizer Traps I’ve Actually Hit

Trap 1: Tokenizer drift — changing the tokenizer silently changes everything

The vocab is baked into the model weights; the tokenizer must match the checkpoint exactly or every id points at the wrong piece. Rolling a new tokenizer onto a fine-tuned model is the same category of bug as loading mismatched weights — output starts subtly, then catastrophically wrong. cl100k_base and o200k_base both encode hello world — but to different ids. Use the wrong one and the model reads gibberish.

Trap 2: Chunking RAG by characters, not tokens

The classic: “I’ll split my documents into 500-character chunks for retrieval.” But your embedding model and your LLM both work in tokens. A 500-character chunk of English is ~125 tokens; a 500-character chunk of Chinese is ~500. Your retrievers and your context window are silently living on different budgets.

# chunker.py — token-aware splitting (character slicing is a lottery)
import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

def chunk_by_tokens(text: str, max_tokens: int = 256):
    ids = enc.encode(text)
    for i in range(0, len(ids), max_tokens):
        yield enc.decode(ids[i:i + max_tokens])   # decode back to clean text

Trap 3: Truncation that isn’t

Models truncate at token limits, and they truncate at token boundaries. If you pre-truncate by characters — worse, by slicing a UTF-8 string mid-multibyte — you’re handing the tokenizer half a rocket emoji and a prefix the model never saw in training. Always truncate at the tokenizer level, using the model’s own max_length.

Trap 4: Tiny edits kill the prompt cache

Remember the leading-space and case facts above? "Bye" and "Bye." tokenize to different id sequences. Change one space, one period, one re-generated timestamp, and every downstream token id shifts — which means the prefix no longer matches what the serving layer cached, and you pay a full prefill again. The formatting traps from the prefix-caching post are tokenizer traps first.

Trap 5: Byte fallback means emoji-shaped surprises

“Everything is in the vocab” isn’t true. Rare or novel glyphs decompose to single bytes, and you can’t predict by looking at text which glyphs got merged. 🎉 costs 3 tokens on cl100k_base but only 2 on the newer o200k_base — while a glyph the tokenizer rarely met can cost 4–6. This is why emoji-heavy product copy costs more than it looks like it should — measure, don’t guess.

TL;DR

• LLMs read tokens (subword chunks), not words or characters; your text is shredded into pieces, then ids, then vectors.
• BPE greedily merges the most frequent adjacent pairs; WordPiece, Unigram, and SentencePiece are cousins with different scoring and space-handling.
• Spaces glue to the front of pieces, and byte-level fallback keeps any Unicode from ever being “unknown” — at the price of emoji and accents costing multiple tokens.
• Tokens are your real budget: ~4 chars/token in English, far worse for emoji and non-Latin scripts, and newer 200k-vocab tokenizers exist precisely to fix that.
• Always count, truncate, and chunk with the exact tokenizer the model ships with — never a proxy.

The next time a “hallucination” looks weird, or a bill is mysteriously 3x your estimate, or a fine-tune produces nonsense: check the tokens first. The model isn’t reading what you think you wrote.

# last_one.py — always decode before you despair
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
ids = [3404, 2065, 374, 7155]
print(enc.decode(ids))   # "Tokenization is cool" — everything is reversible