What a Language Model Actually Does

Ask it how many r's are in "strawberry"

For a long time, this was the question that broke language models. A system that could write working code and explain quantum mechanics would confidently answer two.

People took this as proof the whole thing was a con. It isn't. It's the single most useful clue you can get about how the machine actually works — because the reason it failed is the reason it does everything else it does.

It never saw the letters.

Why nobody explains this

Most guides skip straight to "here are 50 prompts that will change your life." They treat the model as a magic box: put a wish in, get a result out. That works right up until it doesn't, and then you have no idea why — so you conclude the AI is stupid, or you conclude it's infallible. Both are expensive mistakes.

You don't need the maths. You need a mental model accurate enough to predict when it will fail. That's what this lesson builds.

Mechanism 1 — It doesn't read letters. It reads tokens.

Before your text reaches the model, it gets chopped into tokens: chunks of characters, roughly word-sized but not exactly. Common words are usually one token. Rarer words split into pieces.

"strawberry"     →  ["str", "aw", "berry"]        (three tokens, approximately)
"I love you"     →  ["I", " love", " you"]        (three tokens)
"antidepressant" →  ["anti", "depress", "ant"]    (three tokens)

Each token becomes a number. The model only ever sees numbers.

So when you ask it to count the r's, it isn't looking at s-t-r-a-w-b-e-r-r-y. It's looking at something closer to [6039, 675, 15717]. Asking it to count letters is like asking you to count the brushstrokes in a painting you've only ever had described to you over the phone.

This explains a whole family of failures: counting letters, reversing words, rhyming reliably, counting words in a sentence, and arithmetic on long numbers. None of these are "the AI being dumb." They're all the same blind spot.

And it explains the fix. Ask it to write the word out spaced — s t r a w b e r r y — and now each letter is its own token, and it can count them. Newer models often do this automatically, which is why the strawberry question works today. The blind spot didn't go away; the model learned to work around it.

Mechanism 2 — It predicts the next token. That's the whole job.

Here is the entire operation:

Given all the text so far, produce a probability distribution over every token it knows, for what comes next. Pick one. Add it to the text. Repeat.

That's it. Everything else — reasoning, translation, code, a poem about your cat — is that loop running thousands of times.

Give it The capital of France is and the distribution might look like:

" Paris"      92%
" located"     3%
" a"           2%
" one"         1%
...everything else

It picks, appends, and asks again. There is no separate step where it looks something up, and no moment where it decides what it thinks and then writes it down. The writing is the thinking.

This is why "let's think step by step" changes the answer so much. The reasoning it writes out becomes part of the text it's predicting from. More useful text in front of it means better predictions after it. It isn't a magic phrase — you're giving it more to work with.

Mechanism 3 — The context window is everything it can see

The model has no memory. What it has is a context window: a fixed budget of tokens holding everything it can currently see — your instructions, the conversation so far, files you uploaded.

Two consequences that matter every single day:

Nothing survives between conversations. Start a new chat and it has no idea who you are. When a product does seem to remember you across sessions, that's a feature built around the model — saved instructions, a project file, a memory store — quietly pasting your information back into the window each time. The model itself is amnesiac by construction.

Long conversations degrade. As the window fills, detail from early on gets less influence, and past the limit it drops out entirely. The symptoms are recognisable once you know them:

• It forgets an instruction you gave at the start
• It claims something isn't in a document when it is
• The answers turn vague and generic

That's not it losing interest. That's the window.

The fix is mechanical. One conversation, one task. Start fresh when the topic changes. For long documents, anchor the question — "looking only at section 3…"

Mechanism 4 — Same question, different answer

Ask the identical question twice and you'll often get two different replies. That's deliberate.

Instead of always taking the highest-probability token, the model samples from the distribution. A setting usually called temperature controls how adventurous that sampling is: low temperature sticks to the safe, high-probability choice; high temperature reaches further down the list.

Low temperature gives you consistency and blandness. High temperature gives you variety and mistakes. Neither is "correct" — it's a dial, and which end you want depends on whether you're writing a poem or extracting data from an invoice.

Practical consequence: if a first answer disappoints, ask again before concluding it can't do it. You may simply have drawn a bad sample.

Mechanism 5 — Why it invents things, and why that won't be patched out

Now the important part.

The model is optimised to produce plausible continuations, not true ones. Facts aren't stored in a table it can consult; they're compressed into billions of numerical weights, lossily, alongside everything else. Common, often-repeated facts survive that compression well. Rare ones — a specific author's name, an exact date, a particular paper's title — survive it badly.

But here's the thing: a wrong answer and a right answer are produced by exactly the same process. There's no internal alarm that fires on the way out. So a fabricated citation arrives in the same confident sentence shape as a real one. The fluency is constant; only the accuracy varies.

This is called hallucination, and it is not a bug awaiting a fix. It's a direct consequence of how the thing works. Models are getting better at it — and tools that let a model search or run code sidestep it by fetching real data instead of recalling it — but "produces plausible text" remains the underlying job.

Therefore, permanently: numbers, dates, citations, legal, medical and tax claims get verified. Not because the model is untrustworthy in general, but because those are precisely the categories where compression fails and confidence stays high.

Full protocol: AI-07 — Hallucination Hunting.

Mechanism 6 — Two different trainings

Worth knowing, because it explains the personality.

Pre-training is the big one: predict the next token across an enormous pile of text. This is where knowledge, grammar, code and reasoning patterns come from. It ends at a knowledge cutoff — a date after which the model simply hasn't seen the world. Ask about something more recent and it will either say so or, worse, guess.

Post-training shapes behaviour: being helpful, following instructions, refusing certain requests, having a consistent tone. A raw pre-trained model doesn't answer questions — it just continues text. The assistant you talk to is a pre-trained model that was taught to behave like an assistant.

This is why models from different labs feel different despite similar underlying technology. You're mostly feeling post-training.

What this means for you

Not advice — just what follows mechanically from the above.

Try it: three experiments (25 min)

Run these on a real model and write down what happened. This is the output of the lesson — the mental model isn't yours until you've watched it behave.

Experiment 1 — Tokens. Ask: "How many times does the letter 'a' appear in 'banana bandana'?" Note the answer. Then ask: "Write the phrase with a space between every letter, then count the a's one by one." Note whether the answer changed. → What you're watching: the blind spot, and the workaround.

Experiment 2 — Sampling. Open three separate new chats. Paste the identical prompt into each: "Write a two-sentence opening for a story about a lighthouse." → What you're watching: three different samples from the same distribution.

Experiment 3 — Compression. Ask: "Give me three peer-reviewed papers on the effect of sleep on memory consolidation, with authors and years." Then take each one and search for it. → What you're watching: how a real citation and an invented one arrive in identical packaging.

✅ Finish check: you have written answers to all three, and you can state in one sentence why Experiment 1 changed when you asked it to space the letters out.

Summary card

• Text becomes tokens, not letters. Letter-counting, rhyming and long arithmetic fail for this one reason.
• The entire operation is predict the next token, append, repeat. The writing is the thinking.
• The context window is all it can see. No memory between chats. Long chats decay.
• It samples, so the same question gives different answers. Ask twice.
• Hallucination is structural, not a bug: right and wrong answers come out of the same process, with the same confidence.
• There is a knowledge cutoff. Recent events need search or need pasting in.
• Fluency is constant. Accuracy varies. Confidence tells you nothing.

Sources

1. Vaswani, A. et al. — Attention Is All You Need, 2017 (the transformer architecture)
2. Bender, E. et al. — On the Dangers of Stochastic Parrots, 2021 (on plausibility vs truth)
3. Anthropic — Claude documentation: context windows, token counting
4. OpenAI — tokenizer documentation and the tiktoken library (for seeing tokenization directly)

Next lesson: AI-02 — How to Use Claude, Step by Step (L1) Related: AI-04 The Anatomy of a Prompt · AI-05 Which Model for Which Job · AI-07 Hallucination Hunting Path: The Rehearsal Room — prerequisite (required before AI-02) · AI Operator — 1/6