Why AI Can't Count Letters: The Strawberry Problem Explained

Why AI can't count letters comes down to what a language model actually reads. Ask a chatbot how many r's are in "strawberry" and many models have answered two, not three. The model does not see the word as ten letters in a row. It sees a few chunks of text called tokens, and it was trained to predict the next chunk, not to tally characters.

The How Modern AI Actually Works course on Age of AI Entrepreneur explains next-token prediction and what AI gets wrong in short lessons with their sources; sign in with your email and an emailed code to open it. This post covers the letter-counting failure, what researchers think causes it, and how to build a product that never depends on it.

Why AI can't count letters in a word

A person counting the r's in "strawberry" looks at each letter and keeps a running total. A large language model does something else. Before it reads your question, a tokenizer cuts the text into tokens. A common word can be one token. A longer or rarer word is split into two or three pieces. The model then works with those pieces, each turned into a list of numbers.

So the question "how many r's?" asks about a level of detail the model never looks at directly. It has to recall, from training, which letters each token contains, and then add them up across the pieces. Both steps can go wrong, and the second one goes wrong more often than you might expect.

This is the same machinery covered in Is ChatGPT Just Autocomplete?: the model produces the most likely next token given everything before it. A confident "two" is a likely-sounding answer, not the result of a count.

Tokens: what the model actually sees

Most modern tokenizers use a version of byte-pair encoding. The method was first described in 1994 by Philip Gage as a way to compress data. In language model tokenizers it starts from single characters and keeps merging the most frequent pair of neighbouring pieces into a new, longer token, until the vocabulary reaches a set size.

The result is efficient. Frequent words and word parts get their own tokens, so a sentence needs far fewer tokens than characters. That keeps requests cheaper and lets more text fit in the model's window, as the posts on LLM token pricing and the context window explain. The cost is that single letters disappear from view inside each chunk.

The split in the picture is an illustration: each tokenizer cuts words in its own way, and the same word can split differently with a capital letter or a space in front of it.

Three explanations researchers give

A 2024 study on arXiv, Why Do Large Language Models (LLMs) Struggle to Count Letters?, sums up the explanations earlier work has offered. There are three, and they are not exclusive.

• Tokenization. The model reads chunks, not letters, so a count has to be rebuilt from what it remembers about each chunk.
• Limits of attention. The attention mechanism, the part that relates tokens to each other, may not be well suited to keeping an exact running count.
• Little character-level training data. Text on the internet rarely spells words out letter by letter, so the model sees few examples of the skill.

The study then tested a group of models on a large number of words. Its findings point in more than one direction. Models could recognise which letters a word contains but failed to count them. How common a word or its tokens were made no significant difference to the errors, which does not fit a pure tokenization story. Errors rose with the number of letters or tokens in a word, and most models could not get the count right when a letter appeared more than twice.

Researchers still weigh these causes differently, and a single study does not settle it. What is clear for a builder is the practical result: exact counting is a weak spot, and it gets weaker as words get longer and letters repeat.

Why this matters for an AI product

Few products ask a model to count the r's in a fruit. Many ask it for something close: keep a headline under 60 characters, write an SMS that fits one message, fill a form field with a hard limit, check that a code has the right number of digits, or match a pattern exactly. Each of these depends on precise counting or precise character work, the same weak spot.

The failure is also quiet. The model does not say "I am unsure". It gives a number in the same calm tone it uses for everything else, which is why AI hallucination catches teams out. If your product passes that number straight to a user or another system, the error passes with it.

A worked example: a headline length checker

Say you are building a small tool that writes search-friendly page titles for small shops. The rule is simple: every title must be 60 characters or fewer. Here is how to split the job so counting never rests on the model.

1. Let the model write. Prompt: "Write five page titles for a shop that sells handmade ceramic mugs. Keep each short. Return one title per line."
2. Let code count. Run each line through your language's length function, such as title.length in JavaScript. This takes no model call and is always right.
3. Filter and retry. Keep titles of 60 characters or fewer. If fewer than three survive, send the long ones back with the exact number: "This title has 74 characters. Rewrite it in 60 or fewer."
4. Check again. Count the rewritten titles in code before you show them. Never trust the model's own claim that a title "is now 58 characters".

The model does what it is good at, writing natural titles, and code does what code is good at, exact counting. The same split works for word limits, digit checks and any rule you can test with a few lines of code.

Other ways to work around it

• Ask the model to spell it out. Asking for the word one letter per line, then counting, often helps, because each letter then gets its own place in the text. It often helps; it does not always work.
• Give the model a tool. Many model APIs let the model call a function you provide. A count_characters tool turns a guess into a lookup.
• Test it on purpose. Put words with repeated letters, long words and odd spellings in your test set, since the study found those cause the most errors.
• Say what the feature cannot do. If users can ask free-form questions, tell them exact counts come from your code, not the chatbot.

Frequently asked questions

Why does ChatGPT say strawberry has two r's?

Many models have given that answer because they read the word as a few tokens rather than single letters, and then have to rebuild the count from memory. Newer models get this exact example right more often, but the underlying weakness with counting remains.

Is tokenization the only reason AI can't count letters?

No. Researchers also point to limits of the attention mechanism and to how rarely training text spells words out. One 2024 study found word and token frequency made no significant difference, so the debate is open.

Can I trust an AI model to check a character limit?

Not on its own. Let the model write and let a few lines of code count, then send anything too long back with the exact number.

Do bigger models fix the problem?

They can do better on familiar examples, but the same study found most models it tested still failed when a letter appeared more than twice. Treat exact counting as a job for code.

Get started

Open Age of AI Entrepreneur, sign in with your email and an emailed code, and take the How Modern AI Actually Works course: next-token prediction, how a raw model becomes a helpful assistant, and what AI gets wrong. Each lesson takes about nine minutes and lists its sources.