What is tokenizer fertility?
// definition
Tokenizer fertility is the average number of tokens a tokenizer produces per word of input text. Lower is better: a fertility of 1.24 means roughly one token per word, while a fertility of 13.30 means each word is being split into more than thirteen pieces.
Fertility is a property of the tokenizer, not of the model built on it, and it can be measured without running the model at all.
// why it matters
Almost everything in a language model pipeline is priced and limited in tokens. API billing, context windows, and inference latency all scale with token count rather than word count.
So a tokenizer that spends five times as many tokens on Telugu as on English makes every Telugu request roughly five times more expensive, five times more likely to overflow a context window, and slower to generate. The user writing in Telugu pays that premium on every single request, permanently, regardless of how good the model is.
// what we measured
Sthānika AI measured 13 tokenizers across English, Hindi, Marathi, Telugu and Kannada.
| Tokenizer | Indic average | English |
|---|---|---|
| Sarvam-1 2B | 1.92 | 1.44 |
| Sarvam 30B | 2.06 | 1.24 |
| Gemini 3.6 Flash | 2.31 | 1.19 |
| Gemma 3 4B | 2.36 | 1.24 |
| GPT-4o (o200k_base) | 2.64 | 1.23 |
| Llama 3.1 8B | 8.73 | 1.24 |
| GPT-3.5/4 (cl100k_base) | 10.13 | 1.24 |
The full table of 13 is on the tokenizer fertility 2026 report.
The spread is the finding. English fertility is essentially identical across every tokenizer measured, between 1.19 and 1.87. Indic fertility ranges from 1.92 to 10.13, a fivefold difference driven entirely by tokenizer design choices.
// Dravidian scripts cost the most
Telugu and Kannada are consistently the most expensive languages to tokenize. On the cl100k_base tokenizer they reach 13.30 and 14.95 tokens per word against 1.24 for English.
// fertility sets the price, not the outcome
A low-fertility tokenizer is a cost advantage, not a capability advantage, and our data shows the two coming apart clearly. Sarvam-1 2B has the least fertile tokenizer of the 13 measured and finishes last of 16 on the MILU knowledge benchmark. Qwen3 8B has the eleventh worst tokenizer and posts the highest native-language reasoning average of the four models tested end to end.
The one place the two connect: models with efficient Indic tokenizers show smaller gaps between reasoning natively in an Indic language and reasoning through English translation. Sarvam 30B, second best on fertility, shows a gap of -1.15 points. Llama 3.1 8B, twelfth of 13, shows +17.73.
// where the term comes from
Fertility is a standard term in tokenization research. Sthānika AI's contribution is the measurement across Indian languages and the public release of the sentence set used.
// citation
Sthānika AI (2026). The tokenizer sets the price. It doesn't decide the outcome. https://sthanika.ai/research/tokenizer-fertility-2026