The Tokenizer Sets the Price. It Doesn't Decide the Outcome.

Large language models don't read raw text — they read tokens, the subword chunks their tokenizer cuts a sentence into. Many widely used tokenizers were trained on a corpus that is overwhelmingly English and Latin-script, so when one of them meets Hindi, Marathi, Telugu, Kannada or any other Indic language, it frequently gives up on whole-word or whole-syllable chunks and falls back to small subword or raw-byte fragments. The word doesn't get smaller; it gets cut into more pieces.

We built the upstream half of this study first: a frozen, parallel, 1,012-sentence corpus (FLORES+ devtest) run through 13 tokenizers spanning open-weight Indic-aware models, generalist open models, and closed frontier APIs. It confirmed the fertility gap is real and large — see the tables below. What it couldn't tell us is whether that gap actually breaks a model's ability to think, or whether it's a cost that shows up on the invoice and nowhere else. That's the question this downstream extension exists to answer.

Of the 13 tokenizers in Phase 1, 10 are open — Sarvam-1, Sarvam-30B, Google Gemma 2 9B, Google Gemma 3 4B, Meta Llama 3.1 8B, Qwen3 8B, Mistral Nemo (Tekken), BrahmicTokenizer-131K, OpenAI GPT-4o (o200k_base), and OpenAI GPT-3.5/GPT-4 (cl100k_base). 3 are API-only, with no public tokenizer — Claude Sonnet 4.5, Claude Opus 4.8, and Google Gemini 3.6 Flash (see methodology for how those were measured).

downstream models (Phase 2) — all four are open-weight, self-hosted

We deliberately kept Phase 2 to open-weight models only, so every accuracy number reflects the model itself and not a provider's server-side prompt wrapping or hidden reasoning mode.

Ranked by Indic-average fertility across Hindi, Marathi, Telugu and Kannada. Lower is better. The "API" models (Claude Sonnet 4.5, Claude Opus 4.8, Gemini 3.6 Flash) were measured through their chat completions endpoint by counting output tokens for deterministic prompts; see methodology for the exact procedure.

IndicMMLU-Pro mathematics, averaged across Hindi, Marathi, Telugu, Kannada. The four strategies are: native_cot (question and chain-of-thought both in native script), native_english_cot (question in native script, reasoning in English), trans_google (question translated to English before the model sees it, reasoning in English), and trans_sarvam (same, but translated with Sarvam-1 instead of Google Translate).

Using the Phase-1 token counts and published API prices, the same 1,000-character input can cost very different amounts depending on which tokenizer is charging for it. The table below is for the four tokenizers with a direct owner-published price; the open models are shown for comparison using the same token count but no dollar rate attached.

For a 1,000-character Hindi prompt, the cheapest API option (Gemini 3.6 Flash) is roughly one-thirtieth the price of the most expensive (Claude Opus 4.8). Fertility is the mechanism that turns the same sentence into a different number of tokens, and token count is what most APIs bill by.

The four downstream models split cleanly into two groups. Gemma-3-4B and Llama-3.1-8B lose large accuracy points when asked to reason in native script; Qwen3-8B and Sarvam-30B do not. The upstream fertility table alone cannot predict this split: Gemma has a much better tokenizer than Qwen, yet Gemma suffers the larger reasoning tax. Llama and Qwen have nearly identical fertility, yet Llama's tax is more than ten times larger.

This suggests the protecting factor is not the tokenizer itself but something correlated with it at the model-family level — most likely the language distribution and fine-tuning data the base model was exposed to. Models that have seen more native-script reasoning during training appear able to route around tokenizer fragmentation; models that haven't seen enough fall back to English-like reasoning patterns and pay the price when the input is forced to stay in script.