REPORTtokenization · reproducible
Telugu, Hindi, Kannada, Marathi, Hinglish, and English across Gemma, Qwen, Mistral-Tekken, and OpenAI's cl100k/o200k. Modern tokenizers sit within ~10% of each other on Indic — but the previous generation pays 2–4×. Dravidian languages cost ~4–5 tokens per word everywhere; plan serving budgets accordingly. Sentence set and scripts released.
fig 1 — tokens, same Telugu sentence
వర్షాలు ఆలస్యంగా వచ్చాయి...
# Dravidian scripts: ~4–5 tokens/word on every tokenizer.
# cl100k pays 4.3× Gemma.